Mender server GUI not displaying application (.py) update artifact using self hosted production server

Update Modules
8

I’m new to mender and encountered this problem yesterday. By this problem I mean I’m running self-hosted mender and attempting to create single-file artifacts using the GUI (like we learn to do in the tutorial).

Uploads work fine, but no release appears. The files I’m using aren’t overly large, have normal filenames and destination paths, and the artifact names don’t have spaces in them (typical things I’d look for in a tool like this). The developer tools network pane in my browser shows a couple of 4xx errors at .../inventory/filters and .../useradm/roles, but I assume those are just enterprise features I don’t have.

So I looked at the logs and found one error about unparseable XML:
mender-deployments_1 | time="2021-04-24T02:22:46Z" level=error msg="error reaching artifact storage service: SerializationError: failed to decode REST XML response\n\tstatus code: 200, request id: \ncaused by: XML syntax error on line 8: element <link> closed by </head>" file=response_helpers.go func=rest_utils.restErrWithLogMsg line=110 request_id=95786559-f302-4bb0-9e18-4d59933a5421

Google pointed me at an issue in minio, so I figured ah, maybe I’ll just try mender 2.6 since this could be a new bug in 2.7. The error message disappeared from the logs but I had the same problem. Was this maybe escaping the test suite because no “serious users” are going to need the single file upload? Same problem in 2.5, and again in 2.4.

I started poking around further, eventually looking at mongodb in the menderproduction_mender-mongo_1 container. Finally a clue!

adrian@somewhere: docker exec -it menderproduction_mender-mongo_1 /usr/bin/mongo
...
> use workflows
> db.job_queue.find().pretty()
{
	"_id" : "60837a5232849e3b5254cbbb",
	"workflow_name" : "generate_artifact",

# Zillions of these stuck in the job queue :)

Looking at the jobs collection I see this: "...x509: certificate signed by unknown authority". So it’s simply my self-signed certificate. Well I can’t get the certificate I need for some bureaucratic reason, and self-signed will have to work for now. Running the create-artifact tool manually within menderproduction_mender-create-artifact-worker_1 I found this:

/ # create-artifact single-file --help

Besides command line args, supports the following env vars:

CREATE_ARTIFACT_SKIPVERIFY skip ssl verification (default: false)
CREATE_ARTIFACT_WORKDIR working dir for processing (default: /var)
CREATE_ARTIFACT_DEPLOYMENTS_URL internal deployments service url
...

Cool, I don’t have to hack this tool to make it work!

CREATE_ARTIFACT_SKIPVERIFY was my solution. I added it to my prod.yml like so:

    mender-create-artifact-worker:
        command: --automigrate
        environment:
            CREATE_ARTIFACT_SKIPVERIFY: "true"

I hope this is helpful to someone else scratching their head because they lack a “proper” TLS certificate.

So far I’m quite pleased with the quality of this software. As a new user, I was able to navigate the well-defined services, track down my problem, and solve it without too much trouble. Keep up the good work.

2 Likes