Mender server GUI not displaying application (.py) update artifact using self hosted production server

@drewmoseley

I am using self-hosted on AWS EC2 mender server (version 2.4.1). For application update, I have selected a test.py file at the device path “/opt/” to upload as below.

Since UI is showing notification as “upload successfully” But the artifact is not displaying in UI.

Device type: x86_64

image
image

Please can anyone assist me to find out why the application artifact is not showing on UI under the RELEASE tab.

Note: Other .mender artifacts are uploaded successfully.

@mzedel any ideas on this?

Hello @Rohita83,
sorry for the delay in the response - today I finally got around to test 2.4.1 for that aspect and it works fine with the regular setup. The notification indicates that the UI aspect of the upload was completed successfully and the UI doesn’t modify the release list in any way. The next best place to check might be the workflow that is responsible for the artifact generation -> @merlin, @tranchitella do you maybe have more insights what could go wrong there?
Just to be sure: did the release name differ from the name of the other artifacts you uploaded? Different artifacts with the same might be grouped into the same release and might not immediately be obvious…

Hello @Rohita83, can you please check the logs of the create-artifact-worker container and post them here?

I faced a same issue with the Mender server v2.5 Production setup.

@tranchitella
Could you please assist me to get logs of the create-artifact-worker container?

@Rohita83

Get the ID of your create-artifact-worker container running this command:
$ docker ps | grep create-artifact-worker

Then run:
$docker logs id-of-your-container

I’m new to mender and encountered this problem yesterday. By this problem I mean I’m running self-hosted mender and attempting to create single-file artifacts using the GUI (like we learn to do in the tutorial).

Uploads work fine, but no release appears. The files I’m using aren’t overly large, have normal filenames and destination paths, and the artifact names don’t have spaces in them (typical things I’d look for in a tool like this). The developer tools network pane in my browser shows a couple of 4xx errors at .../inventory/filters and .../useradm/roles, but I assume those are just enterprise features I don’t have.

So I looked at the logs and found one error about unparseable XML:
mender-deployments_1 | time="2021-04-24T02:22:46Z" level=error msg="error reaching artifact storage service: SerializationError: failed to decode REST XML response\n\tstatus code: 200, request id: \ncaused by: XML syntax error on line 8: element <link> closed by </head>" file=response_helpers.go func=rest_utils.restErrWithLogMsg line=110 request_id=95786559-f302-4bb0-9e18-4d59933a5421

Google pointed me at an issue in minio, so I figured ah, maybe I’ll just try mender 2.6 since this could be a new bug in 2.7. The error message disappeared from the logs but I had the same problem. Was this maybe escaping the test suite because no “serious users” are going to need the single file upload? Same problem in 2.5, and again in 2.4.

I started poking around further, eventually looking at mongodb in the menderproduction_mender-mongo_1 container. Finally a clue!

adrian@somewhere: docker exec -it menderproduction_mender-mongo_1 /usr/bin/mongo
...
> use workflows
> db.job_queue.find().pretty()
{
	"_id" : "60837a5232849e3b5254cbbb",
	"workflow_name" : "generate_artifact",

# Zillions of these stuck in the job queue :)

Looking at the jobs collection I see this: "...x509: certificate signed by unknown authority". So it’s simply my self-signed certificate. Well I can’t get the certificate I need for some bureaucratic reason, and self-signed will have to work for now. Running the create-artifact tool manually within menderproduction_mender-create-artifact-worker_1 I found this:

/ # create-artifact single-file --help

Besides command line args, supports the following env vars:

CREATE_ARTIFACT_SKIPVERIFY skip ssl verification (default: false)
CREATE_ARTIFACT_WORKDIR working dir for processing (default: /var)
CREATE_ARTIFACT_DEPLOYMENTS_URL internal deployments service url
...

Cool, I don’t have to hack this tool to make it work!

CREATE_ARTIFACT_SKIPVERIFY was my solution. I added it to my prod.yml like so:

    mender-create-artifact-worker:
        command: --automigrate
        environment:
            CREATE_ARTIFACT_SKIPVERIFY: "true"

I hope this is helpful to someone else scratching their head because they lack a “proper” TLS certificate.

So far I’m quite pleased with the quality of this software. As a new user, I was able to navigate the well-defined services, track down my problem, and solve it without too much trouble. Keep up the good work.

2 Likes

@adriaticc Thanks for describing your debug process and sharing it with the community!

The env variable is used in the demo setup which, obviously, we don’t recommend using in a production setup. I am not an expert in the area, but you might find other things not working with self-signed certificates, if so the demo file is a good place to look for the configuration for self-signed certificates setup.