Mender minio: trouble connecting with other containers

[genedupre]

Hello everyone,
i created several mender-production servers, but none of them have a working ‘release’ tab.
this has probably to do with the minio and the mender-deployment container not been able to communicate & when i go to releases, i get an empty white screen

now the question i’m having is why, maybe ive done something wrong with the dns names or certificates, as these are the only things i’m aware of that could affect this situation.
loggin into the minio itself is not a problem, ofcourse ignoring to cert prompt from the browser & also the containers are working like intended,
in ./run up does does it show normal http request without an error

–logs
prod.yml can be found here
and for the key gen i used

CERT_API_CN=mender.inuits.io CERT_STORAGE_CN=mender.inuits.io ../keygen

As always thanks!

[drewmoseley]

Hi @genedupre, nothing immediately jumps out at me. We will definitely need the logs to help diagnose this. The following will pull all the logs into individual files:

$ for i in $(./run ps | grep menderprod | cut -d_ -f2); do ./run logs $i > $i.log.txt; done

I did see that you are using the same FQDN for both the API gateway and the storage gateway. I have not tried that so it may be related.

[genedupre]

hello @drewmoseley thank you for answering,
i also think it will have to do with the FQDN, but maybe it is because i dont fully understand the documentation,

in the production documentation, there is an example of s3.

    storage-proxy:
        networks:
            mender:
                aliases:
                    - s3.example.com
    ...

i assume, this is the place where the ‘minio’ connects to, now we will be hosting the server ourselves, so i assume IRL this would be mender.inuits.com the same goes for

 ...
    mender-deployments:
        ...
        environment:
            DEPLOYMENTS_AWS_AUTH_KEY: mender-deployments
            DEPLOYMENTS_AWS_AUTH_SECRET: ahshagheeD1ooPaeT8lut0Shaezeipoo
            DEPLOYMENTS_AWS_URI: https://mender.inuits.com:9000
    ...

and

    ...
    mender-api-gateway:
        ...
        environment:
            ALLOWED_HOSTS: mender.inuits.com
    ...

as everything will be put inside this one server, i assume everything should also be pointing to the same place, thus all having the same FQDN

what does the api ALLOWED_HOSTS actually do?

as for your command @drewmoseley, this may be a novice question, but i assume this is a line for bash and i insert this line inside the run file?

Thanks for the help!

[drewmoseley]

Yes, the alias under the storage-proxy setting the the URL that will be used for the Minio instance. Basically, the Mender server will store artifacts there and the URL generated when a deployment is created will be based off of that. That URL will be passed to the client for download of the new artifact. It seems like mender.inuits.com is the URL you are using for the Mender API gateway (ie where the web UI shows up and also where the clients connect). In that case, I think you want something with a different name such as s3.mender.inuits.com. I’m not sure if using a different port (9000) is enough here.

(Note that even if you use s3.example.com, it is not going to AWS but rather using an S3-compatible API.)

The ALLOWED_HOSTS entry under the mender-api-gateway specifies the FQDN of the API gateway itself. This is for security purposes. More details here: https://docs.mender.io/2.0/administration/production-installation#api-gateway

The command I recommended needs to be run manually from a login shell on the server running your Mender instance. It will dump the logs for review.

But at the moment my best suggestion is to setup a different FQDN for the storage proxy and see if that helps.

Drew

[MBConsultingUK]

FWIW, this is identical to the issue that I’m seeing at https://hub.mender.io/t/issue-with-mender-2-0-ui-blank-release-list-page/ I’d not thought to look at the storage yet, but I’ll do that now, please let me know if it makes a difference for you!

[MBConsultingUK]

Try the solution at Working with S3 instead of Minio - no artifacts found and blank releases page - it fixed my issue completely.

[genedupre]

hello @drewmoseley
thanks for informing me about that, i had the presumption that s3 was always meant using an amazon service.

i edited the prod.yml to let the storage.proxy point to

s3.mender.inuits.io

and mender-deployments to

DEPLOYMENTS_AWS_URI: https://s3.mender.inuits.io:9000

and remade the keys using

sudo CERT_API_CN=mender.inuits.com CERT_STORAGE_CN=s3.mender.inuits.com …/keygen

this however did not fix the problem, now i did a docker log, and saw that mender-deployments was complaining about multiple certificates, which surprises me since i removed the docker containers & volumes & rebuild them but maybe i’m still having residu files that keep pilling up each other
( docker rmi $(docker images -q) & docker volume rm & rm -rf /var/lib/docker/volumes/)

i have made an log of the containers that showed errors, at the moment i assume that it is a certificate problem that i dont yet understand

@MBConsultingUK
thank you for the suggestion
i was already watching your posts, will keep an eye on it as things develop

edit: made an complete version of the logs using your shell command, there to be found here

[genedupre]

so i was not finding the solution to the problem at all, so what ive done is make an exact copy of the
Production installation | Mender documentation doc’s
using the mender.example.com FQDN and by adding both (s3)mender.example.com in my /etc/hosts of my server & workstation

using standard

CERT_API_CN=mender.example.com CERT_STORAGE_CN=s3.example.com …/keygen
for the key-gen,

after running these containers, there still seems to be no connection to the storage

connection to the minio with an browser works fine tough

ive made a complete logs list here
if somebody would take a look at them & point me to what i’m doing wrong, i would really appreciate that

Thanks!

edit: i made an 1.7 mender-server, and this one seems te work fine,i cant upload artifact to it though, but this is probably because of been out-of-date in reference with my ‘fresh’ artifacts & not because of minio,

Artifact couldn’t be uploaded. unsupported version: 3; supported versions are: 1, 2 [Request ID: ff21af7b]

[MBConsultingUK]

Dd you add the - GATEWAY_IP=my.mender.server to your config under the mender-gui section?

This is what solved it for me - everything was connecting fine with Minio apart from the UI container, and adding this in fixed the issue.

Note that you can set - GATEWAY_IP to a hostname as well as an IP address.

[genedupre]

the problem was indeed as @drewmoseley suggested to do with the FQDN