Working with S3 instead of Minio - no artifacts found and blank releases page

General Discussions
1

Hey folks,

Running Mender 2.1 via the docker-compose scripts for the production environment as documented, using LetsEncrypt certificates for the TLS connections.

In order to see if I can fix the issues I’m encountering here and here I thought I’d skip the storage completely and move over to S3.

I followed the (very brief!) instructions at https://docs.mender.io/2.1/administration/storage#s3-storage-backend and have created the appropriate YAML config, updated the run script to use that instead of the Minio one, and run my image build script.

Checking the S3 bucket, I can see that there is an object in there and when I download it and unpack it then it is the correct Mender Artifact, however when I try and create a deployment I get an error saying that there aren’t any releases, and when I browse to the “releases” page then it is still blank with a load of javascript errors in the console.

The docker logs show nothing of relevance (all the logs when I access the page are server 200 status for the Javascript files), and I’m starting to tear my hair out about this!

Does anyone know why this might be happening?

The Javascript errors are as follows:

Artifacts.js:

Error: Request has been terminated
Possible causes: the network is offline, Origin is not allowed by Access-Control-Allow-Origin, the page is being unloaded, etc.
    at b.crossDomainError (client.js:621)
    at XMLHttpRequest.t.onreadystatechange (client.js:703)

Other errors:

GET https://localhost/api/management/v1/deployments/deployments/releases net::ERR_CONNECTION_REFUSED

GET https://localhost/api/management/v1/deployments/artifacts net::ERR_CONNECTION_REFUSED

Invariant Violation: Minified React error #31; visit https://reactjs.org/docs/error-decoder.html?invariant=31&args[]=Error%3A%20Request%20has%20been%20terminated%0APossible%20causes%3A%20the%20network%20is%20offline%2C%20Origin%20is%20not%20allowed%20by%20Access-Control-Allow-Origin%2C%20the%20page%20is%20being%20unloaded%2C%20etc.&args[]= for the full message or use the non-minified dev environment for full errors and additional helpful warnings. 
    at https://MY.MENDER.SERVER/ui/main.js:47:424

I’m not sure why it’s trying to access localhost (although apparently that’s something to do with an internal system on hosted mender), and I can’t find anything more on the potential cross-origin issue either :frowning:

2

Hi @MBConsultingUK,

Would it be possible for you to share your prod.yml, so we can take a closer look if there is anything obvious missing.

3

@mirzak Sure, here it is (but with some of the sensitive data such as hostnames and auth keys replaced):

# this is a template file for production setup, consult
# https://docs.docker.com/compose/compose-file/ for details on syntax and usage
#
# Notes:
# - integration/docker-compose.yml file is assumed to be included
# - integration/docker-compose.storage.minio.yml is assumed to be included
# - all services are part of `mender` network (service names are unchanged)
# - keys and certificates are generated using keygen utility from integration
#   repository, keys and certificates are stored in ./keys-generated directory
# - certificates and key are mounted into containers using volumes
# - minio artifacts are stored in a named volume `mender-artifacts`; volume
#   needs to be created manually using `docker volume create mender-artifacts`
# - paths need to be adjusted, ie, replace /production/ with production directory
#   (see list of compose bugs)

# related compose bugs:
# - https://github.com/docker/compose/issues/3874
# - https://github.com/docker/compose/issues/3568
# - https://github.com/docker/compose/issues/3219

version: '2'
services:

    mender-useradm:
        command: server --automigrate
        volumes:
            - /etc/letsencrypt/live/my.mender.server/privkey.rsa:/etc/useradm/rsa/private.pem:ro
        logging:
            options:
                max-file: "10"
                max-size: "50m"

    mender-device-auth:
        command: server --automigrate
        volumes:
            - /etc/letsencrypt/live/my.mender.server/privkey.rsa:/etc/deviceauth/rsa/private.pem:ro
        logging:
            options:
                max-file: "10"
                max-size: "50m"
        environment:
            DEVICEAUTH_MAX_DEVICES_LIMIT_DEFAULT: 15

    mender-inventory:
        command: server --automigrate
        logging:
            options:
                max-file: "10"
                max-size: "50m"

    mender-api-gateway:
        ports:
            # list of ports API gateway is made available on
            - "443:443"
        networks:
            - mender
        volumes:
            - /etc/letsencrypt/live/my.mender.server/fullchain.pem:/var/www/mendersoftware/cert/cert.crt:ro
            - /etc/letsencrypt/live/my.mender.server/privkey.rsa:/var/www/mendersoftware/cert/private.key:ro
        logging:
            options:
                max-file: "10"
                max-size: "50m"
        environment:
            ALLOWED_HOSTS: my.mender.server

 #   storage-proxy:
 #       ports:
  #          # outside port mapping for artifact storage (note that storage-proxy listens on port 9000)
  #          - "9000:9000"
  #      networks:
  #          mender:
  #              aliases:
  #                  # change the alias to DNS name that storage will be
  #                  # available on, for instance if devices will access storage
  #                  # using https://s3.acme.org:9000, then set this to
  #                  # s3.acme.org
  #                  - my.minio.server
  #      environment:

  #          # use nginx syntax for rate limiting, see
  #          # https://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate
  #          # Examples:
  #          #   1m - 1MB/s
  #          #   512k - 512kB/s
  #          DOWNLOAD_SPEED: 1m
  #          MAX_CONNECTIONS: 100
  #      volumes:
  #          - /etc/letsencrypt/live/my.minio.server/fullchain.pem:/var/www/storage-proxy/cert/cert.crt:ro
  #          - /etc/letsencrypt/live/my.minio.server/privkey.rsa:/var/www/storage-proxy/cert/private.key:ro
  #          - ./storage-proxy/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf

    mender-deployments:
        command: server --automigrate
        volumes:
            - /etc/letsencrypt/live/my.mender.server/fullchain.pem:/var/www/storage-proxy/cert/cert.crt:ro
        environment:
            STORAGE_BACKEND_CERT: /etc/ssl/certs/storage-proxy.crt
            DEPLOYMENTS_AWS_AUTH_KEY: MY-AUTH-KEY
            DEPLOYMENTS_AWS_AUTH_SECRET: MY-SECRET-KEY

            # deployments service uses signed URLs, hence it needs to access
            # storage-proxy using exactly the same name as devices will; if
            # devices will access storage using https://s3.acme.org:9000, then
            # set this to https://s3.acme.org:9000
            DEPLOYMENTS_AWS_URI: https://s3-eu-west-2.amazonaws.com
        logging:
            options:
                max-file: "10"
                max-size: "50m"

   # minio:
   #     environment:
   #         # access key
   #         MINIO_ACCESS_KEY: MINIO-ACCESS-KEY
   #         # secret
   #         MINIO_SECRET_KEY: MINIO-SECRET-KEY
   #     volumes:
   #         # mounts a docker volume named `mender-artifacts` as /export directory
   #         - mender-artifacts:/export:rw
   #         - /etc/letsencrypt/live/my.minio.server/fullchain.pem:/root/.minio/certs/public.key:ro
   #         - /etc/letsencrypt/live/my.minio.server/privkey.rsa:/root/.minio/certs/public.key:ro

    mender-conductor:
        volumes:
            - ./conductor/server/config:/app/config
        environment:
            - CONFIG_PROP=config.properties

    mender-mongo:
        volumes:
            - mender-db:/data/db:rw
    mender-elasticsearch:
        volumes:
            - mender-elasticsearch-db:/usr/share/elasticsearch/data:rw
            - ./conductor/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    mender-redis:
        volumes:
            - mender-redis-db:/var/lib/redis:rw
            - ./conductor/redis/redis.conf:/etc/redis/redis.conf
            - ./conductor/redis/entrypoint.sh:/redis/entrypoint.sh
        entrypoint: /redis/entrypoint.sh

volumes:
    # mender artifacts storage
    mender-artifacts:
      external:
          # use external volume created manually
          name: mender-artifacts
    # mongo service database
    mender-db:
      external:
          # use external volume created manually
          name: mender-db
    # elasticsearch database
    mender-elasticsearch-db:
      external:
          # use external volume created manually
          name: mender-elasticsearch-db
    # redis database
    mender-redis-db:
      external:
          # use external volume created manually
          name: mender-redis-db
4

Hello @MBConsultingUK,

I would like to verify the environment passed to gui container, one way of doing that:

# docker ps | grep gui
13b4880ca1b3 mendersoftware/gui:2.0.1 “/entrypoint.sh” 19 minutes ago Up 19 minutes 80/tcp menderproduction_mender-gui_1

# docker exec -it 13b4880ca1b3 ps axuw
PID USER TIME COMMAND
1 root 0:00 httpd -f -p 80 -c /etc/httpd.conf
38 root 0:00 ps axuw

# docker exec -it 13b4880ca1b3 cat /proc/1/environ
HOSTNAME=SHLVL=1HOME=/rootPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binPWD=/var/www/mender-gui/dist

peter

5

Thanks @peter

PID   USER     TIME  COMMAND
    1 root      0:00 nginx: master process nginx -g daemon off;
    8 nginx     0:00 nginx: worker process
    9 root      0:00 ps auxw

docker exec -it 4bb98bf8e48b cat /proc/1/environ
 nginx -g daemon off;

I’m now concerned that I’m running NGinx whereas you’re running Apache…

6

I’m now concerned that I’m running NGinx whereas you’re running Apache…

I am running an older version, those commands were just an example.

docker exec -it 4bb98bf8e48b cat /proc/1/environ

so, the environment is empty? is it also empty for /proc/8/environ ?

peter

7

I just get “access denied” when I try and run it against /proc/8, but no, no environment set for /proc/1

8

could you please run one more command in the gui container, and check what rootUrl value is?

docker exec 4bb98bf8e48b cat /var/www/mender-gui/dist/env.js

peter

9

as a workaround you could add to gui section in docker-compose.yml (in the top level directory) environment variable GATEWAY_IP=yourdomain

so it looks something like this:

    mender-gui:
        image: mendersoftware/gui:2.1.0b1
        extends:
            file: common.yml
            service: mender-base
        networks:
            - mender
        environment:
            - INTEGRATION_VERSION
            - MENDER_ARTIFACT_VERSION
            - MENDER_VERSION
            - MENDER_DEB_PACKAGE_VERSION
            - GATEWAY_IP=my.mender.server

peter

10

Had a similar rapport from a user on IRC,

I ran out of disk space on my mender server. I increased the disk space and recreated the services but now my mender-ui is now trying to access the api by localhost instead of my server uri. I upgraded from 2.0.1 to 2.1.x to try and fix the issue but no luck
15:11 The only error i see in my log is mender-conductor_1      | Exception in thread "main" java.io.FileNotFoundException: /app/config/log4j.properties (No such file or directory)
15:20 I went back to 2.0.x which seems fine
11

Yup, that was it, I added this and all of a sudden all my releases are available to me, my devices register correctly, it’s working perfectly!

Thanks @peter and @mirzak for your help - I’ll mark the other posts as closed too.