Issue with x509 certificate on Ubuntu

Hi,

I got the following error after a OS production installation with this command
CERT_API_CN=$API_GATEWAY_DOMAIN_NAME CERT_STORAGE_CN=$STORAGE_PROXY_DOMAIN_NAME …/keygen
Full procedure listed here: https://docs.mender.io/2.2/administration/production-installation
root@ec2:/usr/local/mender-server/production# ./run logs --tail 10 mender-deployments
Attaching to menderproduction_mender-deployments_1

RequestError: send request failed
mender-deployments_1 | caused by: Put https://ec2-12-23-34-45.eu-west-1.compute.amazonaws.com:9000/mender-artifact-storage: x509: certificate is not valid for any names, but wanted to match ec2-12-23-34-45.eu-west-1.compute.amazonaws.com
mender-deployments_1 | WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
mender-deployments_1 | time=“2019-12-02T09:26:38Z” level=info msg=“Deployments Service, version unknown starting up”
mender-deployments_1 | time=“2019-12-02T09:26:38Z” level=info msg=“automigrate is ON, will apply migrations” file=migrations.go func=mongo.Migrate line=48
mender-deployments_1 | time=“2019-12-02T09:26:38Z” level=info msg=“migrating deployment_service” file=migrations.go func=mongo.MigrateSingle line=70
mender-deployments_1 | time=“2019-12-02T09:26:38Z” level=info msg=“migration to version 1.2.1 skipped” db=deployment_service file=migrator_simple.go func=“migrate.(*SimpleMigrator).Apply” line=125
mender-deployments_1 | time=“2019-12-02T09:26:38Z” level=info msg=“DB migrated to version 1.2.1” db=deployment_service file=migrator_simple.go func=“migrate.(*SimpleMigrator).Apply” line=140
mender-deployments_1 | RequestError: send request failed
mender-deployments_1 | caused by: Put https://ec2-12-23-34-45.eu-west-1.compute.amazonaws.com:9000/mender-artifact-storage: x509: certificate is not valid for any names, but wanted to match ec2-12-23-34-45.eu-west-1.compute.amazonaws.com
https://docs.mender.io/2.2/administration/production-installation

uname -a
Linux ec2.eu-west-1.compute.amazonaws.com 4.15.0-1054-aws #56-Ubuntu SMP Thu Nov 7 16:15:59 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
docker -v
Docker version 19.03.5, build 633a0ea838
docker-compose -v
docker-compose version 1.17.1, build unknown

I have tried this install 2 times now, same thing happens

Is the API_GATEWAY_DOMAIN_NAME variable setup correctly? prior to running:

CERT_API_CN=$API_GATEWAY_DOMAIN_NAME CERT_STORAGE_CN=$STORAGE_PROXY_DOMAIN_NAME ../keygen

The error message you are getting hints that it might not be.

@peter, any input?

Indeed. Would check the values of $API_GATEWAY_DOMAIN_NAME and especially $STORAGE_PROXY_DOMAIN_NAME you are passing to the keygen tool.

Do they both contain ec2-12-23-34-45.eu-west-1.compute.amazonaws.com ?

You can also parse the generated certificates and check the CN to do this, using the openssl tool.

Hi,

Thanks for helping troubleshoot this issue my prod.yml looks like this.

version: '2'
services:

mender-useradm:
    command: server --automigrate
    volumes:
        - ./production/keys-generated/keys/useradm/private.key:/etc/useradm/rsa/private.pem:ro
    logging:
        options:
            max-file: "10"
            max-size: "50m"

mender-device-auth:
    command: server --automigrate
    volumes:
        - ./production/keys-generated/keys/deviceauth/private.key:/etc/deviceauth/rsa/private.pem:ro
    logging:
        options:
            max-file: "10"
            max-size: "50m"

mender-inventory:
    command: server --automigrate
    logging:
        options:
            max-file: "10"
            max-size: "50m"

mender-api-gateway:
    ports:
        # list of ports API gateway is made available on
        - "443:443"
    networks:
        - mender
    volumes:
        - ./production/keys-generated/certs/api-gateway/cert.crt:/var/www/mendersoftware/cert/cert.crt:ro
        - ./production/keys-generated/certs/api-gateway/private.key:/var/www/mendersoftware/cert/private.key:ro
    logging:
        options:
            max-file: "10"
            max-size: "50m"
    environment:
        ALLOWED_HOSTS: ec2-12-23-34-45.eu-west-1.compute.amazonaws.com

storage-proxy:
    ports:
        # outside port mapping for artifact storage (note that storage-proxy listens on port 9000)
        - "9000:9000"
    networks:
        mender:
            aliases:
                # change the alias to DNS name that storage will be
                # available on, for instance if devices will access storage
                # using https://s3.acme.org:9000, then set this to
                # s3.acme.org
                -  ec2-12-23-34-45.eu-west-1.compute.amazonaws.com
    environment:

        # use nginx syntax for rate limiting, see
        # https://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate
        # Examples:
        #   1m - 1MB/s
        #   512k - 512kB/s
        DOWNLOAD_SPEED: 1m
        MAX_CONNECTIONS: 100
    volumes:
        - ./production/keys-generated/certs/storage-proxy/cert.crt:/var/www/storage-proxy/cert/cert.crt:ro
        - ./production/keys-generated/certs/storage-proxy/private.key:/var/www/storage-proxy/cert/private.key:ro
        - ./storage-proxy/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf

mender-deployments:
    command: server --automigrate
    volumes:
        - ./production/keys-generated/certs/storage-proxy/cert.crt:/etc/ssl/certs/storage-proxy.crt:ro
    environment:
        STORAGE_BACKEND_CERT: /etc/ssl/certs/storage-proxy.crt
        # access key, the same value as MINIO_ACCESS_KEY
        DEPLOYMENTS_AWS_AUTH_KEY: mender-deployments
        # secret, the same valie as MINIO_SECRET_KEY
        DEPLOYMENTS_AWS_AUTH_SECRET: abcdefpassword

        # deployments service uses signed URLs, hence it needs to access
        # storage-proxy using exactly the same name as devices will; if
        # devices will access storage using https://s3.acme.org:9000, then
        # set this to https://s3.acme.org:9000
        DEPLOYMENTS_AWS_URI: https://ec2-12-23-34-45.eu-west-1.compute.amazonaws.com:9000
    logging:
        options:
            max-file: "10"
            max-size: "50m"

minio:
    environment:
        # access key
        MINIO_ACCESS_KEY: mender-deployments
        # secret
        MINIO_SECRET_KEY: abcdefpassword
    volumes:
        # mounts a docker volume named `mender-artifacts` as /export directory
        - mender-artifacts:/export:rw

mender-mongo:
    volumes:
        - mender-db:/data/db:rw
mender-elasticsearch:
    volumes:
        - mender-elasticsearch-db:/usr/share/elasticsearch/data:rw
        - ./conductor/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
mender-redis:
    volumes:
        - mender-redis-db:/var/lib/redis:rw
        - ./conductor/redis/redis.conf:/etc/redis/redis.conf
        - ./conductor/redis/entrypoint.sh:/redis/entrypoint.sh
    entrypoint: /redis/entrypoint.sh

volumes:
# mender artifacts storage
mender-artifacts:
external:
# use external volume created manually
name: mender-artifacts
# mongo service database
mender-db:
external:
# use external volume created manually
name: mender-db
# elasticsearch database
mender-elasticsearch-db:
external:
# use external volume created manually
name: mender-elasticsearch-db
# redis database
mender-redis-db:
external:
# use external volume created manually
name: mender-redis-db

Hi @leonsgithub. I think the question is related to the shell environment where you invoked keygen. It assumes there is a shell variable called API_GATEWAY_DOMAIN_NAME which will be used in the generation of the keys. I don’t think there is anything in the prod.yml related to that. Can you regenerate the keys ensuring that you set that shell variable appropriately?

Drew

[ Solved ]
rm -rf keys-generated/certs/*

before you trigger the keygen script:
CERT_API_CN=$API_GATEWAY_DOMAIN_NAME CERT_STORAGE_CN=$STORAGE_PROXY_DOMAIN_NAME …/keygen