Issue with x509 certificate on Ubuntu

General Discussions
,
1

Hi,

I got the following error after a OS production installation with this command
CERT_API_CN=$API_GATEWAY_DOMAIN_NAME CERT_STORAGE_CN=$STORAGE_PROXY_DOMAIN_NAME …/keygen
Full procedure listed here: https://docs.mender.io/2.2/administration/production-installation
root@ec2:/usr/local/mender-server/production# ./run logs --tail 10 mender-deployments
Attaching to menderproduction_mender-deployments_1

RequestError: send request failed
mender-deployments_1 | caused by: Put https://ec2-12-23-34-45.eu-west-1.compute.amazonaws.com:9000/mender-artifact-storage: x509: certificate is not valid for any names, but wanted to match ec2-12-23-34-45.eu-west-1.compute.amazonaws.com
mender-deployments_1 | WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
mender-deployments_1 | time=“2019-12-02T09:26:38Z” level=info msg=“Deployments Service, version unknown starting up”
mender-deployments_1 | time=“2019-12-02T09:26:38Z” level=info msg=“automigrate is ON, will apply migrations” file=migrations.go func=mongo.Migrate line=48
mender-deployments_1 | time=“2019-12-02T09:26:38Z” level=info msg=“migrating deployment_service” file=migrations.go func=mongo.MigrateSingle line=70
mender-deployments_1 | time=“2019-12-02T09:26:38Z” level=info msg=“migration to version 1.2.1 skipped” db=deployment_service file=migrator_simple.go func=“migrate.(*SimpleMigrator).Apply” line=125
mender-deployments_1 | time=“2019-12-02T09:26:38Z” level=info msg=“DB migrated to version 1.2.1” db=deployment_service file=migrator_simple.go func=“migrate.(*SimpleMigrator).Apply” line=140
mender-deployments_1 | RequestError: send request failed
mender-deployments_1 | caused by: Put https://ec2-12-23-34-45.eu-west-1.compute.amazonaws.com:9000/mender-artifact-storage: x509: certificate is not valid for any names, but wanted to match ec2-12-23-34-45.eu-west-1.compute.amazonaws.com
https://docs.mender.io/2.2/administration/production-installation

uname -a
Linux ec2.eu-west-1.compute.amazonaws.com 4.15.0-1054-aws #56-Ubuntu SMP Thu Nov 7 16:15:59 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
docker -v
Docker version 19.03.5, build 633a0ea838
docker-compose -v
docker-compose version 1.17.1, build unknown

I have tried this install 2 times now, same thing happens

2

Is the API_GATEWAY_DOMAIN_NAME variable setup correctly? prior to running:

CERT_API_CN=$API_GATEWAY_DOMAIN_NAME CERT_STORAGE_CN=$STORAGE_PROXY_DOMAIN_NAME ../keygen

The error message you are getting hints that it might not be.

@peter, any input?

3

Indeed. Would check the values of $API_GATEWAY_DOMAIN_NAME and especially $STORAGE_PROXY_DOMAIN_NAME you are passing to the keygen tool.

Do they both contain ec2-12-23-34-45.eu-west-1.compute.amazonaws.com ?

You can also parse the generated certificates and check the CN to do this, using the openssl tool.

4

Hi,

Thanks for helping troubleshoot this issue my prod.yml looks like this.

version: '2'
services:

mender-useradm:
    command: server --automigrate
    volumes:
        - ./production/keys-generated/keys/useradm/private.key:/etc/useradm/rsa/private.pem:ro
    logging:
        options:
            max-file: "10"
            max-size: "50m"

mender-device-auth:
    command: server --automigrate
    volumes:
        - ./production/keys-generated/keys/deviceauth/private.key:/etc/deviceauth/rsa/private.pem:ro
    logging:
        options:
            max-file: "10"
            max-size: "50m"

mender-inventory:
    command: server --automigrate
    logging:
        options:
            max-file: "10"
            max-size: "50m"

mender-api-gateway:
    ports:
        # list of ports API gateway is made available on
        - "443:443"
    networks:
        - mender
    volumes:
        - ./production/keys-generated/certs/api-gateway/cert.crt:/var/www/mendersoftware/cert/cert.crt:ro
        - ./production/keys-generated/certs/api-gateway/private.key:/var/www/mendersoftware/cert/private.key:ro
    logging:
        options:
            max-file: "10"
            max-size: "50m"
    environment:
        ALLOWED_HOSTS: ec2-12-23-34-45.eu-west-1.compute.amazonaws.com

storage-proxy:
    ports:
        # outside port mapping for artifact storage (note that storage-proxy listens on port 9000)
        - "9000:9000"
    networks:
        mender:
            aliases:
                # change the alias to DNS name that storage will be
                # available on, for instance if devices will access storage
                # using https://s3.acme.org:9000, then set this to
                # s3.acme.org
                -  ec2-12-23-34-45.eu-west-1.compute.amazonaws.com
    environment:

        # use nginx syntax for rate limiting, see
        # https://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate
        # Examples:
        #   1m - 1MB/s
        #   512k - 512kB/s
        DOWNLOAD_SPEED: 1m
        MAX_CONNECTIONS: 100
    volumes:
        - ./production/keys-generated/certs/storage-proxy/cert.crt:/var/www/storage-proxy/cert/cert.crt:ro
        - ./production/keys-generated/certs/storage-proxy/private.key:/var/www/storage-proxy/cert/private.key:ro
        - ./storage-proxy/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf

mender-deployments:
    command: server --automigrate
    volumes:
        - ./production/keys-generated/certs/storage-proxy/cert.crt:/etc/ssl/certs/storage-proxy.crt:ro
    environment:
        STORAGE_BACKEND_CERT: /etc/ssl/certs/storage-proxy.crt
        # access key, the same value as MINIO_ACCESS_KEY
        DEPLOYMENTS_AWS_AUTH_KEY: mender-deployments
        # secret, the same valie as MINIO_SECRET_KEY
        DEPLOYMENTS_AWS_AUTH_SECRET: abcdefpassword

        # deployments service uses signed URLs, hence it needs to access
        # storage-proxy using exactly the same name as devices will; if
        # devices will access storage using https://s3.acme.org:9000, then
        # set this to https://s3.acme.org:9000
        DEPLOYMENTS_AWS_URI: https://ec2-12-23-34-45.eu-west-1.compute.amazonaws.com:9000
    logging:
        options:
            max-file: "10"
            max-size: "50m"

minio:
    environment:
        # access key
        MINIO_ACCESS_KEY: mender-deployments
        # secret
        MINIO_SECRET_KEY: abcdefpassword
    volumes:
        # mounts a docker volume named `mender-artifacts` as /export directory
        - mender-artifacts:/export:rw

mender-mongo:
    volumes:
        - mender-db:/data/db:rw
mender-elasticsearch:
    volumes:
        - mender-elasticsearch-db:/usr/share/elasticsearch/data:rw
        - ./conductor/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
mender-redis:
    volumes:
        - mender-redis-db:/var/lib/redis:rw
        - ./conductor/redis/redis.conf:/etc/redis/redis.conf
        - ./conductor/redis/entrypoint.sh:/redis/entrypoint.sh
    entrypoint: /redis/entrypoint.sh

volumes:
# mender artifacts storage
mender-artifacts:
external:
# use external volume created manually
name: mender-artifacts
# mongo service database
mender-db:
external:
# use external volume created manually
name: mender-db
# elasticsearch database
mender-elasticsearch-db:
external:
# use external volume created manually
name: mender-elasticsearch-db
# redis database
mender-redis-db:
external:
# use external volume created manually
name: mender-redis-db

5

Hi @leonsgithub. I think the question is related to the shell environment where you invoked keygen. It assumes there is a shell variable called API_GATEWAY_DOMAIN_NAME which will be used in the generation of the keys. I don’t think there is anything in the prod.yml related to that. Can you regenerate the keys ensuring that you set that shell variable appropriately?

Drew

6

[ Solved ]
rm -rf keys-generated/certs/*

before you trigger the keygen script:
CERT_API_CN=$API_GATEWAY_DOMAIN_NAME CERT_STORAGE_CN=$STORAGE_PROXY_DOMAIN_NAME …/keygen