Redis:5-alpine3.8 container failing during mender prod deployment

Hi,
Im trying to deploy mender prod server locally in my office dev network.
I followed the process mentioned in https://docs.mender.io/2.3/administration/production-installation.
All containers seems to be up except for redis:5-alpine3.8 which is restarting continuously and mendersoftware/mender-conductor:1.6.0 containers which keeps on checking health.

Log for redis

Log for mender-conductor

# this is a template file for production setup, consult
# https://docs.docker.com/compose/compose-file/ for details on syntax and usage
#
# Notes:
# - integration/docker-compose.yml file is assumed to be included
# - integration/docker-compose.storage.minio.yml is assumed to be included
# - all services are part of `mender` network (service names are unchanged)
# - keys and certificates are generated using keygen utility from integration
#   repository, keys and certificates are stored in ./keys-generated directory
# - certificates and key are mounted into containers using volumes
# - minio artifacts are stored in a named volume `mender-artifacts`; volume
#   needs to be created manually using `docker volume create mender-artifacts`

# related compose bugs:
# - https://github.com/docker/compose/issues/3874
# - https://github.com/docker/compose/issues/3568
# - https://github.com/docker/compose/issues/3219

version: '2.1'
services:

    mender-workflows-server:
        command: server --automigrate

    mender-create-artifact-worker:
        command: --automigrate

    mender-useradm:
        command: server --automigrate
        volumes:
            - ./production/keys-generated/keys/useradm/private.key:/etc/useradm/rsa/private.pem:ro
        logging:
            options:
                max-file: "10"
                max-size: "50m"

    mender-device-auth:
        command: server --automigrate
        volumes:
            - ./production/keys-generated/keys/deviceauth/private.key:/etc/deviceauth/rsa/private.pem:ro
        logging:
            options:
                max-file: "10"
                max-size: "50m"

    mender-inventory:
        command: server --automigrate
        logging:
            options:
                max-file: "10"
                max-size: "50m"

    mender-api-gateway:
        ports:
            # list of ports API gateway is made available on
            - "443:443"
        networks:
            - mender
        volumes:
            - ./production/keys-generated/certs/api-gateway/cert.crt:/var/www/mendersoftware/cert/cert.crt:ro
            - ./production/keys-generated/certs/api-gateway/private.key:/var/www/mendersoftware/cert/private.key:ro
        logging:
            options:
                max-file: "10"
                max-size: "50m"
        environment:
            ALLOWED_HOSTS: srsdc107557l601.tnext.loc

    storage-proxy:
        ports:
            # outside port mapping for artifact storage (note that storage-proxy listens on port 9000)
            - "9000:9000"
        networks:
            mender:
                aliases:
                    # change the alias to DNS name that storage will be
                    # available on, for instance if devices will access storage
                    # using https://s3.acme.org:9000, then set this to
                    # s3.acme.org
                    - srsdc107557l601.tnext.loc
        environment:

            # use nginx syntax for rate limiting, see
            # https://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate
            # Examples:
            #   1m - 1MB/s
            #   512k - 512kB/s
            DOWNLOAD_SPEED: 1m
            MAX_CONNECTIONS: 100
        volumes:
            - ./production/keys-generated/certs/storage-proxy/cert.crt:/var/www/storage-proxy/cert/cert.crt:ro
            - ./production/keys-generated/certs/storage-proxy/private.key:/var/www/storage-proxy/cert/private.key:ro
            - ./storage-proxy/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf

    mender-deployments:
        command: server --automigrate
        volumes:
            - ./production/keys-generated/certs/storage-proxy/cert.crt:/etc/ssl/certs/storage-proxy.crt:ro
        environment:
            STORAGE_BACKEND_CERT: /etc/ssl/certs/storage-proxy.crt
            # access key, the same value as MINIO_ACCESS_KEY
            DEPLOYMENTS_AWS_AUTH_KEY: mender-deployments
            # secret, the same valie as MINIO_SECRET_KEY
            DEPLOYMENTS_AWS_AUTH_SECRET: Virender4182020

            # deployments service uses signed URLs, hence it needs to access
            # storage-proxy using exactly the same name as devices will; if
            # devices will access storage using https://s3.acme.org:9000, then
            # set this to https://s3.acme.org:9000
            DEPLOYMENTS_AWS_URI: https://srsdc107557l601.tnext.loc:9000
        logging:
            options:
                max-file: "10"
                max-size: "50m"

    minio:
        environment:
            # access key
            MINIO_ACCESS_KEY: mender-deployments
            # secret
            MINIO_SECRET_KEY: Virender4182020
        volumes:
            # mounts a docker volume named `mender-artifacts` as /export directory
            - mender-artifacts:/export:rw

    mender-mongo:
        volumes:
            - mender-db:/data/db:rw
    mender-elasticsearch:
        volumes:
            - mender-elasticsearch-db:/usr/share/elasticsearch/data:rw
    mender-redis:
        volumes:
            - mender-redis-db:/var/lib/redis:rw
            - ./conductor/redis/redis.conf:/etc/redis/redis.conf
            - ./conductor/redis/entrypoint.sh:/redis/entrypoint.sh
        entrypoint: /redis/entrypoint.sh

volumes:
    # mender artifacts storage
    mender-artifacts:
      external:
          # use external volume created manually
          name: mender-artifacts
    # mongo service database
    mender-db:
      external:
          # use external volume created manually
          name: mender-db
    # elasticsearch database
    mender-elasticsearch-db:
      external:
          # use external volume created manually
          name: mender-elasticsearch-db
    # redis database
    mender-redis-db:
      external:
          # use external volume created manually
          name: mender-redis-db

Hi Vikram!

hello again! nice to hear from you.
could you please get me the output of:

cat ../conductor/redis/entrypoint.sh

from the place you run ./run up -d ?

peter

cat …/conductor/redis/entrypoint.sh

#!/bin/ash
redis-server /etc/redis/redis.conf

It should be #!/bin/bash above

No file at /etc/redis/redis.conf :neutral_face:


I have moved redis.conf file located at conductor/redis to /etc/redis
later when i ran ./entrypoint.sh in redis directory, i got below error that redis-server command not found

image

ok thanks, one more thing I would require:

docker inspect id-of-redis-container

pg

[
    {
        "Id": "48cff944cd633cc58edbb76a2e3400d6259f74e40e9b6dc7dcf86b9223b55003",
        "Created": "2020-04-16T18:12:59.411092889Z",
        "Path": "/redis/entrypoint.sh",
        "Args": [],
        "State": {
            "Status": "restarting",
            "Running": true,
            "Paused": false,
            "Restarting": true,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 1,
            "Error": "",
            "StartedAt": "2020-04-16T18:57:12.92949515Z",
            "FinishedAt": "2020-04-16T18:57:12.999161013Z"
        },
        "Image": "sha256:95f0c755feb65af441c0cb8727bb13b35517977a3bed06e4a3456c95d6a69dd4",
        "ResolvConfPath": "/DockerApps/dockerhome/containers/48cff944cd633cc58edbb76a2e3400d6259f74e40e9b6dc7dcf86b9223b55003/resolv.conf",
        "HostnamePath": "/DockerApps/dockerhome/containers/48cff944cd633cc58edbb76a2e3400d6259f74e40e9b6dc7dcf86b9223b55003/hostname",
        "HostsPath": "/DockerApps/dockerhome/containers/48cff944cd633cc58edbb76a2e3400d6259f74e40e9b6dc7dcf86b9223b55003/hosts",
        "LogPath": "/DockerApps/dockerhome/containers/48cff944cd633cc58edbb76a2e3400d6259f74e40e9b6dc7dcf86b9223b55003/48cff944cd633cc58edbb76a2e3400d6259f74e40e9b6dc7dcf86b9223b55003-json.log",
        "Name": "/menderproduction_mender-redis_1",
        "RestartCount": 53,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "mender-redis-db:/var/lib/redis:rw",
                "/home/vikram/mender-server/conductor/redis/entrypoint.sh:/redis/entrypoint.sh:rw",
                "/home/vikram/mender-server/conductor/redis/redis.conf:/etc/redis/redis.conf:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "menderproduction_mender",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "on-failure",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": [],
            "CapAdd": null,
            "CapDrop": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/asound",
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/DockerApps/dockerhome/overlay2/298807fee560282737d88e13c24a5e910dd23fe4854f372ed1c2d34c5f9c7eb7-init/diff:/DockerApps/dockerhome/overlay2/7240105b4b5bb75b5a34011760fcacabc3cd84ef0065c1246c117f496d29859c/diff:/DockerApps/dockerhome/overlay2/55e36f9cbb9d39e476e00df21bb87ee1d3439f621f176ae53fa12de539d34603/diff:/DockerApps/dockerhome/overlay2/55412f7bde47d724cc91c8fe9d5a273d8eeb50b4b15fba05f1631e008a7f40d2/diff:/DockerApps/dockerhome/overlay2/63d09b715549716e8c9580c78736f42cd694e3968d31b7d37b2b3df80bd1b50f/diff:/DockerApps/dockerhome/overlay2/4b4858664a966900617d9ebe1835353d4864c44166c811847f1389efbcfd95d6/diff:/DockerApps/dockerhome/overlay2/d13f11c139f59ba76419339da5ba2f562d9dda28cecd71e01007c7d3d00aee74/diff",
                "MergedDir": "/DockerApps/dockerhome/overlay2/298807fee560282737d88e13c24a5e910dd23fe4854f372ed1c2d34c5f9c7eb7/merged",
                "UpperDir": "/DockerApps/dockerhome/overlay2/298807fee560282737d88e13c24a5e910dd23fe4854f372ed1c2d34c5f9c7eb7/diff",
                "WorkDir": "/DockerApps/dockerhome/overlay2/298807fee560282737d88e13c24a5e910dd23fe4854f372ed1c2d34c5f9c7eb7/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/home/vikram/mender-server/conductor/redis/entrypoint.sh",
                "Destination": "/redis/entrypoint.sh",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/home/vikram/mender-server/conductor/redis/redis.conf",
                "Destination": "/etc/redis/redis.conf",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "e92dbbee354e0f7e1185095e8597e903f51c08fec425f73ce7a5b6cebe0185f6",
                "Source": "/DockerApps/dockerhome/volumes/e92dbbee354e0f7e1185095e8597e903f51c08fec425f73ce7a5b6cebe0185f6/_data",
                "Destination": "/data",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "mender-redis-db",
                "Source": "/DockerApps/dockerhome/volumes/mender-redis-db/_data",
                "Destination": "/var/lib/redis",
                "Driver": "local",
                "Mode": "rw",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "48cff944cd63",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "6379/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "REDIS_VERSION=5.0.3",
                "REDIS_DOWNLOAD_URL=http://download.redis.io/releases/redis-5.0.3.tar.gz",
                "REDIS_DOWNLOAD_SHA=e290b4ddf817b26254a74d5d564095b11f9cd20d8f165459efa53eb63cd93e02"
            ],
            "Cmd": null,
            "Image": "redis:5-alpine3.8",
            "Volumes": {
                "/data": {},
                "/etc/redis/redis.conf": {},
                "/redis/entrypoint.sh": {},
                "/var/lib/redis": {}
            },
            "WorkingDir": "/data",
            "Entrypoint": [
                "/redis/entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "bf0b3fbd010cd66129786fb8f5e2c2d946d0c24645346ff4d7c32f69a34a1aba",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "menderproduction",
                "com.docker.compose.service": "mender-redis",
                "com.docker.compose.version": "1.24.0"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "0136210d077ec696b7c56c8eff654c0f721a2d7311eef35130643ccf6c2354de",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/0136210d077e",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "menderproduction_mender": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "48cff944cd63",
                        "mender-redis"
                    ],
                    "NetworkID": "3ef64d1e19e41ab2a9c693320191fdf1b4ad0f9ebb1c7a38788327805b653fa4",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]

Also i think redis-server is not installed in my server.
Also this server has different docker home location which is /DockerApps

You should not need local redis installation, and since docker works, I would put that on a side for now.
Could we try to run docker redis container yb hand using:

docker run -v mender-redis-db:/var/lib/redis -v /home/vikram/mender-server/conductor/redis/redis.conf:/etc/redis/redis.conf -v /home/vikram/mender-server/conductor/redis/entrypoint.sh:/redis/entrypoint.sh -it --entrypoint=/redis/entrypoint.sh redis:5-alpine3.8

peter

P.S.
I just edited the paths, /tmp was my local path, yours is /home/vikram/mender-server/

Hi Peter,
Its the same error again!

I would like to add some info here

  1. I didn’t use ./run pull to pull the images as both git and internet access is not available to the server, I pulled the images in my windows pc and pushed them into our own hosted repo and pulled those images back from repo into my dev server and tagged them properly according to compose file image names
  2. Python is not installed or path may be wrong. so while using ./run up -d or ./run pull,
    the initial commands in the run file may fail like:
    export INTEGRATION_VERSION=(git describe --tags --abbrev=0) export MENDER_ARTIFACT_VERSION=(…/extra/release_tool.py -g mender-artifact)
    export MENDER_VERSION=$(…/extra/release_tool.py -g mender)

Hope these are not an issue

I see. If I were you I would go to the machine which has the network access and all other things, and run:

docker save ready-and-correct-redis-image -o redis.tar

and then transfer the redis.tar to the server, and use:

docker load -i redis.tar

peter

One thing i found today is if i skip using git commands during the mender deployment we get this issue. I check this in one EC2 instance, when i skip using git commands like add and commit i got this issue. But if use git commands as well i didnt get this issue and all containers are up

Hello Peter,
I have build the whole project in EC2 with all the changes required with respect to my internal servers. Just copied this to my local servers and now all containers are up!!! :wink:

Hi Vikram,

good to hear you solved the issue.
as usual, reach out anytime.

peter