Error in launching containers

Hi,

I have set up the production mode of mender. These days it was fine. I had added around 500 simulated devices. After a week server had run out of disk memory. So, I tried to cleanup some docker images and stopped containers. But, after this when started the server with ./run up command I am getting this error. Help me understand what I have broken in my setup?

ERROR: for mender-elasticsearch Cannot create container for service mender-elasticsearch: open /var/lib/docker/volumes/mender-elasticsearch-db/_data: no such file or directory

ERROR: for minio Cannot start service minio: OCI runtime create failed: container_linux.go:345: starting container process caused “process_linux.go:430: container init caused “rootfs_linux.go:58:
mounting \”/var/lib/docker/volumes/mender-artifacts/_data\” to rootfs \"/var/lib/docker/overlay2/4f0b102a6c9d184bb3c1910c9c73bd55811aedfcd5f4ada5de496492df29e1d9/merged\" at \"/export\" caused \“stat /var/lib/docker/volumes/mender-artifacts/_data: no such file or directory\”"": unknown

Why /var/lib/docker/volumes/mender-redis-db/_data folder are not there? need to create manually? have removed accidentally?

Hi @pei.cei

Looks like you deleted docker volumes which was created during installation.

I’m able to reproduce this as the following:

# docker inspect menderproduction_mender-elasticsearch_1 | jq '.[].Mounts[].Source'
"/var/lib/docker/volumes/mender-elasticsearch-db/_data"
# docker stop menderproduction_mender-elasticsearch_1
menderproduction_mender-elasticsearch_1
# rm -rf /var/lib/docker/volumes/mender-elasticsearch-db/
# docker start menderproduction_mender-elasticsearch_1
Error response from daemon: OCI runtime create failed: container_linux.go:346: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/docker/volumes/mender-elasticsearch-db/_data\\\" to rootfs \\\"/var/lib/docker/overlay2/ef51ac8aab3da876f1066ef5357d652abbce4411db08d59ac759128044cfca1a/merged\\\" at \\\"/usr/share/elasticsearch/data\\\" caused \\\"stat /var/lib/docker/volumes/mender-elasticsearch-db/_data: no such file or directory\\\"\"": unknown

docker volume create commands from installation manual created folders in /var/lib/docker/volumes and if you manually delete some of them then the data are lost. If loosing data from mender-elasticsearch-db and mender-redis-db volumes is not critical but without data from mender-db and mender-artifacts volumes it’s impossible to restore previous service’s state.