I have set up the production mode of mender. These days it was fine. I had added around 500 simulated devices. After a week server had run out of disk memory. So, I tried to cleanup some docker images and stopped containers. But, after this when started the server with ./run up command I am getting this error. Help me understand what I have broken in my setup?
ERROR: for mender-elasticsearch Cannot create container for service mender-elasticsearch: open /var/lib/docker/volumes/mender-elasticsearch-db/_data: no such file or directory
ERROR: for minio Cannot start service minio: OCI runtime create failed: container_linux.go:345: starting container process caused “process_linux.go:430: container init caused “rootfs_linux.go:58:
mounting \”/var/lib/docker/volumes/mender-artifacts/_data\” to rootfs \"/var/lib/docker/overlay2/4f0b102a6c9d184bb3c1910c9c73bd55811aedfcd5f4ada5de496492df29e1d9/merged\" at \"/export\" caused \“stat /var/lib/docker/volumes/mender-artifacts/_data: no such file or directory\”"": unknown
Why /var/lib/docker/volumes/mender-redis-db/_data folder are not there? need to create manually? have removed accidentally?
Looks like you deleted docker volumes which was created during installation.
I’m able to reproduce this as the following:
# docker inspect menderproduction_mender-elasticsearch_1 | jq '.[].Mounts[].Source'
"/var/lib/docker/volumes/mender-elasticsearch-db/_data"
# docker stop menderproduction_mender-elasticsearch_1
menderproduction_mender-elasticsearch_1
# rm -rf /var/lib/docker/volumes/mender-elasticsearch-db/
# docker start menderproduction_mender-elasticsearch_1
Error response from daemon: OCI runtime create failed: container_linux.go:346: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/docker/volumes/mender-elasticsearch-db/_data\\\" to rootfs \\\"/var/lib/docker/overlay2/ef51ac8aab3da876f1066ef5357d652abbce4411db08d59ac759128044cfca1a/merged\\\" at \\\"/usr/share/elasticsearch/data\\\" caused \\\"stat /var/lib/docker/volumes/mender-elasticsearch-db/_data: no such file or directory\\\"\"": unknown
docker volume create commands from installation manual created folders in /var/lib/docker/volumes and if you manually delete some of them then the data are lost. If loosing data from mender-elasticsearch-db and mender-redis-db volumes is not critical but without data from mender-db and mender-artifacts volumes it’s impossible to restore previous service’s state.