Hello everyone,
I have installed Mender on my own server, I create an image from a raspbian one and manage to accept device on server.
However if I try to Decommission device it returns me following error
There was a problem decommissioning the device: cannot DELETE /api/management/v2/devauth/devices/5ca6332f6a029b000101d59e (500)… [Request ID: 7aab2882]
Is there something wrong with permissions?
Best regards
Which version of the server are you running?
Can you also post the output of docker ps to check the health of the different services.
Hello Mirzak,
I’m using latest version 2.0
root@MENDER:/usr/bin/mender-server# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1abfa46f8053 mendersoftware/api-gateway:1.6.0 "/entrypoint.sh" 3 hours ago Up 3 hours 0.0.0.0:443->443/tcp menderproduction_mender-api-gateway_1
c21e1e648237 mendersoftware/deviceauth:2.0.0b1 "/usr/bin/deviceauth…" 3 hours ago Up 3 hours 8080/tcp menderproduction_mender-device-auth_1
2e11165c6abc mendersoftware/mender-conductor:1.3.0b1 "/srv/start_conducto…" 3 hours ago Up 31 seconds 8080/tcp, 8090/tcp menderproduction_mender-conductor_1
61e85ee099ca mendersoftware/useradm:1.7.0 "/usr/bin/useradm --…" 3 hours ago Up 3 hours 8080/tcp menderproduction_mender-useradm_1
e527433ec589 mendersoftware/deployments:1.7.0b1 "/entrypoint.sh --co…" 3 hours ago Up 3 hours 8080/tcp menderproduction_mender-deployments_1
2dcdd5013f36 mendersoftware/inventory:1.6.0b1 "/usr/bin/inventory …" 3 hours ago Up 3 hours 8080/tcp menderproduction_mender-inventory_1
56ff4575e3a6 openresty/openresty:1.13.6.2-0-alpine "/usr/local/openrest…" 3 hours ago Up 3 hours 0.0.0.0:9000->9000/tcp menderproduction_storage-proxy_1
73aaab21e8f2 mendersoftware/gui:2.0.0b1 "/entrypoint.sh" 3 hours ago Up 3 hours 80/tcp menderproduction_mender-gui_1
7c2354c66da2 minio/minio:RELEASE.2018-09-25T21-34-43Z "/usr/bin/docker-ent…" 3 hours ago Up 3 hours (healthy) 9000/tcp menderproduction_minio_1
2b7e9a8ba074 mongo:3.4 "docker-entrypoint.s…" 3 hours ago Up 3 hours 27017/tcp menderproduction_mender-mongo_1
1e30624393d6 redis:5-alpine3.8 "/redis/entrypoint.sh" 3 hours ago Up 3 hours 6379/tcp menderproduction_mender-redis_1
f9a6f6d14f6a elasticsearch:5-alpine "/docker-entrypoint.…" 3 hours ago Restarting (1) 29 seconds ago menderproduction_mender-elasticsearch_1
Edit: @mirzak, prettified docker ps output
Mender 2.0 is still in beta. Can you reproduce this on Mender 1.7? It’d be interesting to know if it is a beta-specific issue.
I have hosted 2.0.0b1 setup and able to decommission device.
Based on your log @zumiani, these two services seem to be restarting/crashing,
2e11165c6abc mendersoftware/mender-conductor:1.3.0b1 "/srv/start_conducto…" 3 hours ago Up 31 seconds 8080/tcp, 8090/tcp menderproduction_mender-conductor_1
f9a6f6d14f6a elasticsearch:5-alpine "/docker-entrypoint.…" 3 hours ago Restarting (1) 29 seconds ago menderproduction_mender-elasticsearch_1
You can extract additional logs
https://docs.mender.io/1.7/troubleshooting/mender-server#container-logs
Which might provide additional insights on to why.
I’m going to install mender 1.7 instead of 2.0 and I’ll report results.
Thank you
I have installed version 1.7, following guide at
https://docs.mender.io/1.7/administration/production-installation
Deployments docker container continues to restart
root@VPS-CHECKPT-RASP:/usr/local/bin/mender-server-1.7/production# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
87fa5e48e757 mendersoftware/api-gateway:1.6.0 "/entrypoint.sh" 11 minutes ago Up About a minute 0.0.0.0:443->443/tcp menderproduction_mender-api-gateway_1
1e7dd5a0b058 mendersoftware/inventory:1.5.0 "/usr/bin/inventory …" 11 minutes ago Up About a minute 8080/tcp menderproduction_mender-inventory_1
602543fa3eb1 mendersoftware/deployments:1.6.0 "/entrypoint.sh --co…" 11 minutes ago Restarting (4) 18 seconds ago menderproduction_mender-deployments_1
909fafbf014a mendersoftware/deviceauth:1.7.0 "/usr/bin/deviceauth…" 11 minutes ago Up About a minute 8080/tcp menderproduction_mender-device-auth_1
63c942151f61 mendersoftware/useradm:1.7.0 "/usr/bin/useradm --…" 11 minutes ago Up About a minute 8080/tcp menderproduction_mender-useradm_1
f6cdebded239 mendersoftware/mender-conductor:1.2.0 "/srv/start_conducto…" 11 minutes ago Up 4 minutes 8080/tcp menderproduction_mender-conductor_1
a1f6aa69492e openresty/openresty:1.13.6.2-0-alpine "/usr/local/openrest…" 11 minutes ago Up About a minute 0.0.0.0:9000->9000/tcp menderproduction_storage-proxy_1
d3e4fe5f6016 mendersoftware/gui:1.7.0 "/entrypoint.sh" 11 minutes ago Up 4 minutes 80/tcp menderproduction_mender-gui_1
4dfe597fed3a minio/minio:RELEASE.2018-09-25T21-34-43Z "/usr/bin/docker-ent…" 11 minutes ago Up About a minute (healthy) 9000/tcp menderproduction_minio_1
b5b224d67bfe redis:3.2.11-alpine "/redis/entrypoint.sh" 11 minutes ago Up 4 minutes 6379/tcp menderproduction_mender-redis_1
76b8d661c1be mendersoftware/elasticsearch:2.4 "/docker-entrypoint.…" 11 minutes ago Up 4 minutes 9200/tcp, 9300/tcp menderproduction_mender-elasticsearch_1
68035912c932 mongo:3.4 "docker-entrypoint.s…" 11 minutes ago Up About a minute 27017/tcp menderproduction_mender-mongo_1
Error log from docker logs command
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2019-04-09T07:16:30Z" level=info msg="Deployments Service, version unknown starting up" file=main.go func=main.cmdServer line=103
time="2019-04-09T07:16:31Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=migrations.Migrate line=48
time="2019-04-09T07:16:31Z" level=info msg="migrating deployment_service" file=migrations.go func=migrations.MigrateSingle line=70
time="2019-04-09T07:16:31Z" level=info msg="migration to version 1.2.1 skipped" db="deployment_service" file="migrator_simple.go" func="migrate.(*SimpleMigrator).Apply" line=125
time="2019-04-09T07:16:31Z" level=info msg="DB migrated to version 1.2.1" db="deployment_service" file="migrator_simple.go" func="migrate.(*SimpleMigrator).Apply" line=140
time="2019-04-09T07:16:31Z" level=info msg="Deployments Service, version unknown starting up" file=main.go func=main.cmdServer line=123
InvalidAccessKeyId: The access key ID you provided does not exist in our records.
status code: 403, request id: 1593BD14B331CAD8, host id:Sorry… I missed MINIO_ACCESS_KEY and MINIO_SECRET_KEY
Decommission on 1.7 works like a charm
In order to user Update Modules available on 2.0 I switch back from 1.7 to 2.0.
I have a couple of problem with that:
There was a problem updating the device authorization status: cannot PUT /api/management/v2/devauth/devices/5caf32b95f919b00018f0c70/auth/5caf32b95f919b00018f0c71/status (500)... [Request ID: 0574f6c9]./run ps
menderproduction_mender-api-gateway_1 /entrypoint.sh Up 0.0.0.0:443->443/tcp
menderproduction_mender-conductor_1 /srv/start_conductor.sh Up 8080/tcp, 8090/tcp
menderproduction_mender-deployments_1 /entrypoint.sh --config /e … Up 8080/tcp
menderproduction_mender-device-auth_1 /usr/bin/deviceauth --conf … Up 8080/tcp
menderproduction_mender-elasticsearch_1 /docker-entrypoint.sh elas … Restarting
menderproduction_mender-gui_1 /entrypoint.sh Up 80/tcp
menderproduction_mender-inventory_1 /usr/bin/inventory --confi … Up 8080/tcp
menderproduction_mender-mongo_1 docker-entrypoint.sh mongod Up 27017/tcp
menderproduction_mender-redis_1 /redis/entrypoint.sh Up 6379/tcp
menderproduction_mender-useradm_1 /usr/bin/useradm --config … Up 8080/tcp
menderproduction_minio_1 /usr/bin/docker-entrypoint … Up (healthy) 9000/tcp
menderproduction_storage-proxy_1 /usr/local/openresty/bin/o … Up 0.0.0.0:9000->9000/tcp
`
time="2019-04-11T14:54:08Z" level=error msg="internal error: submit device decommissioning job error: failed to submit decommissioning job: Post http://mender-conductor:8080/api/workflow/decommission_device: dial tcp 172.18.0.5:8080: connect: connection refused" file=response_helpers.go func=rest_utils.RestErrWithLogMsg line=62 request_id=54bf4729-ce28-4e4c-ac0c-1f3317bb2f88 user_id=440b9415-d1b7-4615-ad6e-b550a8f21092
Does running the cleanup documented here help?
Database clean works: it removes devices from database.
However still error during Accept and Decommission.
During Accept following error appears
mender-device-auth_1 | time="2019-04-12T06:48:21Z" level=error msg="internal error: submit device provisioning job error: failed to submit provision device job: Post http://mender-conductor:8080/api/workflow/provision_device: dial tcp 172.18.0.5:8080: connect: connection refused" file=response_helpers.go func=rest_utils.RestErrWithLogMsg line=62 request_id=db351606-8776-4816-8053-97033ec90ec2 user_id=440b9415-d1b7-4615-ad6e-b550a8f21092
I tried setting up a 2.0.b1 server today, and I am not able to reproduce your issues.
You could also try a refresh of the containers,
./run pull
./run stop
./run rm
./run up -d
My installation is a brand new one.
So containers come from ./run pull command.
If it is possible to retrieve more logs from containers in order to debug this issue I’ll e glad to retrieve them and post to this conversation.
Thank you
Logs from conductor and elasticsearch could be useful.
These are logs file from elastic and conductor
Looks like your elasticsearch dies because of lack of memory and conductor doesn’t start because of unavailable elasticsearch. It tries to allocate 2Gb of RAM (from provided elasticsearch log: jvm_args: -Xms2g -Xmx2g) but you have only 0.96Gb available (MemTotal: 1015716 kB).
You need to change Xms and Xmx JVM params to for instance 512Mb or increase RAM on your server.
Thank you 0lmi,
where do I have to set those parameters?
Thank you