# Self hosted Mender - error during device decommission

**URL:** <https://hub.mender.io/t/self-hosted-mender-error-during-device-decommission/460>\
**Category:** General Discussions\
**Created:** [April 8, 2019, 9:18am UTC](https://hub.mender.io/t/self-hosted-mender-error-during-device-decommission/460 "2019-04-08T09:18:17Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![zumiani](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/zumiani/32/185_2.png) [@zumiani](https://hub.mender.io/u/zumiani)\
**Post date:** [April 8, 2019, 9:18am UTC](https://hub.mender.io/t/self-hosted-mender-error-during-device-decommission/460/1 "2019-04-08T09:18:17Z")

</div>

Hello everyone,

I have installed Mender on my own server, I create an image from a raspbian one and manage to accept device on server.  
However if I try to Decommission device it returns me following error

There was a problem decommissioning the device: cannot DELETE /api/management/v2/devauth/devices/5ca6332f6a029b000101d59e (500)… [Request ID: 7aab2882]

Is there something wrong with permissions?  
Best regards

---

<div class="post-metadata">

**Author:** ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)\
**Post date:** [April 8, 2019, 12:00pm UTC](https://hub.mender.io/t/self-hosted-mender-error-during-device-decommission/460/2 "2019-04-08T12:00:45Z")

</div>

Which version of the server are you running?

Can you also post the output of `docker ps` to check the health of the different services.

---

<div class="post-metadata">

**Author:** ![zumiani](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/zumiani/32/185_2.png) [@zumiani](https://hub.mender.io/u/zumiani)\
**Post date:** [April 8, 2019, 12:09pm UTC](https://hub.mender.io/t/self-hosted-mender-error-during-device-decommission/460/3 "2019-04-08T12:09:41Z")

</div>

Hello Mirzak,

I’m using latest version 2.0

```auto
root@MENDER:/usr/bin/mender-server# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1abfa46f8053 mendersoftware/api-gateway:1.6.0 "/entrypoint.sh" 3 hours ago Up 3 hours 0.0.0.0:443->443/tcp menderproduction_mender-api-gateway_1
c21e1e648237 mendersoftware/deviceauth:2.0.0b1 "/usr/bin/deviceauth…" 3 hours ago Up 3 hours 8080/tcp menderproduction_mender-device-auth_1
2e11165c6abc mendersoftware/mender-conductor:1.3.0b1 "/srv/start_conducto…" 3 hours ago Up 31 seconds 8080/tcp, 8090/tcp menderproduction_mender-conductor_1
61e85ee099ca mendersoftware/useradm:1.7.0 "/usr/bin/useradm --…" 3 hours ago Up 3 hours 8080/tcp menderproduction_mender-useradm_1
e527433ec589 mendersoftware/deployments:1.7.0b1 "/entrypoint.sh --co…" 3 hours ago Up 3 hours 8080/tcp menderproduction_mender-deployments_1
2dcdd5013f36 mendersoftware/inventory:1.6.0b1 "/usr/bin/inventory …" 3 hours ago Up 3 hours 8080/tcp menderproduction_mender-inventory_1
56ff4575e3a6 openresty/openresty:1.13.6.2-0-alpine "/usr/local/openrest…" 3 hours ago Up 3 hours 0.0.0.0:9000->9000/tcp menderproduction_storage-proxy_1
73aaab21e8f2 mendersoftware/gui:2.0.0b1 "/entrypoint.sh" 3 hours ago Up 3 hours 80/tcp menderproduction_mender-gui_1
7c2354c66da2 minio/minio:RELEASE.2018-09-25T21-34-43Z "/usr/bin/docker-ent…" 3 hours ago Up 3 hours (healthy) 9000/tcp menderproduction_minio_1
2b7e9a8ba074 mongo:3.4 "docker-entrypoint.s…" 3 hours ago Up 3 hours 27017/tcp menderproduction_mender-mongo_1
1e30624393d6 redis:5-alpine3.8 "/redis/entrypoint.sh" 3 hours ago Up 3 hours 6379/tcp menderproduction_mender-redis_1
f9a6f6d14f6a elasticsearch:5-alpine "/docker-entrypoint.…" 3 hours ago Restarting (1) 29 seconds ago menderproduction_mender-elasticsearch_1

```

Edit: @mirzak, prettified `docker ps` output

---

<div class="post-metadata">

**Author:** ![drewmoseley](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/drewmoseley/32/47_2.png) [@drewmoseley](https://hub.mender.io/u/drewmoseley)\
**Post date:** [April 8, 2019, 4:37pm UTC](https://hub.mender.io/t/self-hosted-mender-error-during-device-decommission/460/4 "2019-04-08T16:37:26Z")

</div>

Mender 2.0 is still in beta. Can you reproduce this on Mender 1.7? It’d be interesting to know if it is a beta-specific issue.

---

<div class="post-metadata">

**Author:** ![pei.cei](https://avatars.discourse-cdn.com/v4/letter/p/77aa72/32.png) [@pei.cei](https://hub.mender.io/u/pei.cei)\
**Post date:** [April 9, 2019, 6:38am UTC](https://hub.mender.io/t/self-hosted-mender-error-during-device-decommission/460/5 "2019-04-09T06:38:02Z")

</div>

I have hosted 2.0.0b1 setup and able to decommission device.

---

<div class="post-metadata">

**Author:** ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)\
**Post date:** [April 9, 2019, 6:42am UTC](https://hub.mender.io/t/self-hosted-mender-error-during-device-decommission/460/6 "2019-04-09T06:42:38Z")

</div>

Based on your log @zumiani, these two services seem to be restarting/crashing,

```auto
2e11165c6abc mendersoftware/mender-conductor:1.3.0b1 "/srv/start_conducto…" 3 hours ago Up 31 seconds 8080/tcp, 8090/tcp menderproduction_mender-conductor_1
f9a6f6d14f6a elasticsearch:5-alpine "/docker-entrypoint.…" 3 hours ago Restarting (1) 29 seconds ago menderproduction_mender-elasticsearch_1 

```

You can extract additional logs

[https://docs.mender.io/1.7/troubleshooting/mender-server#container-logs](https://docs.mender.io/1.7/troubleshooting/mender-server#container-logs)

Which might provide additional insights on to why.

---

<div class="post-metadata">

**Author:** ![zumiani](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/zumiani/32/185_2.png) [@zumiani](https://hub.mender.io/u/zumiani)\
**Post date:** [April 9, 2019, 6:49am UTC](https://hub.mender.io/t/self-hosted-mender-error-during-device-decommission/460/7 "2019-04-09T06:49:29Z")

</div>

I’m going to install mender 1.7 instead of 2.0 and I’ll report results.  
Thank you

---

<div class="post-metadata">

**Author:** ![zumiani](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/zumiani/32/185_2.png) [@zumiani](https://hub.mender.io/u/zumiani)\
**Post date:** [April 9, 2019, 7:31am UTC](https://hub.mender.io/t/self-hosted-mender-error-during-device-decommission/460/8 "2019-04-09T07:31:58Z")

</div>

I have installed version 1.7, following guide at  
[https://docs.mender.io/1.7/administration/production-installation](https://docs.mender.io/1.7/administration/production-installation)

Deployments docker container continues to restart

```
root@VPS-CHECKPT-RASP:/usr/local/bin/mender-server-1.7/production# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
87fa5e48e757 mendersoftware/api-gateway:1.6.0 "/entrypoint.sh" 11 minutes ago Up About a minute 0.0.0.0:443->443/tcp menderproduction_mender-api-gateway_1
1e7dd5a0b058 mendersoftware/inventory:1.5.0 "/usr/bin/inventory …" 11 minutes ago Up About a minute 8080/tcp menderproduction_mender-inventory_1
602543fa3eb1 mendersoftware/deployments:1.6.0 "/entrypoint.sh --co…" 11 minutes ago Restarting (4) 18 seconds ago menderproduction_mender-deployments_1
909fafbf014a mendersoftware/deviceauth:1.7.0 "/usr/bin/deviceauth…" 11 minutes ago Up About a minute 8080/tcp menderproduction_mender-device-auth_1
63c942151f61 mendersoftware/useradm:1.7.0 "/usr/bin/useradm --…" 11 minutes ago Up About a minute 8080/tcp menderproduction_mender-useradm_1
f6cdebded239 mendersoftware/mender-conductor:1.2.0 "/srv/start_conducto…" 11 minutes ago Up 4 minutes 8080/tcp menderproduction_mender-conductor_1
a1f6aa69492e openresty/openresty:1.13.6.2-0-alpine "/usr/local/openrest…" 11 minutes ago Up About a minute 0.0.0.0:9000->9000/tcp menderproduction_storage-proxy_1
d3e4fe5f6016 mendersoftware/gui:1.7.0 "/entrypoint.sh" 11 minutes ago Up 4 minutes 80/tcp menderproduction_mender-gui_1
4dfe597fed3a minio/minio:RELEASE.2018-09-25T21-34-43Z "/usr/bin/docker-ent…" 11 minutes ago Up About a minute (healthy) 9000/tcp menderproduction_minio_1
b5b224d67bfe redis:3.2.11-alpine "/redis/entrypoint.sh" 11 minutes ago Up 4 minutes 6379/tcp menderproduction_mender-redis_1
76b8d661c1be mendersoftware/elasticsearch:2.4 "/docker-entrypoint.…" 11 minutes ago Up 4 minutes 9200/tcp, 9300/tcp menderproduction_mender-elasticsearch_1
68035912c932 mongo:3.4 "docker-entrypoint.s…" 11 minutes ago Up About a minute 27017/tcp menderproduction_mender-mongo_1

```

Error log from docker logs command

```
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2019-04-09T07:16:30Z" level=info msg="Deployments Service, version unknown starting up" file=main.go func=main.cmdServer line=103
time="2019-04-09T07:16:31Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=migrations.Migrate line=48
time="2019-04-09T07:16:31Z" level=info msg="migrating deployment_service" file=migrations.go func=migrations.MigrateSingle line=70
time="2019-04-09T07:16:31Z" level=info msg="migration to version 1.2.1 skipped" db="deployment_service" file="migrator_simple.go" func="migrate.(*SimpleMigrator).Apply" line=125
time="2019-04-09T07:16:31Z" level=info msg="DB migrated to version 1.2.1" db="deployment_service" file="migrator_simple.go" func="migrate.(*SimpleMigrator).Apply" line=140
time="2019-04-09T07:16:31Z" level=info msg="Deployments Service, version unknown starting up" file=main.go func=main.cmdServer line=123
InvalidAccessKeyId: The access key ID you provided does not exist in our records.
        status code: 403, request id: 1593BD14B331CAD8, host id:
```

---

<div class="post-metadata">

**Author:** ![zumiani](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/zumiani/32/185_2.png) [@zumiani](https://hub.mender.io/u/zumiani)\
**Post date:** [April 9, 2019, 7:40am UTC](https://hub.mender.io/t/self-hosted-mender-error-during-device-decommission/460/9 "2019-04-09T07:40:16Z")

</div>

Sorry… I missed MINIO\_ACCESS\_KEY and MINIO\_SECRET\_KEY

---

<div class="post-metadata">

**Author:** ![zumiani](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/zumiani/32/185_2.png) [@zumiani](https://hub.mender.io/u/zumiani)\
**Post date:** [April 9, 2019, 8:52am UTC](https://hub.mender.io/t/self-hosted-mender-error-during-device-decommission/460/10 "2019-04-09T08:52:38Z")

</div>

Decommission on 1.7 works like a charm

---

<div class="post-metadata">

**Author:** ![zumiani](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/zumiani/32/185_2.png) [@zumiani](https://hub.mender.io/u/zumiani)\
**Post date:** [April 11, 2019, 12:30pm UTC](https://hub.mender.io/t/self-hosted-mender-error-during-device-decommission/460/11 "2019-04-11T12:30:48Z")

</div>

In order to user Update Modules available on 2.0 I switch back from 1.7 to 2.0.  
I have a couple of problem with that:

1. Accept device returns me an error  
`There was a problem updating the device authorization status: cannot PUT /api/management/v2/devauth/devices/5caf32b95f919b00018f0c70/auth/5caf32b95f919b00018f0c71/status (500)... [Request ID: 0574f6c9]`  
I think it is related also to Decommission issue
2. elasticsearch docker continuosly restart  
`./run ps`

## ` Name Command State Ports

menderproduction\_mender-api-gateway\_1 /entrypoint.sh Up 0.0.0.0:443-\>443/tcp  
menderproduction\_mender-conductor\_1 /srv/start\_conductor.sh Up 8080/tcp, 8090/tcp  
menderproduction\_mender-deployments\_1 /entrypoint.sh --config /e … Up 8080/tcp  
menderproduction\_mender-device-auth\_1 /usr/bin/deviceauth --conf … Up 8080/tcp  
menderproduction\_mender-elasticsearch\_1 /docker-entrypoint.sh elas … Restarting  
menderproduction\_mender-gui\_1 /entrypoint.sh Up 80/tcp  
menderproduction\_mender-inventory\_1 /usr/bin/inventory --confi … Up 8080/tcp  
menderproduction\_mender-mongo\_1 docker-entrypoint.sh mongod Up 27017/tcp  
menderproduction\_mender-redis\_1 /redis/entrypoint.sh Up 6379/tcp  
menderproduction\_mender-useradm\_1 /usr/bin/useradm --config … Up 8080/tcp  
menderproduction\_minio\_1 /usr/bin/docker-entrypoint … Up (healthy) 9000/tcp  
menderproduction\_storage-proxy\_1 /usr/local/openresty/bin/o … Up 0.0.0.0:9000-\>9000/tcp  
`

---

<div class="post-metadata">

**Author:** ![zumiani](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/zumiani/32/185_2.png) [@zumiani](https://hub.mender.io/u/zumiani)\
**Post date:** [April 11, 2019, 2:55pm UTC](https://hub.mender.io/t/self-hosted-mender-error-during-device-decommission/460/12 "2019-04-11T14:55:00Z")

</div>

`time="2019-04-11T14:54:08Z" level=error msg="internal error: submit device decommissioning job error: failed to submit decommissioning job: Post http://mender-conductor:8080/api/workflow/decommission_device: dial tcp 172.18.0.5:8080: connect: connection refused" file=response_helpers.go func=rest_utils.RestErrWithLogMsg line=62 request_id=54bf4729-ce28-4e4c-ac0c-1f3317bb2f88 user_id=440b9415-d1b7-4615-ad6e-b550a8f21092`

---

<div class="post-metadata">

**Author:** ![drewmoseley](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/drewmoseley/32/47_2.png) [@drewmoseley](https://hub.mender.io/u/drewmoseley)\
**Post date:** [April 11, 2019, 5:17pm UTC](https://hub.mender.io/t/self-hosted-mender-error-during-device-decommission/460/13 "2019-04-11T17:17:13Z")

</div>

Does running the cleanup [documented here](https://docs.mender.io/2.0/troubleshooting/mender-server#cleaning-up-the-deviceauth-database-after-device-decommissioning) help?

---

<div class="post-metadata">

**Author:** ![zumiani](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/zumiani/32/185_2.png) [@zumiani](https://hub.mender.io/u/zumiani)\
**Post date:** [April 12, 2019, 6:51am UTC](https://hub.mender.io/t/self-hosted-mender-error-during-device-decommission/460/14 "2019-04-12T06:51:10Z")

</div>

Database clean works: it removes devices from database.  
However still error during Accept and Decommission.  
During Accept following error appears

`mender-device-auth_1 | time="2019-04-12T06:48:21Z" level=error msg="internal error: submit device provisioning job error: failed to submit provision device job: Post http://mender-conductor:8080/api/workflow/provision_device: dial tcp 172.18.0.5:8080: connect: connection refused" file=response_helpers.go func=rest_utils.RestErrWithLogMsg line=62 request_id=db351606-8776-4816-8053-97033ec90ec2 user_id=440b9415-d1b7-4615-ad6e-b550a8f21092`

---

<div class="post-metadata">

**Author:** ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)\
**Post date:** [April 12, 2019, 3:44pm UTC](https://hub.mender.io/t/self-hosted-mender-error-during-device-decommission/460/15 "2019-04-12T15:44:19Z")

</div>

I tried setting up a `2.0.b1` server today, and I am not able to reproduce your issues.

You could also try a refresh of the containers,

```auto
./run pull
./run stop
./run rm
./run up -d

```

---

<div class="post-metadata">

**Author:** ![zumiani](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/zumiani/32/185_2.png) [@zumiani](https://hub.mender.io/u/zumiani)\
**Post date:** [April 12, 2019, 4:03pm UTC](https://hub.mender.io/t/self-hosted-mender-error-during-device-decommission/460/16 "2019-04-12T16:03:15Z")

</div>

My installation is a brand new one.  
So containers come from `./run pull` command.  
If it is possible to retrieve more logs from containers in order to debug this issue I’ll e glad to retrieve them and post to this conversation.

Thank you

---

<div class="post-metadata">

**Author:** ![0lmi](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/0lmi/32/133_2.png) [@0lmi](https://hub.mender.io/u/0lmi)\
**Post date:** [April 15, 2019, 6:34am UTC](https://hub.mender.io/t/self-hosted-mender-error-during-device-decommission/460/17 "2019-04-15T06:34:29Z")

</div>

Logs from conductor and elasticsearch could be useful.

---

<div class="post-metadata">

**Author:** ![zumiani](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/zumiani/32/185_2.png) [@zumiani](https://hub.mender.io/u/zumiani)\
**Post date:** [April 15, 2019, 7:03am UTC](https://hub.mender.io/t/self-hosted-mender-error-during-device-decommission/460/18 "2019-04-15T07:03:23Z")

</div>

These are logs file from elastic and conductor

> **[TransferNow](https://www.transfernow.net/files/?utm_source=17gli193aq31&utm_medium=&utm_content=en)**
>
> Download your files now with TransferNow.net, the simple and free service for sending and sharing large files.

---

<div class="post-metadata">

**Author:** ![0lmi](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/0lmi/32/133_2.png) [@0lmi](https://hub.mender.io/u/0lmi)\
**Post date:** [April 15, 2019, 8:14am UTC](https://hub.mender.io/t/self-hosted-mender-error-during-device-decommission/460/19 "2019-04-15T08:14:07Z")

</div>

Looks like your elasticsearch dies because of lack of memory and conductor doesn’t start because of unavailable elasticsearch. It tries to allocate 2Gb of RAM (from provided elasticsearch log: `jvm_args: -Xms2g -Xmx2g`) but you have only 0.96Gb available (`MemTotal: 1015716 kB`).

You need to change `Xms` and `Xmx` JVM params to for instance 512Mb or increase RAM on your server.

---

<div class="post-metadata">

**Author:** ![zumiani](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/zumiani/32/185_2.png) [@zumiani](https://hub.mender.io/u/zumiani)\
**Post date:** [April 15, 2019, 9:39am UTC](https://hub.mender.io/t/self-hosted-mender-error-during-device-decommission/460/20 "2019-04-15T09:39:26Z")

</div>

Thank you 0lmi,  
where do I have to set those parameters?  
Thank you

[Next page](https://hub.mender.io/t/self-hosted-mender-error-during-device-decommission/460.md?page=2)