# Mender rootfs group update issue

**URL:** https://hub.mender.io/t/mender-rootfs-group-update-issue/2664
**Category:** General Discussions
**Created:** [October 13, 2020, 4:23pm UTC](https://hub.mender.io/t/mender-rootfs-group-update-issue/2664 "2020-10-13T16:23:57Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![anishmonachan](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/anishmonachan/32/802_2.png) [@anishmonachan](https://hub.mender.io/u/anishmonachan)
#### Post date: [October 13, 2020, 4:23pm UTC](https://hub.mender.io/t/mender-rootfs-group-update-issue/2664/1 "2020-10-13T16:23:57Z")

</div>

I am performing an ota rootfs update on the system. I have the following configuration. Six jetson tx2 mender clients are connected through a single network interface sharing the network. These mender clients are running on yocto based image.

I’m performing updates via [hosted.mender.io](http://hosted.mender.io) UI. Three devices updates are successful and remaining three updates are failed. I had to create another deployment to update failed updates and I have all of my jetsons updated. This is the constant behavior for all updates I have tried.

I couldn’t make a group update successful at one deployment.

---

<div class="post-metadata">

### Author: ![drewmoseley](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/drewmoseley/32/47_2.png) [@drewmoseley](https://hub.mender.io/u/drewmoseley)
#### Post date: [October 13, 2020, 5:55pm UTC](https://hub.mender.io/t/mender-rootfs-group-update-issue/2664/2 "2020-10-13T17:55:59Z")

</div>

Hi @anishmonachan all device updates are completely independent with Mender. We do support retries at the deployment level so that if a deployment fails, it will automatically be retried. We have found that with larger device fleets, it is likely there will be spurious failures due to power issues and such, requiring retries to get the entire fleet updated. That said, updating 6 devices should not have these kinds of issues. I suspect there is some other reason that the devices are failing to update. Do you have client logs (journalctl -u mender-client) from the failed deployments?

Drew

---

<div class="post-metadata">

### Author: ![anishmonachan](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/anishmonachan/32/802_2.png) [@anishmonachan](https://hub.mender.io/u/anishmonachan)
#### Post date: [October 16, 2020, 7:51am UTC](https://hub.mender.io/t/mender-rootfs-group-update-issue/2664/3 "2020-10-16T07:51:50Z")

</div>

There is not much information about the cause of failure with the mender logs. Today, all updates are failed while I reproduce the errors. Mender logs are below

> – Logs begin at Fri 2020-10-16 07:39:36 UTC. –  
> – Logs begin at Fri 2020-10-16 07:39:36 UTC, end at Fri 2020-10-16 07:44:54 UTC. –  
> Oct 16 07:39:39 j140-tx2 systemd[1]: Started Mender OTA update service.  
> Oct 16 07:39:44 j140-tx2 mender[4850]: time=“2020-10-16T07:39:44Z” level=info msg=“Loaded configuration file: /var/lib  
> /mender/mender.conf” module=config  
> Oct 16 07:39:44 j140-tx2 mender[4850]: time=“2020-10-16T07:39:44Z” level=info msg=“Loaded configuration file: /etc/men  
> der/mender.conf” module=config  
> Oct 16 07:39:44 j140-tx2 mender[4850]: time=“2020-10-16T07:39:44Z” level=info msg=“Mender running on partition: /dev/m  
> mcblk0p1” module=cli  
> Oct 16 07:39:44 j140-tx2 mender[4850]: time=“2020-10-16T07:39:44Z” level=info msg=“State transition: init [none] → in  
> it [none]” module=mender  
> Oct 16 07:39:44 j140-tx2 mender[4850]: time=“2020-10-16T07:39:44Z” level=info msg=“State transition: init [none] → id  
> le [Idle]” module=mender  
> Oct 16 07:39:44 j140-tx2 mender[4850]: time=“2020-10-16T07:39:44Z” level=info msg=“authorization data present and vali  
> d” module=mender  
> Oct 16 07:39:44 j140-tx2 mender[4850]: time=“2020-10-16T07:39:44Z” level=info msg=“State transition: idle [Idle] → ch  
> eck-wait [Idle]” module=mender  
> Oct 16 07:39:44 j140-tx2 mender[4850]: time=“2020-10-16T07:39:44Z” level=info msg=“State transition: check-wait [Idle]  
> → inventory-update [Sync]” module=mender  
> Oct 16 07:39:45 j140-tx2 mender[4850]: time=“2020-10-16T07:39:45Z” level=info msg=“State transition: inventory-update  
> [Sync] → check-wait [Idle]” module=mender  
> Oct 16 07:39:45 j140-tx2 mender[4850]: time=“2020-10-16T07:39:45Z” level=info msg=“State transition: check-wait [Idle]  
> → update-check [Sync]” module=mender  
> Oct 16 07:39:45 j140-tx2 mender[4850]: time=“2020-10-16T07:39:45Z” level=info msg=“Correct request for getting image f  
> rom: [https://s3.amazonaws.com](https://s3.amazonaws.com)” module=client\_update

############## Logs from UI

2020-10-16 07:40:03 +0000 UTC info: Running Mender version 2.2.0b1  
2020-10-16 07:40:03 +0000 UTC debug: handle update fetch state  
2020-10-16 07:40:03 +0000 UTC debug: status reported, response 204 No Content  
2020-10-16 07:40:04 +0000 UTC debug: Received fetch update response &{200 OK 200 HTTP/1.1 1 1 map[Accept-Ranges:[bytes] Content-Length:[806929408] Content-Type:[application/vnd.mender-artifact] Date:[Fri, 16 Oct 2020 07:40:04 GMT] Etag:[“fb0c2697919ac2684b5d6234d87575bd-77”] Expires:[Tue, 13 Oct 2020 13:51:06 GMT] Last-Modified:[Tue, 13 Oct 2020 13:41:07 GMT] Server:[AmazonS3] X-Amz-Id-2:[a4hOUccdvbluWmFqLdQ7gf9eU7vtTA/KsEaDhYRJX23YnTeGQkWq2TUoK5CxSzQ7/f0ru5OVkvg=] X-Amz-Request-Id:[A816970751C77AB3]] 0x40003606a0 806929408 false false map 0x400049a500 0x4000566d10}+  
2020-10-16 07:40:04 +0000 UTC info: State transition: update-fetch [Download\_Enter] → update-store [Download\_Enter]  
2020-10-16 07:40:04 +0000 UTC debug: handle update install state  
2020-10-16 07:40:04 +0000 UTC debug: status reported, response 204 No Content  
2020-10-16 07:40:04 +0000 UTC debug: Read data from device manifest file: device\_type=j140-tx2  
2020-10-16 07:40:04 +0000 UTC debug: Current manifest data: j140-tx2  
2020-10-16 07:40:04 +0000 UTC info: no public key was provided for authenticating the artifact  
2020-10-16 07:40:04 +0000 UTC info: Update Module path “/usr/share/mender/modules/v3” could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available  
2020-10-16 07:40:04 +0000 UTC debug: checking if device [j140-tx2] is on compatible device list: [j140-tx2]  
2020-10-16 07:40:04 +0000 UTC debug: installer: processing script: ArtifactReboot\_Enter\_50  
2020-10-16 07:40:04 +0000 UTC debug: installer: successfully read artifact [name: v5.2.0; version: 3; compatible devices: [j140-tx2]]  
2020-10-16 07:40:04 +0000 UTC debug: Active partition: /dev/mmcblk0p1  
2020-10-16 07:40:04 +0000 UTC debug: Detected inactive partition /dev/mmcblk0p30, based on active partition /dev/mmcblk0p1  
2020-10-16 07:40:04 +0000 UTC info: opening device /dev/mmcblk0p30 for writing  
2020-10-16 07:40:04 +0000 UTC debug: Open block-device for installing update of size: 3078619136  
2020-10-16 07:40:04 +0000 UTC debug: Device: /dev/mmcblk0p30 is a ubi device: false  
2020-10-16 07:40:04 +0000 UTC info: native sector size of block device /dev/mmcblk0p30 is 512, we will write in chunks of 1048576  
2020-10-16 07:40:04 +0000 UTC debug: Opening device: /dev/mmcblk0p30 for writing with flag: 2  
2020-10-16 07:41:58 +0000 UTC info: Running Mender version 2.2.0b1  
2020-10-16 07:41:58 +0000 UTC error: Update was interrupted in state: update-store  
2020-10-16 07:41:58 +0000 UTC info: Update Module path “/usr/share/mender/modules/v3” could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available  
2020-10-16 07:41:58 +0000 UTC info: State transition: init [none] → cleanup [Error]  
2020-10-16 07:41:58 +0000 UTC debug: transitioning to error state  
2020-10-16 07:41:58 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds  
2020-10-16 07:41:58 +0000 UTC debug: Handling Cleanup state  
2020-10-16 07:41:58 +0000 UTC info: State transition: cleanup [Error] → update-status-report [none]  
2020-10-16 07:41:58 +0000 UTC debug: statescript: timeout for executing scripts is not defined; using default of 1h0m0s seconds  
2020-10-16 07:41:58 +0000 UTC debug: handle update status report state  
2020-10-16 07:41:59 +0000 UTC debug: status reported, response 204 No Content  
2020-10-16 07:41:59 +0000 UTC debug: attempting to upload deployment logs for failed update

---

<div class="post-metadata">

### Author: ![anishmonachan](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/anishmonachan/32/802_2.png) [@anishmonachan](https://hub.mender.io/u/anishmonachan)
#### Post date: [October 16, 2020, 9:26am UTC](https://hub.mender.io/t/mender-rootfs-group-update-issue/2664/4 "2020-10-16T09:26:26Z")

</div>

## Update

I think it is due to timesync issue. I was restarting all mender client to force mender service to check for update. So mender service was starting before timesync. The mender update had been successful with normal `updatepollinginterval` behavior.

---

<div class="post-metadata">

### Author: ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)
#### Post date: [October 16, 2020, 9:30am UTC](https://hub.mender.io/t/mender-rootfs-group-update-issue/2664/5 "2020-10-16T09:30:57Z")

</div>

> [@anishmonachan](#):
>
> I was restarting all mender client to force mender service to check for update.

Doing this during an update cycle will cause the Mender client to think that it got interrupted during the deployment (e.g same as a powerless/reset of device).

This is also the error you see:

```auto
2020-10-16 07:41:58 +0000 UTC error: Update was interrupted in state: update-store

```

---

<div class="post-metadata">

### Author: ![anishmonachan](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/anishmonachan/32/802_2.png) [@anishmonachan](https://hub.mender.io/u/anishmonachan)
#### Post date: [October 16, 2020, 9:37am UTC](https://hub.mender.io/t/mender-rootfs-group-update-issue/2664/6 "2020-10-16T09:37:10Z")

</div>

Thank you @mirzak

---

<div class="post-metadata">

### Author: ![anishmonachan](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/anishmonachan/32/802_2.png) [@anishmonachan](https://hub.mender.io/u/anishmonachan)
#### Post date: [October 16, 2020, 11:17am UTC](https://hub.mender.io/t/mender-rootfs-group-update-issue/2664/7 "2020-10-16T11:17:08Z")

</div>

@mirzak just a follow-up question. From your reply I understood update fails at boot if we create deployment while the mender client is offline. So is that necessary to create deployment during the mender client is up? Mender has any solution to update at boot?
