# Slow deployment

**URL:** <https://hub.mender.io/t/slow-deployment/378>\
**Category:** General Discussions\
**Created:** [March 15, 2019, 5:13pm UTC](https://hub.mender.io/t/slow-deployment/378 "2019-03-15T17:13:15Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![mterwoord](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mterwoord/32/100_2.png) [@mterwoord](https://hub.mender.io/u/mterwoord)\
**Post date:** [March 15, 2019, 5:13pm UTC](https://hub.mender.io/t/slow-deployment/378/1 "2019-03-15T17:13:16Z")

</div>

Since we moved the mender server into a datacenter, deployments have been really slow. I applied the tactics described in [1], which make upgrade times acceptable. (1 to 1,5 hour update time, mender file is about 80MB)

Today, we pushed the first update to a device in the field. The update completed in 4 minutes.  
Something must be wrong on our network, but we dont experience any other issues.

Can anyone suggest any things to investigate this?

[1] [Speed up mender upgrade](https://hub.mender.io/t/speed-up-mender-upgrade/268/12)

---

<div class="post-metadata">

**Author:** ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)\
**Post date:** [March 18, 2019, 7:58am UTC](https://hub.mender.io/t/slow-deployment/378/2 "2019-03-18T07:58:38Z")

</div>

I am bit confused here by these two lines,

> which make upgrade times acceptable. (1 to 1,5 hour update time

and

> The update completed in 4 minutes.

First you mention hours to complete an update and then 4 minutes? Could you clarify this.

---

<div class="post-metadata">

**Author:** ![mterwoord](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mterwoord/32/100_2.png) [@mterwoord](https://hub.mender.io/u/mterwoord)\
**Post date:** [March 18, 2019, 9:43am UTC](https://hub.mender.io/t/slow-deployment/378/3 "2019-03-18T09:43:14Z")

</div>

These are on different networks: At our office network, things take hours, at a customer network, its in 4 minutes.

I’m pretty sure its something on our network, but no clue on how to find out what…

I turned on debug logging of the mender service, but no clues there.

---

<div class="post-metadata">

**Author:** ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)\
**Post date:** [March 18, 2019, 9:48am UTC](https://hub.mender.io/t/slow-deployment/378/4 "2019-03-18T09:48:05Z")

</div>

> I turned on debug logging of the mender service, but no clues there.

I would just try fetching an artifact with “wget” from the device, if it is still slow then Mender can be eliminated and could be “general network” issue.

Could you try that?

---

<div class="post-metadata">

**Author:** ![mterwoord](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mterwoord/32/100_2.png) [@mterwoord](https://hub.mender.io/u/mterwoord)\
**Post date:** [March 18, 2019, 10:01am UTC](https://hub.mender.io/t/slow-deployment/378/5 "2019-03-18T10:01:48Z")

</div>

How do I get the URL to use? I guess the mender server is secured for unauthorized downloading?

---

<div class="post-metadata">

**Author:** ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)\
**Post date:** [March 18, 2019, 10:04am UTC](https://hub.mender.io/t/slow-deployment/378/6 "2019-03-18T10:04:44Z")

</div>

You can check the Mender client log, the URL that is generated is printed and is pre-signed URL and is usable for 24 hours.

---

<div class="post-metadata">

**Author:** ![mterwoord](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mterwoord/32/100_2.png) [@mterwoord](https://hub.mender.io/u/mterwoord)\
**Post date:** [March 23, 2019, 7:59am UTC](https://hub.mender.io/t/slow-deployment/378/7 "2019-03-23T07:59:48Z")

</div>

I did some assumption-checking:

- wget downloads as fast as I expected

- dd-ing the passive root partition takes 4 minutes

- ran wireshark on the traffic. (but due to https being used, i cannot see the contents) Not too much interesting stuff there, but I’m not too familiar with things

Any suggestions I could try to further pinpoint this?

---

<div class="post-metadata">

**Author:** ![mterwoord](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mterwoord/32/100_2.png) [@mterwoord](https://hub.mender.io/u/mterwoord)\
**Post date:** [March 23, 2019, 4:04pm UTC](https://hub.mender.io/t/slow-deployment/378/8 "2019-03-23T16:04:20Z")

</div>

Made some big breakthrough:

At our network, we use a pfsense-based firewall. THe underlying firewall software (packet fence) has an option to pre-filter tcp packets, for bad combinations of flags, for example. Disabling that functionality solves my speed issues…

Now imo, the big question is, what is mender doing to trigger this…

I could offer a wireshark trace of the data during a (slow) upgrade…

---

<div class="post-metadata">

**Author:** ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)\
**Post date:** [March 25, 2019, 7:45am UTC](https://hub.mender.io/t/slow-deployment/378/9 "2019-03-25T07:45:11Z")

</div>

Hard to provide anything specific about the why, but there is one difference between doing a `wget` and downloading with the Mender client and that is that the Mender client utilizes HTTP range requests. One of the reasons is to support resume of downloads in case of network interruption.

HTTP range requests is standard HTTP feature and I do not really think that this is causing the “red flags” but it is notable difference in how content is moved on the network.

---

<div class="post-metadata">

**Author:** ![mterwoord](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mterwoord/32/100_2.png) [@mterwoord](https://hub.mender.io/u/mterwoord)\
**Post date:** [March 25, 2019, 8:12am UTC](https://hub.mender.io/t/slow-deployment/378/10 "2019-03-25T08:12:50Z")

</div>

It seems the scrubbing is not something that’s really working great with tcp windows, which probably is being used.

See [http://openbsd-archive.7691.n7.nabble.com/Scrub-reassemble-tcp-td259581.html](http://openbsd-archive.7691.n7.nabble.com/Scrub-reassemble-tcp-td259581.html)
