Mender demo server sudden slow and stuck

General Discussions
1

Hi,

I have been using Mender for couple of weeks now, suddenly Mender server GUI is stuck and slow, its staggering. No errors in the logs. Could you let me know how to resolve this.

Im running the server in a 2 cpus, 8GB RAM. So server requirements also good rgt? or is it less?

2

What does docker stats give you on the containers?

3

Hi

image
image1347×652 25.3 KB

Looks fine with here,

4

But the Server has gone extremly slow, Gui, if i click anything, it isn’t smooth anymore. DOnt knw what hapnd suddenly.

5

and free ?

6

image
image706×48 1.71 KB

7

Its a 4 CPU and 16 GB RAM server.

8

Can you open the developer tools in your browser and check in the network tab if there are any requests that get sent excessively?
Apart from that: how many devices are you serving in the environment and how are their update intervals configured?

9

Hi @mzedel,

I checked the network tab, It fine I guess, no excessive requests made, It makes request as per we view the tab. and I run only three devices and five releases. Its a demo server. It started behaving like this since yesterday. in the device tab, it is sending 8 requests every 3 seconds, maybe thats default for refreshing the device tab.

10

Hi @oleorhagen and @mzedel

The only thing which I found off was this logs of elastic search docker, but I couldn’t get to a conclusion to what I should infer from this log:

image
image1350×200 12.4 KB

11

Hi @mzedel , Do you know what has happened? Cuz Im not able to figure out the reason? Is this becuase of the demo server?

12

based on this information alone I’m not sure what to focus on unfortunately… maybe we can track it down :slight_smile:

  • does it slow down on all UI pages? specifically releases, settings & help pages should be independent from most updating related actions
  • the 8 requests every 3 seconds for three devices seem too much - details on these requests might help (maybe you can copy some of the response bodies and send them across)
  • some info on the device intervals… 3 demo devices with demo intervals should not cause problems at all…
13

Hi @mzedel, Yes it slows down the entire UI, every tab, releases, devices. But settings and help is fine, maybe its cuz is independent,

I can attach the request sent when I click device tab, (with three devices)

image
image955×449 24.7 KB

14

this looks quite normal, so apart from the other info can you get me the response body from the following calls:

  • count?status=accepted
  • devices?status=accepted&per_page=20&page=1

it might be possible that these two calls return different values - even though I can’t see any reason why this would happen on the device groups tab, let alone any of the other areas…

and for context: what browser version are you on + what Mender version are you running?

15

Hi @mzedel,

The mender version is 2.3.0 and Chrome browser is Version 83.0.4103.97.

response of count?status=accepted is
{“count”:3}

response of * devices?status=accepted&per_page=20&page=1 is:
[{“id”:“17a0c1dc-a425-4e79-bb51-87a068edba63”,“identity_data”:{“mac”:“3e:3e:44:dd:33:4a”},“status”:“accepted”,“decommissioning”:false,“created_ts”:“2020-06-10T11:51:36.896Z”,“updated_ts”:“2020-06-10T15:21:10.221Z”,“auth_sets”:[{“id”:“56c1101e-b5f1-424b-9ef7-99c5fcf3e3df”,“identity_data”:{“mac”:“3e:3e:44:dd:33:4a”},“pubkey”:"-----“ts”:“2020-06-10T11:51:36.904Z”,“status”:“accepted”}]}

and you gotto see this, I just went to Device tab and around 1000 request in couple of mins, simply getting accumulated, why?
image

16

Hi @mzedel,

Im facing another issue, One of my deployment is failed and current software I can see “Inconsistent”, after this Mender deployment isnt happening, deploy is stuck in ‘pending’.

17

@mzedel Do you knw why this is happenining?

18

The number of requests seems to be a reason for the slow down - this number of requests for this device count should maybe be achieved after an hour, not at all within minutes.
I can only imagine this is due to faulty authentication data on the devices - even though I don’t really know why this would be the case. But to say anything more specific I would need at least the full body for the requests, better yet the entire response. And the device intervals from the 3 devices.

Another potential slow down: if you have 3 virtual demo devices (the qemu client) this will slow down your system considerably.

19

Yes, It is indeed strange to me too @mzedel considering the fact Mender is realible.

Yes, the request are happening in few minutes, not hour. Im going to setup a new server of Demo version and see if its behaving this way again.

20

Hi @mzedel,
I reinstalled the Mender server, Now its all good and smooth. But a major query I have.,

So one of my deployment is failed\aborted, and therefore I understand the release became “Inconsistent”, and So I created a new artifact, But after it shows ‘Incoinsistent’, The device doesnt sync nor I cant deploy any further updates, Unless I restart the device manually. This cant happen in real scenario rgt, I cant go and reboot devices in production if the release fails. How to resolve this? or Is this how Mender works.?