Mender demo server sudden slow and stuck

Hi @peter,

Yes, I was checking and CPU and Mem Usage in my laptop is going between 65- 90 % …

Hi @mzedel,

It returns the list of my 8 accepted devices from Mender server… That api u were asking isn’t it?

ok, thanks, thats pretty high! @nishad1092 does the browser consume lot of CPU and mem?

peter

Yes it does, around 65-85 % everytime I go to mender UI. Do you it has anything to do with its a self-signed certificate but not a trusted one.

Whats the limit of devices we can onboard to the mender server? If I could manage the database, how to get into the Mongo db (not just the container, that we have already discussed), Could you tell me once again please Peter.

Hey @nishad1092

I am sure that load generated by the browser has nothing to do with the certificate. I am also sure that it is not normal and that this is the main problem. the only thing to find out is: why it happens. You mentioned somewhere that you are seeing lots of requests in the web console. could you share with me the output of:

c=`docker ps | grep deviceauth | awk '{print($1);}'`
docker logs $c > deviceauth.log
c=`docker ps | grep inventory | awk '{print($1);}'`
docker logs $c > inventory.log

(You can send them over email)
There is no limit on the devices. You can get to the db with:

c=`docker ps | grep mongo: | awk '{print($1);}'`
docker exec -it  $c mongo

make a backup first, before playing there.
Maybe you can send the counts:

use deviceauth
db.devices.count()
use inventory
db.devices.count()

peter

1 Like

and one more thing, did you upgrade this installation? I mean: did you have an older version of Mender installed? what is the current version you have?

pg

Hi @peter,
I’m very sorry that I couldnt get back to you earlier., I was really busy now that we are closing the deployment date.

  1. I shall send you the output of those docker via mail today…

  2. I did the docker exec -it $c mongo, and I got inside the db, but I was thinking more of an interactive way to get inside db using Robo 3T or any studio tool (But still I couldnt since db is a container), if u have any way to connect this db running in the host (server) to a studio tool (Robo 3T which I use) in my laptop, it will be great…

  3. Once I get connected to the db, I can tell you the count and etc. But one thing I have made sure Peter, If I have 3 or 4 devices, the browser is smooth, UI has not sluggish, but when more devices onboard, then UI is sluggish and yes, I’ll also point out that the system CPU usage is above 65 - 80 percent from the Google chrome browser.

  4. Very importantly Peter, I’m still using 2.4.1b version. I will be backing up and upgrading to 2.4.1 today, I hope all data persist and everything gets better after upgrade.

I’ll be staying in update from this today…

Thanks for all of your support :slight_smile:

Hi @peter,

I have upgraded the mender server to 2.4.0 now, And its pretty smooth, It isnt sluggish, I really wish it stays like this, But a small issue again, I have accepted device and It is getting accepted but It isn’t showing in the device API,
https://server/api/management/v1/inventory/devices,
But I can see the same accepted device in this API:
https://server/api/management/v2/devauth/devices

It is indeed strange,because I can see all other devices which were approved before mender upgrade ?

Did the server got messed up ?

Hi @drewmoseley @peter @tranchitella

Its been working so good so far, But I’m facing slow and lags in the server since today,

I can only snip the image, sorry for that:

Mender page can’t be opened, it is slow and even after it comes, when i click device tab, it is not opening. I have around 30 devices.

Errors are accumulating, It is not able to fetch the devices I guess, It shows pending in the ‘Network’ console of chrome for many requests… and slowly it gets responded with status 200.

I use a 4 vCPUs, Azure server, with 16GB RAM,
Please suggest me a solution to this…

I’m using a General Purpose server from this category, tell me which would be best suited for a mender production server please:

  • General Purpose (Av2, B, DC, Dsv3, Dv3, DSv2, Dv2) – Balanced ratio of CPU and memory
  • Compute optimized (Fsv2, Fs, F) – High CPU
  • Memory optimized (Esv3, Ev3, M, GS, G, DSv2, Dv2) – High memory
  • Storage optimized (Lsv2, Ls) – High disk throughput and IO
  • GPU (NV, NVv2, NC, NCv2, NCv3, ND, Ndv2) – Specialized workloads for graphics or AI
  • High performance (H) – These are the fastest (and most expensive) VMs with high CPU and memory for complex computational problems

Hi @peter @drewmoseley

I have upgraded my server too, to a Memory Optimized one, specific to handle traffic and better memory.

Still mender server is extremely slow,
I get these from the UI inspect:

{error: Error: Response timeout of 10000ms exceeded
at g.a._timeoutError (https:/mender.eastus…, res: undefined}
authorized-devices.js:154 {error: Error: Response timeout of 10000ms exceeded
at g.a._timeoutError (https://mender.eastus…, res: undefined
at retrytimer.js:12
authorized-devices.js:154 {error: Error: Response timeout of 10000ms exceeded
at g.a._timeoutError (https://mender, res: undefined}
4helpers.js:96 failed to extract request id: TypeError: Cannot read property ‘body’ of undefined
at yi (helpers.js:90)
at retrytimer.js:12

I get : Page unresponsive alert popup and I had to close the page…

@merlin can you help here?

hello @nishad1092

so, you managed to get rid of 500 errors? or are they still visible?
can you run network diagnostic tools from your desktop (the machine you run browser from) to mender.eastus… host?
For example use this: https://iperf.fr/ you need to install it both on the server and on your desktop machine. Then on the server run:

iperf -s -p 6724 

and on the desktop run:

iperf -c mender.eastus.... -p 6724 -n 32M

peter