@imort to be fully honest, I have no idea why you are facing these issues, but ECS is not a deployment set up we officially support. I would suggest you to try 2.7, and if Traefik is the issue, you can still use the old nginx-based API gateway with minor modifications to the configuration file to expose the new micro services available in 2.7
Sorry if it looks like I blaming anyone or expecting some free support in my specific case 
It’s not, I was really surprized by strange issue, while v2.3 deployment was relatively smooth, that’s all.
Working on bringing v2.7 running on ECS right now, I’ll update that thread later if you don’t mind.
Thank you for all your kind help!
@imort sure, let me know how it goes.
Well, 2.7 works as expected, just finished basic tests.
I’ll add a few pull requests, mostly updating config.yaml docs for services. There are services which using a default inventory address, and no sign of that in config.yaml. Has to look to the config.go to make it rolling in the end.
Thank you for all your help here!
@imort glad to read Mender 2.7 worked fined, ping me in the PRs so I can prioritize them.
Migrated from v2.3 to v2.7.
Find the cause of my workflow duplicate record ID issue btw: the race condition between create-artifact-worker and workflow-worker.
Solution: add --excluded-workflows=<all_workflow_queues> for create-artifact-worker.
Have issues after the migration though: no existing devices found after migration.
Tried,
$ deviceauth propagate-inventory
$ deviceauth propagate-inventory-statuses
$ deviceauth propagate-inventory-id-data
Now I can see rejected devices, that’s all.
A new device was accepted successfully: pending → accepted, all good.
I can see that old and new devices have different documents scheme in inventory.devices:
Old device: Mender old device record - Pastebin.com
New device: Mender new device inventory record - Pastebin.com
Can you please advice?
@imort the old devices should have been migrated by the inventory service when running migrations. Are you running the services in automigrate mode? If not, can you please run “inventory migrate” in the inventory container?
It’s on, and I can see it in the inventory container logs:
time=“2021-04-30T05:52:04Z” level=info msg=“migration to version 0.2.0 skipped” db=inventory file=migrator_simple.go func=“migrate.(*SimpleMigrator).Apply” line=125
time=“2021-04-30T05:52:04Z” level=info msg=“migration to version 1.0.0 skipped” db=inventory file=migrator_simple.go func=“migrate.(*SimpleMigrator).Apply” line=125
time=“2021-04-30T05:52:04Z” level=info msg=“migration to version 1.0.1 skipped” db=inventory file=migrator_simple.go func=“migrate.(*SimpleMigrator).Apply” line=125
time=“2021-04-30T05:52:04Z” level=info msg=“migration to version 1.0.2 skipped” db=inventory file=migrator_simple.go func=“migrate.(*SimpleMigrator).Apply” line=125
time=“2021-04-30T05:52:04Z” level=info msg=“DB migrated to version 1.0.2” db=inventory file=migrator_simple.go func=“migrate.(*SimpleMigrator).Apply” line=140
time=“2021-04-30T05:52:04Z” level=info msg=“Inventory Service, version unknown starting up”
time=“2021-04-30T05:52:04Z” level=info msg=“setting up prod middleware” file=middleware.go func=main.SetupMiddleware line=130
time=“2021-04-30T05:52:04Z” level=info msg=“listening on :8080”
time=“2021-04-30T05:52:04Z” level=info msg=“automigrate is ON, will apply migrations” file=migrations.go func=“mongo.(*DataStoreMongo).Migrate” line=100
time=“2021-04-30T05:52:04Z” level=info msg=“migrating inventory” file=migrations.go func=“mongo.(*DataStoreMongo).Migrate” line=106
time=“2021-04-30T05:52:04Z” level=info msg=“migrating inventory” file=migrations.go func=“mongo.(*DataStoreMongo).MigrateTenant” line=43
@imort were those “old device” records in the mongodb when the migrations ran the first time? Mender runs migration script only once, when it detects the migration has not been applied. If you restore/add records after the migration was executed, those records won’t be automatically migrated.
The migration responsible for updating the schema is 1.0.0, you could try removing the corresponding document in migration_info and run migrations again, but this could lead to undesired effects if you are mixing old-style and new-style documents in the devices collection.
See: inventory/migration_1_0_0.go at master · mendersoftware/inventory · GitHub
Rolled back to the MongoDB backup before migration, restarted all services:
time=“2021-04-30T06:28:52Z” level=info msg=“Set revision to 0 for 93 devices” file=migration_1_0_2.go func=“mongo.(*migration_1_0_2).Up” line=44
All documents in inventory.devices now have revision=0, can it be the problem?
Checked a device before and after, scheme was changed: attributes now have inventory-* and system-* keys inside.
Still no old devices in the UI (
Devices migrated from v2.3 to v2.7 didn’t have a few fields in the inventory.devices:
Added manually, device appears in the inventory.
Last check-in was before the migration, so looks like I’ll have to deal with deviceauth too.
Confirming: adding two keys above works.
Device appears in the inventory, Mender receiving updates, deployment works.
Thank you very much @tranchitella for all your help and patience here!
I hope this thread will help someone else later to save some time
@imort glad to know you solved your issues!