Hey, has someone integrated Mender with the end-to-end test automation framework Robot Framework already? It would be possible to interface with the Mender API using a custom keyword library.
Hi @fkromer,
Thanks for reaching out! Technically speaking, there shouldn’t be a major problem in running the a Robot test against Mender, but given the nature of OTA solutions, my first hunch would be that this would exclude the client, right? I mean, we definitely do exercise the API (don’t know which framework we use for it, but it also doesn’t matter), but the real challenge in OTA end to end testing is to run the client on a form of device, have it interact with the management backend, and then check several scenarios. Bonus points for involving real hardware devices.
So taking that into account, what would the value proposition be here, over other testing approaches?
Greetz,
Josef
Hi Josef,
I’ve not given enough context. Thanks for moderating this thread.
… the real challenge in OTA end to end testing is to run the client on a form of device, have it interact with the management backend, and then check several scenarios. Bonus points for involving real hardware devices.
So taking that into account, what would the value proposition be here, over other testing approaches?
In the end, it all boils down to those statements I fully agree with.
Context:
Whenever I join a new company or project I try to push system level acceptance test driven development (ATDD) forward or at least evaluate and mention what would be possible. The reason for this is to provide a lot of value for the company/project in the mid/long term. Acceptance/end-to-end test automation is a de-facto standard in non-cyber physical system development (data domain, web domain, …) actually. This thread is not about testing Mender itself. It’s about utilities to enable ATDD and system level acceptance test automation in the (OTA) software update context of distributed cyber physical systems using the Robot Framework (RF). As far as I know the RF is the most flexible test automation framework available for distributed cyber physical systems.
Value reasoning (robot framework library for Mender):
In a staging environment including the overall system from a software update perspective (device with integrated mender client, mender, RF including libraries used for applying test config/input/output) it would be possible to fully automate the testing of (OTA) system software update features. The kind of RF libraries needed besides a RF mender library depends on the system at hand. There would not be an immediate value from Mender organization point of view. But there would be a value from company/project point of view: Test driving OTA integration into a system and securing against regressions after it has been integrated.
Cheers,
Florian