30 Jun EOSS2023 – Test-driven development challenges

The trials and tribulations of test-driven development on real hardware

In our line of work, we very often find ourselves developing test systems for the hardware we develop for.
Even the most careful development practices sometimes result in bugs that won’t be detected – until you try to run your code on real hardware. For this reason, testing with actual hardware boards is essential.
A recent talk at the Embedded Open Source Conference in Prague (https://sched.co/1LcQH) also talked about this.
It seems the author and I concur: developing an integrated DevOps strategy involving hardware testing is really difficult to get right.

There really is a common thread for all companies developing a board farm for testing on hardware: these systems are fragile and require a lot of attention. The main point of his talk that I wish to highlight is about reliability. 99% reliability just isn’t enough. Not even close.

The pursuit of perfection

It would seem that a 99% success rate is really good for testing. If you pass 99 out of every 100 tests, what is there to worry about?
The problem is when you do a lot of tests, very often, these failures really start to add up quickly. A 1% failure rate can mean multiple failures per day in your test setup. So for all of these failures, who will look at them? Who will diagnose them?
Any unreliability in your test setup that leads to a false negative undermines the confidence of the developer that looked at the test. Soon, they will stop looking at the test results, expecting another hardware failure – which makes the tests pretty pointless.
In addition, any failure in a CI pipeline can grind your build and release process to a halt, so it’s really important to get on top of these failures and examine them. No developer likes to debug failures in the test setup itself (or bugs in the tests), so these really need to be made to an entirely different standard of reliability.

In general, I don’t like to pursue perfection in code. No code is ever really “perfect”. But the cost of imperfections in test systems is often too high. Don’t accept that 99% success rate. Push for 100%. Regardless of how much effort it takes.

If you too are struggling to verify software on a range of different hardware platforms, Mind can help! Contact Mind today!