Just after earning my degree, I was fortunate to land a position in a small but solid R&D department as an embedded software engineer. My first assignment was to create a Unit Test project and have it run in our build pipeline after compilation. I must admit, I wasn’t more than enthusiastic about this task. To me, it seemed like busywork. I held the belief that Unit Tests are mostly useless throwaway code—I figured they likely wanted me to hold off on diving into production code right away. Which seems fair, I guess. As the newcomer, I had to respect the request of the more experienced engineers. So, for now, I thought I’d just go along with it, even if I felt it’s just a waste of time.
And so I went ahead with it. I put together the test suite and some basic tests, including a few that I brought over from our old, deprecated test framework to googletest. The test was set off by our build pipeline after compilation; it actually transferred the test binary to a physical device situated in our office, running the QNX operating system to conduct on-target tests. This was imperative due to the code’s usage of QNX primitives and the necessity to test it on that operating system. So, the test was a mix between a Unit Test and an Integration Test. The process functioned well and ran multiple times a day. Because the test coverage was quite limited and no one ever tampered with that code, it always passed—another reason why I considered it a waste of time. Plus, adding new tests was a rare occurrence. Writing Unit Tests for embedded firmware and hardware abstractions is quite challenging because you want to test the interaction with the hardware, which is something you can’t do in pure code. With the test running automatically and consistently passing, we soon put it out of mind and moved on to other matters.
Fast forward about a year. The test ran hundreds, if not thousands of times successfully. Talk about a waste of time. But then, one day, we started noticing test failures. Admittedly, not many, maybe three over the course of a few weeks. The test actually crashed with a Segmentation Fault, indicating a severe error. What caught our attention was that none of the code under test had changed. So, we definitely needed to look into it! I’ll spare you the details of the error search, but eventually, I was able to replicate the problem while a debugger was attached, which allowed me to grasp the entire context of the problem.
The problem was related to how our threading abstraction, which operated through inheritance, was utilized in the test framework. There is a base class that initializes the thread with a virtual function, and the user of the abstraction is supposed to override that function, kind of like this:
