Have you ever considered where the I is in your Continuous Integration (CI) solution?
The challenge in CI is what it is you are continually doing. This is important because the integration is the key and a good manual integration setup is more effective than a poor continuous something-or-other.
Continuous Integration has been a part of software development for decades, but more recently has become a buzz word and hence a must for larger software teams or you may be seen as not doing software correctly! The unfortunate side effect can be that something is put in place, to ensure the implicit kpi is met, and it stops there.
The C in CI
The C in CI is simple, as continually doing something does not really involve much more than say configuring a Jenkins environment or online pipeline to kick off some scripts to build something. This is probably why often enough, teams develop a CI which is effectively a build setup to produce whatever monolithic blob of software or embedded image that is needed for the end product. This is important naturally, but if this is where the CI stops, then the integration is limited or non existent, and you have a continuous build (CB) setup rather than a continuous integration setup.
A fundamental problem with these monolithic builds, built regularly, often as nightly builds, is the ramifications of a single failure. If your build takes hours to perform, and a line of code in your build fails towards the end of the build, the build that a bunch of team members have been waiting for, to verify their integration, becomes effectively unusable, and is delayed by another day.
Hence trying to do your integration using these continuous builds eventually grind to a halt as a useful tool, and become a continuous annoyance and time eater rather than a useful tool to improve your development.
Some primary reasons for this to happen, would be because you need to have a monolithic build of some sort to make a release. So this is done, and once its done it is understandable that one thinks CI is done and no actual integration is preformed. Furthermore, anything monolithic allows easy addition of non modular components (for example through software with excessive dependency chains), so its ‘quick and easy’ to add, and before long untangling is too hard, so any attempt to modularise and improve integration is seen as too time consuming and the train continues along its path of least resistance.
The I in CI
Unlike the Continuous, unfortunately the Integration is more involved and does require some thinking and planning, not something all engineers are fond of or interested in. Some build systems, like the Yocto Project do lend themselves to helping you in achieving more modular integration setups, but it does require, skill, effort and the desire to actually do it, so even setups using advanced build systems can fall into the trap of not utilising well targeted integration.
The integration needs to be multi-fold, while targeting the end product, by splitting the device software into manageable sets of sub components which can work and be tested independently to the final system. They should also leverage off the individual software component tests, to ensure only integration testing is required in integration rather than dilution of these if software tests are lumped into the integration effort.
Put the I back in CI
So, taking a step back, for successful software development, including embedded development, you need these things.
- A build setup to generate your releases and system test images, with emphasis here on the word system. This does not need to be continuous, and probably should be less continuous and instead timed to suit effective iteration.
- A build setup for individual software components, with emphasis here on individual. This should be very often and very fast, and whenever possible for embedded device be performed on the development host rather than the embedded or emulated embedded device (or you risk losing the fast). This should also include effective unit and component tests.
- Multiple integration setups, simply because you can not effectively integrate if you do everything at once. This should involve:
3a. groups of components built and tested on the development host.
3b. groups of component built and tested on an emulated embedded device
3c. groups of components built and tested on a physical embedded device
Regular Iteration
We will call 1 and 2 Regular Iteration. These steps should be done at regular intervals, feed from individual software builds and into system builds. It is tempting to call this Continuous Iteration, just to put an annoying twist on CI :-), but RI is maybe a little less confusing.
Continuous Integration
Our last group, 3, will be what we are going to consider as Continuous Integration, with the integration fairly and squarely at the centre because the emphasis of this is to integrate in numerous ways to target specific feature sets in order to ensure a robust system release by early detection of issues before system testing.
The Why
The positive side effects of multiple integration setups are:
- Finding issues will be faster
- previously simple build issues would be found late in a slow large build, now most build issues will be found in fast individual small build steps
- Isolating issues will be easier
- with targeted integration setups, triaging of many issues will be limited to those components, and many subtle system errors will be ironed out at this stage
- Less time is spent building huge software blobs/releases
- as we have actively decided that building large blobs really does not have to happen all the time, we open resources for faster ways to find issues
- More time is available to test your huge software blob/release
- because we spend less time building large blobs, we open resources to test those blobs which are built at less regular intervals.
Yocto Project ® and all related marks and logos are trademarks of The Linux Foundation. This website is not, in any way, endorsed by the Yocto Project or The Linux Foundation.
