Down a rabbit hole: The importance of CI/CD

Originally posted by Ed Vielmetti on August 10 2020

Open-source developers working with Arm have to go down a lot of rabbit holes, including the intricate (and often unforeseen) dependencies and interdependencies that can pop up during a build.

With build systems, the thing you do first is in almost every system is compute dependency analysis. Then once you have some understanding of all there is to do, you decide what tasks to perform in parallel — especially on multiprocessor systems. 

The problem is, what to do first in a complex environment is not always obvious. What you need for success is an evolved approach — one that does not depend on proof of concept (POC), but that focuses on repeatable success and scalability.

Once things work, if you want them to rinse and repeat, then you enter into a world of CI/CD. Here, not only does the project have to work during POC, but it also has to continue to work over time for your architecture. 

I have collaborated on more than 100 projects, some of the CI/CD challenges specific to Arm. Others, that I have run into include:

• Lack of native support
• imperfect emulation in QEMU
• complexity of cluster build environments
• a poor understanding of dependencies up and down the chain

Did someone say rabbit?

Pulling a rabbit out of your hat can get tricky to say the least.

When I think of rabbits, I think of creatures that are fast and quick to multiply. Fast-growing active projects are similar in nature, and when combined with a finite amount of resources can lead to endless tunneling in deep, dark rabbit holes. 

These can be overcome with much diligence and effort (so much digging!) — plus a few juicy carrots (read: tools that help make the journey more enjoyable) to enable fast-path growth and help make solving all those build problems much easier. 

Here are five of my favorite carrots for accelerating tricky projects:

• Use modern tooling that supports multi-architecture builds directly. For example, Docker’s buildx lets you build multi-arch images, link them together with a manifest file, and push them all to a registry using a single command. Artur Klauser has a nice tutorial on building multi-architecture Docker images with buildx that he published to Medium earlier this year.
• Keep your Dockerfiles under control with hadolint, a smart lint tool that helps you keep your code within standard best practices. Hadolint will warn you when your Dockerfile uses constructions that are ambiguous, risky, or unportable, and keeping it happy will keep you happy in the end. Bas Harenslack wrote an introduction to Linting your Dockerfile with Hadolint for Go Data Driven that is a good place to start.
• Another lint tool for your build system is shellcheck, an open-source static analysis tool that automatically finds bugs in your shell scripts. Debugging shell code can be awkward because of its venerable syntax, and shellcheck remembers for you the various quoting conventions necessary to avoid surprises. Andy Crouch has a useful review of shellcheck, showing some common errors that it catches. 
• Take advantage of hosted build systems that have native support for multiple architectures, like Drone Cloud and Travis CI. It is usually faster and more reliable to build and test your code on native hardware than to rely on emulation. (For extra credit, read more about Drone in A Blank Check for Open Source on the Packet blog, and read more on building on multiple CPU architectures from Travis.)
• Learn about NixOS, a purely functional operating system that has a build system (nixpkgs) with extensive caching, reproducible results, and fast and exact repeatability. NixOS deals with dependency conflicts by letting you pin every build to a precise description of what it depends on. Graham Christensen’s weblog is an ever-entertaining set of stories about NixOS, including his recent Erase Your Darlings about keeping your work environment in order by erasing it and recreating it, quickly, whenever you get started.

There is a lot to learn about build systems, and dozens of examples of good practices and innovative tools that I did not include here. If you have a favorite, please share it with me (ed@packet.com) to include in a future feature or on social media.