A build system around nix-shell
I finally got my CI/CD pipeline moved over to Forgejo actions. As part of my migration, I had to learn a lot about Forgejo (since it’s my first time using Foregejo), but I also needed to learn a lot about how a nixos host works with Forgejo too. Mostly since I’m still new to NixOS and there’s a fairly steep learning curve.
As such, nothing here is “idiomatic” nix or Forgejo. It’s just what I managed to cobble together after several days of trial and error (which you’ll see reflected in my action run history - there was a lot of trial and error).
That said, I did get something working, and I’m pretty happy with it and just wanted to share it briefly.
Part 1: The Project
The project I chose to work on first is my most recent C project, which is basically a collection of functions, macros, etc. for me to use in other C projects. It has it’s own unit test library, syscall wrappers, threading primitives, allocators, etc. In short, it’s trying to be my own “standard” library that I can use to minimize my reliance on other standard (or platform-specific) libraries. It also adds in additional functionality that I find standard libraries match.
I chose this project as a starting point for a few reasons. Namely:
It uses C99 with no 3rd-party libraries outside of standard and OS-libraries (which it wraps), so it’s very simple to get a build environment setup.
I’ve done the most experiments with using nix-shell for setting up the build system in this project, so I already have a good starting point.
I’ve done a lot of tests to make sure that the code compiles with many different tool-chains. This means that if I run into any errors it’s most likely to be a configuration issue not a code issue.
I have minimal artifacts being produced right now (pretty much just documentation), which means I don’t have to setup an artifact repository quite yet. This helps narrow the scope of my experiment (and the work needed to get off the ground)
It’s a library not an application, so there’s no deploy pipeline to worry about which greatly reduces the scope of my experiment
My experiments in this project included using nix-shell to spin up different versions of GCC and Clang, as well as cross-compile for Windows and Linux on different architectures - and run those cross-compiled versions in virtual machines. So, I had a pretty good starting point to say the least. Before we go too far into the CI/CD part, let’s first dive into my starting point a little more.
Nix-Shell for local builds
I had three parts1 for my nix-shell builds:
A build.sh file that took in a lot of environment variables to tweak things as needed
A folder of *.nix files with instructions on how to run build.sh for every tool-chain and platform
A nix-build.sh file which ran through every one of my nix files in order and ran them
The build.sh file was a little complicated - not because building my library is complicated (it’s pretty much just compiling my peaksc.c file which then includes all my other .c files). Rather, it’s because I have code generation for my utility executables. This code generation handles things like turning text and data files into C-style byte arrays, or generating type-specialized data structures. These utilities then have their outputs used for further steps, like running additional tests or even as a prerequisite for the other utilities.
What makes this more complex is that not every one of my tool chains can reliably run these executables - or at least not as part of a “build” step. Emscripten has a lot of restrictions where it can’t directly access the disk but uses a virtual file system, at least with default build settings (due to how WASM works and the need to “expose” disk access through a JS API). This means that generating code files doesn’t really work since changes would be lost.
I also run into issues with Wine and MinGW since MinGW and Wine don’t like being loaded as part of the same nix-shell (from what I can tell it has something to do with the “crossSystem” config in my nix-shell file).
So, this means that my build.sh file needs to be able to not run the utilities. However, I still want to compile them to make sure that I didn’t break anything there (like use a linux-only API in a windows executable) - so I still need a way to get the generated files.
On my local machine, how I get those generated files is I simply run a non-emulated version of my generators before I run anything that can’t generate those files. That way they live on-disk already and I can just reuse them.
Overall, this system works great - although it’s really slow especially as configurations have grown a lot over time. It turns out running 9 different emulation layers (some for different versions of windows, others for different CPU architectures) and 12 non-emulated toolchains (including different sanitizers, older versions of compilers, and some esoteric compilers) just takes time. I am glad that I’m using C for doing this crazy experiment - trying to run all of these setups in C++ or Rust would be painfully slow!
The other benefit is that running just one tool-chain was really easy to do, just run “nix-shell nix/<toolchain>-shell.nix" and suddenly you had a reliable, reproducible run on that tool-chain - even if it needed emulation!
Part 2: Plan vs Reality
My initial plan was actually really simple. I have NixOS installed. I can simply setup a Forgejo runner with my host NixOS (I know it’s not “secure” - but this is self hosted for me with no one else able to contribute or trigger or login), and then I would just run my “nix-build.sh” script and be done. Simple, right?
Well, it would be, if there weren’t safety mechanisms built in where the runner actually ran in a virtual Nix environment and didn’t have access to the nix-shell command - which is what my entire local build system was built around. Argh.
Okay, new plan. I had two options:
Rebuild my entire build system so that I have all the tools globally available and I just run those specific tools
Figure out how to get nix-shell to work
Option 1 is what I’ve seen in most enterprise places I’ve worked at. The build system is setup fundamentally differently than the local environment. And it’s awful. Things work in the build that don’t work locally and vice versa. A true pain. Also, I don’t think it would be possible to do easily given that I have multiple versions of the same tool-chain. I don’t want to figure out how to keep gcc-9 and gcc-12 versions straight. With all of this, Option 1 felt like it wasn’t a real option - at least not for long-term success. So I crossed it off quickly.
So, Option 2 it is then. So, how do I get nix-shell to work?
Well, I tried a lot of things. Exposing programs, installing the “nix” package manager directly on the hidden VM, etc. Eventually, what I settled on was to not run on the host machine at all and instead run everything in a nixos docker container. That way it was a “clean” NixOS, not a nix VM in NixOS. Of course, the base nix container doesn’t quite have everything installed that Forgejo needs. Forgejo is built for NodeJS devs, and so they assume NodeJS is installed when they do a git checkout. This meant I had to update my run steps to install NodeJS before I did a checkout. Also, docker runs in headless mode by default, but Wine (and winecfg) don’t. Which caused some interesting issues. Fortunately, there’s xvfb-run which will stub out the GUI stuff so I can effectively run GUI apps in headless mode. But after all that, it worked!
Except, it was slow.
The hardware I’m using for my server isn’t bad, but it is older and energy efficient - so it’s not all that quick. Plus, now I was running inside a container, and the container was acting - funny. I hadn’t quite nailed it down yet (I would soon). Plus, everything was running in-serial, when it didn’t really have to.
Part 3: Optimizing
Fortunately, I’ve had to fix a lot of build systems at work. So I had a really good idea on what was going on, and how to fix it.
The first issue was everything was serial - which was needed for the dependencies between generating files. Well, that’s okay. First step is to simply break it from one command to a series of jobs to run each nix-shell, and then I can reorder, parallelize, or remove really slow bits that I don’t care about. But once I get it into jobs, it’ll be easier to work with.
So I did, and then very quickly realized what that funky behavior was I was seeing. Every job that couldn’t generate code was now failing. And this was because every job was getting a new docker container. It was never reusing a container - even between jobs. This was different than what I’m used to where a clean step was mandatory since the same git repo would be reused between runs.
Okay, so quick patch is to just run a job that can generate files in every step. No sweat.
Once that was done, I started parallelizing (which just meant spinning up new runners).
But, the fact I was regenerating files everywhere bugged me. I only really needed to generate once and download everywhere it was needed. So, I took a look at Forgejo artifacts, realized it was easy to upload and then download output between steps, and then proceeded to add uploading of generated sources followed by downloading of those sources everywhere it was needed. I then had a parallel build system with only a few sections in serial.
I then noticed something interesting: Forgejo runs in the order declared whenever it can. So I did one more trick, I put the blocking steps first. That way the blocked steps would be unblocked as quickly as possible.
And with that, I had my build system!
Where I’m headed
So far, I only have Linux and Windows support. And right now, Windows builds are only automated through MinGW - I still have to do MSVC builds manually. What I want to do is automated MSVC builds somehow (maybe through a VM or wine or native windows box, not sure yet).
Additionally, I want to add support for OSX. There are a few places that will need to be updated, but it shouldn’t be too hard. I have a Mac laying around somewhere - I just need to get it dusted off and add in the platform-specific wrappers where they’re needed.
After that, we’ll I’ll probably keep working on the library again. I have a lot of functionality I’m trying to add as I build my way up to an application. Right now I’m focused on adding in more testing functionality. I’ll then want to add a network stack and a window management system - which will bring oh so many levels of “fun” when trying to merge it with the current build pipeline.
At some point I’ll start adding support for more platforms. Arduino is pretty high on my list (I already have Raspberry Pi), as well as probably another micro-controller or single-board computer or two. Android and iOS support would be nice at some point, but I’m not really big into mobile development so it’s not a high priority for me yet. FreeBSD support would also be interesting - but again it’s not very high on my list. I’ve only used FreeBSD a handful of times, but that’s always been exclusively in a VM. I know that some work is needed since currently I have a few Linux-specific APIs instead of only generic POSIX APIs (yes, there’s a difference).
At some point, I’ll get more of my other projects moved over. So far I’m liking the nix-shell system, especially now that I’ve gotten a lot of the tool-chains working.
Technically, I also have a CMake setup. However, I mostly use that for IDE support (like CLion and Visual Studio). I do try to keep the build.sh and CMake systems in-sync. However, I chose not to build my nix-shell stuff around CMake since that’s just a lot more complexity, and honestly bash scripting this stuff isn’t that hard.