Glibc incompatibility in Nix builds 

Setup

The test machine used runs Debian 11 with kernel 5.10.0-35-amd64 and glibc 2.31.
Nix was installed on the machine with the latest stable nixpkgs channel:

$ sh <(curl --proto '=https' --tlsv1.2 -L https://nixos.org/nix/install) --daemon

Test Program

We will use the memfd_secret system call which was introduced in Linux kernel 5.14 and supported in glibc 2.35.

The test program acts differently whether the __NR_memfd_secret symbol is defined or not. This is a realistic scenario where one might want to enable specific optimizations based on features from more recent Kernel versions and skip these if the kernel functions are not available. 

#include <unistd.h>
#include <sys/syscall.h>
#include <unistd.h>
#include <stdio.h>

int main() {
    #ifndef __NR_memfd_secret
        printf("Doing stuff without memfd\n");
    #else
        printf("Doing stuff with memfd\n");
        long ret = syscall(__NR_memfd_secret, 0);
        if (ret == -1) {
            perror("syscall memfd_secret failed");
            return 1;
        } else {
        printf("syscall memfd_secret succeeded\n");
        }
    #endif

    return 0;
}

The execution should follow the first branch of the #ifndef condition since the system call is not available with our current kernel (not defined with our glibc).

Compiling with system's GCC

The execution is as expected:

$ /usr/bin/gcc main.c 
$ ./a.out 
Doing stuff without memfd

Building with Nix

Here is a default.nix file that defines a nix package for our test program. It builds main.c using tools from our installed nixpkgs:

with import <nixpkgs> {};

stdenv.mkDerivation {
  name = "test-program";
  src = ./.;
  buildPhase = ''
    $CC main.c -o test-program
  '';
  installPhase = ''
    mkdir -p $out/bin
    cp test-program $out/bin/
  '';
}

We get a runtime failure when running the nix package:

$ nix-build
$ ./result/bin/test-program 
Doing stuff with memfd
syscall memfd_secret failed
: Function not implemented

The binary was compiled against glibc 2.40 from Nix, which defines the __NR_memfd_secret symbol but our kernel version doesn't support the memfd_secret system call. We can inspect the binary with ldd:

$ ldd ./result/bin/test-program 
./result/bin/test-program: /lib64/ld-linux-x86-64.so.2: version `GLIBC_2.35' not found (required by /nix/store/cg9s562sa33k78m63njfn1rw47dp9z0i-glibc-2.40-66/lib/libc.so.6)
    linux-vdso.so.1 (0x00007ffcebefe000)
    libc.so.6 => /nix/store/cg9s562sa33k78m63njfn1rw47dp9z0i-glibc-2.40-66/lib/libc.so.6 (0x00007fbe42b20000)
    /nix/store/cg9s562sa33k78m63njfn1rw47dp9z0i-glibc-2.40-66/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fbe42d29000)

Detecting Undefined Symbols on Packages from the Nix Cache

Undefined symbols (marked U) can be listed using the nm tool. These symbols should be dynamically resolved at runtime. Here bash is downloaded from the nix cache and nm is run on the binary:

$ nix build nixpkgs#bash
$ nm result/bin/bash -u | head -n 15
nm: warning: ./bash: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0008002
nm: warning: ./bash: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
nm: warning: ./bash: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
                 U __asprintf_chk@GLIBC_2.8
                 U __ctype_b_loc@GLIBC_2.3
                 U __ctype_get_mb_cur_max@GLIBC_2.2.5
                 U __ctype_tolower_loc@GLIBC_2.3
                 U __ctype_toupper_loc@GLIBC_2.3
                 U __environ@GLIBC_2.2.5
                 U __errno_location@GLIBC_2.2.5
                 U __fdelt_chk@GLIBC_2.15
                 U __fprintf_chk@GLIBC_2.3.4
                 U __fpurge@GLIBC_2.2.5
                 w __gmon_start__
                 U __isoc23_strtoimax@GLIBC_2.38
                 U __isoc23_strtol@GLIBC_2.38
                 U __isoc23_strtoumax@GLIBC_2.38
                 U __libc_current_sigrtmax@GLIBC_2.2.5

Note the unsupported symbols and the "@GLIBC_2.38".

Short Conclusion

Nix is unsuitable for production use as it cannot be used to create portable binaries targeting older systems.

Packages from the binary cache are built using the current stable nixpkgs version: there is no guarantee that they will be built against the same glibc and kernel versions as the target system, resulting in potential runtime crashes.

Long Conclusion: Patching Nixpkgs to Use an Older Glibc

Glibc cannot be overriden like any other dependency using nixpkgs's regular overlay tools or override attributes. The glibc package is built as part of stdenv which is the result of nixpkgs' bootstrap process, the only way to override it is to build a custom stdenv from scratch.

The easy way would be to simply replace the current stdenv with the one from glibc 2.31-era, however, stdenv received major refactors over the years which makes it unusable as is. Even with a patched stdenv, using such an old tool kit causes a lot of packages to fail to build.

The other way is to patch nixpkgs directly, which makes it easier to patch individual failing packages later on. 

Patching Nixpkgs

In the following, we patch nixpkgs 25.05 to use glibc 2.31 and kernel 5.10.0-35.

Reasons for why nixpkgs did not choose this path are multiple:
- It adds a lot of complexity when it comes to keeping nixpkgs building and coherent
- In 2024, nix binary cache was about 770 TB in size, multiply by the number of glibc/kernel pairs you want to support 

Bootstrap Process

Glibc comes from the bootstrap process so we ideally only need to patch that. This involves:

- Replacing the glibc and linux headers built during the bootstrap process with the ones we are targeting
- Replacing the bootstrap files downloaded at stage 0 so we use coherent tools during the bootstrap process

The easiest way is to copy-paste-fix the old nixpkgs packages as you cannot simply edit version numbers (multiple patches might be applied during build for example).

We start encountering problems:

For example, 2020's glibc package (2.31) needs fixes to evaluate within nixpkgs 25.05:
- lib attribute was moved out of stdenv so any stdenv.lib needs to be changed to simply lib
- libgcc was added as an argument to glibc package so it needs to be added to the old package
- libpthread needs to be added to glibc package the same way as in current glibc else we get very hard-to-debug errors later on.

Other fixes may be needed such as adding --disable-werror to configure flags because GCC 14's defaults have changed to be more strict. 

If you did replace the initial bootstrap files, you might have to delete failing assertions such as "Detected Bash version that isn't supported by Nixpkgs".

Beyond stdenv

Now that you have a stdenv that actually builds, you may start building packages. There is no way to know whether a package will or will not build beforehand since we changed stdenv.

Here is a list of steps taken to patch nixpkgs to use glibc 2.28 (chronological order):

• bootstrap-files: use old files from f85b2a1
• glibc: [1/5] copy glibc 2.31 from older nixpkgs f85b2a1
• glibc: [2/5] fix nix errors: lib is no longer in stdenv.lib in recent nixpkgs
• glibc: [3/5] change version to 2.28-10 and remove failing patches
• glibc: [4/5] add libgcc to passthru
• glibc: [5/5] fix libpthread
• linuxHeaders: [1/2] copy linuxHeaders 4.19.16
• linuxHeaders: [2/2] fix nix errors
• stdenv: remove failing assertion when using Bash version lesser than 5
• binutils: [1/4] copy binutils 2.28 from nixpkgs a9eb3ee and fix nix errors
• binutils: [2/4] fix compilation with recent GCC
• binutils: [3/4] patch missing header
• binutils: [4/4] add plugin-api-header to passthru to fix libbfd build
• getent: get getent from stdenv.cc.libc
• libredirect: [1/2] add -ldl flag
• libredirect: [2/2] use old test file
• audit: [1/2] use version 2.8.5 (kernel incompatibility)
• audit: [2/2] fix for recent GCC (allow multiple definitions)
• gnutls: disable ktls (not available on older kernel)
• llvm: disable hanging tests
• linux-pam: patch to include stdio.h
• ffmpeg: don't use libopenmpt
• nix: make stable nix be 2.4

Again, this patch was made with specific packages (~2000) to build in mind (slapos, some python3 packages, gcc, ...), other packages might not build at all.

Conclusion

Nix packages cannot be used reliably to create portable binaries, unless nixpkgs is extended to support for glibc/kernel pairs.

With some work, we managed to patch an important subset of nixpkgs 25.05 to build against glibc 2.28 (2018), 2.31 (2020) and 2.35 (2022). The older the glibc, the harder it is to patch nixpkgs.

If nixpkgs adds support for any glibc/kernel pair, it would become suitable for portable binaries creation and production use but it will require additional development work and computing power to maintain.