NixOS and glibc

The NixOS distribution (on unstable) is really up to date and hip and cool… until a security bug in glibc hits. Because Nix packages are pure functions and different inputs result in different outputs, you can’t easily swap out glibc (even though it is dynamically linked at runtime) because, well, nearly everything depends on it. The NixOS build system needs to rebuild the world to get out a patch. And that seems to take at least a few days.

I asked around on #nixos and the friendly folk there pointed me to the nix-dev mailing list, where a workaround was posted, using system.replaceRuntimeDependencies, a NixOS option I wasn’t aware of.

Basically, after a complete build of NixOS, Nix can then traverse the dependency graph of a particular package and sed references to it with a different reference. In this case, the insecure glibc with a patched build. This version then outputs a new build of NixOS, with any binaries depending on glibc patched.

The mailing list solution is here. (It’s worth reading Russell O’Connor’s followup as well which explains in detail how this works.)

OK, great. I added that to my configuration.nix and ran nixos-rebuild switch and watched the new glibc compile and then lots of new derivations get built. My little chromebook barely has any free space so I frantically deleted stuff as my /nix/store doubled (more or less) in size. Which makes sense: any package that ultimately depends on glibc must be copied, modified, and given a new /nix/store build output.

Let’s try to trace this work in the store. First, find the patch:

ls /nix/store | grep cve-2015-7547

We can inspect this derivation if we’re interested, and make sure it references the github link to the patch:

I’ve edited the output a little so that it will fit in the page.

cat $patch | tr ',' '\n' | grep -C 3 github

Now let’s check what references this patch:

nix-store -q --referrers $patch

There’s the derivation for a patched glibc. And let’s see what the output for that derivation is:

nix-store -q --outputs $glibc_drv

So: there is our patched glibc. Our programs should all reference this glibc now, not other ones. I make sure, first, to reboot the machine to reload all running programs. Now let’s check lsof to make SURE this glibc is in use.

This command first lists all occurrences of a ** file being opened by a program, and then prints only the unique paths that are referenced. (By the way, I first checked for other versions of libc and found none, and including 2.21 in the grep removes other libc-related things.)

sudo lsof | grep | cut -c 89- | sort | uniq

There at the top of the output is our 5nd78r... glibc… and two others! So other programs are still referencing other (I’m assuming, unpatched) glibc libraries. Let’s try to figure out what programs have opened those:

for x in $(sudo lsof | grep | cut -c 89- | sort | uniq | xargs);
  do echo $x; echo "---------"; sudo lsof | grep $x | cut -c -12; echo;

[...truncating output, lots more here...]



Interesting. Most of the programs on my system reference the patched version of glibc; and then there’s: lsof itself (!!) which references another, and then emacs and gmain (??) which reference yet another glibc.

To do: figure out why that is, fix it.

Read NixOS and glibc part 2.