One Billion Objects and No Database: Inside cache.nixos.org

Amazon Web Services (AWS) has renewed its sponsorship for cache.nixos.org, the Nix public binary cache, enabling the operation of this foundational piece of infrastructure for another year.

AWS’s support started in 2023, when its Open Source Strategy and Marketing team, along with its Open Data team, stepped in when a previous sponsorship was discontinued. At this time, the growth of the Nix ecosystem was reaching an inflection point, outpacing the capacity to sustain it without additional support. The cache is depended on by Nix users worldwide, and its maintenance is therefore a high priority for the entire community. At the time, Ron Efroni, then a NixOS Foundation board member and now its president, in concert with Dave Nalley, director of developer experience at AWS, led a conversation that, in the span of around two weeks, resolved the concerns about the cache’s sustainability. AWS agreed to cover the cache’s hosting costs for 12 months — a substantial contribution which was renewed in 2024, 2025, and again for 2026.

Not just any package repository

Users most often encounter the cache as the authoritative package repository for both NixOS and the Nix package manager. Without a cache, every Nix client would have to build every package from source, which can take hours or even days and needs hardware many users do not have. Most Nix workflows would grind to a halt.

In serving prebuilt packages, the cache resembles the repositories maintained by Debian, Fedora, and Homebrew. For downloading and running software, the Nix archive (NAR) fetched from cache.nixos.org is similar to a .deb, .rpm, or Homebrew bottle. But underneath, the Nix cache differs fundamentally in its implementation, reflecting Nix’s radically different approach to software packaging.

Traditionally, installing software on Linux meant extracting the necessary files from a package archive and distributing them to conventional locations around the filesystem. While simple, this approach has led to a number of inconveniences. For example, when packages depend on conflicting versions of the same library, they cannot be installed at the same time, since the two versions’ files have to be placed at the same paths. This is a problem especially for developers, who routinely need to switch between different versions of their toolchains.

Nix solves this problem by doing away with conventional locations: each package or dependency is placed in its own dedicated path under /nix/store/. A NAR fetched from the cache contains the compressed contents of a unique Nix store path: the directories and files that make up a specific package, dependency, or build artifact. This store path is always unique, because it includes a hash derived from all the inputs used to build its contents. Nix prepends this input-addressed hash to the package’s name and version, like so: /nix/store/4lbicqd70qlfdw2l5wbq23c9qj1m5nww-cowsay-3.8.4.

By using input-addressed store paths, Nix guarantees a set of properties that are analogous to the atomicity, consistency, isolation, and durability (ACID) guarantees of a relational database.

These properties are what give you reproducible build- and run-time behavior, reliable rollbacks, automatic garbage collection, and safe multi-user installations. This means that users can install, upgrade, or remove packages on the same system at the same time, including packages that pull in conflicting versions of the same dependencies. For example, if you need to run kubectl 1.27 (built from an older package set that uses glibc 2.35) alongside Git 2.51 (built from a newer package set that uses glibc 2.40), both programs can coexist on the same machine thanks to Nix’s hashed, input-addressed store paths. And once store objects (packages, build inputs, etc.) are no longer referenced by the global system or individual users’ profiles, Nix can automatically reclaim that disk space.

Populating the cache

The Nix binary cache is populated by an automated build pipeline centered around Nix’s Hydra CI system, which directs a pool of geographically distributed builder machines.

When a maintainer makes a commit that updates a Nix expression in the Nixpkgs GitHub repo, either by changing its version, dependencies, patches, build instructions, tests, or post-build actions, this triggers Hydra to:

• evaluate the commit’s Nix expressions across the whole jobset (more than 211,000 packages);
• generate derivations from those expressions;
• compare them to what’s already been built and cached;
• queue only the new or changed derivations for builds.

Hydra runs many jobsets several times per day as new commits land on active branches, although especially large and expensive jobsets (such as nixos/unstable) tend to run less frequently. It signs and uploads successful builds to cache.nixos.org, where they’re then available as prebuilt binaries.

Interestingly, since around July of 2026, Hydra runs nearly 40,000 fewer build jobs than it did just a year ago. The reason for this is that Nixpkgs recently dropped support for macOS platforms running on x86_64. This coincides with Apple’s planned transition away from Intel-based Macs — starting with the upcoming Golden Gate release, new versions of macOS will run only on Apple Silicon.

A number of artifacts from this CI pipeline end up in the cache.

• Nix Archive (NAR) files

  These nar.xz or nar.zst files, stored at nar/, contain the compressed contents of a single /nix/store/ path. The NAR format is comparable to tar, but much simpler: NARs don’t need to support long filenames, POSIX attributes, a wide range of extensions, or rich metadata. They’re designed to serialize a filesystem object tree, including bare directory structure, file contents, symlinks, and basic metadata.

  The NAR format is an excellent example of simple, stable design: it hasn’t been revised or extended since its introduction — more than 20 years ago.

• Archive information files

  These are top-level files named <store-hash>.narinfo. Think of them as tiny text records that point to the compressed NAR, record its sizes and hashes, list the store paths it references as runtime dependencies, and carry the Hydra build farm’s signature so clients can verify the artifact. Despite each individual file’s negligible size, there are more than 90 GB of narinfo files in the S3 bucket used to host the Nix binary cache.

• Build logs

  The log/ folder stores the output of Nix build jobs for debugging and auditing. This allows anyone to fetch the build log for a cached store path with nix log /nix/store/<store-hash>-package-name.

• Content-addressed mappings

  Metadata in the realisations/ folder maps derivations to their content-addressed store paths. This is relevant for builds using content-addressed derivations (CA-derivations).

• Store metadata

  nix-cache-info is a small metadata file that describes the basic design of the cache as a whole, such as the cache version and its supported features.

BYOBC

The official cache.nixos.org lives in S3, but if you happen to have half a petabyte of available storage, you could definitely roll your own. As we explained previously, each package in cache.nixos.org has been built from source by a pool of geographically distributed builders — bare-metal and cloud alike, spanning most of Europe — using Nix’s own CI.

This build-time reproducibility is possible because the recipes Nix uses to build software, derivations and the Nix language, don’t imperatively prescribe anything. Build recipes are expressed in the Nix language; Nix evaluates these to produce a derivation, or a low-level build specification. The derivation records every declared input; so as long as inputs remain available (via cache.nixos.org, in Nixpkgs, or locally) Nix can reproduce any build, anywhere, at any time.

This makes it so anyone can host their own Nix binary cache if they wanted, just by cloning the Nixpkgs GitHub repo and building every package from source. They would need at least a terabyte of space just for the build step, and they will probably want to provision a pool of concurrent builders. And they would need to foot the bill for the data egress charges associated with serving millions of users. But this isn’t just technically possible, it’s doable. The Flox Binary Cache is one such example: it applies the same principle of reproducible Nix builds to provide prebuilt packages (including CUDA-accelerated packages like PyTorch and TensorFlow) directly from canonical Nixpkgs definitions.

And the best part is that, when built against the same Nixpkgs commits, packages in Flox’s or anyone/anywhere’s binary cache are almost always bit-for-bit identical to those in cache.nixos.org. That’s because Nix builds run inside an isolated sandbox. This means that only explicitly declared inputs are available, so your toolchain can’t silently pull in host libraries, environment variables, arbitrary metadata, or other kinds of state. If the inputs match, the outputs match too.

Building deterministically at this scale is surprisingly difficult to achieve. In practice, Nix’s functional build pattern nails it for the overwhelming majority of packages (about 99.9%), with the remaining sliver being tracked (and continuously chipped away at) as non-deterministic outliers.

Operating on an enormous scale

The overwhelming majority of Nix clients fetch and use software packages from the Nix binary cache. It fields approximately 5,700 requests per second — almost 15 billion requests per month — and accounts for the vast majority of traffic handled by the public Nix ecosystem’s CDN endpoints. It stores 636 TB of data, with around 545 TB in AWS’s infrequent-access storage tier and 91 TB in standard storage. Just as impressive, cache.nixos.org, contains more than 1 billion objects, making it one of the largest public software caches on the Internet.

Such a feat requires careful design. Faced with this challenge, one may be tempted to start diagramming complex architectures with eventually-consistent distributed databases and the like.

In a scene from American novelist Cormac McCarthy’s Blood Meridian, a character exhibits a giant petrified dinosaur bone before a group of mystified, stupefied cowboys. “Your heart’s desire is to be told some mystery. The mystery is that there is no mystery”, he tells them.

There is no mystery to the Nix binary cache: it is just a huge, signed key–value store hosted in an Amazon S3 bucket. Every key is derived from a unique hash, and because there’s no database lookup required, there is no backing database. This avoids a slew of wicked problems that tend to arise when you’re attempting to scale a massive project backed by a database.

The “fresh” contents of cache.nixos.org live in a single S3 bucket in the AWS us-east-1 region. The Nix binary cache also contains packages that are at more than a decade old, software that in some cases has disappeared from the Internet. Because the Nix community has preserved its source code and build definitions, it’s possible to rebuild it today, on modern systems, using its required dependencies. As a cost optimization, these older, historical packages typically live in cheaper AWS storage tiers. The bucket for less-frequently used packages recently moved to S3 Intelligent-Tiering, which automatically shifts objects among “Frequent”, “Infrequent”, “Archive Instant”, and “Archive” tiers, based on how often they’re accessed. This is expected to reduce storage costs, as well as the effort required of the Nix Infrastructure team to manage archived packages.

If the design of the Nix binary cache is conceptually simple, distributing cached artifacts efficiently and reliably to consumers spread out across the world is a much more difficult problem. To this end, a content distribution network, Fastly, sits in front of cache.nixos.org as its global caching layer. If a Nix package isn’t already cached, Fastly streams it from S3 as it arrives, so users don’t have to wait for it to download before starting. For very large files, Fastly splits them up — ironically, to get around its own size limitations. It also transparently rewrites certain kinds of S3 permission errors, so users see 404: Not Found errors, rather than foreboding 403: Forbidden messages. Finally, Fastly handles TLS for the cache.nixos.org domain at the edge, providing secure HTTPS connections to clients while communicating with S3 over TLS.

Amazon S3 just works

Even though the binary cache is simple enough to be host on many different architectures, S3 is particularly suited to the task for a few specific reasons. At the time cache.nixos.org was created, there was basically no other option. In 2007, if you wanted access to distributed, highly available, pay-as-you-go object storage, your choices were Amazon S3 or … running Hadoop and HDFS yourself. But unlike S3, HDFS isn’t a key–value store — it isn’t even an object store! It’s a hierarchical, POSIX-like filesystem, not a global bucket that anyone can fetch from over HTTP.

For the Nix community’s requirements, S3’s key-addressed object storage was the perfect substrate for cache.nixos.org: .nar and .narinfo files could be persisted as object keys and exposed via S3’s internet-facing REST APIs. S3 could store a virtually unlimited number of blobs, serve them up directly over HTTP, replicate them for durability, and scale to meet growing demand. This was revolutionary at the time, and perfect for the “not trying to do too much” design ethos of cache.nixos.org.

Today, the Nix binary cache could live in virtually any other hyperscale cloud. Its design ethos focused on simplicity means that it is not locked into a specific platform. The cache is in S3 because S3 works, and the rational choice is to keep it where it works — this is how cache.nixos.org has been hosted for almost two decades, after all.

This status quo was challenged in 2023, when one of the NixOS Foundation’s principal financial sponsors discontinued its support. During this time, Nix saw enormous growth, meaning that stable infrastructure was a priority concern. The NixOS Foundation responded to this challenge with radical transparency, and issued a call for assistance from the Nix community and commercial vendors. Our employer, Flox, was one of the vendors that responded to this call, along with others in the Nix-adjacent space. AWS Open Source also responded to this call, pledging credits to help fund cache.nixos.org for the fiscal year — a contribution which proved central to the sustainability of the Nix ecosystem and which started what has become a proven track record of support for the binary cache.

Maintaining the cache

At its core, the binary cache is just a signed collection of blobs in S3, which is itself a basic key–value store, with no backing database and almost no service-dependent logic. This primitivity has the advantage of making the cache comparatively simple to maintain.

But there’s an absolute difference between theory and practice. In theory cache.nixos.org is an ideal design, implemented in an ideal cloud service with ideal users and ideal usage characteristics. In practice, and at scale, the binary cache is low- but by no means no-maintenance.

Even the best, most thoughtful designs need upkeep, after all. For the most part, the Infrastructure team supporting the binary cache does its work behind the scenes, to the point that the cache itself seems to “just work”. Nevertheless, this team has had to put out its share of fires, as well as deal with occasional crises. The fact that it manages to keep a low profile is a testament to the efforts of its members, present and past. While Amazon and the folks at AWS kindly help offset the fiscal (and, to some extent, technological) burden of hosting the Nix binary cache; the infra team shoulders the operational burden of keeping it available, responsive, and up-to-date.

Acknowledgments

On behalf of the NixOS Foundation, we want to express our gratitude to AWS for its ongoing support. Their help frees the Infra team to focus on durable fixes rather than day-to-day firefighting. It also gives the team headroom for much more ambitious work, like better garbage collection, new mirroring options, and partnering opportunities.

If you’d like to learn more about Nix infrastructure (along with the people who support and maintain it), check out their team page.

Special thanks to Nix-whisperer Tom Bereknyei for consultation, feedback, and hugely constructive criticism during the drafting of this article.