Increasing the Integrity of the Open Source Software Supply Chain with Reproducible Builds

CEA List, BINSEC seminar
2022-05-06

Stefano Zacchiroli — Télécom Paris, IP Paris
stefano.zacchiroli@telecom-paris.fr
https://upsilon.cc/zack
@zacchiro | mastodon.xyz/@zacchiro

Open Souce Software Supply Chain Attacks

[Ohm20]: Marc Ohm, Henrik Plate, Arnold Sykosch, Michael Meier. Backstabber’s Knife Collection: A Review of Open Source Software Supply Chain Attacks. DIMVA 2020: 23-43.

The software supply chain

• Supply chain: the set of activities required by an organization to deliver goods or services to consumers.

• Software supply chain: the set of software components and software services required to deliver an IT product or service to users.

  • libraries, runtimes, and other software component dependencies
  • base system (operating system, package manager, compiler, …)
  • development tools and platform (e.g., IDEs, build system, GitHub/GitLab, CI/CD, …)
  • etc.

• Key artifact for audits: SBOM = Software Bill of Materials

Supply chain attacks

A software supply chain attack is a particular kind of cyber-attack that aims at injecting malicious code into an otherwise legitimate software product.

Notable examples

• NotPetya (2017): ransomware concelaed in an update of a popular accounting software, hitting Ukranian banks and major corps (B$)
• CCleaner (2017): malicious version of a popular MS Windows maintenance tool, distributed via the vendor website
• SolarWinds (2020): malicious update of the SolarWinds Orion monitoring software, shipping a delayed-activation trojan. Breached into several US Gov. branches as well as Microsoft

Open source supply chain attacks

• Is this specific to Free/Open Source Software (FOSS)? No.

• But modern FOSS package ecosystems are heavily intertwined.

  • Examples: NPM (JavaScript), PyPI (Python), Crates (Rust), Gems (Ruby), etc.
  • 100-10’000x packages, depending on each other due to code reuse opportunities.
  • Reverse transitive dependencies grow fast. A single package could be required by thousands of others.

left-pad (2016)

function leftpad (str, len, ch) {
    str = String(str);
    var i = -1;
    if (!ch && ch !== 0) ch = ' ';
    len = len - str.length;
    while (++i < len) { str = ch + str; }
    return str;
}

• Maintainer: “I think I have the right of deleting all my stuff”. “Unpublish” package.
• Impact: “many thousands of projects”, including major ones like babel and atom.
• NPM operators forcibly “ununpublish” package.

Open source supply chain attacks (cont.)

• For an attacker, code injection into (transitively) popular leaf packages has a low opportunity cost.
• Also, entirely open FOSS package ecosystems (≠ Linux distros) could be easy to infiltrate.

(An) open source development workflow

(image from [Ohm20])

Attack tree — Injection

(image from [Ohm20])

Attacker’s goal: package P containing malicious code is available from download from a distribution platform and P is a reverse transitive dependency of a legitimate package.

Injection vector — Typosquatting

Injection → Create New Package → Typosquatting

1. Create a new package with a name similar (e.g., Levenshtein distance <= 2) to an existing popular package, including malicious code. Examples:

   • squat on PyPI the Debian package name (“python-sqlite” v. “sqlite”)
   • English variants (“color” v. “colour”)
   • Unicode tricks

2. Upload it to a distribution platform (e.g., PyPI)

3. Wait for users to mistype (e.g., pip install python-sqlite)

Related attack vector: Use After Free

Injection vector — Become maintainer

Injection → Infect Existing Package → Inject into Source → Commit (as maintainer) → Social Engineering to become Maintainer

1. Package maintainer: “I no longer have time for this project, who wants to take over its maintenance?”
2. Attacker: raises hand
3. Attacker: releases new version including malicious code

Might require early investment to accrue enough “street credibility” to win over maintenance at the right moment. For popular packages with low bus factor it could be worth it.

Injection example — Compromise build system

Injection of Malicious Code → Infect Existing Package → Inject during the Build → Compromise Build System

• Often, the code run by users run is written but not built by maintainers

• Rather, it is built by 3rd-party vendors

  • e.g., GNU/Linux distros, app store operators, arch “porters”

• It becomes attractive to break into vendor build systems, compromising binaries “downstream”, without anybody looking merely at source code noticing

Related attack vectors: Inject into [Package] Repository System (≠ VCS)

Reproducible Builds

https://reproducible-builds.org/

[Lamb22]: Chris Lamb, Stefano Zacchiroli. Reproducible Builds: Increasing the Integrity of Software Supply Chains. IEEE Softw. 39(2): 62-70 (2022).

On untrusted code

“You can’t trust code that you did not totally create yourself. […] No amount of source-level verification or scrutiny will protect you from using untrusted code.”
— Ken Thompson, Reflections on Trusting Trust, Turing Lecture 1984

• 40 years later nobody “totally creates” code they run

• Reuse of open source software (FOSS) is everywhere in IT

  • “99% of audited code bases contain FOSS components” (Synopsis 2020)

• Also, the FOSS we run is often not built by its developers

Problem statement

A reproducible build (r-b) process

Precondition/hypothesis: we can “reproducibly build” all relevant (FOSS) products, i.e.:

The build process of a software product is reproducible if, after designating a specific version of its source code and all of its build dependencies, every build produces bit-for-bit identical artifacts, no matter the environment in which the build is performed. — [Lamb22]

(we’ll verify later how realistic this is)

R-B approach

Making Debian reproducible

• Let’s try a large-scale experiment: making all Debian packages build reproducibly from source

  • Debian: one of the largest and most popular GNU/Linux distro, esp. in the server/cloud market
  • 30’000+ (source) packages, 1+B lines of code
  • initial goal of the reproducible-builds.org initiative, est. 2014

• Goals:

  1. empirical experiment to map the origins of non-reproducibility
  2. real impact (if successful) due to Debian popularity in the industry

Build reproducibility in the small

Let’s see a bestiary of real-world examples…

Build timestamps

void usage() {
    fprintf(stderr,
            "foo-utils version 3.141 (built %s)\n",
            __DATE__);
}

• The __DATE__ C preprocessor macro “expands to a string constant that describes the date on which the preprocessor is being run.”

• Fix: SOURCE_DATE_EPOCH envvar (standardized by r-b) to enable controlling for this

Build paths

fprintf (stderr,
    "DEBUG: boop (%s:%s\n",
    __FILE__, __LINE__);

• The __FILE__ C preprocessor macro “expands to the name of the current input file”. This results in non reproducibility when the program is built from different directories, e.g., /home/lamby/tmp vs. /home/zack/tmp.
• Fix: introducted gcc -ffile-prefix-map option (and related -fdebug-prefix-map) to support embedding relative (rather than absolute) paths

Filesystem ordering

NAME
    readdir - read a directory

SYNOPSIS
    #include <dirent.h>
    struct dirent *readdir(DIR *dirp);

[…] The order in which filenames are read by successive calls to
readdir() depends on the filesystem implementation; it is unlikely
that the names will be sorted in any fashion. […]

• Fix: impose a deterministic order in build systems/recipes, e.g., via an explicit sort()

Archive metadata

• Archive formats like .zip and .tar embeds various kinds of metadata by default

  • user/group ownership (e.g., zack v. lamby)
  • file modes (umask)
  • timestamps

• Fix: control for this, e.g.:

  • tar --owner=0 --clamp-mtime=T
  • touch --date=$SOURCE_DATE_EPOCH

Randomness

Even when the entire environment inputs are controlled for, many builds remain non-deterministic. For instance due to randomness in unexpected places.

my %h = ( a => 1, b => 2, c => 3);
foreach my $k (keys %h) {
    print "$k\n";
}

Perl’s hash type does not define an ordering of its keys, so a call to sort should be inserted before keys %h to make it deterministic.

Uninitialized memory

• Many data structures have undefined areas that do not affect their operation, but could end up being serialized in build artifacts.
• Padding for natural memory alignment can also be filled with random content.
• Fix: explicitly zero-out memory.

--- a/direntry.c
+++ b/direntry.c
@@ -24,6 +24,7 @@

  void initializeDirentry(
      direntry_t *entry, Stream_t *Dir) {
+         memset(entry, 0, sizeof(direntry_t));
          entry->entry = -1;
          entry->Dir = Dir;

• A patch for GNU mtools to ensure a direntry_t struct does not contain uninitialized memory.

Build reproducibility in the large

• Now let’s assume we know how to fix all micro-issues that affact build reproducibility.
• How do we go about making large FOSS software collections reproducible?
• Use case: Debian
• Approach: establish a corresponding Quality Assurance process and soft-enforce it using Continuous Integration (CI)

Adversarial rebuilding

How do you find build reproducibility issues, at scale?

• mass-rebuild all packages…

• …building each of them twice…

• …in two build environments configured to differ as much as possible

  • clock set 18 months in the future in 2nd build
  • changing: hostname, locales, kernel
  • reverse filesystem ordering using disorderfs
  • 30+ variations in total

Recording build information

• According to our definition of a reproducible build, legitimate build inputs should be controlled for and replicated identical in the 2nd build

  • source version of product under build
  • ditto for all transitive build dependencies
  • toolchain version

• To that end, the .buildinfo file format has been standardized to capture these information

.buildinfo — Example

Source: black
Version: 20.8b1-1
Checksums-Sha1:
  9915459ae7a1a5c3efb984d7e5472f7976e996b1 2584 black_20.8b1-1.dsc
  14bfd3011b795f85edbc8cc4dc034a91cfaa9bcd 111096 black_20.8b1-1_all.deb
  69c3d4ae7115c51e7b00befe8b4afd5963601d66 285684 python-black-doc_20.8b1-1_all.deb
Checksums-Sha256: [...]
Build-Architecture: amd64
Installed-Build-Depends: autoconf (= 2.69-11.1), automake (= 1:1.16.2-4), […], gcc (= 4:10.2.0-1), […], python3 (= 3.8.2-3), […], xz-utils (= 5.2.4-1+b1), zlib1g (= 1:1.2.11.dfsg-2)

An example .buildinfo file, recording both the environment and results of building Debian’s black package. (See full version.)

Build attestations

• .buildinfo files also contain the cryptographic checksums of final build artifacts, acting as build attestations

I, Alice, given source X, build dependencies Y_1,…,Y_n and toolchain Z, have conducted a build run obtaining a set of artifacts with checksums K_1,…,K_m.

• Anyone (for QA or independent verification purposes) can rerun the build and publish their own build attestations

.buildinfo — Usage

• Before installation, users verify package checksums against published build attestations
• Published by either vendors they trust; or relying on some consensus within a network of independent rebuilders
• Debian publishes 27+M build attestations at https://buildinfo.debian.net

Reproducible Debian — Evolution over time

Quality assurance synergies

• systematic R-B testing ⇒ systematic build testing, catching any FTBFS (Fail To Build From Source) bug

• some software will only FTBFS in the extreme R-B build environment; fixing it will make the software more robust in general

  • e.g., expired SSL certificates at +18 months, or unusual timezone offsets

• R-B testing can detect user-level breakages by serendipity

  • e.g., HTML documentation pointing to /tmp/build/foo/usage.html instead of /usr/share/doc/foo/usage.html

Quality assurance synergies — security

• Security issues can also be spotted during R-B testing by serendipity

{
  'cgibin' => '/usr/lib/cgi-bin/gbrowse',
  'conf' => '/etc/gbrowse',
  'databases' => '/var/lib/gbrowse/databases',
  'htdocs' => '/usr/share/gbrowse/htdocs',
  'OpenIDConsumerSecret' => '639098210478536',
  'tmp' => '/var/cache/gbrowse'
},

An example ConfigData.pm. As it was created at build time, all users shared the same OpenIDConsumerSecret. (See: Debian bug #833885.)

The Reproducible Builds ecosystem

https://reproducible-builds.org/

• 2014: project kickstarted by Debian folks for Debian needs fun
• joined since: Arch Linux, coreboot, F-Droid, Fedora, FreeBSD, Guix, NixOS, openSUSE, Qubes, Tails
• 2017 milestone: Tails (live distro used by Snowden to exfiltrate NSA documents) publishes a fully reproducible ISO to improve end-user verifiability
• independent project hosted by Software Freedom Conservancy + corporate sponsors (e.g., Google, The Linux Foundation, Ford Foundation, Siemens)

Challenges

Takeaways

• Open source software supply chain attacks are a hot topic in cybersecurity right now.
• Several ACES team members have started working in this space.
• Reproducible Builds help countering build/distribution injection attacks.

Appendix

Root cause analysis — Diffoscope