Software EngineeringDevOps & Deployment

Moving AuthOS Rust Builds Out of Docker with cargo-zigbuild

How we cross-compiled three AuthOS binaries with cargo-zigbuild, kept Docker for packaging, and cut our warm publish builds from about 15 minutes per variant to 90 seconds.

Rust source passing through an optimize-and-build step into a Docker image
Lead image Rust source passing through an optimize-and-build step into a Docker image
On this page

Audio edition

0:00 / 10:03

Ready

Our AuthOS release process built three Rust binaries: one each for SQLite, PostgreSQL, and MySQL. We built every variant in a multi-stage Dockerfile, so publishing a version meant waiting for the same expensive compilation boundary three times.

A typical build took 15–20 minutes. Across the three variants, a release could occupy 45–60 minutes.

The problem was not that Docker cannot cache Rust builds. BuildKit supports persistent cache mounts for Cargo’s registry and target directory. Our Dockerfile simply did not use them effectively: it copied the project and ran a release build in a layer that source changes invalidated.

We could have improved that Docker build with cache mounts, but our development machine already had a useful Cargo cache. We chose to compile the Linux artifacts on the host with cargo-zigbuild, then give Docker only the finished binary and its runtime files.

That choice cut a warm build for one variant to about 90 seconds. This article describes that specific pipeline, including the parts that did not become portable merely because Zig was involved.

The boundary we changed

The original Dockerfile combined two jobs:

# Simplified version of the original build
FROM rust:1.83-alpine AS builder
RUN apk add --no-cache musl-dev openssl-dev
WORKDIR /app
COPY . .
RUN cargo build --release

FROM alpine:3.20
COPY --from=builder /app/target/release/myapp /app/myapp
ENTRYPOINT ["/app/myapp"]

It produced a deployable image, but a source change invalidated the layer containing cargo build. In our setup, that often meant rebuilding dependencies as well as the application.

The replacement has a narrower contract:

  1. Cargo and cargo-zigbuild produce an x86_64-unknown-linux-musl binary on the build host.
  2. The publish script copies that artifact into target/dist/ and optionally strips and compresses it.
  3. Docker packages the artifact, entrypoint, certificates, and any remaining runtime libraries.
  4. The script tags and pushes the resulting image.

The host-side cross-compilation architecture Fig 1: Compilation moved out of Docker; Docker still defines the runtime image.

This is not Docker-free deployment. Docker still gives us a repeatable filesystem and entrypoint. It just no longer owns the expensive Rust compilation step.

Why cargo-zigbuild fit

cargo-zigbuild invokes Zig as the linker for Cargo targets. For this pipeline, that made a Linux musl target practical from our development host without maintaining a separate Linux builder image.

The target matters. A binary built for x86_64-unknown-linux-musl is not automatically compatible with every machine or image:

  • the CPU architecture must still match;
  • native dependencies may need to be built for the target;
  • dynamically linked libraries must exist in the runtime image;
  • the result must be tested inside the exact image we intend to publish.

Zig simplified the linker and libc side of the cross-build. It did not remove those runtime obligations.

Building one variant

The build host needs Zig, the Rust target, and cargo-zigbuild. UPX is optional.

# macOS with Homebrew
brew install zig upx
cargo install --locked cargo-zigbuild
rustup target add x86_64-unknown-linux-musl

On Linux, install Zig using an appropriate package or the official Zig download , then install the same Cargo command and Rust target:

cargo install --locked cargo-zigbuild
rustup target add x86_64-unknown-linux-musl

For the SQLite variant, the essential command was:

cargo zigbuild \
  --release \
  --locked \
  --target x86_64-unknown-linux-musl \
  --no-default-features \
  --features db_sqlite

--no-default-features is important in this project. It makes the selected database backend explicit instead of allowing a default feature to leak another backend into the build.

The packaging Dockerfile is deliberately small:

FROM --platform=linux/amd64 alpine:3.20

ARG BINARY_NAME=sso_sqlite

# These were the runtime packages required by our artifact.
RUN apk add --no-cache libgcc ca-certificates

WORKDIR /app
COPY target/dist/${BINARY_NAME} /app/sso
COPY docker-entrypoint.sh /app/docker-entrypoint.sh
RUN chmod +x /app/sso /app/docker-entrypoint.sh

EXPOSE 80
ENTRYPOINT ["/app/docker-entrypoint.sh"]

Do not copy the libgcc line into another project without checking it. A fully static artifact may not need it; an artifact with additional native dependencies may need more. file, readelf, and a container smoke test are better evidence than the target name alone.

The publish script

We wrapped the sequence in a Node.js script, docker-publish.js. Its main jobs were to check prerequisites, build the selected feature set, prepare the artifact, build the image, and push its tags.

The core looked like this:

// Core build logic (simplified)
async function buildBackend(backend, imageName, version) {
    exec(
        `cargo zigbuild --release --locked ` +
        `--target=${TARGET} --no-default-features --features=db_${backend}`
    );

    const binaryPath = `target/dist/sso_${backend}`;
    fs.copyFileSync(
        `target/${TARGET}/release/sso_${backend}`,
        binaryPath
    );

    exec(`strip ${binaryPath}`);

    try {
        exec(`upx --best --lzma ${binaryPath}`);
    } catch (error) {
        console.warn("UPX failed; publishing the uncompressed binary");
    }

    exec(
        `docker build -f Dockerfile ` +
        `--build-arg BINARY_NAME=sso_${backend} ` +
        `-t ${imageName}:${backend}-${version} .`
    );
    exec(`docker push ${imageName}:${backend}-${version}`);
}

The example is intentionally incomplete. A production script should stop on compilation, copy, strip, Docker, or push failures. Only UPX is optional. A compressed binary is useful; an image containing the wrong or stale binary is not.

Cross-platform scripts also need a strip tool that understands the target object format. If the host’s strip cannot process the Linux artifact, use a target-aware tool or omit stripping rather than treating a failed strip as success.

What changed in our measurements

These are observations from the AuthOS build machine and project, not a general Rust benchmark:

MeasurementMulti-stage Docker buildHost cargo-zigbuild pipeline
Cold buildabout 18 minutesabout 4 minutes
Warm buildabout 15 minutesabout 90 seconds
Published imageabout 29 MBabout 11 MB

The warm result mattered most. Cargo could reuse artifacts already present in the host target directory, so a normal source change did not require the pipeline to reconstruct the dependency graph from scratch.

Image size changed for more than one reason: the new runtime image contained only the prepared binary and runtime files, and we stripped and compressed the binary. The table does not isolate how much each change contributed.

Across three backend variants, the release workflow fell from roughly 45–60 minutes to around five minutes when the relevant caches were warm. Cold builds remained slower, as expected.

Problems we had to solve

Native dependencies crossed the boundary too

Some crates discover headers and libraries on the build host. That becomes fragile when the artifact targets a different libc or operating system. openssl-sys was the example in this project, and enabling the vendored OpenSSL build avoided depending on the host’s OpenSSL installation:

[dependencies]
openssl = { version = "0.10", features = ["vendored"] }

Vendoring is not a universal fix. It increases build work and still requires the dependency’s build scripts to support the target. When a cross-build fails, inspect the crate’s documented target support before adding more linker flags.

UPX had to remain optional

UPX reduced these binaries further, but it was not required to produce a correct image. We kept its failure non-fatal and published the uncompressed artifact when necessary.

That fallback needs to be visible in the logs, and the resulting image still needs a smoke test. Compression changes the executable that will run in production; it should not be treated as a harmless file-copy optimization.

The Docker context needed an explicit allow-list

Running docker build . from a Rust workspace can send a large target directory to the builder. We excluded its contents, then admitted only the distribution directory:

target/*
!target/dist/
!target/dist/**
.git
node_modules
*.md

The order matters because the last matching .dockerignore rule wins. Before relying on this, inspect Docker’s reported context size and confirm the expected binary reaches the COPY instruction.

Parallel commands did not mean parallel compilation

We tried launching the SQLite, PostgreSQL, and MySQL builds together. Cargo serialized work while they contended for its package and build locks, so the apparent concurrency did not shorten compilation in the way we expected.

We kept compilation orderly and allowed independent Docker and registry work to overlap where useful. If parallel compilation becomes important, separate target directories can remove lock contention, but they also reduce cache sharing and increase disk use.

Every feature set deserves its own smoke test

A successful compile only proves that one feature combination produced an artifact. The publish workflow should start each image and exercise enough of the selected database path to catch a mismatched binary, missing library, or incorrect entrypoint.

That check is especially important when a script names binaries dynamically. A stale target/dist/sso_mysql can look perfectly valid to Docker even if the MySQL compile failed before the copy.

Kitchen made the pipeline easier to operate

We exposed the script through a small web dashboard called Kitchen. It offered backend selection, live logs over Server-Sent Events, build history in SQLite, and a way to stop a running build.

Kitchen dashboard with build controls and live output

Kitchen also detected Docker Hub credentials and handled version tags. The useful part was not the button itself; it was that the button invoked one pipeline and exposed its output. The same script could still run without the dashboard.

When I would keep compilation in Docker

Moving compilation to the host traded some build isolation for direct access to a warm local cache. That was sensible for our single-machine publish workflow, but it is not the only way to solve slow Rust builds.

I would keep a multi-stage Docker build when the build environment must be identical across developer machines, when native dependencies are hard to cross-compile, or when the builder already has a durable BuildKit cache. Docker documents cache mounts for Cargo’s registry and target directory, and external caches can carry BuildKit state into ephemeral CI jobs.

Other reasonable options include:

  • BuildKit cache mounts inside the Dockerfile;
  • an external BuildKit cache for CI;
  • sccache when compiler results need to be shared across machines;
  • a dedicated builder image for complex native dependencies.

For AuthOS, the simplest boundary was host compilation followed by container packaging. The result was not “Docker builds are slow.” It was narrower and more useful: our Dockerfile was repeatedly paying compilation costs that the host had already cached. Once Docker received a prepared artifact instead, a warm three-variant release took about five minutes—and we retained Docker for the job it was still doing well.

Resources

Continue reading

Complete index →