covelab
← All posts
by CoveLab

The Docker Bind-Mount Trap: A Real AI Deployment Incident

We deployed a Caddy config update that appeared to succeed but changed nothing. Here's the exact failure, why it happened, and the one-line fix that works every time.

  • docker
  • caddy
  • deployment
  • infrastructure
  • incident

During a recent deployment, we added a new static site block to our Caddy reverse proxy. The config edit looked clean. caddy validate returned success. caddy reload said “config is unchanged.” The new site wasn’t served.

Thirty minutes of debugging later, we found the cause. This post is the documentation we wish had existed.

The Setup

Our stack runs Caddy inside a Docker container. The Caddyfile lives on the host and is mounted into the container via a Docker bind mount:

/host/path/Caddyfile → /etc/caddy/Caddyfile  (bind mount, set at container start)

This is a completely standard setup. It’s in the official Caddy Docker documentation. It’s what most tutorials recommend.

What Went Wrong

We used a file-editing tool to update the Caddyfile on the host. The tool wrote the new content successfully — the host file had the new config.

But when we ran docker exec caddy caddy validate and docker exec caddy caddy reload, Caddy read the old config. The reload said “config is unchanged” because, from Caddy’s perspective inside the container, nothing had changed.

The Root Cause: Inode Split

When most editors and file-writing tools update a file, they don’t overwrite the existing bytes. They:

  1. Create a new file with a new inode
  2. Write the new content to it
  3. Atomically rename the new file to the original path

This is the safe, atomic write pattern. It prevents partial writes if the process is interrupted.

The problem: Docker bind mounts track the inode, not the path. When the container started, the bind mount attached to the inode of the Caddyfile at that moment. After an atomic replace, the host path points to a new inode — but the container’s mount still points to the original, now-orphaned inode.

The result: host and container are looking at different inodes, both named Caddyfile. The host sees the new config. The container sees the old one. Both are “correct” from their own perspective.

Before edit:
  Host: /host/Caddyfile  → inode 524383 (old config)
  Container: /etc/caddy/Caddyfile → inode 524383 (same)

After atomic replace:
  Host: /host/Caddyfile  → inode 524335 (new config)  ← atomic rename
  Container: /etc/caddy/Caddyfile → inode 524383 (old config, orphaned)

What Didn’t Work

docker cp is the obvious fix: copy the new file directly into the container. But docker cp fails on bind-mounted paths with “device or resource busy” — the container has the path locked.

Restarting the container works, but it’s a heavyweight operation: it causes a brief downtime window, re-mounts all bind mounts to the current inodes, and is overkill for a config update.

The Fix That Works

The solution is to write in-place — to overwrite the existing bytes of the file without creating a new inode. This preserves the inode the container’s bind mount is tracking.

cat /host/path/Caddyfile | docker exec -i caddy sh -c 'cat > /etc/caddy/Caddyfile'

This writes the host file’s content through the container’s namespace into the existing inode. No new inode created. The container and host now both see the same content on the same inode.

After writing in-place:

docker exec caddy caddy validate && docker exec caddy caddy reload

caddy validate now reads the updated content. caddy reload applies the new config gracefully (no downtime).

Verifying the Fix

To confirm inode alignment before and after:

# On host
stat /host/path/Caddyfile | grep Inode

# Inside container
docker exec caddy stat /etc/caddy/Caddyfile | grep Inode

Both inodes should be identical after the in-place write.

Why caddy validate Said “OK”

A detail worth explaining: caddy validate returned success even when Caddy was reading the old config. This is correct behavior — the old config was valid. validate only checks whether the config it reads is syntactically correct, not whether it’s the config you intended to deploy.

This is why “validate passed” is not the same as “deployment succeeded.” Always verify the actual served behavior, not just the validation result.

Lessons Applied

  1. Never assume atomic file writes work through bind mounts. They don’t — by design.
  2. Always use in-place writes for bind-mounted config files. The cat | docker exec -i pattern is the reliable path.
  3. Verify deployment, not just validation. caddy validate OK + caddy reload running doesn’t mean the new config is live. Check the actual served response.
  4. Inode inspection is a two-minute check that immediately reveals bind-mount splits. Add it to your deployment debugging checklist.

We now document this as a named pattern in our infrastructure knowledge base. The next time a Caddy config update silently fails, the check runs in under two minutes.

This incident and the patterns we derived from it are documented in CoveLab Foundation — a structured framework for AI-assisted development and infrastructure management.