Reproducing Stale Nginx DNS with Docker Compose
A deterministic Docker Compose reproduction of Nginx proxying to an old backend IP, including the misleading first test and two runtime DNS fixes.

On this page
Audio edition
0:00 / 4:09
Ready
A literal hostname in proxy_pass can outlive the IP address it first resolved to. The configuration starts cleanly and requests succeed, but replacing the backend later leaves Nginx connecting to the old address. A reload appears to fix the problem because it triggers another lookup.
I wanted a local reproduction that separated this DNS behaviour from the rest of a production system. The useful part of the exercise was not just making Nginx return 502 Bad Gateway; it was discovering why my first test could not expose the bug.
The configuration under test
This was the deliberately problematic configuration:
events { worker_connections 1024; }
http {
server {
listen 80;
location / {
proxy_pass http://my-backend-service:8000;
proxy_set_header Host $host;
}
}
}
In the Nginx 1.23 image used for this experiment, the hostname in that literal proxy_pass value is resolved while the configuration is loaded. Nginx does not periodically replace the address just because DNS later returns a different one.
That distinction matters when a stable service name points to containers or hosts whose addresses can change. Restarting or reloading Nginx refreshes the address, but it also hides the configuration problem.
Why the first reproduction passed
My first Compose file gave two backend services the same network alias:
services:
nginx_proxy:
# ...
backend_v1:
# ...
networks:
my_app_net:
aliases:
- my-backend-service
backend_v2:
# ...
networks:
my_app_net:
aliases:
- my-backend-service
I started Nginx with backend_v1, confirmed the response, stopped V1, and started backend_v2. I expected the next request to fail. Instead, V2 answered successfully.
Docker had reused the old container address. V1 returned its address to the network pool, and V2 received the same one moments later. Nginx still held a stale DNS result, but that result happened to point to the replacement container.
The successful request did not disprove the stale-address problem. It showed that the test did not require the address to change.
Make the address change explicit
I assigned fixed, different addresses to the two backends so the transition could no longer depend on Docker’s address allocation:
services:
nginx_proxy:
image: nginx:1.23-alpine
container_name: nginx_proxy
ports:
- "8080:80"
volumes:
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
networks:
- my_app_net
backend_v1:
image: python:3.9-alpine
container_name: backend_v1
command: >
sh -c "echo '<h1>Response from Backend V1 at 172.20.0.10</h1>' > index.html && python -m http.server 8000"
networks:
my_app_net:
aliases:
- my-backend-service
ipv4_address: 172.20.0.10
backend_v2:
image: python:3.9-alpine
container_name: backend_v2
command: >
sh -c "echo '<h1>Response from Backend V2 at 172.20.0.11</h1>' > index.html && python -m http.server 8000"
networks:
my_app_net:
aliases:
- my-backend-service
ipv4_address: 172.20.0.11
networks:
my_app_net:
driver: bridge
ipam:
config:
- subnet: 172.20.0.0/24
Starting the backend before the proxy also removes a startup race in the demonstration:
docker compose up -d backend_v1
docker compose up -d nginx_proxy
curl http://localhost:8080
# <h1>Response from Backend V1 at 172.20.0.10</h1>
Nginx has now resolved my-backend-service to 172.20.0.10. Replacing the backend moves that name to .11:
docker compose stop backend_v1
docker compose up -d backend_v2
curl http://localhost:8080
# 502 Bad Gateway
The fixed addresses make the result unambiguous. DNS points to .11, while the running Nginx configuration still tries .10.
Resolve the variable at request time
One fix, including for the Nginx version in this experiment, is to put the backend address in a variable and configure a runtime resolver:
events { worker_connections 1024; }
http {
resolver 127.0.0.11 valid=5s;
set $backend "my-backend-service:8000";
server {
listen 80;
location / {
proxy_pass http://$backend;
proxy_set_header Host $host;
}
}
}
Containers on a custom Docker network use Docker’s embedded DNS server at 127.0.0.11. Nginx’s proxy_pass documentation
explains that a domain name supplied through a variable is resolved using the configured resolver.
Here, valid=5s overrides the DNS answer’s cache lifetime for a short demonstration cycle. It is not a universal production value. The resolver should be local and trusted, and the cache duration should reflect how the service is expected to move.
After loading this configuration, I repeated the V1-to-V2 switch, allowed the five-second cache window to pass, and requested the page again. Nginx resolved the alias to 172.20.0.11, and V2 answered.
A current Nginx alternative
Open-source Nginx 1.27.3 added dynamic resolution for servers in an upstream group. On that version or later, the same intent can be expressed without putting the proxy_pass target in a variable:
events { worker_connections 1024; }
http {
resolver 127.0.0.11 valid=5s;
upstream backend {
zone backend 64k;
server my-backend-service:8000 resolve;
}
server {
listen 80;
location / {
proxy_pass http://backend;
proxy_set_header Host $host;
}
}
}
The shared zone stores the upstream’s runtime state, and resolve tells Nginx to monitor DNS changes. The upstream module documentation
records the version boundary and the requirement for both a resolver and shared memory.
Whichever form matches the installed Nginx version, the operational requirement is the same: if a service name is expected to move while Nginx remains running, make that DNS refresh explicit and test with addresses that actually change.

