One Mattermost Bridge, Two Hot-Swappable Agents

My Mattermost bridge began with one simple assumption: a message arrived for Codex.

That assumption leaked everywhere. The bridge knew how to start Codex, resume its threads, translate its events, post its progress, deliver its files, and run scheduled Codex routines. It worked, but adding a second agent—Pi—turned every one of those details into an architectural question.

This article continues a Mattermost setup I documented in three earlier pieces:

• How I Run Codex from Mattermost on My Phone covers Mattermost, the original local bridge, its security boundary, and the path from a phone to the development machine.
• Schedule Codex Routines as Finite Runs explains why recurrence belongs to the bridge rather than an agent conversation.
• Replacing a VPS Reverse SSH Tunnel Without Losing Rollback treats the public network route as a separate operational concern.

Those pieces establish the transport and scheduler. This article is about what happened when the process behind them stopped serving only one kind of agent.

I did not want a second bridge. That would duplicate Mattermost connections, routing, delivery, scheduling, and operational state. I also did not want one giant switch statement whose Codex and Pi branches grew independently inside the same process.

The boundary I eventually wanted was more precise:

• one persistent process owns Mattermost and routines;
• each agent owns its runtime-specific state and protocol;
• both agents expose the same small contract to the bridge; and
• either agent runtime can be replaced while the bridge keeps running.

The first extraction was still asymmetric

My first attempt moved Pi into a child process but left Codex embedded in the bridge.

That improved Pi’s lifecycle, but it did not solve the design problem. Pi had a worker host, IPC messages, health checks, and shutdown behavior. Codex still called directly into the bridge. Every improvement to the Pi path raised the question of whether I should duplicate it for Codex.

The asymmetry also made the bridge misleading. It appeared to support agents, but its real abstraction was “Codex plus a Pi exception.”

The useful correction was to treat Codex as an agent in exactly the same architectural sense as Pi. Their internal protocols are different, but their relationship to Mattermost is not:

Mattermost
    |
    v
+-------------------------------------------+
| Persistent bridge core                    |
| routing | thread queues | stream | files  |
| slash commands | routine scheduler        |
+---------------------+---------------------+
                      |
                AgentRegistry
                  /         \
                 v           v
          Codex worker    Pi worker
          generation N    generation N

The core no longer needs to know how a Codex app-server turn differs from a Pi SDK session. It asks the selected agent to execute a turn and consumes normalized events.

Give the bridge one agent contract

The shared boundary is deliberately small. Conceptually, an agent host needs to support four operations:

await agent.executeTurn(request, { onEvent })
await agent.reload()
agent.status()
await agent.dispose()

An execution request contains the prompt, Mattermost thread identity, working directory, attachments, and any agent-specific resume information. The result is not a Codex event or a Pi event. It is a stream of bridge events:

worker_assigned
turn_started
text
progress
artifact

That vocabulary is enough for the bridge to update a working message, post a final answer, and attach generated files. Runtime-specific event parsing stays on the worker side.

For Codex, the worker owns the app-server client, durable Codex thread and turn identifiers, CLI fallback, and conversion of app-server events into bridge events.

For Pi, the worker owns Pi sessions, provider and model configuration, and the decision about which Pi events are safe and useful to publish. Tool calls and raw tool output are filtered before they cross IPC; Mattermost receives narrative thinking, lifecycle summaries, text, and artifacts rather than a transcript of every internal operation.

The distinction matters. Normalization should reduce coupling, not erase useful differences. Both workers speak the same transport contract, while each remains responsible for the semantics of its own runtime.

Remove the plumbing once, not twice

Moving both agents out of the bridge could easily have produced two almost-identical worker servers. Instead, I extracted a shared worker harness.

The harness owns the repetitive process mechanics:

• initialization and readiness;
• request and response correlation;
• event forwarding;
• health reporting;
• drain and shutdown messages; and
• structured error replies.

The Codex and Pi entrypoints are then thin adapters around their runtimes. In this implementation, those entrypoints are 43 and 32 lines respectively. The point is not the line count by itself. It is that adding another lifecycle signal or fixing an IPC error path now happens in one place.

The resulting source layout makes ownership visible:

bridge.mjs
lib/
  agent-factory.mjs
  agent-registry.mjs
  hot-swap-worker-host.mjs
  agent-worker-server.mjs
  codex-worker-host.mjs
  codex-app-server-runtime.mjs
  codex-event-formatters.mjs
  pi-worker-host.mjs
  pi-agent-runtime.mjs
  pi-worker-events.mjs
  mattermost-client.mjs
  mattermost-streamer.mjs
  routine-coordinator.mjs
  routine-store.mjs
  session-store.mjs
workers/
  codex-agent-worker.mjs
  pi-agent-worker.mjs

bridge.mjs fell from 2,769 lines to 1,502. More importantly, those remaining lines describe bridge behavior instead of mixing transport, scheduling, two agent protocols, and two process lifecycles in one file.

Hot swapping is a handover, not a restart

A reload creates a candidate worker before touching the current one:

start candidate
      |
      v
wait for ready ---- failure ----> reject candidate
      |                             keep current
      v
atomically select candidate
      |
      v
drain old generation
      |
      v
stop old generation

The order is the feature.

If the candidate cannot initialize, the current worker remains selected. If it becomes ready, new turns go to the new generation immediately. Existing turns stay with the old generation until they finish, after which that process can exit.

This is why a plain child-process restart was not enough. Killing the old process before the replacement is healthy creates an avoidable interruption. Switching all traffic at once can also strand a turn whose in-memory state belongs to the old runtime.

Codex needs one additional affinity rule. An active turn—or a goal continuing across turns—remains attached to the generation that owns it while that generation drains. The replacement handles new work, but it does not steal live state merely because its process is newer.

From Mattermost, the explicit controls are:

/reload codex
/reload pi

File watchers can request the same guarded swap when agent-specific source changes. They are enabled by default and can be governed independently:

CODEX_WORKER_HOT_RELOAD=1
PI_WORKER_HOT_RELOAD=1

Core changes are different. Mattermost routing, scheduler logic, or the shared delivery path still belong to the persistent process and therefore still require a bridge deployment. Hot swapping is a boundary, not a promise that every file can change without a restart.

Serialize sessions, not the whole server

I did not add a global concurrency limit.

The correctness requirement is narrower: two turns must not write to the same agent session at the same time. The bridge already knows the Mattermost thread, so it serializes work per thread. Independent threads can run concurrently, and their workers can make progress independently.

Thread A: turn 1 -> turn 2 -> turn 3
Thread B: turn 1 -> turn 2
Thread C: turn 1

Threads A, B, and C may overlap.
Turns within A remain ordered.

This avoids both failure modes: concurrent mutation of one session and an unnecessary server-wide bottleneck. If a provider later imposes a real capacity limit, that can be represented where it belongs. It does not need to be invented pre-emptively in the bridge.

Keep routines in the process that does not move

Scheduled routines were the main reason I refused to restart the live bridge while developing this change.

The routine scheduler owns timing, run receipts, Mattermost destinations, and delivery. Those are bridge concerns, so it remains in the persistent core. In this design, the coordinator dispatches each finite routine turn to Codex through AgentRegistry.

Reloading Pi therefore has no effect on a Codex routine. Reloading Codex sends new routine work to the new generation while an already-running routine drains on the old one. The scheduler itself never stops.

This preserves the ownership rule from my earlier routine design: the bridge decides when a finite run begins, while the selected agent does the work. A worker replacement should not move the clock, duplicate a run, or erase its receipt.

Configure identity separately from transport

Mattermost channel routing selects an agent name. The registry turns that name into a worker configuration. Provider credentials, model choice, and working directory remain properties of the selected runtime.

A minimal environment might look like this:

CODEX_WORKDIR="$HOME"
PI_WORKDIR="$HOME/Desktop/pi"

CODEX_WORKER_HOT_RELOAD=1
PI_WORKER_HOT_RELOAD=1

Secrets are supplied through the service environment rather than committed beside this configuration. In my setup, Pi can reuse an existing provider credential already available to another local automation, but that does not make the key part of Mattermost routing or session state.

The default Pi workspace is intentionally separate:

~/Desktop/pi

That gives Pi a predictable filesystem boundary without forcing Codex into the same directory. A request can still provide a more specific working directory when the workflow requires one.

Test the lifecycle, not only the answer

A chatbot smoke test can prove that an agent answered once. It cannot prove that the bridge survives a worker replacement.

The validation suite for the refactor contained 23 tests. I added cases around the handover itself:

I also ran each real worker through a two-generation smoke test.

Pi moved from generation 1 to generation 2 and reopened the same durable session without sending a model prompt. Codex moved through the same handover and resumed the same durable Codex thread, also without prompting the model. A fake Codex app-server test covered overlapping turns, drain behavior, normalized streaming, rate-limit events, and CLI fallback without consuming model work.

Finally, I checked for leaked worker processes after the suite. Lifecycle tests that leave children behind are testing a different system from the one I intend to operate.

What remains deliberately shared

Hot swapping stops at the agent boundary. A change to Mattermost routing, thread queues, streaming, file delivery, or scheduling still changes the persistent core and requires that process to be redeployed. Moving those components into replaceable workers would weaken the ownership model merely to increase the amount of code labelled hot-reloadable.

The same rule limits what a third agent should add. It needs a runtime adapter, an event filter, and a thin worker entrypoint. It should not bring another Mattermost connection, scheduler, session queue, or attachment pipeline.

Codex and Pi do not become identical under this design. The shared contract keeps their process lifecycle out of the bridge; the workers give their differences somewhere specific to live.