Fast Rust Exposed a Race the Handler Was Hiding

Fast code does not forgive sloppy state changes. It exposes them.

That is how a SAML certificate rotation feature turned into a concurrency bug. The API was written in Rust with axum, SeaORM, and PostgreSQL as the source of truth. The handler looked normal. The logic was easy to read.

Under concurrent pressure, it failed immediately.

The bug was the old one: Check-Then-Act.

Read state. Make a decision. Write new state. Somewhere between those steps, another request sneaks in and makes the decision stale.

The Rule

Organization admins can generate new X.509 certificates for SAML authentication. The rule is strict:

1. A service can have many signing keys for historical verification.
2. Only one key can be active at a time.
3. When generating a new key, the system must deactivate the old one.

The database enforces the rule. That part matters. If the application lies, the database should still refuse bad state.

The Handler That Read Well and Failed

The first handler was ordinary CRUD:

Find the active key. Deactivate it. Insert the new active key.

// 1. Check: Find the currently active key
let active_key = saml_keys::Entity::find()
    .filter(saml_keys::Column::ServiceId.eq(service_id))
    .filter(saml_keys::Column::IsActive.eq(true))
    .one(db).await?;

// 2. Act: Deactivate it (if it exists)
if let Some(key) = active_key {
    let mut active = key.into_active_model();
    active.is_active = Set(false);
    active.update(db).await?;
}

// 3. Act: Insert the new key
let new_key = saml_keys::ActiveModel {
    service_id: Set(service_id),
    is_active: Set(true),
    ..Default::default()
};
new_key.insert(db).await?;

It reads well. That is the problem. Race conditions often do.

The Crash Was the Database Doing Its Job

The integration test suite sends concurrent requests to simulate a double-click on “Generate Certificate” or two admins rotating keys at the same time.

Under load, the handler returned 500 Internal Server Error. PostgreSQL reported a UniqueViolation.

What happened?

The requests made the same decision from the same old snapshot.

1. Request A reads the table. It finds no active key (or finds one).
2. Request B reads the table before Request A finishes writing. It sees the exact same state as Request A.
3. Request A inserts a new key with is_active = true. This succeeds.
4. Request B tries to insert a new key with is_active = true. This fails.

Postgres did its job. The application left a gap between “check” and “act”, and another request walked through it.

Lock the Shared Parent

To serialize certificate rotation, both requests need to contend on the same resource.

We cannot reliably lock the keys, because the key might not exist yet. The shared resource is the Parent Service.

The fix is a transaction with SeaORM’s .lock_exclusive():

use sea_orm::{TransactionTrait, QuerySelect, Set, EntityTrait, QueryFilter};

// Start a transaction
db.transaction::<_, (), Error>(|txn| {
    Box::pin(async move {
        // 1. THE LOCK: Select the Parent Service "FOR UPDATE"
        // This effectively creates a mutex for this specific service_id in Postgres.
        // Request B will hang here until Request A commits.
        let _service_lock = service::Entity::find_by_id(service_id)
            .lock_exclusive() 
            .one(txn)
            .await?;

        // 2. Now we have exclusive access. Deactivate old keys safely.
        saml_keys::Entity::update_many()
            .col_expr(saml_keys::Column::IsActive, sea_orm::sea_query::Expr::value(false))
            .filter(saml_keys::Column::ServiceId.eq(service_id))
            .filter(saml_keys::Column::IsActive.eq(true))
            .exec(txn)
            .await?;

        // 3. Insert the new key
        let new_key = saml_keys::ActiveModel {
            service_id: Set(service_id),
            is_active: Set(true),
            ..Default::default() // Set your keys/certs here
        };

        new_key.insert(txn).await?;

        Ok(())
    })
})
.await?;

.lock_exclusive() generates a SELECT ... FOR UPDATE on the Service row. Request B stops at the lock until Request A commits.

The gap is gone. There is no stale decision window.

The Constraint Has To Match The Rule

The lock fixes the handler. The schema still has to express the business rule.

Do not use a standard composite unique constraint like UNIQUE (service_id, is_active) here. It sounds right, then blocks the history table you actually need.

That constraint allows one active key, but it also allows only one inactive key. The system needs a full history of rotated certificates.

The correct shape is a Postgres Partial Index:

CREATE UNIQUE INDEX unique_active_saml_key 
ON saml_keys (service_id) 
WHERE is_active = true;

It enforces uniqueness only when is_active = true. Inactive rows are ignored by the uniqueness rule, so the audit trail can grow.

Locks Are Not Free

Pessimistic locking is not magic. It buys correctness by reducing concurrency on the locked resource.

Use it deliberately.

1. The “Hot Row” Bottleneck

Put this lock on a hot row and you will destroy throughput.

Imagine a counters table where every API request updates a global view count. If every request locks the same row, the application becomes a queue behind that row.

2. Connection Pool Exhaustion

This is the quieter failure. Waiting for a Postgres lock still holds a database connection.

If the pool has 100 connections and 100 requests are stuck waiting on Service A, the pool is empty. Request 101 might only need Service B, but it cannot get a connection. One tenant’s contention can starve unrelated work.

Why it works here

For SAML certificate rotation, the lock is acceptable:

• Frequency is Low: Admins rotate certificates once a year, not 1,000 times a second.
• Scope is Isolated: We are locking a specific service_id. Locking “Service A” does not block “Service B”.

If this were a high-frequency feature, such as updating a session timestamp, this would be the wrong tool. Use Atomic Updates (UPDATE table SET val = val + 1) or Optimistic Locking with a version column and retries.

The Takeaway

1. Trust the Database: Application-level checks are race-prone when they decide future writes.
2. Lock Early: If a write depends on a read, lock the shared resource before the decision.
3. Know the Cost: Pessimistic locking guarantees correctness by making contenders wait.
4. Test Concurrency: Normal integration tests prove logic. Concurrent tests prove the logic survives reality.