What the Gateway Taught Us After Burning 800ms Per Request

The final fix was tiny.

We created an internal Auth Loopback API, slapped a Cache policy in front of the real auth server, pointed our existing HTTP Callout policies to the loopback, and reused the cached JSON response until it expired.

It sounds like a five-minute hack. It was not.

It took a full day of trying reasonable things that refused to work. That day was not wasted. It showed us the real platform: what the gateway allowed, what the sandbox blocked, and which abstractions were imaginary.

The Dead Ends Were Honest

Every failed attempt started from a reasonable assumption.

That table is the real story.

We did not need to be more clever. We needed a design that lined up with the gateway’s actual extension points.

The Numbers Justified the Detour

Before we added caching, every single API request paid a heavy tax for a token callout.

After we put the loopback in place, the numbers looked like this:

That first cold request still had to call the real auth server. Expected.

But every “warm” request after that completely skipped the auth server, grabbing the cached JSON response straight from the loopback API. We successfully turned the auth server from a per-request dependency into a once-per-cache-window dependency.

That is the operational shift:

Before the fix:
requests hitting auth server = total requests hitting your business API

After the fix:
requests hitting auth server = cache misses

If the API handles thousands of requests during an eight-hour token window, this is not a cute micro-optimization. It reduces latency, removes load from the auth server, and cuts off a source of cascading failures.

The Final Architecture Was Small on Purpose

Our final request path ended up looking like this:

1. The client calls the business API.
2. The business API fires an HTTP Callout to http://localhost:8082/loopback.
3. The Auth Loopback API checks its Cache policy.
4. On a cache hit: The loopback instantly returns the cached auth JSON.
5. On a cache miss: The loopback calls the real auth server, stores the response, and then returns it.
6. The business API extracts the access_token from the JSON.
7. The business API sets the backend Authorization header.
8. The backend receives the authenticated request.

Yes, this added one internal hop. In exchange, the common path removed a slow remote auth callout. That trade is easy.

Lessons Learned the Hard Way

1. Gateway policies have strong personalities

Policy names are often deceptively broad.

The word “Cache” might mean response caching, token caching, arbitrary key-value storage, distributed caching, or something else entirely depending on the gateway and version. Read the behavior, not the label.

In our older version of Gravitee, the core behavior of the Cache policy was strict response caching. Once we finally accepted that, the loopback pattern became the obvious choice.

2. Sandboxes are intentional product decisions

The Groovy sandbox didn’t block our hacks by accident. It represented a deliberate security posture.

Could an administrator have just changed the whitelist for us? Probably. Would that have made our script work? Maybe. But it also would have introduced a hard production configuration dependency, required a gateway restart, and left us managing a script with dangerously privileged behavior.

The loopback pattern avoided that whole conversation.

3. “Compatibility” is a strict contract, not a vibe

Our auth server definitely looked OAuth2-ish, but the gateway’s native OAuth2 resource expected a much stricter, standard shape than our endpoint actually provided.

This happens constantly in enterprise systems. An internal service can use familiar words like “token”, “grant type”, “bearer”, and “scope,” while still being different enough to break a product-native plugin.

When that happens, do not argue with the plugin. Write down the contract mismatch and move on.

4. Cache keys are literal security boundaries

We only got away with a static cache key (service-token-general) because the specific service credential and permission set we were using was shared across all our APIs.

If your tokens differ by tenant, backend, scope, environment, region, or calling user, your cache key must reflect that. If you get this wrong, you will accidentally reuse a token in the wrong context, and you will leak data.

Good cache keys are boring and incredibly explicit:

service-token:{environment}:{backend}:{scope}

5. The common path deserves all your attention

Our cold cache request still took about 1.4s. That was totally acceptable because it only happened once every eight hours.

The warm request was the product experience. That path needed to be fast, observable, and easy to explain.

Performance engineering is rarely about making everything perfect. It is usually about moving heavy costs out of the common path without hiding failure.

Check Newer Gravitee Before Copying This

If you are starting fresh, do not copy the loopback pattern blindly.

First, check exactly which Gravitee version you’re on, and look at your available policies.

The official Gravitee Cache policy documentation describes the Cache policy exactly as we experienced it: it’s strictly for upstream response caching. However, newer Gravitee documentation also introduces a very handy Data Cache policy . That newer policy is specifically built for arbitrary key-value operations—including storing authentication tokens right before you fire an HTTP callout.

That means your decision tree today should look something like this:

Need to cache a service token?
  -> Do I have the Data Cache policy available?
      -> yes: evaluate Data Cache first.
      -> no: can response caching solve my problem using the loopback pattern?
          -> yes: use an internal loopback API.
          -> no: start looking at a custom policy or requesting a gateway config change.

The lesson is not “always use the loopback pattern.”

The real lesson is:

Always use the smallest, simplest gateway-native mechanism that actually matches the behavior you need.

The Production Checklist

Before shipping any gateway token cache, verify this:

• Your token TTL is definitely longer than your cache TTL.
• Cache failures are highly observable in your logs.
• You are strictly avoiding caching auth failure responses.
• Your cache key includes every single dimension that could change the permissions (tenant, environment, scope, etc.).
• Your loopback endpoint is locked down and not exposed to the public internet.
• You know exactly how the cache behaves across multiple gateway nodes (local vs. distributed).
• You understand (and accept) the cold-cache latency cost that happens during deployments and restarts.
• Your logs clearly distinguish between a cache hit, a cache miss, and a hard auth server failure.
• Your rollback plan is incredibly boring: just changing the HTTP Callout URL back to the real auth server.

That last point is so important. A good production fix shouldn’t require a dramatic rollback.

The Pattern in One Sentence

The Auth Loopback pattern simply turns a response cache into a token cache by sliding a tiny internal API between your business API and the auth server, forcing the cache to return the exact JSON response your business API already knows how to parse.

Is it the most modern solution if your gateway already has a dedicated data-cache policy? No.

But it is a clean answer when you are stuck with a strict response cache, locked inside a rigid sandbox, and working on a production system that cannot wait for a custom plugin.

That is the kind of architecture worth keeping: not fancy, not fragile, just aligned with the natural grain of the platform.