Two API pods died the same afternoon. Over about three days their memory went from roughly 550Mi to about 4Gi, then Linux killed the process. Usage climbed by tens of megabytes per hour and never dropped between requests. Workers on the same cluster stayed flat.

The model was fine. Redis was fine. Only the long-lived API process was growing, and both replicas were the same age, so they OOM'd together and took the API down.

I had already moved session state into Redis on purpose. So where was the memory going?

Why Redis was there

A support chat needs memory. If the user says "yes", that only makes sense if you know what you asked last turn. Transfer? Keep debugging? Close the chat? Same word, different meaning. Check Chat history is not enough for multi-turn support.

So each chat got a few keys in Redis.

History, "did I already offer live support?", how many bad turns in a row, "did they already ask for a human?". Both API pods read the same keys. Restart one pod and the chat is still there.

I thought that covered it.

What still goes wrong

Redis holds the durable facts. But every request still runs inside an API pod, and that pod is a process that can stay alive for days. On each turn the pod needs a small Python helper so it can ask Redis for the last few messages and hand them to the model.

Without a cache, every turn looks like this:

def get_history(session_id):
    helper = build_langchain_helper(session_id)  # new object every time
    return helper.load_from_redis()              # real messages still in Redis

Chat_42 sends two messages a second apart. That builds the same helper twice. I didn't like that, so I kept the helper on the pod:

cache = {}  # lives for the life of the API process

def get_history(session_id):
    if session_id not in cache:
        cache[session_id] = build_langchain_helper(session_id)  # once
    helper = cache[session_id]                                  # reuse next turn
    return helper.load_from_redis()                             # still from Redis

Turn 1 for chat_42 builds and stores. Turn 2 finds it in cache and skips build_langchain_helper. The messages never moved out of Redis. The dict was only meant to avoid rebuilding the wrapper.

The helpers were not thin, and the cache had no cleanup. Every new chat added a row. Finished chats never left. After a few days the pod still held an object for every session it had ever seen, including ones that ended hours ago.

Redis still had the real chat. The pod also kept its own map forever. In code review that map looked like the optimization above. It was a second store with no eviction.

Both pods started at the same time, so they grew up together and died in the same window.

The decision that caused the OOM

I used LangChain's ConversationBufferWindowMemory on top of its Redis chat history (RedisChatMessageHistory). The messages already lived in Redis. The wrapper only existed so I could read the last k turns for a session.

I didn't build that wrapper fresh on every request. I kept a process-wide dict keyed by session id and reused each ConversationBufferWindowMemory for the life of the API process. Create once per session, skip rebuilding LangChain every turn. Seemed fine.

The cache is the problem. Every new chat adds an entry. Nothing removes old ones.

LangChain made it worse, and harder to spot, because of what happens when you pass url=...:

RedisChatMessageHistory(
    session_id,
    url=redis_url,
)

That url does more than answer "where is Redis?" Inside LangChain's helper it means: create and hold a Redis client on that history instance. Connection pool, sockets, file descriptors.

The type says "chat history." The object also owns a private Redis connection. I already had a shared Redis client on the process for flags and counters. Chat history didn't use it. I handed a URL to the library and assumed I was caching a thin reader.

Caching the wrapper is not sharing Redis

This is the mix-up that made the bug hard to see. Caching felt like connection reuse. In my head: one Redis pool, many sessions.

It was the opposite. Each session's history object got its own client from url=..., and caching kept those clients alive for the life of the pod.

Sharing means every history object holds a reference to the same client.

So I got both problems at once: an unbounded session-to-memory dict with no eviction, and a brand new Redis client on first create for each session. Heap objects and open connections piled up until the pod died.

Under load you leak memory and file descriptors together. There's a smaller bug from the same cache too. Change k, the number of recent turns to keep, and a session that already has a wrapper keeps the old k until restart.

The Redis design held up. I broke it by keeping a second, unbounded copy of session state in the API heap, without knowing the cached library object owned a connection.

What I do now

If you cache an abstraction, you cache everything its constructor allocates. Durable session data belongs outside the request process. Share connections. Don't cache session worlds.

Sharing Redis means passing the same client in. Don't pass url= and let the library invent a new one per session.

Concrete changes:

1. One shared Redis client for chat-history reads. Every per-session history helper gets a reference to that client. Never url=... per session.
2. No process-wide map of session to LangChain memory. The wrapper is a disposable per-request view. Redis holds the truth, and building it is cheap. If you still want to cache history objects, they need that shared client and some eviction.
3. After deploy, watch RSS and file descriptors. Flat RSS with climbing FDs still means you're leaking connections.
4. When you review any "reuse this per session" cache in a long-lived API, open the constructor. Does it build a client, a pool, a thread, a file handle? If yes, caching the object caches that resource too.

The OOM was unbounded process-local caching of an object whose real cost sat behind a type name.

Same habit, quieter places

Once you ask whether a map grows with sessions forever inside a process that outlives the request, you see the habit elsewhere. These didn't kill the pods. They were still a second copy in RAM after the real data already lived somewhere else.

Metrics that never left the API

After a normal chat turn, the API enqueues a background job to score the answer and write analytics. The metrics object (dozens of fields, including the user's question text) went into a class-level dict on the API process, keyed by trace id. The worker was supposed to look up the same key later, enrich it, and insert the row.

That can't work across processes. The worker has its own empty dict, so get(trace_id) always missed. Analytics for ordinary turns quietly never landed. Every successful API turn left another entry behind, because remove only ran in the worker.

Same shape as the LangChain cache. Data I treated as shared was trapped in one process's heap. Fix: put the metrics on the task payload. Pass by message, not by shared heap. The worker already gets the job, so send the numbers with it.

A flag that already lived in Redis

I already had a Redis key for "has this user asked for a human?" so both API replicas agree. A guardrail also kept a chat_id -> bool dict on the pod. Never cleared.

The fact had two homes. Pod A might think the user already asked. Pod B might not. And the local dict grew with every chat that tripped that path.

Delete the local dict. One source of truth: the Redis key.

Neither of these needed a new architecture. Same rule as the OOM. If the durable fact already has a home outside the request process, whether that's Redis or the message you send the worker, don't keep an unbounded copy in the API heap because it felt convenient.

Closing

Session memory makes short replies possible. Putting it in Redis makes replicas and restarts safe. Keeping a second unbounded copy inside the API undoes that, sometimes as a quiet correctness bug, sometimes as an OOM that takes the site down.

If you want the why behind the Redis keys for classifier-driven decisions, there's a separate piece: Chat history is not enough for multi-turn support.