Context Window Management: LLM Memory in Production

The art of forgetting what doesn't matter.

The Infinite Context Myth

Gemini 1.5 Pro has a 2 million token window. Claude 3.5 has 200k.
Great, you think. I'll just stuff the whole database in there.

Do not do this.

1. Cost: 1M tokens cost real money ($5-$10 per call).
2. Latency: Processing 1M tokens takes 30-60 seconds.
3. Lost in the Middle: Research shows models are worse at retrieving info buried in the middle of a massive prompt.

Strategy 1: The Rolling Window (FIFO)

Keep the last N messages. Drop the rest.

• Pros: Simple, cheap.
• Cons: Loses long-term context. My name is Bob (said 20 turns ago) is forgotten.

Strategy 2: Summarization-based Memory

Periodically ask the LLM to summarize the conversation so far.

• Prompt: Summarize the key facts from this chat: Users name is Bob, he likes Red widgets.
• Context: [Summary] + [Last 5 Messages].
• Pros: Retains key facts indefinitely.

Strategy 3: Vector Memory (Long-Term)

Store every message in a Vector DB. Retrieve only relevant past messages based on the current query.

• User: What did we decide about the budget?
• System: Searches Vector DB for budget. Retrieves message from 3 days ago.
• Pros: True long-term memory.

Conclusion

Treat tokens like RAM. Just because you have 64GB doesn't mean you should write inefficient code. Optimize for the Working Set of memory.

---

*Next: [Prompt Engineering Patterns That Scale](/blog/prompt-engineering-patterns-scale)*

How This Article Was Made

This article is a live example of the AI-enabled content workflow we build for clients.

Stage	Who	What
Research	Claude Opus 4.5	Analyzed current industry data, studies, and expert sources
Curation	Tom Hundley	Directed focus, validated relevance, ensured strategic alignment
Drafting	Claude Opus 4.5	Synthesized research into structured narrative
Fact-Check	Human + AI	All statistics linked to original sources below
Editorial	Tom Hundley	Final review for accuracy, tone, and value

The result: Research-backed content in a fraction of the time, with full transparency and human accountability.

Why We Work This Way

We're an AI enablement company. It would be strange if we didn't use AI to create content. But more importantly, we believe the future of professional content isn't AI vs. Human—its AI amplifying human expertise.

Every article we publish demonstrates the same workflow we help clients implement: AI handles the heavy lifting of research and drafting, humans provide direction, judgment, and accountability.

Want to build this capability for your team? Let's talk about AI enablement →

---