Part 2 of 4
🤖 Ghostwritten by Claude · Curated by Tom Hundley
This article was written by Claude and curated for publication by Tom Hundley.
Garbage Chunks In, Garbage Answers Out.
You have a 50-page PDF. You cant feed the whole thing into the embedding model at once (usually). You have to cut it up into chunks.
Most developers use the default: CharacterTextSplitter(chunk_size=1000).
This is why their RAG apps fail.
Imagine a sentence spans across the cut:
If the user asks What was 2023 revenue?, the search engine might find Chunk 2, but without the context of revenue for 2023, its meaningless. Or it finds Chunk 1, which has no answer.
Instead of cutting blindly at 1000 characters, try to cut at paragraphs (\n\n). If thats too big, cut at sentences (. ). If thats too big, cut at words.
Using an embedding model to scan the document sentence by sentence. When the topic changes (cosine similarity drops), we make a cut.
Always overlap your chunks (e.g., 10-20%). This ensures that sentences on the boundary arent sliced in half.
Chunking is an art. For contracts, we chunk by Clause. For code, we chunk by Function. For manuals, we chunk by Header.
Dont use the default. Know your data.
This article is a live example of the AI-enabled content workflow we build for clients.
| Stage | Who | What |
|---|---|---|
| Research | Claude Opus 4.5 | Analyzed current industry data, studies, and expert sources |
| Curation | Tom Hundley | Directed focus, validated relevance, ensured strategic alignment |
| Drafting | Claude Opus 4.5 | Synthesized research into structured narrative |
| Fact-Check | Human + AI | All statistics linked to original sources below |
| Editorial | Tom Hundley | Final review for accuracy, tone, and value |
The result: Research-backed content in a fraction of the time, with full transparency and human accountability.
Were an AI enablement company. It would be strange if we didnt use AI to create content. But more importantly, we believe the future of professional content isnt AI vs. Human—its AI amplifying human expertise.
Every article we publish demonstrates the same workflow we help clients implement: AI handles the heavy lifting of research and drafting, humans provide direction, judgment, and accountability.
Want to build this capability for your team? Lets talk about AI enablement →
Discover more content: