
chopper
io.github.wiiiktor/chopper
Cuts the exact quotes that answer a question out of long text and reranks passages, in one pass.
Documentation
70-second video: youtu.be/KMvmdXCEOJU
Try it in your browser, nothing to install: English demo ยท Polish demo
They run the recommended base chopper (0.31B) on your CPU, with small (0.14B) as a faster option.
See it on real examples: 20 English questions, Chopper vs Opus 5.5 vs gold (PDF)
Where this is going. The standalone chopper is a first, practical step. The main idea is an integrated chopper: chopping built into the answering LLM itself, reading the model's own cache of the session instead of re-encoding the text in a second model, which should give the highest possible quality and speed. A proof of concept with MiniCPM3-4B is described here.
Chopper 0.3B close to Opus 5.5 in cutting out the best pieces
Chops are pieces cut out of the original text snippets retrieved from a database or a long conversation: exact quotes, never rewritten. The chopper (mmBERT, 0.31B) makes them; for comparison, Claude Opus 5.5 makes them too, also forced to quote directly.
What is new: a new method, invented in November 2026 in Poland (EU). The chopper is not an LLM summarising the snippets: a small encoder reads the question and all 20 snippets together in one pass and marks every word keep or drop. There is no text generation, so it is fast: 50 ms per question (20 snippets, ~2,400 tokens) on a consumer-grade GPU (RTX 3090), 36 ms with mmBERT-small.
One database retrieval per question, even at 4 hops. Every result below uses a single retrieval: each question gets its 20 snippets once, the chopper cuts them once, and Qwen3-14B answers once. There are no follow-up searches and no second rounds, so a 4-hop question has to be chained from one cut.
A reranker that also gives you the chops
One model replaces both the reranker and the compressor. Full comparison: below.
Qwen3-14B answering from full snippets or just short chops
400 MuSiQue questions, 100 each at 1, 2, 3 and 4 hops, each with its own 20 retrieved snippets. Qwen3-14B (4-bit, no fine-tuning, no thinking mode) answers from the given text only; gpt-5.6-luna judges. Length = prompt tokens. Each score counts every correct answer, including those from incomplete chops; the number in brackets counts only questions whose chops kept every hop and the answer.
Qwen3-14B: all snippets vs chops from the chopper vs chops from Opus 5.5
Read each diagram like website traffic: questions flow left to right, and at every hop the ones whose fact the chops missed drop out; the last stage is Qwen3-14B answering correctly. Top: all 20 retrieved snippets. Middle: the chopper's chops. Bottom: Opus 5.5's chops.
The last stage is the table's score; the lighter block under it counts answers Qwen got right from incomplete chops, and the number in brackets leaves them out. More: docs/DETAILS.md ยง7.
Also a reranker: ranking and chopping in one pass
The chopper scores every passage while it chops (mean p(KEEP) of the passage's 16 most likely tokens). On the 8-language rerank holdout (400 queries, 16 passages each, never trained on), ranked against dedicated rerankers:
| ranking 16 passages | nDCG@10 | Hit@1 | GPU time per query (RTX 3090) |
|---|---|---|---|
| chopper, mmBERT-base (0.31B), one pass | 0.973 | 0.972 | 0.20 s, chops + ANSWER/SEARCH included |
| bge-reranker-v2-m3 (0.57B) | 0.984 | 0.985 | 0.20 s |
| Qwen3-Reranker-0.6B | 0.977 | 0.972 | 0.77 s |
| Qwen3-Reranker-4B | 0.986 | 0.990 | 3.3 s |
The chopper was never trained to rank, yet it is within about one point of the best rerankers at the speed of the fastest,
so a pipeline can drop its reranker: retrieve โ chopper gives the ranking and the quotes. Caveats (this test is easy
for every model; the labels favour Qwen3-Reranker-0.6B) in the details.
Against other context compressors
Same 400 questions, same 20 snippets, same Qwen3-14B prompt; each compressor cuts the snippets once:
| Qwen3-14B answering from | tokens | 1-hop | 2-hop | 3-hop | 4-hop |
|---|---|---|---|---|---|
| all 20 snippets (no compression) | 2,620 | 84 | 28 | 22 | 12 |
| LLMLingua-2 (Microsoft), rate 0.1 | 374 | 28 | 6 | 2 | 4 |
| Provence (Naver), its default threshold 0.1 | 263 | 77 | 27 | 26 | 16 |
| Provence, threshold 0.5 | 152 | 71 | 24 | 21 | 11 |
| chops from the chopper, mmBERT-small (0.14B) | 217 | 72 | 49 | 32 | 23 |
| chops from the chopper, mmBERT-base (0.31B) | 228 | 83 | 48 | 45 | 29 |
LLMLingua-2 drops tokens without looking at the question, so the facts a chain needs rarely survive. Provence reads the question but prunes each snippet on its own. To be fair to them: the chopper was trained on MuSiQue's train split (this test uses its dev split) and on other multi-hop data, while both were trained for general single-passage use.
Chopper ๐ช keep only the part of RAG that answers the question
Standard RAG (retrieval-augmented generation) pastes whole passages into your model's context, and they stay there. About 95% of those tokens don't help. After a few lookups, a long session is mostly retrieval spam.
Chopper is a small model that reads the retrieved passages first and keeps only the fragments that matter. Your model then sees ~100 tokens instead of ~3,000 per lookup, and a session can carry dozens of lookups.
question โโบ retrieve 16 passages โโบ ๐ช rank + chop in one pass โโบ model answers, or asks one follow-up search
- ๐ฏ Clean context: only useful fragments enter the session; the raw passages never stay.
- โก Less compute: the main model reads 2โ5% of the retrieved text.
- ๐ Also a reranker: the same pass ranks the passages almost as well as bge-reranker-v2-m3 (nDCG@10 0.973 vs 0.984, 8 languages), at the same speed. No separate reranker needed.
- ๐ Multi-hop works: the model writes a follow-up question when a fact is missing, and searches again.
- ๐ Built on one home GPU: every dataset and model here was made on a single RTX 3090 (some labels came from APIs).
Results at a glance
| retrieved text the main model reads | 2โ5% (about 20โ40ร less); the answer is still in the kept text for 85% of HotpotQA and 69% of TriviaQA questions |
| "nothing here": pools whose answer was deleted (8 languages, business documents) | asks to search again 67% of the time, keeps 1.4% of the text |
| ranking 16 passages (8-language holdout, 400 queries) | nDCG@10 0.973, vs bge-reranker-v2-m3 0.984 and Qwen3-Reranker-4B 0.986; 0.20 s like bge, 16ร faster than Qwen-4B, chops included (details) |
| chopping time per lookup (RTX 3090) | 155 ms (separate mmBERT) ยท 277 ms (built into MiniCPM3-4B) |
| 4-hop MuSiQue questions answered correctly by Qwen3-14B, one retrieval | 12% โ 29% with the chopper's chops instead of all 20 snippets (11ร fewer tokens); with Opus 5.5's chops: 43% |
| Polish questions (PoQuAD, PolQA) answered correctly by Qwen3-14B | 85.6% from the Polish chopper's chops (255 tokens) vs 85.2% from all 16 snippets (3,856 tokens) |
| license | MIT: free for commercial use, weights and code. The closest alternatives, Provence and XProvence (Naver, sentence pruning + reranking), are CC BY-NC-ND 4.0: non-commercial only |
All quality numbers in this repository (F1, recall, answer accuracy, nDCG) were measured on raw chops, without word or sentence snapping. Since chopper-rag 0.1.7 the package extends chops to whole sentences by default (97% of chops then end at a sentence end, vs 36% raw; about 4.1% of the text kept instead of 3.4%; details).
In training (Nov 2026), suggested by the OPI PIB AI Lab: the same Polish chopper recipe on two Polish encoders, pl-ModernBERT-base from OPI PIB (results expected about 16 Nov) and polish-roberta-base-8k (8k context, about 18 Nov), compared with mmBERT-base-pl on the same 856 Polish questions. Restarted 15 Nov with the training data fitted to the Polish tokenizers (why).
Long-conversation memory results (LoCoMo, LongMemEval) are in the details: there the 4B answering model is the limit.
Pick your chopper
| ๐ง universal | ๐ช integrated | |
|---|---|---|
| what it is | a separate small encoder (mmBERT-base, 0.31B, recommended; or mmBERT-small, 0.14B; both also in Polish versions) | a 20% slice of MiniCPM3-4B's own weights + a small LoRA |
| works with | any answering model: text in, text out | MiniCPM3-4B |
| extra memory | 0.31B / 0.14B weights | 45M parameters; reads the model's own cache |
| weights | ๐ค wiiiktor/chopper_mmbert | ๐ค wiiiktor/chopper_minicpm3 |
Start with the universal one. The integrated one is the long-term bet: trained together with a model from the start, it should become almost free. If you pretrain models and want to build that, contact wiiiktor.
Quick start
No install: try the in-browser demo (po polsku). It runs base (0.31B) by default, like the code below, or small (0.14B).
In your own RAG pipeline (LangChain, LlamaIndex, Haystack or plain Python):
pip install chopper-rag # or "chopper-rag[langchain]", [llama-index], [haystack], [all]
As a tool for Claude (Claude Code, Claude Desktop or any MCP client): pip install "chopper-rag[mcp]", then
claude mcp add chopper -- chopper-mcp. Claude can then call chop_passages / chop_files to read a few quotes
instead of whole documents, or rank_passages to rerank; setup in integrations/chopper-rag.
from chopper_rag import Chopper
chopper = Chopper.from_pretrained(variant="base") # or "small"; for Polish "base-pl", "small-pl"
result = chopper.run("When was the Eiffel Tower completed?", passages) # passages: list of retrieved texts
print([c.text for c in result.chops], result.decision) # exact quotes + ANSWER / SEARCH
print(result.ranking(top_k=5)) # the same pass as a reranker: [(passage_index, relevance), ...]
Framework examples: integrations/chopper-rag. Listed in the LangChain integrations (ChopperCompressor). Or try the demo in this repo:
pip install -r requirements.txt
# your documents (a folder of .txt / .md, or a .jsonl with "title" and "text") + any Hugging Face causal LM
python operator-for-llm-with-mmbert.py --docs my_docs/ --model <hf model id> \
--ask "Who founded the company that distributed the film UHF?"
# leave out --ask for an interactive session; it shows how many tokens plain RAG would have added
Weights download from Hugging Face on first use. Training data: ๐ค wiiiktor/chopper_demo.
Want the details?
Everything else is on one page, docs/DETAILS.md:
- how it works: double prefill, rollback, the answer-or-search decision;
- datasets and training on one GPU;
- all results, including the search loop and the 8-language rerank pilot;
- rewiring a model into its own chopper and what comes next;
- credits, licenses and related work.
Credits
Built on mmBERT (JHU CLSP), MiniCPM3-4B (OpenBMB), Qwen3, Qwen3-Embedding and Qwen3-Reranker (Qwen team), bge-reranker (BAAI); labels and judgements also came from Claude Opus 5.5 (Anthropic) and gpt-5.6-luna / GPT-6 Luna (OpenAI). Compared with LLMLingua-2 (Microsoft) and Provence (Naver). Data: the MuSiQue, HotpotQA, 2WikiMultiHopQA, TriviaQA, QASPER, Multi-Session Chat, WildChat, FineWeb, Wikipedia, LoCoMo and LongMemEval datasets; for Polish, PoQuAD and PolQA (IPI PAN). Our memory evaluation follows Memory Tax (He et al., 2026) and compares with MemT-1.0 (MemTensor, 2026). Full table with licenses: docs/DETAILS.md.
Licenses
Commercial use is allowed. The mmBERT choppers and all code are MIT. For comparison, Naver's Provence and XProvence context pruners are licensed CC BY-NC-ND 4.0 (no commercial use, no derivatives).
| what | license |
|---|---|
mmBERT choppers, incl. Polish (mmbert-base, mmbert-small, mmbert-base-pl, mmbert-small-pl) | MIT (as mmBERT) |
code in this repo (LICENSE) and pip install chopper-rag | MIT |
| MiniCPM3 chopper | MiniCPM Model License (as MiniCPM3-4B) |
training data wiiiktor/chopper_demo | CC BY-SA 4.0 (share-alike from HotpotQA, Wikipedia and PolQA); Polish parts: PoQuAD CC BY 4.0, PolQA CC BY-SA 4.0 |
Some labels came from Claude Opus 5.5 and OpenAI Luna models, used to train a non-competing extraction tool; see docs/DETAILS.md.
Bored with reading LLMs' outputs? See also my other project, control_room: a VS Code panel for Claude Code that shows each answer as cards, charts and clickable options instead of a wall of text.
chopper-ragpip install chopper-rag


