# MemoryAgentBench benchmark — Hanzo AI

> MemoryAgentBench benchmark: Hanzo Context doubles multi-hop substring EM from 33 to 67 on 262k haystacks without LLM memory reads.

[Benchmarks](https://hanzo.ai/benchmarks) / MemoryAgentBench

measured

# MemoryAgentBench

FactConsolidation: haystacks of 6k to 262k tokens where a later line silently overwrites an earlier one. A typed search over parsed facts resolves the question with no language model reading anything.

dataset

[MemoryAgentBench](https://huggingface.co/datasets/ai-hyz/MemoryAgentBench)

licence

MIT

introduced by

[Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions](https://arxiv.org/abs/2507.05257) — Hu, Wang, McAuley, UC San Diego, 2025

harness

hanzoai/benchmarks · brain

our paper

[Linkage, Not Similarity](https://github.com/hanzoai/papers/blob/main/site/public/pdfs/hanzo-context-memory.pdf)

`node results.mjs`regenerates every table on this page from the raw runs

## What is measured

FactConsolidation. A haystack is a numbered list of templated facts in which a later line silently replaces an earlier one about the same subject and relation, and nothing marks the update. Questions ask for the current value, single-hop and multi-hop, over haystacks of 6k to 262k tokens. One hundred questions per haystack. The metric is substring exact match, the benchmark’s own; an unanswered question scores zero. Dev is the 6k haystacks, test is 32k, 64k and 262k.

No language model reads the memory or writes the answer in any row here. The question is compiled into a plan sketch and a deterministic typed search over the parsed fact store resolves it — 45 templates, no line unparsed at any size, the supersession chain keyed by subject and relation. That is why these rows are not comparable to a pipeline that reads the haystack with a frontier model, and it is also the point: the retrieval is doing the work.

## Results

substring exact match · 6k from dev, 32k / 64k / 262k from test · no reader · pooled = the mean of the four haystacks

### single-hop

configuration

6k

32k

64k

262k

pooled

plain resolver

96.0

97.0

95.0

94.0

95.5

typed search

100.0

98.0

98.0

97.0

98.3

### multi-hop

configuration

6k

32k

64k

262k

pooled

plain resolver

76.0

59.0

59.0

33.0

56.8

typed search

93.0

89.0

89.0

67.0

84.5

Typed search pools to 98.3 single-hop and 84.5 multi-hop against the plain resolver’s 95.5 and 56.8. The largest single move is 262k multi-hop, from 33.0 to 67.0.

## Numbers other systems publish

Quoted with their protocols, because the protocols differ, and none of them re-run here. CAR (arXiv 2606.01435) reports 78.0 single-hop / 30.2 multi-hop pooled over 6k–262k with gpt-4o-mini and 94.8 / 51.5 with gpt-4o, and 82 / 27 and 93 / 41 respectively at 262k. Its own baselines at 262k multi-hop: HippoRAG-v2 5, gpt-4o whole-context 5, BM25 3, Cognee and MemGPT 3, Mem0 2, Zep 3. That is the nearest thing to a like-for-like comparison anywhere on this site — same data, same split, same substring-EM metric — and it is still not one, because CAR reads the haystack with a frontier model and the rows above use no model at all.

Pith publishes per-lane multi-hop figures on a vendor page: 83 / 84 at 32k and 64k, where typed search scores 89.0 / 89.0.

Hanzo AI Cloud

### Build on Hanzo Cloud

[Start on Hanzo Cloud](https://cloud.hanzo.ai)[View Models & Pricing](https://hanzo.ai/pricing)

[PreviousLoCoMo](https://hanzo.ai/benchmarks/locomo)

[All Benchmarks Board](https://hanzo.ai/benchmarks)

[NextLongMemEval](https://hanzo.ai/benchmarks/longmemeval)

### Related in memory

[LoCoMo](https://hanzo.ai/benchmarks/locomo)[LongMemEval](https://hanzo.ai/benchmarks/longmemeval)[LoCoMo · subject scope](https://hanzo.ai/benchmarks/locomo-subject-scope)[LoCoMo-Conv](https://hanzo.ai/benchmarks/locomo-conv)

Other benchmarks[CronQuestions](https://hanzo.ai/benchmarks/cronquestions) · [RepoBench-R](https://hanzo.ai/benchmarks/repobench-r) · [Fleet residency](https://hanzo.ai/benchmarks/fleet) · [Live agent footprint](https://hanzo.ai/benchmarks/goroutine) · [Sandbox cold start](https://hanzo.ai/benchmarks/sandbox) · [Engine prefill on CUDA vs llama.cpp](https://hanzo.ai/benchmarks/inference) · [GPQA-Diamond](https://hanzo.ai/benchmarks/gpqa)

raw runs, plans, traces and scoring: hanzoai/benchmarks · brain · every table above regenerated by `node results.mjs`