Benchmarking Bitemporal Agent Memory in Production — Including the Bugs It Caught

Most "AI memory" benchmarks are synthetic: a curated corpus, cherry-picked queries, one flattering run. We did the opposite — we benchmarked FlukeBase's memory layer against our **own live production store** (3,418 memory records in May; 5,700+ today, roughly 2,000 of them active and searchable — the rest archived or superseded, with full bitemporal history preserved), using strict single-target probes, and committed up front to publishing whatever came out.

What came out was better than a clean scorecard: **the benchmark caught two real bugs and a data-quality regression before we shipped a single marketing number.** This article reports all three runs, the failures, and the fixes.

Methodology

- **Corpus**: the real FlukeBase production memory store — incident post-mortems, strategy decisions, CI diagnoses, competitor intel, baselines. No synthetic documents. Recall probes search the **active** set (~2,000 records at the July runs); archived and superseded records remain queryable through bitemporal history.

- **Probes**: 7 labeled queries, each with exactly one pre-registered target memory. Paraphrased phrasing (no keyword copying from the target).

- **Metric**: strict single-target rank in hybrid recall (vector + full-text merge, composite scoring 0.7 semantic / 0.2 recency / 0.1 importance), top-10 window. recall@k and MRR over the 7 probes.

- **Integrity rules** (set before the first run): no fabricated numbers, no fake head-to-head against products that don't have a memory layer, no cherry-picking runs, and *strict-target misses count as misses even when topically-correct sister memories rank #1*.

Strict single-target scoring deliberately **understates** quality: when four memories all correctly describe the same incident, six of them are "wrong answers." We report it anyway because it's the honest lower bound.

Results across three runs

| Metric | May 24 (3,418 records) | Jul 2 pre-remediation (5,709 records) | Jul 2 post-remediation (~2,000 active) |

|---|---|---|---|

| recall@1 | 0.43 | 0.14 | 0.14 |

| recall@5 | 0.71 | 0.57 | **0.71** |

| recall@10 | 1.00 | 0.57 | **0.86** |

| MRR | 0.57 | 0.24 | **0.33** |

Store→recall round-trip latency: **4 ms** (system eval suite, 7/7 pass, May run). Query embedding on self-hosted TEI (BAAI/bge-m3, 1024-dim): ~1–2 s CPU inference.

What the benchmark caught

Run 1 (May): two real bugs → publication blocked

The May run's headline capability was **bitemporal point-in-time recall** — "what did the agent know on March 1st?" Live testing it failed: `as_of_date` was silently ignored, returning May memories for a March query. A second probe silently degraded from hybrid to full-text with no signal to the caller.

We deferred publication. Both bugs got fixed the proper way:

- `as_of_date` is now parsed and threaded into both the vector and full-text paths, with a regression test that stores memories at t0/t1/t2 and asserts future-dated exclusion.

- Recall responses now return `search_mode_used` (`"text_fallback"` + a reason note whenever the vector path could not run) — degradation is a first-class, caller-visible signal, covered by regression tests.

Run 2 (July, pre-remediation): store pollution measured in lost recall

Between runs, an automated competitor-monitoring loop flooded the store with content-free stub memories ("Competitor content changes detected (66 total in this run)…" — 681 active stubs, 6–119 characters each). Because composite scoring rewards recency, fresh junk outranked real content: two probes returned *ten junk results and no target*.

This is the finding we'd want a memory platform to surface: **recall quality is a data-hygiene property, not just an algorithm property.** A memoryless stack can't even have this problem — or measure it.

Remediation (same day, through the platform's own tools)

1. Embedding backfill: every active memory now has a current-model vector (the store had accumulated a blind spot of un-embedded rows after an embedding-model migration).

2. Archived the 681 content-free stubs via the audited admin path — archived, not deleted; the bitemporal history is preserved.

3. recall@5 returned to May parity; recall@10 recovered to 0.86.

The bitemporal demo (live, unedited)

Query: *"flukebase-tei outage embeddings"*

- **Unfiltered**: top results are the June 21 incident post-mortem and the June 30 re-enablement notes — correct and current.

- **With `as_of_date=2026-06-01`**: every June memory is excluded; results are only what the system knew before June 1. Point-in-time recall, on live data, with version history intact.

This is the capability gap we'd point auditors at: agent decisions can be replayed against *what the agent knew at the time* — not reconstructed from guesswork.

Honest caveats

- n=7 probes is a smoke-scale benchmark, not a leaderboard. Its value is that it runs against production data and is re-runnable after every incident.

- Strict single-target scoring punishes near-duplicate correct answers (probe 2's true answer ranked #3 as a sister memory; strict scoring counts that a miss).

- recall@1 remains the weakest axis — near-duplicate real memories compete for rank 1. That's the next data-hygiene target (memory compaction), not a retrieval change.

- The May latency/eval numbers (4 ms, 7/7) are from the May run; July re-verified retrieval quality, TEI-backed semantic mode, and bitemporal filtering live.

Why we publish it this way

Every number above is reproducible against our own production system, failures included. If a benchmark can't embarrass you, it can't inform you either. Supervised autonomy is our whole thesis — see [/trust](https://flukebase.me/trust) — and it applies to marketing claims too: measured, attributable, and auditable.