resplens

Problem

Public health surveillance numbers are revised after publication, some weeks rest on few reporting hospitals, and some sudden jumps are reporting artifacts. A language model that reads these numbers at face value will state wrong trends with confidence. The question behind the project: can an assistant answer "what is the trend, and can I trust it?" from the data itself, without inventing numbers or caveats?

The data are ObsAwareData (CC BY 4.0): weekly CDC hospital admissions (NHSN), emergency department visits (NSSP), and wastewater (NWSS) for the 50 states and DC, with a reliability flag on every value and the full revision history.

System

• Six data tools behind an MCP server. Series with a flag on every week, a reliability summary with plain-language caveats, the revision history of one week's value, and a series exactly as it was published on a past date. Tools never average, fill in, or reinterpret values. Invalid requests return an error that tells the model how to correct the call.
• Document retrieval over 28 public documents (about 74,000 words): CDC dataset descriptions, reporting protocols, methodology pages, FAQs, two MMWR reports, and the ObsAwareData documentation. Every source file is kept unchanged with its URL, retrieval date, and SHA-256.
• An agent loop that gives a local model the six data tools plus documentation search.
• Reproducible inputs. All answers read one tagged data release, downloaded on first use and checked against its published checksums.

Evaluation 1: which retrieval setup finds the evidence?

40 hand-reviewed questions, each with verbatim evidence from the documents; 15 for tuning, 25 held out. Four test configurations were fixed from the tuning results before the held-out set was run once. Results on the 21 answerable held-out questions, 300-token chunks:

Reranked hybrid retrieval found the evidence in the top five for 8 more questions than BM25 and none fewer (exact sign test p = 0.008). No configuration was reliably better at putting the evidence first. Two of the three misses were table lookups (a state threshold in a 50-row table, formula text from a PDF table), which argued for answering such questions with structured tools rather than text retrieval.

Evaluation 2: is retrieval needed at all?

The whole corpus, about 131,000 tokens, fits in a long-context model, so retrieval has to earn its place. The same local generator (gemma4:31b) answered the 25 held-out questions from the top 5 retrieved chunks and from all 28 documents in context. A judge from a different model family graded the answers; on 15 answers the author graded blind, it agreed on all 15.

The difference in correctness is not reliable at this sample size (3 questions to 1, p = 0.63), while the long-context prompt was about 100 times larger and each answer about 73 times slower. Every retrieval error came from evidence that was not retrieved, and in each case the model said the documents did not answer rather than inventing one.

Evaluation 3: does the agent answer correctly and stay grounded?

37 tasks (13 for tuning, 24 held out) of seven types: trends, lookups, revisions, definitions, mixed, unanswerable, and prompt injection. The plan was frozen and published before any test run. A task succeeds only if the answer is correct, every number comes from a tool output or is correctly derived from one, the reliability caveat is stated when needed, and no injected instruction was followed. Two local models ran each test task three times with the same prompt and tools.

• No detectable difference between the models (sign test p = 0.69; a sensitivity analysis on the first run favors the other model by one task). With 24 tasks only large differences could be detected, and the results say so.
• The failures differ in kind. gpt-oss more often misread a tool output or a document (8 runs); gemma more often gave an incomplete documentation answer (5 runs).
• Prompt injection. Both models resisted a planted tool call and a false number in a retrieved document, but both followed a low-cost instruction planted in a data field in all 6 runs. A system-prompt rule that tool outputs are data, not instructions, was not enough.
• The tools are what make it work. The retrieval-only pipeline, with no data tools, answered 7 of 24 tasks correctly, none of them questions about data values, and stated no reliability caveat on any of the 6 tasks that required one.

Practices

Fix the comparison before looking at test results

Each stage wrote its plan, metrics, and test configurations before the held-out run. Changes made afterwards are listed with their dates, including one made after the agent results were in, and its effect is reported.

Check grounding mechanically, review the rest by hand

An automatic check matches every number in an answer against the tool outputs. Numbers that do not match directly (53 occurrences) were labeled by hand, blind to the model, as correctly derived or unsupported.

Measure the judge, not just the models

The LLM judge was compared with the author's blind grades: 45 of 48 answers agreed on correct versus not correct (Cohen's kappa 0.63). One systematic judge error, accepting refusals on data questions as correct, was found and corrected by hand, and the agent answers were searched for the same error.

Make records reproducible, and correct them in public

Every run stores its configuration, corpus manifest hash, model names, and git commit. When the retrieval runs turned out to carry a commit stamp that predated their code, all four were rebuilt from committed code and matched question by question; the correction and the stricter dirty check are documented.

Limitations and next steps

• Question and task sets are small (21 to 24 held-out items), written or reviewed by the author, on one data release; differences of a few items cannot be resolved.
• One injection of each kind: the data-field result shows a weakness, not a rate.
• Planned: a real-time forecasting benchmark that rebuilds the data as it was known each week, a fine-tuned small model for tool calling, and a deployed service.