When a cancer is unusual or advanced, the decision about how to treat it often goes to a molecular tumor board: a room of oncologists, pathologists, and geneticists who read the scans, the biopsy, and the tumor's DNA together and agree on a plan. It is slow, expensive, expert work. So it is a natural target for AI, and several groups have now built agents that try to do the same job: read a patient's file, use tools, consult treatment guidelines, and write a recommendation.
These agents are rarely a single model and are instead reasoning systems with the ability to collect information via tool calls ranging from medical document searches to medical image processing. Many engineering decisions come together to make such a system. To explore these design decisions, we build an AI tumor board model and examine three subsystems within it: the agent harness, the guideline search, and the image processing. This piece walks through what we found, and ends with an interactive viewer where you can open any of the 20 cases, pick any version of the agent, and read exactly what it did and how it was graded.
Everything here is measured on ferber20: 20 real molecular-tumor-board cases published with a study from a clinical group. Each case gives the agent what the board would have: a patient history, a molecular report listing the tumor's mutations, and radiology scans, ending with an open question like "what are the next steps in her treatment?"
Grading a free-text treatment plan is the hard part. For each case, the original clinicians wrote down the specific things a good answer must contain, and across the 20 cases there are 109 such criteria in total. A criterion is a concrete clinical point, for example:
Four of the seven things a complete answer should mention:
The agent's answer is scored on how many of a case's criteria it actually covers, we call this its completeness. Because reading a long clinical answer against a checklist is itself a judgment task, we use a large language model as the grader (an "LLM-as-judge": one AI model reading another's answer and deciding, criterion by criterion, whether it was addressed).
How consistent is the LLM judge with expert oncologist? On the criteria the original study had four oncologists vote on, our grader matches the doctors' consensus 92.6% of the time. For comparison, the individual oncologists agreed with their own consensus 94.0% of the time, and agreed with each other 88.7% of the time.
Completeness is the one measure checked against oncologists. We also report four secondary measures, correctness, safety, citation grounding, and helpfulness, but these are approximate AI-graded proxies.
The agent, which is based off the Ferber et al original, runs in two stages. First it gathers evidence: it reads the scans, looks up the tumor's mutations in a cancer-genetics database, searches the medical literature, and does any needed calculations. Then it retrieves guidelines and writes: it pulls relevant passages from a library of oncology treatment guidelines and drafts a recommendation that cites them, with a self-check pass at the end.
The three subsystems we take apart are highlighted below. Click any one to jump to what we found.
The harness is the scaffolding around the model: the tools it can reach, and the script that decides when to call them. We test the utility of the harness and different ways of wiring up tools. The full agent scores +15 percentage points higher on completeness than the same strong base model answering with no tools at all (79% versus 64%).
Three findings fall out of this. First, general web search adds nothing: bolting a search box onto the model left completeness unchanged. What helps is not access to the open web but access to the right domain tools. Second, which model API you drive the tools through made no measurable difference (a 2.6-point gap that is well within noise). Third, and least expected: letting the model's own agent runtime drive the loop, instead of an explicit script that says "read the scan, then look up the mutation, then retrieve guidelines," made the agent worse by about 9 points. More autonomy was not better here; the model skipped steps it should have taken.
The harness is a plain server-side loop, not the model's own agent runtime. It builds the first message from the patient case, the clinical question, and a list of the imaging files on disk, then hands the model a fixed menu of tools (catalogued below). Each turn the model either emits a tool call, which the harness executes and feeds back as the next message, or writes its final free-text plan, which ends the loop. Imaging is gated behind an explicit request protocol: the model asks for a file with [REQUEST: <filename>] and the harness returns the tool's read of it, so the model never sees a scan it did not ask for.
The two stages are just an ordering the script enforces, gather evidence (imaging read, genetics lookup, literature, calculator), then retrieve guidelines and write. The faithful arm runs exactly this script. The two API-transport arms run the same script and tools but route calls through the chat completions API versus the Responses API; they tie. The model-driven arm removes the script: it exposes the same tools but lets the model's built-in tool-use runtime decide the order and when to stop. That is the arm that loses ~9 points, because the model ends the loop early and skips evidence steps the script would have forced. The web-search arm is a vanilla model given only a web-search tool, no domain tools, no script, and it lands at the bare-model level.
Stage 2 depends on pulling the right passages from a library of treatment guidelines, the classic "retrieval" problem. There are many ways to do it, and strong opinions about which is best: search by meaning (vector search) or by keyword; add a reranking step or not; use a home-built index or a hosted one; feed the model a short snippet of each passage or the whole thing. We tried variations across these options and did not find they affected the model's score substantially. However, this may result from the benchmark evaluation saturating, whether because the rubric is too coarse or because the case studies do not need granular search capabilities.
Swapping the search engine did nothing, plain vector search, vector search with a reranker, and two hosted search services all landed within a couple of points of each other. Searching the guideline library by keyword (as if grepping a folder of files) tied with searching it by meaning on completeness, though the keyword version was a touch weaker on correctness and citation quality. And how much of each guideline passage the model reads, a truncated snippet, the full chunk, or the whole surrounding section, made no difference either.
Retrieval is a single tool the model calls in Stage 2. When it fires, the model first proposes a handful of sub-queries (it decides how many) rather than reusing the raw question; each sub-query is run against the guideline corpus, and the merged hits become the passages the model may cite. The corpus is 413 guideline documents across four sources (catalogued below), chunked and, for the vector engines, embedded into a Chroma store.
The knobs we varied all sit inside this one tool, as a swappable engine setting. The engine decides how a sub-query finds passages: cosine vector search over an embedding store (chroma_cosine), cosine plus a Cohere reranking pass, or a hosted file-search index (tried over two different APIs). The paradigm swaps that whole vector path for a lexical one: the filesystem variant keeps the same rag tool but sets its engine to fs_bm25, a keyword BM25 search over the corpus files, instead of embeddings. The grounding window controls how much of each hit the model actually sees: a truncated snippet (~700 characters), the full retrieved chunk, or the chunk expanded to its parent document section. After drafting, a citation self-evaluation pass re-reads each cited passage, flags any citation the text does not support, and revises. Every one of these lands in the same 78–82% band.
The third subsystem lets the agent use the patient's scans. Giving the agent image access lifts completeness by +8.4 points. But the interesting part is where that lift lands.
Of the 109 grading criteria, 25 are things you can only say if you actually looked at the scan (a description of a lesion, whether disease has progressed). The imaging lift lands almost entirely on those:
Two design questions follow. One: is a measurement tool, software that outlines and measures the lesion, worth adding on top of a plain radiology read? Barely. Its extra contribution was about 2 points and not statistically distinguishable from zero; the qualitative read already captures most of what the scan offers, unless you specifically need the tumor measured. Two: should the model read the scan pixels directly, or go through a separate vision tool that writes a text report? On answer quality the two tie, and reading pixels directly is simpler and cheaper.
One qualitative pattern is worth flagging, with a heavy caveat. When the model read the scan pixels directly, it sometimes stated an imaging-derived conclusion, such as the cancer's stage, as established fact rather than a provisional read, and then let that conclusion drive a major recommendation. In one case the pixel-reading agent opened with a firm stage-IV diagnosis taken straight from its own look at the CT slices:
Doe, 53, has newly diagnosed colorectal carcinoma with radiologic stage IV disease: a large, multifocal metastatic lesion occupying much of the right hepatic lobe and multiple bilateral pulmonary nodules, consistent with liver and lung metastases.
In another, it turned a single small lung nodule that had been under watchful waiting into confirmed metastatic disease:
Follow-up CT imaging initially showed a single small lung nodule; the most recent CT now demonstrates multiple bilateral, round, solid pulmonary nodules consistent with hematogenous metastases.
Passages like these read less like a hedged radiologist ("findings suggestive of…, recommend confirmation") and more like a committed diagnosis off a single slice. It seemed to come up more when the model read pixels directly than when it worked from a written radiology report, which at least puts an intermediate description in between, and adding a "confirm before you commit" step (asking the model to re-check any image-derived claim before stating it) seemed to make it rarer. We want to be careful here, though: these are a handful of examples surfaced by an LLM judge, not a calibrated safety measurement, so we offer them as a possibility worth watching rather than a settled finding. You can read the flagged runs in full in the case viewer.
There are three distinct ways the agent can see a scan, and they are separate code paths. Radiology read: the model calls radiology_report(path_to_img_folder, query) and a vision model returns a written report answering the query, the model works from that text, never the pixels. Measurement: the model calls medsam(path_to_img, bbox_coordinates), which runs a MedSAM segmentation over the box it supplies and returns the outlined lesion's size. Native multimodal: no imaging tool at all, the scan is attached to the message and the model reads the pixels directly. The five imaging arms are the on/off combinations of these plus a no-imaging floor. A separate histology_classifier(patient_id, targets) tool replays the case's pre-computed MSI / KRAS / BRAF calls; it is not scored as an imaging arm but is part of the evidence stage.
The safety arm changes only the loop, not the tools. The standard imaging agent states what it read straight into its plan. The confirm variant inserts one extra pass before the answer is finalized: the model must re-examine every claim it drew from an image and either back it with a second source or soften it, which is what drives over-trust to zero. Because reading pixels directly (native multimodal) has no intermediate text report to sanity-check against, it was the mode most prone to over-trust before the confirmation step was added.
Here is the exact toolbox the agent could reach and the exact corpus it retrieved from.
Eleven tools the model can call. Each row is the tool's name, what it does, and the arguments the model fills in.
| Tool | What it does | Inputs (arguments) | Calls |
|---|---|---|---|
| Stage 1 — gather evidence | |||
| oncokb | Looks a tumor variant up in the OncoKB cancer-genetics knowledge base (is it oncogenic, is it a drug target). | hugo_symbol, alteration, change | 3,180 |
| medsam | Runs a MedSAM segmentation over a bounding box on a scan and returns the outlined lesion's size (the measurement tool). | path_to_img, bbox_coordinates | 2,335 |
| calculate | A guarded two-operand arithmetic calculator (doses, intervals, simple ratios). | a, b, operator | 2,261 |
| pubmed | Searches the PubMed literature and returns matching abstracts. | pubmed_search_terms, query | 1,769 |
| histology_classifier | Replays the case's pre-computed molecular calls (MSI status, KRAS, BRAF) for the requested targets. | patient_id, targets | 1,735 |
| radiology_report | Sends a scan folder to a vision model and returns a written radiology report answering the query (the qualitative imaging read). | path_to_img_folder, query | 1,628 |
| Stage 2 — retrieve guidelines & write | |||
| rag | Guideline retrieval. The model proposes sub-queries and the tool pulls matching passages from the corpus, then a citation self-check re-reads and revises. Its engine is the swappable retrieval backend, and is where the engine / paradigm / grounding-window studies live: chroma_cosine (vector), fs_bm25 (lexical BM25 over the corpus files — the “filesystem grep/BM25” variant), or a hosted file-search index. | mode, engine, subqueries, n_subqueries, citation_selfeval, n_reads | 1,700 |
| Filesystem primitives — direct file access over the same guideline corpus | |||
| list_dir | Lists the guideline corpus as a directory tree. | path | — |
| grep | Keyword-greps the guideline files for a pattern. | pattern, path, max_results | — |
| read_file | Opens a slice of a specific guideline file. | path, offset, limit | — |
| Web search — the tool behind the harness “web search adds nothing” comparison | |||
| web_search | OpenAI-native web search: makes a nested Responses-API web_search call and returns a source-cited summary of current authoritative oncology information (FDA labels, NCCN / ESMO / ASCO guidance, pivotal trials). This is the tool the harness section’s “web search adds nothing” finding refers to; in the web-search arm the model actually searched in ≈93% of rollouts and completeness still did not improve. (The chat-completions arm exposes it as this function tool; the Responses-backbone arms attach OpenAI’s hosted web_search tool directly.) | query | ≈93%* |
The lexical filesystem grep/BM25 retrieval variant is the rag tool run with its fs_bm25 engine (a BM25 keyword search over the corpus files), a drop-in alternative to vector search inside the otherwise-fixed pipeline. The three standalone filesystem primitives above (list_dir, grep, read_file) let a model open and search corpus files directly; they are part of the toolbox but were not invoked by any of the variants shown here, so their call counts are blank. The original paper’s google_search tool (a nested Google Custom Search agent) is not listed: its whole-web endpoint was discontinued, so it was replaced by the OpenAI-native web_search above and left gated off and unused in our runs. * The web_search figure is the share of the harness web-search arm’s rollouts in which the tool was invoked; the other counts are absolute call totals from the imaging and retrieval rollouts, where web search is disabled.
The rag tool searches a fixed library of 413 guideline documents drawn from four sources. This is four of the six sources the original paper used: the two others, MDCalc and UpToDate, are paywalled and were left out.
| Source | Documents | What it is |
|---|---|---|
| MEDITRON | 175 | A large general-medical guideline collection (immunization guides and other clinical documents); the broadest and most heterogeneous source. |
| Onkopedia (DE) | 118 | The German-language Onkopedia oncology guidelines. |
| Onkopedia (EN) | 44 | The English-language Onkopedia oncology guidelines. |
| ESMO | 50 | European Society for Medical Oncology clinical practice guidelines. |
| ASCO | 26 | American Society of Clinical Oncology clinical practice guidelines. |
| Total | 413 | Onkopedia appears in two languages, so the four sources span five folders. |
Each document carries a <!-- source | title --> header and is cited as [source: title]; the case viewer shows the exact passages each run retrieved and how the model cited them.
Every number above comes from running these agents on the 20 cases, five times each. The case viewer is its own page: pick a patient case and a version of the agent, and it shows exactly what was fed to the model (the patient case, the prompt, the imaging), the agent's evidence-gathering loop, its full written recommendation, and every rubric score with the judge's reasoning.
Interactive Open the case viewer → 20 cases × 22 versions of the agent × 5 runs — inputs, trace, answer, and every score.A note on what's shown: the completeness grader saved which criteria were met but not a sentence of reasoning per criterion, so you'll see the checklist of hits and misses. The other axes carry the grader's written rationale. A few older variants (the harness-component runs) saved their scores but not their full transcripts; those show the scores and reasoning and mark the answer as unavailable rather than reconstructing it.