For MLSys'26 we deployed an artifact-evaluation agent on HotCRP as a real reviewer. That deployment was a field experiment: AE submission, author discussions. Most people who want to know whether a paper's code actually runs have none of that. They have a PDF.

So we isolated the core around reproducibility. The result is a paper reproducibility agent: a standalone tool that takes just a paper PDF (local path or URL) and returns a reproducibility report—find the open-source repo, clone it, install and run it, and optionally verify the paper's claims claim by claim.1

Architecture#

The code is three layers:

  • repro_cli.py — the entry point. It resolves the PDF (path or URL), picks a mode, and owns report I/O. Like the AE agent's CLI, it is the boundary: the model proposes; the harness executes and writes.
  • paper_agent/ — the reproduction logic. Model-facing tool definitions live here, along with a tool session that keys every action to one paper: PDF reading with link extraction, repo search, and web search.
  • agent_core/ — the machinery shared with the AE agent: provider-agnostic LLM backends, normalized tool transcripts, history eviction, checkpoint/resume, and the low-level executors. VMExecutor runs commands as a local subprocess or over SSH on a remote GPU box; pdf, github, and web handle extraction, inspection, and fetch.

A run is a pipeline, not one long conversation:

  1. Prep sub-agents, in parallel: summarize the paper, find the repository, extract the claims (deep mode), and inspect the environment to draft an install-and-run plan.
  2. Main loop: the ReproAgent executes the plan—clone, install, run, read logs—with checkpoints so a crash never erases hours, and history eviction plus token budgets so a long run stays inside the context window.
  3. Review: per-claim adjudicators judge each claim against the execution evidence, an integrity auditor hunts for artifacts engineered to merely look reproducible, and a report critic attacks overconfident verdicts before anything is written.
  4. Report: Markdown, JSON, and PDF, plus a timestamped action log. A dependency-free local web dashboard (dashboard.py) launches, watches, and stops runs.
Architecture of the paper reproducibility agent: PDF input, parallel prep sub-agents, ReproAgent main loop with VMExecutor and checkpoints, claim adjudicators and report critic, and Markdown/JSON/PDF reports
Prep in parallel, one main loop, adversarial review

The shape is the AE post-mortem turned into structure. Prep runs as parallel sub-agents because the AE agent burned its context on one long loop. The critic gate exists because the AE agent once submitted text that lagged its own comment thread. Budgets and checkpoints exist because unbounded retry is not diligence. And every session is keyed to a single paper because cross-paper bleed is worse than silence.

Sub-agents: narrow remit, strict output#

The main loop is a generalist; everything around it is a specialist. Seven sub-agents share one harness (run_tool_subagent in paper_agent/subagents.py), and the harness—not the prompt—enforces the discipline:

  • Bounded by construction. A sub-agent gets a short tool loop (at most a dozen turns, 2k tokens each), so it cannot wander the way the AE agent's one long loop did.
  • A tool allowlist enforced at dispatch. The paper summarizer can only read the PDF; the repo finder adds web and GitHub search; the report critic gets no tools at all. A disallowed call is answered with an error result, so the model self-corrects instead of escaping its remit.
  • Strict tagged-JSON output. Each specialist must end with a single <paper_summary>, <repository>, <verdict>, … block. Narrow remit plus schema-checked output produces far cleaner intermediate artifacts than one prompt asked to do everything.
  • Degrade, never block. A failing sub-agent returns a safe default instead of raising, so no specialist can kill a run. An adjudicator that never emits a verdict is marked unresolved rather than erasing the main run's own finding—and it gets one retry with zero tools advertised, which reliably makes a weaker model answer instead of browsing again.
  • One shared trace. All sub-agents share the run's tool session and append to the same action timeline, tagged by sub-agent, so the report shows exactly who did what.

The most interesting specialist is the integrity auditor. The main loop is deliberately charitable: follow the README, run what the authors say to run. The auditor assumes the opposite—that the artifact may be engineered to look reproducible—and tries to disprove the headline claims. It greps for the paper's literal numbers in committed files, checks whether "reproduce" scripts recompute results or just re-plot cached ones, diffs the baseline's invocation against the method's, and compares the paper's stated scale with the script's actual flags. Reporting zero concerns is a valid outcome; unfounded suspicion is treated as a bug, just like unfounded confidence. The charitable pass plus the adversarial pass is the AE badge separation carried one level deeper.

What upgraded#

AE agent (conference) Paper reproducibility agent (general)
Needs HotCRP account + assignment Needs a PDF
Posts comments / submits badges Writes Markdown, JSON, and PDF reports
Badge criteria (Available / Functional / Reproduced) Verdicts: reproducible / partial / not / inconclusive
One long review loop per paper Prep sub-agents in parallel, then a main loop, then adjudicators, an integrity audit, and a report critic
Human-in-the-loop for author Q&A Local web dashboard to launch, watch, and stop runs

Two modes cover different depths:

  • reproduce — find and clone the repo, follow its own README/Dockerfile/scripts, report whether it builds and runs.
  • reproduce-deep — everything above, plus an evaluation plan derived from the paper's claims and per-claim adjudication against execution evidence.

That is the AE lesson applied outside the AEC: separate environment archaeology (can we install and run it?) from claim checking (do the numbers hold?), and do not collapse them into a single “LGTM.”

# PDF only — no conference login
python repro_cli.py run --pdf paper.pdf --mode reproduce-deep

# Or a URL; optional remote GPU via SSH
python repro_cli.py run --pdf https://arxiv.org/pdf/….pdf --mode reproduce \
    --vm-host gpu.example.edu --vm-user me --vm-key ~/.ssh/id_rsa

Using the dashboard#

The CLI is scriptable, but the easiest way to drive the agent is the local web dashboard—Python stdlib only, no extra dependencies:

python dashboard.py                          # serves http://127.0.0.1:8765
python dashboard.py --port 9000 --pdf-dir ~/papers
  1. Pick a paper and a depth. Choose a PDF, reproduce or reproduce-deep, and a model—the model dropdown is filled live from your provider's /models endpoint, using the same .env config the CLI reads.
  2. Choose where it executes. The local shell by default, or an SSH GPU host, mirroring the CLI's --vm-* flags.
  3. Watch the run. A stage tracker follows the pipeline above—prep, main loop, review, report—with iteration counts, tool-call totals, and a live tail of every shell command as it executes.
  4. Stop or collect. Stop kills the run's whole process group; a finished run links its Markdown, JSON, and PDF reports. Each run is isolated under its own repro_workspace/ directory.

Screenshots and a walkthrough are on the project page.

Powered by FreeInference#

Long agent runs are token-hungry, and paying frontier API prices for every repro attempt is a tax on exactly the research this tool is for. We run development and paper runs on FreeInference — a free, OpenAI-compatible inference API built at Harvard SEAS (MadSys Lab) for open-source, research, and education, serving frontier open models behind one endpoint. The backend adapters in agent_core/ auto-detect OpenAI-compatible, Gemini, and Anthropic endpoints, so pointing the agent there is configuration, not a fork:

export OPENAI_BASE_URL=https://freeinference.org/v1
export OPENAI_API_KEY=…          # from freeinference.org after signup
export OPENAI_MODEL=…            # any model id the endpoint lists

For long runs, set request pacing and a max-completion-token cap—shared free capacity needs the same rate-limit hygiene as any endpoint.

Why this matters#

The AE deployment showed that an agent can run the review protocol end to end but spends most of its effort on environment archaeology—and that this is exactly what agents are good at when tools and logs are available. The paper reproducibility agent removes the conference from the loop: no credentials, no badge forms, no assignment. PDF in, reproducibility report out, running on free open-model inference when you want it. Those are ordinary systems problems. That is why they are worth solving.

Code and endpoints:

If you are trying paper-level repro and want to compare notes, I am happy to talk.

Footnotes

  1. A caveat: the repo is mostly vibe-coded at the moment—written at speed with heavy agent assistance and validated by using it rather than by careful review. A working prototype whose architecture we stand behind, not polished library code. ↩