Deploying an artifact-evaluation agent at MLSys'26

Artifact evaluation has a scaling problem. Systems papers come with code, data, and scripts that reviewers must install, run, and compare with the paper's claims. The work is repetitive, hardware-intensive, and well suited to a tool-using agent.

It is also unforgiving: a useful reviewer must distinguish a broken artifact from a broken environment, keep track of multi-day conversations, and turn partial evidence into a defensible badge decision.

At MLSys'26, we tested that idea in a live conference workflow. We gave a language model the same tools as a human AE reviewer, put it on HotCRP under its own identity, and let it follow the protocol end to end.

This is a field report on what worked, what broke, and what we would change—not a claim that AE is solved or a disclosure of confidential reviews.[^scope]

The setup#

We connected the agent to HotCRP, the paper PDF, GitHub, the open web, and an H100 review VM on the Chameleon testbed, which hosted the MLSys'26 AE environment.

The control plane mattered as much as the model: the CLI owns every HotCRP write. Comments and reviews pass through a sanitizer and an optional human approval gate. The model investigates and proposes actions, but never sees the conference password. Keeping credentials and writes outside the tool loop made a real reviewer account practical and gave us one place to enforce conference policy.[^arch]

Surface	What the agent does
HotCRP	Read paper metadata, read comment threads, post author-visible questions, submit badge reviews
Paper PDF	Extract abstract, claims, and the artifact appendix
GitHub	Inspect tree, README, license, releases, DOI / archive links
Review VM	Clone, install, run commands, read logs and outputs

We used three run modes:

1. Comment (default) — investigate, stop at a blocker, ask one precise question, and save a local report.
2. Draft — complete every phase and save an unsubmitted review.
3. Submit — complete every phase and submit the badge verdict.

Comment mode ran on a schedule, so the agent could revisit papers after authors replied. A human could then promote a paper to submit mode. The agent submitted only for papers short on human reviewers, filling coverage gaps rather than displacing people.

This separation let the agent investigate broadly while keeping final review submission narrow and human-controlled.

What happened in practice#

A read-only snapshot after the AE period showed:

Metric	Count
Papers with at least one agent comment	25
Author-visible comments by the agent	62
Comments on those threads (all parties)	391
Completed reviews submitted by the agent	8

The agent could flag missing or broken artifacts even when a human would submit the review, but it submitted only where human coverage was short. Of those eight reviews, about half confidently recommended all three badges. The rest scored Available highly but were lower and less confident on Functional or Reproduced. An archive and README are much easier to verify than H100-scale results.

In other words, the gap between 25 commented papers and eight submitted reviews was intentional. Discussion was broad; submission was reserved for gaps in human coverage.

The review loop was the real product#

The clearest wins followed a multi-day loop:

1. The agent identified a blocker: a missing DOI or license, broken install, SSH-only submodule, or missing offline path.
2. The authors fixed or clarified it.
3. The agent checked again.
4. Only then did it score the badges and write the review.

This is standard human AE practice. Making it repeatable worked when authors were responsive and artifacts offered an offline or pre-run verification path. Several strong reviews analyzed author-provided logs instead of rerunning full training jobs—the same tradeoff human reviewers make under time constraints. The useful unit of work was not a one-shot verdict, but a blocker, an author reply, and a verified fix.

Reviewer-grade forensics#

When the environment cooperated, the agent found specific problems: the wrong Python minor version broke Dynamo, a Makefile had commented-out baseline targets, and a dataset loader required trust_remote_code. That is more useful to authors than a score alone. AE is partly environment archaeology, and agents can be good at it when they have tools and logs.

Did it agree with human reviewers?#

The same eight papers had human co-reviewers using the 1–5 HotCRP form for Available, Functional, and Reproduced. That gives 28 agent–human pairs and a human–human baseline of 38 pairs. We report aggregates only.[^agree]

We compare both exact matches and scores within one point. The latter matters because human reviewers also disagree on the precise 1–5 score.

Exact match on all three badges in one agent–human pair: 11% (3/28).

The main takeaways:

1. Available was the easiest badge to align on. The agent’s exact agreement with humans was 71%, versus 55% between humans. Both were about 96–97% within one point.
2. On Functional and Reproduced, the agent lagged humans on exact matches but not within one point. Exact rates were 39% / 29% for agent–human and 74% / 58% for human–human; within-one rates were nearly identical at 96% / 86% and 97% / 84%.
3. On average, the agent was slightly more generous than the human median: +0.4 for Functional and +0.6 for Reproduced. It often chose “Exceeded” where humans chose “Met,” though it was harsher on two difficult papers.

Caveats: eight papers are a field sample, not a formal evaluation; they were short on human reviewers; and HotCRP's labels do not map directly to an internal +1/0/−1 schema.

Where the deployment broke down#

The agent did not know when to stop#

The costliest failures were open-ended infrastructure battles: full disks, missing Docker, Ray initialization failures, timed-out 30+ GB container pulls, and inconsistent GPU access. The agent would try, post an update, receive an author reply, and try again. Days could pass without a review.

Without states such as blocked_infra, blocked_authors, ready_to_submit, and submitted, comment mode becomes an infinite retry loop. The agent needs a stopping policy and a way to file a low-confidence, partial review. Checkpoints can preserve work, but they cannot decide when the work should end.

A human reviewer eventually recognizes diminishing returns. The agent needs that judgment encoded as policy.

The review environment is part of the artifact#

MLSys AE involves multi-GPU containers, CUDA graphs, and hours-long simulations, not just “clone and pytest.” A changing fleet—no GPU on Monday, H100s on Tuesday, full disks on Wednesday—wastes tokens and authors' patience. It needs a pinned image, preflight checks for disk, GPUs, Docker, and Python, and a clear distinction between artifact failure and reviewer-fleet failure. Otherwise, limitations of the review infrastructure can be mistaken for failures of the artifact itself.

Sporadic GPU access was the biggest bottleneck. Hardware-dependent claims stayed in blocked_infra, retries accumulated, and Reproduced scores remained low-confidence. Future deployments should reserve GPU capacity or limit Reproduced to workloads the fleet can run.

Identity could not be an afterthought#

Authors of two papers realized that their reviewer was an agent. Our control plane guarded content—every HotCRP write passed through a sanitizer and approval gate—but not identity. The signals were obvious: long templated comments, replies at machine speed, and no policy for direct questions about the reviewer. Identity and disclosure must be decided with the PC and AE chairs before deployment.

What we would keep—and what we would change#

1. Comment-first is the product, not a fallback. The useful unit of work was “blocker → reply → re-check,” not a one-shot verdict.
2. Long-running tools need budgets and a terminal outcome. Checkpoints preserve work; they do not tell the agent when to stop.

Was it worth doing?#

The agent did not replace the AEC. It stress-tested the process—and our engineering. The next version should post fewer comments, get stuck less often, and reject reviews that still read like status updates. These are ordinary systems problems, which is why they are worth solving.

The experiment also clarified where agents help most: tracing environment failures, maintaining author feedback loops, and turning execution evidence into a structured review.

We also turned the reusable core intoa a paper reproducibility agent, a standalone agent that needs only a paper PDF or URL. It finds the repository, clones and installs the artifact, runs it, and—in deep mode—checks each claim against the execution evidence. Parallel prep agents feed a checkpointed main loop, while per-claim adjudicators and a report critic produce Markdown, JSON, and PDF reports. Runs can use FreeInference for free, open-model inference. The architecture and workflow are covered in a separate post.

If you are running AE for a systems conference and want to compare notes, I am happy to talk.

Acknowledgments#

We thank the Chameleon testbed for sponsoring the H100 review VM. In particular, we thank Kate Keahey (PI, Argonne National Laboratory) and the Chameleon team for supporting this research infrastructure.