Good systems research is often summarized backward. The published paper starts with a problem, presents a mechanism, and ends with an evaluation. This makes the solution look like the creative part and the problem look inevitable.

In practice, finding the right problem is often harder—and more important—than solving it. An elegant solution to the wrong problem has little value. A clear formulation of an important problem can redirect an entire field.

This distinction matters even more in the age of AI. Coding agents can set up an environment, implement a prototype, and explore a well-specified design space at remarkable speed. They are much less reliable at deciding what deserves to be specified. The space of possible problems is effectively infinite, and the evidence needed to judge them is scattered across deployed systems, messy datasets, operational constraints, and the experience of the people who run them.

This is why measurement occupies such a large part of our research. We do not measure only at the end to validate a system. We measure at the beginning to discover what is worth building.

Measure one level deeper#

John Ousterhout made a closely related argument in his 2018 Communications of the ACM article, Always Measure One Level Deeper. Performance measurement, he argues, should not be a last-minute search for a favorable benchmark. Done well, it exposes properties of the system, improves the system itself, and sharpens the developer's intuition.

The carpenter's rule is “measure twice, cut once.” Repeating a measurement establishes confidence in the number. Measuring one level deeper explains why the number exists. That explanation is often where the real problem—and the next design—appears.

Measurement gives us a map, not a single answer#

A good measurement study rarely produces one isolated problem. It produces a map: inefficiencies that share a cause, interactions hidden behind abstraction boundaries, and assumptions that no longer match reality. One observation may become a paper; the surrounding observations may shape years of work.

During my Ph.D., I spent more than two months cleaning and compiling a handful of open-source cache datasets. The work was painfully slow. Compute was limited, coding agents did not exist, and the pipeline was full of bugs and retries—humans are excellent bug generators too. At the time, those months could easily have looked wasted. I was cleaning data rather than building a new system.

They became some of the most important months of my Ph.D.

Living with those traces changed how I understood cache eviction. Conventional algorithms invested heavily in deciding which objects deserved to stay. The workloads showed that two other operations were at least as important: quick demotion, which removes new objects that fail to demonstrate reuse, and lazy promotion, which avoids eagerly moving an object on every hit. We developed these ideas in the HotOS paper FIFO Can Be Better than LRU: The Power of Lazy Promotion and Quick Demotion.

Those observations led to S3-FIFO and SIEVE, two simple eviction algorithms now used in production systems and open-source libraries.

SIEVE began with another result that looked wrong. FIFO-Merge, the eviction algorithm in Segcache, outperformed LRU on many workloads. This contradicted the conventional wisdom that tracking recency should make LRU more effective than FIFO. Digging into the result revealed that FIFO-Merge was capturing a more general principle. Stripping that principle down to its essential mechanism led to SIEVE.

Neither algorithm was the goal when I started cleaning traces. The problems and the designs emerged from the data. Those two months did not produce one answer; they produced a different way to see the field.

The cleaned traces are now available in the cacheMon cache dataset, so other researchers can reuse them without repeating all of that work.

Measurement is not merely the prelude to the research. Often, it is where the research agenda comes from.

The surprising result is usually wrong—until it is not#

Measurements sometimes contradict conventional wisdom. Most of the time, we made a mistake. The instrumentation is broken, the baseline is misconfigured, a column was parsed in the wrong unit, or an uncontrolled variable explains the effect.

The right first response is skepticism. Recheck the pipeline. Reproduce the result. Change one factor at a time. Ask what evidence would make the observation disappear.

Occasionally, the result survives. Those moments are unusually valuable because they expose a mismatch between our mental model and the real system.

One of my students, Akira van de Groenendaal, recently observed that serving a bursty LLM workload could be faster and more efficient than serving a smooth workload with the same amount of traffic. Bursts are normally something a serving system must absorb, not a source of efficiency. If the observation held generally, it would change how we perform load balancing, routing, and traffic shaping for LLM inference.

A closer investigation narrowed the result. The effect appeared under two conditions:

  1. an unoptimized FlashAttention-2 kernel path for batches that mix prefill and decode work, which are frequently encountered on A100 GPUs; or
  2. particular model architectures such as DeltaNet.

The original observation was not a universal law, but it was not simply an error either. Finding its boundary conditions exposed behavior that a high-level model of LLM serving had hidden. The broad claim became smaller, but the result became more useful: we now knew which implementation and architectural details created the effect.

That is what a productive anomaly does. It turns “this cannot be right” into a more precise question.

Measurement is how we learn the system#

Traces and logs do more than report performance. They teach us how a system actually works.

Architecture diagrams show components and intended control flow. A trace shows the interactions the diagram left out: background tasks competing with user requests, retries amplifying load, one layer batching work for another, and tools that look idle while their subprocesses consume resources. Following these interactions is often the fastest way to understand a system whose behavior spans code owned by many teams.

Measurement is therefore a learning process, not a mechanical data pipeline. At the beginning of a study, our vocabulary is usually too crude. We group together behaviors that look similar from a distance. As we inspect more evidence, the categories change. That evolution is not an inconvenience to automate away. It is how our model of the system becomes more accurate.

AI makes measurement easier—and harder#

Coding agents have transformed the mechanics of measurement. They can set up an unfamiliar environment, write scripts and instrumentation, launch experiments, produce plots, and test hypotheses. For some tasks, our productivity has improved by one or even two orders of magnitude.

But faster mechanics do not guarantee better measurement. It is now easy to produce a large volume of plausible-looking results before anyone has examined the assumptions underneath them. A measurement script is not neutral: it is a hypothesis about which events matter and how they should be interpreted.

Every measurement encodes assumptions and judgments:

  • Which records refer to the same entity?
  • Is a missing value zero, unknown, or evidence that collection failed?
  • Are two inconsistent observations errors, or do they expose different system states?
  • Which behaviors belong in the same category?

Real-world data is dirty. Missing fields, inconsistent clocks, changing schemas, and duplicated events are normal. An agent can apply a reconciliation rule at enormous speed. It cannot tell us whether that rule represents reality unless we supply the context and inspect the consequences.

We encountered this recently while measuring user and coding-agent activity on a large shared HPC cluster. Our first AI-assisted classification labeled all VS Code activity as agent activity. The choice looked reasonable: VS Code creates heavy, long-running processes that resemble coding agents.

After checking other properties, I realized the category was wrong. We revised the taxonomy so that agents spawned through VS Code counted as agents, while the editor's remaining processes formed a separate VS Code category. That seemed better because the editor has many internal activities that differ from direct human activity. Later, we learned that this distinction was still incomplete: it missed too many activities that should have been attributed to humans.

Each revision improved the labels. More importantly, each revision improved our understanding of the workload. We learned which signals were ambiguous, how tools created subprocesses, and why our original concepts of “human,” “agent,” and “editor” activity were too simple.

I enjoy this evolution in understanding, and I do not think it should ever be skipped. If we ask an agent for a taxonomy and accept its answer without understanding how or why, we have not automated the research; we have outsourced the judgment. Agents are very good at giving us what we request. Our responsibility is to keep asking whether we requested the right thing. Otherwise, instead of making AI work for us, we risk becoming slaves to its outputs.

What measurement-first research requires#

Being measurement-driven does not mean trusting every pattern in a dataset. A single unusual trace can inspire a useful hypothesis, but a system tuned only to that trace is overfitting. We still have to ask:

  • Does the observation appear across applications, organizations, and time periods?
  • Is it stable across system configurations?
  • Does the proposed mechanism address the cause, or only correlate with it?
  • What happens on workloads where the pattern is weak or absent?
  • Which data-cleaning and classification decisions materially change the conclusion?

Averages are not enough. They can hide a mechanism that helps a few workloads dramatically and hurts many others slightly. We need distributions, outliers, sensitivity analyses, and explicit boundary conditions. Breadth is what turns a workload observation into a systems principle.

The research loop I trust looks like this:

  1. Measure how a real system behaves (via instrumentation and log collection).
  2. Use the evidence to find a problem worth solving.
  3. Build the smallest mechanism that targets the measured gap.
  4. Evaluate it on workloads that can disprove the idea.
  5. Deploy it, measure again, and revise the model.

AI can accelerate every step in this loop, but it cannot take responsibility for it. We still have to notice the inconvenient result, question the easy classification, and decide which problem matters.

The best systems are not imposed on a workload. They are taught by it.