How to tell if a quantum computer is getting better

A quantum announcement arrives with a large number and an invitation to be impressed. The harder question is what improved. Did the computer tackle a more demanding task? Produce a more dependable answer? Finish sooner? A useful benchmark keeps those questions connected. You do not need to read every equation to notice when a headline has separated a performance number from the work that gives it meaning.

A new benchmark offers a timely example

A preprint submitted on September 10 introduces QUOPS, the quantum universal operation performance system. Its authors describe a benchmark combining the size of successfully executed, computationally relevant circuits with execution speed. They report physical-qubit experiments on processors from Quantinuum, Google and IBM, alongside assessment of an encoded logical processor and projections for future systems. These are distinct kinds of evidence within the same research contribution.^[1]

The September paper supplies a useful case study for reading quantum performance claims. Its value for a general reader is the opportunity to ask a better question than which company currently displays the largest number.

First: what job did the computer do?

A benchmark chooses work for the machine. Before comparing the score, identify that work. A test built around a particular circuit can illuminate one capability while leaving other capabilities unanswered. That limitation is normal. Trouble begins when the result is presented as a verdict on every task a prospective user might care about.

Imagine choosing a delivery service by its fastest recorded trip. That record would tell you little about a route with different distances, traffic or loading requirements. In the same way, a quantum result needs its workload attached. Ask whether the task resembles the intended application, and which important parts of that application the test leaves outside its boundary.

Second: what counted as a good answer?

Fast output is not automatically useful output. A comparison needs a stated success condition: what quality of answer the test accepts and how success is judged. Without that condition, readers cannot tell whether a larger or faster result preserves the thing the computation was meant to accomplish.

Consider an invented example. One machine produces an acceptable result in ten minutes. Another produces a rougher result in five. Calling the second twice as fast conceals a change in the answer being accepted. The example is deliberately simple, but the reading habit travels well: keep the quality requirement fixed before treating a performance ratio as an improvement.

Third: when did the clock start and stop?

A timing result needs a boundary as much as an accuracy result does. Was the measurement a device operation, one trial or the time to obtain a usable final answer? The relevant boundary depends on the question being asked. A component test can be valuable even when it does not measure the entire workflow.

For a hypothetical customer choosing between services, preparation, repeated attempts and checking the result might affect the decision. For a scientist studying a particular operation, those steps might properly sit outside the experiment. Neither perspective should silently replace the other. The article should tell you which stopwatch was used before inviting you to compare times.

Check what kind of evidence supports the number

Physical qubits and logical qubits describe different levels of a machine. IBM explains logical information as encoded across physical qubits, with error-detection and correction procedures intended to protect it. A logical-qubit count alone does not describe which protected operations a system can perform.^[2]

There is another distinction to preserve: observation, interpretation and projection. Observing a result reports what occurred. Interpreting it involves a model or argument. Projecting a future result adds assumptions about what will change. A good chart labels those differences in words. A forecast can be useful without inheriting the evidentiary status of an experiment simply because both appear on the same slide.

NVIDIA’s September 14 announcement says a QUOPS reference implementation is available in CUDA-Q. That is software availability, rather than a separate experiment reproducing the paper’s results.^[3]

A paper, repository, institutional announcement and integration note can all describe one contribution. Each helps in a different way, but counting them as four independent confirmations would exaggerate the evidence. Follow the links far enough to see which document contains the measurement.

The next time a quantum headline crosses your feed, keep three questions handy: what task, what acceptable answer and what timing boundary? Then check whether the number was measured or projected. Those questions make enthusiasm more informed. They also make a story worth sharing: you can explain what changed without passing along a conclusion the test never established.