Counting the Wrong Things: A Sampling Story From a Strained Supply Year
In early 2022, a public works department we'll call the District faced a familiar squeeze: vendors were back-ordered, materials cost more every month, and half the maintenance team was rotating back to the depot after a long stretch of remote dispatch. Into that pressure came an alarming claim from the floor supervisor. Pump-seal failures, he said, had roughly doubled. He wanted budget approved that week to switch suppliers.
The numbers behind the claim were real, but the way they had been collected made them almost meaningless. This is the quiet trap of statistical sampling: a measurement can be precise, repeatable, and completely misleading at the same time. Before the District spent money it did not have on a supplier change it might regret, it was worth understanding what its sample actually represented.
What the sample was really measuring
The supervisor's figure came from the seals his crew happened to bring back to the depot for warranty claims. That is a convenience sample, the easiest kind to gather and the easiest to fool yourself with. Two things had changed at once. With return-to-office, more crews were physically passing the depot and dropping off failed parts they would previously have binned in the field. And with prices climbing, the team had started filing warranty claims on parts they used to write off. The count of seals on the depot bench went up. The actual failure rate in the field did not.
A sound sample answers a clearly stated question about a clearly defined population. Here the population was every pump seal in service across the District. The sample was the subset of failed seals that made it onto a bench. Those are not the same thing, and no amount of careful counting closes the gap.
Sampling basics that would have caught it
Define the population first. Decide what you are trying to learn about — all seals in service, this quarter's installs, one manufacturing lot — before you pick up a single data point. The District never wrote this down, so it could not tell what its numbers excluded.
Make the sample representative. Random or systematic selection from the whole population beats whatever is convenient. Pulling every twentieth installed seal for inspection on a fixed schedule would have given a stable picture independent of who walked past the depot.
Keep the sample size honest. Small samples swing wildly. A jump from 6 failures to 12 looks like a doubling but is well within normal variation at that volume. Larger samples, or a control chart over time, separate real signal from noise.
Watch for changes in the measurement, not the process. If your collection method, reporting incentive, or who-records-what shifts mid-stream, your trend reflects the change in counting, not a change in the world.
How it resolved
The District paused the supplier switch and ran a two-week structured check: a fixed number of seals inspected per route, recorded the same way regardless of where the truck ended its day. The corrected failure rate was essentially flat year over year. The supplier was not the problem; the data-collection habit was. The avoided cost of an unnecessary contract change paid for the exercise many times over.
The broader lesson holds in any volatile year. When inputs are scarce and budgets are tight, the temptation to act fast on the first number you see is strongest exactly when that number is least trustworthy. Good sampling is not statistical nicety — it is how you avoid spending real money chasing a phantom.
If your team is making big calls on shaky measurements, XNM's strategic advisory can help you build the sampling discipline that keeps decisions grounded in what is actually happening.