This note is about the method. A companion note describes what the dashboard does and how to use it. Every example below is in the public repository, in the decision record that fixed it.

What the agents were good at

I rebuilt the dashboard with an agentic AI harness. That is a set of instructions telling a coding agent how to work, not only what to build. Write the specification first. Plan before you build. Prove the change with a test. Do not call the work complete until the exit criteria pass.

It worked. Typed code, tests, data validation, and documentation all came out well, and they came out fast. The agents wrote 95 decision records as they went, which is more written reasoning than I would have produced alone.

The value was the limits, not the speed. Every change still went through human review. What follows is what that review had to catch, because nothing else was going to.

A baseline nobody chose

The dashboard tells you how a reservoir compares against normal for the week. To do that it needs a period to average over. For months that period was 2015 to 2025.

Nobody decided that. The pipeline had a line in it that read START_DATE = "20150101". That is the date the pipeline starts asking the federal services for readings. The normal quietly inherited it.

Three things followed. 2015 through 2025 is the driest stretch in the modern record for this region, so every reservoir was measured against a drought and a bad year read as ordinary. The snowpack half of the site had always used 1991 to 2020, the period the World Meteorological Organization defines, so the two halves of one page were answering different questions. And the size of the error was never measured.

It turned out to be large. Lake Powell read 44.6 percent of normal against 2015 to 2025, and 35.0 percent against 1991 to 2020. That is nearly ten points, on the reservoir that dominates every combined figure the site publishes.

The obvious defense is that the older data does not exist. That was assumed rather than checked. When somebody finally asked the services directly, 54 of the 69 reservoirs had readings from 1991 or earlier, and those 54 held 98.2 percent of the combined capacity. The data had been there the whole time.

This is the failure that gives the note its title. Every test passed. The code did exactly what it said. A variable named START_DATE was doing a second job nobody had named, and no test goes red on a misleading name.

The check that already knew

A percentage needs a denominator. The dashboard divides by the conservation pool, which is what a dam operator means by full. That is right for most reservoirs. For thirteen of them it was not, and the published numbers said so out loud.

Detroit Lake was published at 223.7 percent full.

The reason is real and boring. Detroit is a flood-control project. Its reported series is gross storage, which is all the water behind the dam including the flood space. The conservation pool describes only the summer pool underneath that. Two true numbers about two different volumes, divided by each other.

Here is the part worth sitting with. The tool that builds the capacity table had carried a check since the day it was written: a capacity below what we have already seen in the reservoir means the match is wrong. Detroit failed that check by 175 percent. The check ran, noticed, wrote a line into a report, and kept going. Nobody read the report.

An agent will build you the check. It will not build you the habit of reading what the check said.

A line through Flaming Gorge

A grey administrative line was drawing across the reservoirs. It was most obvious at Flaming Gorge, which sits on the Utah and Wyoming border, where the line ran straight through the water.

The fix looked obvious. Move the state boundary layer to the bottom. That was tried, and the line did not move, because the line was never in that stack.

A basemap is two stacks, not one. Base layers draw below everything a map adds. Reference layers draw above everything a map adds, which is the whole point of them, so that place names stay readable over whatever sits on top. The basemap in use carries state boundaries in that second stack along with the labels. No amount of reordering the map's own layers could have fixed it.

Reasoning about the problem produced confident wrong answers for days. Printing the contents of the reference stack ended it in a minute. The durable advantage of an agent is that measuring is cheap for it. Run the thing, query the page, count what came back. A person tends to read the code and trust it.

Environments report success they have not earned

A page that loads, paints a basemap, and draws no reservoirs at all looks completely fine in a screenshot. Headless browsers returned blank canvases and called them passes. Fonts fell back silently. The charts broke under the content security policy and logged nothing.

The fix was to stop trusting the absence of errors. Every page now publishes a readiness object with counted fields: how many reservoirs the data supplied, how many the map actually drew, how many symbols the renderer holds, whether the labels are on. A field is added when a new thing can fail, and a field is never removed. The browser tests read those numbers instead of looking at pictures.

I hit this same failure again while writing this note. The first attempt to photograph the dashboard produced a clean screenshot of a page saying the browser could not display the map.

The ceiling is visual

Almost every problem that survived to the end was one an agent cannot see. Labels collided at one zoom level and not the next. A symbol read as one class on screen and a different class in a screenshot. A legend order looked arbitrary.

An agent can confirm that a color value matches the table it is supposed to match. It cannot see that two labels overlap, or that a map feels wrong. Symbol size and label placement took five separate decision records to settle, and every one of them started with a person looking at a map and saying that is not right.

This is the honest limit of the method for cartography today. It is not a small limit. On a mapping project, the last mile is the visual mile.

What I would tell someone starting

  • Name the things that carry meaning. A variable doing a second, unnamed job is the bug you will not find.
  • Decide what a number is measured against, in writing, before you publish the number.
  • Make every check fail loudly. A warning appended to a report nobody opens is the same as no check.
  • Ask for counts, not for green. Absence of an error is not evidence that the work happened.
  • Look at it yourself. Keep a person between the agent and anything a reader will see.

None of this argues against building with agents. The dashboard exists at a size I could not have reached alone, and the written record of why it works the way it does is better than anything I have produced by hand. The method is good at what can be counted. Everything it got wrong, it got wrong about meaning.

View the Western Water Dashboard, the methods and sources, or the source on GitHub. The decision records quoted here are ADR-041, ADR-072, and the basemap reference note in the repository. A system atlas draws the whole system as it stood on one date. Agent skills: addyosmani/agent-skills. A companion note describes what the dashboard does.