We audited our own portfolio: 149 claims, 11 held

The finding

I graded 149 claims about public data on my old portfolio. Eleven held.

The site looked finished. It had charts, source lines and dataset IDs. On 24 September 2026 I found that its three proof datasets had been largely generated offline, not downloaded¹. The pages presented them as real NOAA, Kansas City 311 and College Scorecard records. Those figures had been live since at least 11 September¹.

That is the exact failure a BI advisor exists to catch. So I ran the audit on my own work first, and I am publishing the result.

How the numbers got there

A generator script built the data. It typed the weather decade averages in as constants. It drew the yearly climate curve from a cosine. It filled a sample year with seeded random noise. It hand-wrote five named extreme weather days, one of them a 1954 day labelled "Dust Bowl Era"².

Every number was plausible. Most had no source. That is the danger: fast tools, AI included, make plausible numbers cheap.

How we checked

"We" means me and the AI agents I build with. They fetched, counted and compared. Three gates stayed with me: what to download, what to correct and when to publish².

1. Approve the sources. Before my yes, the agents ran only header, schema, count and documentation reads. Nothing was saved until I approved each named source².
2. Pull the real record. Six files and 17 Socrata query results, 55,374,240 bytes in all, each hashed with sha256¹.
3. Set the bar before looking. "Narrowing" needed a 95% interval wholly below zero. "Collapse" needed a 50% drop on two measures. A field-beats-school sentence needed a ratio above 1.5¹.
4. Inventory every claim. 149 claims across eight pages, each pinned to an exact text anchor on its live page³.
5. Grade each one. Five verdicts, in the table below.
6. Count twice. A second code path recomputed the headline numbers, and 9 of 9 matched exactly. All 13 grouped 311 totals reconciled to the source's own count¹.
7. Correct and release. The staged rebuild, made with 280 exact edits by its build script, went live at 23:20 CDT that night, on my instruction¹. A separate set of 125 exact edits to the old source pages, labels and axes included, still waits on my sign-off³.

A fidelity check read all 1,007 numbers on the rebuilt pages¹. None was untraced, and none was a retired figure. Its self-test plants a retired number on purpose and catches it.

What we found

Tallies from the before-and-after table³. One more row, a record the old page never stated, is set aside⁵.

Three headlines moved

Weather flipped. The old page said the daily temperature swing was narrowing. The data show no significant trend: −0.018 °F a decade, with a 95% interval of −0.119 to +0.078¹.

311 shrank. The yearly count of request types in use fell 45.1% at the switch, or 20.9% by a traffic-weighted count. Under our rule that is "narrowed"; "collapse" needed 50% on both².

College shrank. Only 8.8% of 21,940 bachelor's programs carry median debt above median first-year pay. Field of study explains somewhat more of that spread than the school: adjusted R² 0.49 against 0.34¹. The resampled range straddles our 1.5 bar, so the page prints no multiplier¹.

Our own process slipped, too

Our rules call for a review before any code runs, through four lenses: skeptic, quant, systems and advisor. This time it ran after. All four lenses said constrain. All ten of their constraints were applied and rechecked¹.

One example: the weather page may say "no significant" narrowing, never "held steady". Another: the 311 page must show the traffic-weighted (entropy) count beside the distinct one.

The correction gate slipped as well. I approved the corrections by approving the release, after the fact, not by signing off the table first. The 125 edits to the old source pages still wait on that direct sign-off.

The publish gate bent too: an agent ran the deploy on my word, where the plan asked for my own command².

The audit also lists what it did not recheck. Career claims on the old home page sat outside its scope. The unpaginated Socrata pull in my old planning brief was not re-run. When it was, on 17 October, it returned 1,000 rows; the brief's 3,000 was a count of text lines, three per row⁶.

What this means for your BI team

Many dashboards earn trust by looking finished. Looking finished is not evidence.

• Audit before you migrate. A Tableau-to-Power BI move copies every claim, true or not. Trace the most-used dashboards first, or you pay to move the errors.
• Set the bar before the result. Decide what counts as a decline, a breach or a win before anyone sees the number.
• Run headline KPIs on two code paths. When they agree, the arithmetic probably holds; the definition and the source still need their own check. When they disagree, you found the problem before your board did.
• Grade, do not just fix. The mix of verdicts tells you whether you have a data problem or a process problem.
• Keep people on the gates. In this audit, AI agents fetched, counted and corrected within a day. A person approved the sources and the release; here the staged corrections rode in with the release, and the old-page edits still wait.

This is why an advisor audits before advising. A BI lead brings one asset into the room: numbers that can be trusted. I would rather show you my 92 than ask you to trust my 11.

Notes

1. Portfolio audit, tableau/real-data-audit.md, dated 2026-09-24, with checks added on 2026-09-25: verdict, sources, checks and the late review. The date the generated figures went live comes from its decisions table. The college figures are its claims 23, 25 and 26. ↩︎
2. Plan plans/2026-09-24_portfolio-real-data.md: the generator findings are its claims 1 to 4, and the three approval gates are G1 to G3. Its claims table records the reads made before G1, and G3 opens on my own deploy command. Its 311 rule reads a drop of 10% to 50% as "narrowed". ↩︎
3. Every before-and-after pair is in tableau/tableau-sources/out/before_after.md. The 125 edits for the old source pages, not yet applied, are in tableau/tableau-sources/out/edit_log.md. Each row's anchor is in tableau/tableau-sources/out/before_after.json; the verbatim check is in the audit's Checks run table. ↩︎
4. Held means the claim matches the real record today. Two of the eleven, the NOAA source line and the Scorecard refresh date, became true only when real data replaced the generated set. ↩︎
5. The audit's table has 150 rows. One of them, the 57.1 °F swing on 1953-01-15, is a record the old page never stated, so this post does not count it. The audit's own "150 of 150 found verbatim" line includes that row. ↩︎
6. Re-run of the unpaginated 2019 pull, 2026-10-17, with raw responses and sha256 hashes: creation-test/danieljin-bi/content/checks/kc311_recheck_2026-10-17/manifest.json. ↩︎