Can a model feel twilight? An AI creativity benchmark

Twelve runs, eleven worlds

Twelve runs made the study. Eleven worlds came back.

The other one drew one line of white text on a white page: "Day 1 ☀️ | FPS: 60"¹. Its code loaded Three.js from a cdnjs address that returns 404, so nothing ever rendered². The frame counter was hard-coded. The number looked healthy anyway.

That is the study in miniature. A number on a screen is not a measurement.

The brief

In September 2026 I gave twelve AI setups the same core brief³. It came in three versions: the first draft, a later one that added a reference rule and a no-sub-agent rule, and a one-shot version with no web access.

Each version set the same task. Build a complete, interactive Minecraft web artifact. Judge it on technical depth, aesthetic taste, and whether it runs.

The taste bar was specific: Minecraft at twilight or sunrise, with a day-night cycle, fog, torchlight and an ambient or generative soundtrack. The run bar was strict: visual impact within three seconds, zero silent crashes, zero empty TODO stubs³.

A 2026-09-17 edit to the main brief named that reference, Vesper Hollow, for every later run on it. The one-shot brief never named it³. One unflagged solo run, Aetheria Hollow, has a spec sheet that matches Vesper Hollow almost point for point².

One solo run drafted three candidates with copies of itself, then picked a winner⁴.

The runs took place from 2026-09-15 to 2026-09-22². Eleven rendered. Their single HTML files range from 26.5 KB to 723 KB⁵.

Three more run folders from those eight days never reached the study page. Gemini 2.5 Flash (2026-09-17) left an empty file. GLM 5.3 Flash and Nemotron 3 Ultra ran on 2026-09-20. No file records why those two were left out⁶.

Two islands that check themselves

Two builds set the ceiling, by different routes.

Vigil, from Claude Opus 5.5 at max effort, is a 330 KB file with no library at all². Its WebGL2 engine is hand-written. Every texture, block, light and note is generated when the page opens. It was built in one shot, with no web access and no sub-agents. It saw no reference build. It knew only a one-line summary of the GPT-6 build below, which I named as the bar to beat⁷.

Stillwater Sanctuary, from GPT-6, is a 723 KB file with Three.js r160 embedded inside it². Nothing downloads at runtime, so it opens with networking off. This was a calibrated run. An earlier build, Vesper Hollow, guided its atmosphere; its code was written independently.

The art is not what sets them apart. They are the only two of eleven that keep a test script and its saved results in their run folders⁸⁹¹⁰.

Vigil keeps a 39-check Playwright suite. Its saved run, made just before a final edit, records a first frame at 1.55 s, 240 fps at 1600×900, and zero console errors⁸. Stillwater keeps its own verification script, run headless at a desktop size and a phone size⁹.

Features are easy to list. These two kept the tests that check theirs.

The cost of beauty

Beauty has a running cost. With Vigil running as a page background, whole-device GPU use on my test machine measured 57%, against about 12% at idle¹¹.

A quiet edition capped it at 30 fps and limited its pixel budget. GPU use fell to 18%, and whole-device GPU power fell from 131 W to 41 W¹¹. The day cycle, the water and the lighthouse beams stayed.

That is one machine and one short test. Still, the trade is real: every visual effect spends someone's hardware.

Four leads, held loosely

More hands, more surface

One orchestra split the work across five subsystem workers, the other across two². The study page's build cards show what the first four runs shipped². The five-worker build shipped five subsystems. The broadest early solo run, Aetheria Voxel, shipped four: terrain, redstone, crafting and a synth. The two-worker build shipped one diorama with a soundtrack. So the page's verdict ranks both orchestras higher than its own cards support. Neither orchestra showed obviously better taste.

A cheaper boss did not sink the team

In one run, gpt-5.6-luna, the cheaper and lower-scoring model, directed two claude-fable-5.1 workers²¹². Their build, Vesper Hollow, shipped and later became the main brief's taste reference³.

The study page counted the Flash orchestra as a second cheap-boss run. Published prices and scores say otherwise: Gemini 3.7 Flash costs and scores more than the gpt-5.6-luna workers the page names¹². So the cheap-boss case rests on one run. It is a lead, not a rule.

The merge still needed testing

In the Flash orchestra, the team logged and fixed two integration bugs after the merge¹³. The study page credits the Flash boss². Either way, split work still needed a test of the whole.

A reference helps, by an unknown margin

Gemini 3.8 Flash built the widest solo feature list: five biomes, redstone, crafting, enchanting and a piano². Shown Vesper Hollow, it rebuilt a similar feature set with visibly better light. That is one pair of runs.

Verdict (Informal)

This is not a controlled study. It is twelve runs, one judge (me), and one specific idea of what twilight should feel like. Every finding above is a lead to rerun, not a result to quote.

The range is the lesson. At the top, two lighthouse islands test themselves. At the bottom, a blank page reports sixty frames a second.

What this means for your BI team

AI tools now draft DAX, SQL and calculated fields¹⁴. I expect the same spread there, and the same habits to catch it.

• Ask for the test, not the demo. Only 2 of 11 builds kept a re-runnable test with its results. An AI-translated measure should ship with a parity check against the source report.
• Distrust a healthy-looking number. The broken build showed "Day 1 ☀️ | FPS: 60" on a blank page. A KPI tile can do the same from a stale extract, so show when each number was last computed.
• Hand over a reference. The calibrated runs borrowed taste from a strong example. In a Tableau to Power BI migration, the source workbook is that example; give it to the model with your target conventions.
• Test the cheaper pipeline. One cheaper boss shipped here, and one run proves little. Run the low-cost setup on your own work before you pay for the premium one by default.
• Set a performance budget first. Vigil's light cost real GPU time until a quiet edition capped it. Agree on a frame-rate and GPU budget, not only a load-time limit, before anyone adds the animation.

That is how I would test AI inside a BI pipeline: one brief, measured output, and a test that ships with the work.

Play the builds

Notes

1. creation-test/deepseek-chat_default/index.html, lines 21-22; the cdnjs three.js r146 URL returned 404 when checked 2026-10-17. ↩︎
2. Study page, creation-test/danieljin-site/claude-opus-5.5_max/site/ai-creativity-study.html; runs dated 2026-09-15 to 2026-09-22. ↩︎
3. First brief, as the early runs saw it: git 395b8dcd:Adam/xp/creation-test/instruction.md (2026-09-15). Commit 676b3abd (2026-09-17) added a Vesper Hollow calibration note, the _cheat rule and a zero-sub-agent rule. One-shot brief: creation-test/instruction_oneshot.md. A 2026-10-08 edit removed the task line and the calibration note from the working file, creation-test/instruction.md. ↩︎
4. creation-test/claude-sonnet-5_ultracode/plan.md, steps 2 and 3: three parallel claude-sonnet-5 sub-agents drafted candidates, then claude-sonnet-5 judged them. ↩︎
5. Byte sizes of the eleven index.html builds in tableau/ai-creativity-study/, measured 2026-10-17. Smallest 26,500 bytes (gemini-3.1-pro), largest 723,126 bytes (gpt-6). The gemini-3.7-flash orchestra build first shipped as index.html plus css/ and js/. The study page's builder inlined its scripts into one file on 2026-09-15 (plans/2026-09-15_danieljin-ai-creativity-study.md, claim 3). ↩︎
6. Run folders creation-test/gemini-2.5-flash_high/, creation-test/glm-5.3-flash_default/ and creation-test/nemotron-3-ultra_default/. Gemini 2.5 Flash left a 5-byte index.html (2026-09-17 23:34), and commit 0fbd9254 (2026-09-19) lists gemini-2.5 among the benchmark's new builds. GLM and Nemotron each have a _patched copy, all dated 2026-09-20. None of the three appears on the study page. ↩︎
7. creation-test/claude-opus-5.5_max/index.md: its opening note and the section "How this run followed the brief". ↩︎
8. creation-test/claude-opus-5.5_max/verification.json (2026-09-22 14:29; 39 of 39 checks passed), recorded on a 329,603-byte build. The shipped file is 329,926 bytes, last written at 14:45, and carries a ?bg backdrop mode; no saved run of the 39 checks covers it. A 2026-09-24 test of an identical copy in ?bg mode booted without errors at 239.8 fps (vigil-quiet/verification.json). ↩︎
9. creation-test/gpt-6_default_cheat/verification.json and verify.cjs (headless Edge, 1440×1000 and 390×844). ↩︎
10. Survey of the twelve runs' files, 2026-10-17. Gemini 3.1 Pro keeps check_errors.py, a console-error script, and the Flash orchestra keeps test-soundscape.html; neither has saved results. Claude Sonnet 5's plan.md logs manual browser checks without a script. No run folder for Vesper Hollow survives under creation-test/; its published copy holds only the page and a screenshot. The published copies in tableau/ai-creativity-study/ hold no tests. ↩︎
11. vigil-quiet/report.md and power-measurements.json, 2026-09-24: whole-device GPU medians in headless Chrome at 1920×1080 on an NVIDIA GeForce RTX 2070 SUPER. The original was loaded with ?bg, with twelve readings per condition; idle readings 12.5% and 12%. ↩︎
12. Checked 2026-10-17; prices are per 1M input and output tokens. OpenAI API pricing lists gpt-5.6-luna at $0.20 and $1.20. On Artificial Analysis, Claude Fable 5.1 costs $10 and $50 and scores 47 to 53 on the Intelligence Index. GPT-5.6 Luna scores 16 to 37 across its effort settings. Gemini 3.7 Flash costs $0.75 and $3.75 and scores 39 at High. ↩︎
13. creation-test/gemini-3.7-flash_high_orchestra/README.md, section "Troubleshooting & Verification": a canvas-append collision and a HUD hidden behind the canvas, each with its fix. It does not say who found or fixed them. Its hierarchy diagram adds a sixth node under the boss, "Master Integrator", and names no worker model. ↩︎
14. Vendor docs, accessed 2026-10-17. DAX: Write DAX queries with Copilot, Microsoft Learn. SQL: Microsoft Copilot in the Data Warehouse Workload Overview, Microsoft Learn. Calculated fields: Explore Your Data with Tableau Agent, Tableau Help, section "Create calculations". ↩︎