Notes /
Vibe Bench: 10 AI models, one music studio
I gave 10 AI models the same home-made music studio and one job: a 3-track album with covers and video loops. No scores.
I built a small music studio on my PC and gave it to 10 AI models. Same prompt, same tools, one job: make a 3-track album, with a cover and a short video loop for each song.
Site: vibe-bench.madeby.dev
The studio runs locally on one RTX 4070: YuE2 for the songs, Qwen Image for the covers, MiniMax H3 for the loops. The model reaches it only through nine MCP tools. It can “listen” to its own renders through a few small analysis models (tags, CLAP, Whisper) and decide to keep a take, fix it or start over. Nobody answers its questions.
13 albums, 39 tracks, 143 minutes. Claude, GPT, Gemini, Grok, DeepSeek and a local Qwen 9B as a baseline. No scores, it’s a vibe check, not a ranking.
The fun bits:
- Nobody saw anyone else’s album, and “Night Ferry” still came up three times: Opus and DeepSeek named their albums that, Sonnet a track. “Harbour Lights” also showed up three times.
- 22 of 39 style prompts ask for night by name. Every model apparently wants to soundtrack a late ferry.
The harder part was the harness, not the music. Each model runs in its vendor’s own CLI from an empty folder. Codex and Grok’s CLIs still read my real home folder and pulled in my personal skills, hooks and MCP servers. Now every run gets an empty home, and isolation is checked by actually trying to read a file, run a shell and fetch a URL, not by asking the model what tools it has.
Everything on the page is AI-generated music, images and video. Full prompts, lyrics, what the ears heard and every tool call are there too.
Go have a listen.