studio-log
I spent $21.10 to find out two of my own skills make the model worse
I write agent skills to make the video pipeline behave. Router skills, a scriptwriter, a personal-brand agent. I had never checked whether any of them helped.
So I built a harness: eight cases, each with a prompt and a set of graders, run with the
skill loaded and again without it, three runs per arm, graded by a judge model against
written rubrics. Model under test was claude-sonnet-5. Total spend, across six runs:
$21.10.
Two of them made the model measurably worse.
| Case | With skill | Without | Delta | n |
|---|---|---|---|---|
| scriptwriter-named-one | 0.222 | 0.444 | −0.222 | 3/3 |
| scriptwriter-format-first | 0.667 | 0.833 | −0.166 | 3/3 |
| gsd-format-first | 1.000 | 1.000 | 0.000 | 3/3 |
| gsd-studio-dry-run-ungated | 0.800 | 0.800 | 0.000 | 3/3 |
| gsd-studio-routes-to-scriptwriter | 0.445 | 0.416 | +0.029 | 3/2 |
| gsd-studio-gate-holds | 0.333 | 0.286 | +0.047 | 3/3 |
| personal-brand-carousel | 0.833 | 0.750 | +0.083 | 3/3 |
| scriptwriter-drafts-two | 0.456 | 0.342 | +0.114 | 3/2 |
Reading this honestly
The two negative results are both scriptwriter. That is the skill I was most confident
in, and it is the only one where I had written detailed formatting instructions. The
pattern I would guess at, and it is a guess rather than a finding, is that the instructions
crowd out the task: the model spends its attention satisfying my format and less of it
writing a good beat.
The positive deltas are small. +0.114 at n=3 is not a result I would defend in a paper.
The honest summary of this table is “four skills do approximately nothing, two help a
little, two hurt”, and the two that hurt are the ones I use most.
Also worth stating plainly: gsd-studio-gate-holds scores 0.333 absolute with the
skill. Whatever the delta says, the gate-holding behaviour is bad in both arms. The eval
found a real problem that has nothing to do with skills.
What I am going to do about it
Not much yet, and I would rather say that than imply a fix I have not shipped.
The immediate value was not a fix, it was calibration. I had a confident belief about which of my tooling worked. The belief was wrong in two places, and it cost twenty-one dollars and an evening to find out. That is cheap for the class of mistake it prevents.
The next step is to cut scriptwriter down and re-run the same two cases. If the score
comes up when I remove instructions, that is the answer.
Why this is on the site
The method argues that a graphic may never claim more certainty than the narration. The same rule has to apply to me. Publishing a system while quietly sitting on the measurement that says two parts of it are counterproductive would be exactly the move the certainty rule exists to stop.