# CRAFT: turning a grading rubric into a model's to-do list > A new method reads the fine print in eval rubrics to find not just where a model fails, but which underlying skill it lacks — then generates practice data aimed straight at that weakness. _Source: arXiv preprint (author-reported experiments) · 2026-07-21 · 7 min read_ Canonical: https://iyu.app/e/craft-rubric-capability-diagnosis ## The 60-second version CRAFT mines the grading criteria inside rubric-based evals to diagnose a model's specific weak skills, then generates fine-tuning data aimed at exactly those gaps. **Key points** - Treats each rubric criterion as a probe of one capability, then clusters them into a hierarchical skill tree. - Scores the model at every node and picks weak spots dynamically across tree levels — the zoom where each failure is clearest. - On 4 open models in finance and legal, models retrained on CRAFT's diagnosis beat prompt-level EvalTree and random practice data (authors' own results). - Caveat: single preprint, self-reported numbers, and it needs an existing rubric-based eval to work. **Verdict.** A sharp reframing — diagnose at the criterion level, not the prompt level — that turns an eval into a targeted training plan. Promising, but unreplicated. ## Full explainer > **⚑ Caveat:** All results below are **self-reported by the authors** in a single arXiv preprint (submitted 17 Jul 2026). There is no independent replication yet, and gains vary by domain — treat the numbers as promising, not settled. ### The problem — Evals say where a model fails, not why Run a model through a benchmark and you learn it scored badly on certain questions, topics, or categories. What you *don't* learn is the underlying reason — was it weak legal reasoning, sloppy arithmetic, a formatting slip? The failure's actual cause stays implicit. And if you can't name the broken capability, you can't reliably fix it or generate the right practice data for the next training run. > Most evals tell you *where* a model fails. They rarely tell you *what it cannot do*. ### The trick — Every grading criterion is a skill probe CRAFT (Clustering Rubrics to diagnose weak capabilities and generate targeted Fine-Tuning data) starts from a simple reframing. Serious evaluations are often **rubric-based**: each prompt ships with grading criteria a judge checks off. CRAFT treats every single criterion — say, *"correctly applies the statute of limitations"* — not as a checkbox but as a measurement of one specific capability. - **Extract** — read every prompt-and-rubric pair and write a plain description of the capability each criterion tests. - **Cluster** — group those thousands of descriptions into a hierarchical **capability tree**: broad skills near the top, narrow ones at the leaves. - **Score** — evaluate the target model at every node of the tree. - **Select** — pick the low-performing nodes *dynamically across levels*, at whatever granularity makes each weakness clearest. - **Generate** — use those weak nodes to direct the creation of targeted fine-tuning data. ### The elegant bit — Weakness lives at different zoom levels A model's blind spots don't all sit at the same resolution. Sometimes a whole broad branch is weak; sometimes only one narrow leaf is. Rather than fix the zoom level in advance — as prompt-level or category-level methods do — CRAFT chooses the altitude per failure, reading the weakness where it shows up most sharply. The result is a model-specific diagnosis, and that diagnosis doubles as an instruction set for what practice data to build. ### Does it work? — Retrained models, held-out tests Holding data-generation, fine-tuning, and evaluation fixed, the authors pit CRAFT against **EvalTree** (clustering at the coarser prompt level) and against plain **untargeted random** data generation. They test four open-source models across finance and legal, scoring on 13 held-out benchmarks deliberately disjoint from the diagnostic data — so the model can't just memorize the test. - **Setup:** What it means - **4 open-source models:** The targets being diagnosed and fine-tuned - **2 domains:** Finance and legal - **13 held-out benchmarks:** Disjoint from the diagnostic data - **Finance result:** CRAFT strongest domain average for all 4 models - **Legal result:** Strongest for 3 of 4; within variance for the 4th The finance sweep is the cleanest win; legal is more mixed but never clearly behind. The consistent story: diagnosing at the **rubric-criterion** level, rather than at prompts or categories, yields both a sharper picture of what a model can't do and measurably better models after fine-tuning on that picture. > **✎** The method needs a good rubric-based evaluation to mine in the first place. If your evals are just accuracy scores with no grading criteria, there's nothing for CRAFT to reverse-engineer. ### Why it matters — From grading the model to writing its homework The bigger shift is closing the loop. An eval that only produces a score is a dead end; an eval that produces a ranked list of missing skills, at the right granularity, becomes the first step of the next training run. CRAFT stops asking *where* a model fails and starts asking *what it cannot do* — then hands you the worksheet to fix it. ## Primary sources - [CRAFT — arXiv:2607.16122](http://arxiv.org/abs/2607.16122v1) --- _Published by iyu (https://iyu.app) — the day's AI news, checked against primary sources and rewritten in plain language. Free to quote with attribution and a link to the canonical URL._