Example rubric
The developer asks the agent to remove the dead RecordDropped exception class from a data pipeline, together with every stale reference to it (the import, the except clause and the docstring and comment mentions), leaving the package importable and consistent.
Eight items. A run is graded on each: met = 1, partial = 0.5, not met = 0. Items marked must weigh 2; items marked should weigh 1. The run's score is the weighted fraction.
Items 1 to 6 were written from the request, the developer's reply and the original agent's attempt. Items 7 and 8 were added because every one of five independent Astra runs on this turn did them, and nothing in the first six covered them. For items 1 to 4 and 7 to 8, every Astra run met the item. On items 5 and 6, the Astra runs differed: one reported that it could not run the tests, and the quality of the file report varied. Those two are where the rubric separates runs.