← All work
Calibrate
An item-response-theory engine that scores post-training tasks by difficulty and discrimination, and flags the broken ones.
2PL
IRT model
211
validated tasks
37 / 40
planted-bad tasks caught
1.8%
false-positive rate
What it does
Not every task in a post-training set measures what you think it does. Calibrate fits a two-parameter-logistic IRT model to scores from a model ladder, so each task gets a difficulty and a discrimination estimate. Tasks that fail to separate stronger models from weaker ones are flagged as broken.
On a simulated model ladder it validated 211 tasks and caught 37 of 40 planted-bad ones at a 1.8% false-positive rate.
Stack
Python · NumPy · SciPy · SQLAlchemy · FastAPI · Subprocess sandbox · Pytest · Hypothesis
Honest notes
The results are on a simulated model ladder. The code is private for now; I'm happy to walk through it.