How the corrections are verified
A correction is published only when the same script works on the version released before the model's knowledge cutoff, fails on the newer release with an error about the changed API, and the corrected script passes on the newer release. Sandbox controls must pass for a run to count.
- Models
- 3
- Selection cutoff
- 2026-04-01
- Known breaks detected
- 10/10
- False breaks
- 0 %
Steps
- Select. Popular PyPI and npm packages with a minor or major release after the earliest knowledge cutoff of the checked models.
- Read. Release notes between the two versions: GitHub releases, changelog files and linked what's-new pages.
- Extract. A helper model lists breaking changes; each must quote the release notes word for word.
- Ask. Each checked model writes a short script for a task that naturally uses the changed API. It is not told which version is installed.
- Run. The script runs in a sandbox without network on both versions, each installed with dependencies as of its release date.
- Fix. A helper model writes the smallest change, seeing the names the new version really exports; the corrected script must pass on the new version.
- Compare. The corrected script's output on the new version is compared with the original's on the old version (proof.equivalent, method stdout_match): same, differs, or not compared (nothing printed, no run on the old version, the original's output varies between runs, the API was removed). A match is observed stdout of one example, not a proof of equivalent behaviour.
- Apply. The published patch is applied word for word to the original example and must parse, drop the old names, run on the new version and print what the corrected script prints (proof.applicability: applies, syntax_error, incomplete, fails, not_applicable, unknown). Automatic application is allowed only when proof.equivalent is true and proof.applicability.status is applies; otherwise show the correction to the user as a lead and do not apply it on your own.
- Publish. Only the changed lines are published as the correction, with the error, the models that wrote the old form in our runs, the release-note quote and both checks.
- Recheck. When the package publishes a newer release, the same scripts run again on it: the old code must still fail with the same error and every fix must still pass. A confirmed entry shows that release in its sandbox proof; one that breaks goes to a person, who decides whether to withdraw it.
| Control | Group | Result |
|---|---|---|
| pos-py-pandas-append | positive | detected |
| pos-py-numpy-float_ | positive | detected |
| pos-py-sqlalchemy-engine-execute | positive | detected |
| pos-py-sklearn-load_boston | positive | detected |
| pos-py-pydantic-basesettings | positive | detected |
| pos-py-werkzeug-url_quote | positive | detected |
| pos-js-node-fetch-require | positive | detected |
| pos-js-chalk-require | positive | detected |
| pos-js-uuid-deep-require | positive | detected |
| pos-js-p-limit-require | positive | detected |
| neg-py-requests-session | negative | passed both versions |
| neg-py-pyyaml-safe_load | negative | passed both versions |
| neg-py-numpy-sum | negative | passed both versions |
| neg-py-pandas-sum | negative | passed both versions |
| neg-py-attrs-define | negative | passed both versions |
| neg-js-lodash-chunk | negative | passed both versions |
| neg-js-uuid-v4 | negative | passed both versions |
| neg-js-semver-gt | negative | passed both versions |
| neg-js-dayjs-year | negative | passed both versions |
| neg-js-ms | negative | passed both versions |
Limits
- Only code that runs offline in one short script is checked.
- A model may answer differently on another day; entries name the models in which the old form was seen and the date of the runs.
- Knowledge cutoffs are the vendors' stated dates; a model may know a little about releases shortly before its cutoff.
- "Fails on" names the newest release checked; the change happened somewhere between the two versions shown.
- Both checks look at one example script: the output comparison sees printed text only (not return values, files, other inputs or exception types), and the patch check applies the published lines to the code the models wrote, not to yours.
kindnames the API change;proofholds the observations; neither is derived from the other.
For agents
- This page as JSON:
https://formachines.ai/errata/index.json - Whole database in plain text:
/errata/llms-full.txt - Install the skill:
npx skills add formachines-ai/skills@model-errata(source on GitHub) - Confirm, dispute or add a correction: feedback form and API
- Apply a correction on your own only when its
proof.auto_applyis true (Automatic application is allowed only when proof.equivalent is true and proof.applicability.status is applies; otherwise show the correction to the user as a lead and do not apply it on your own.)