Controlled multi-task benchmarking
FORGE is expanding from a first external pilot into multi-task evidence across security, database, CI/CD and infrastructure work. Early runs are labeled honestly when they are exploratory instead of fully controlled.
Why this exists
DG-BM-001 proved FORGE works on an external project, but it was a pilot. Controlled benchmarking tightens the method: same base commit, same task brief, same acceptance criteria, same verification commands and visible limitations.
The first exploratory phase is now complete across security, database, CI/CD and Kubernetes/runtime-probe work. It produced public evidence and one product improvement: feature-intent wording such as incident CRUD is classified as feature work while outage or failure language remains bug work.
Benchmark matrix
| ID | Area | Goal | Status |
|---|---|---|---|
| DG-BM-002 | Authentication / security | Protect mutating API behavior and verify rejected/allowed paths. | Exploratory complete |
| DG-BM-003 | Database migration | Add schema-changing incident audit events and verify API behavior. | Exploratory complete |
| DG-BM-004 | GitHub Actions CI/CD | Add pull-request checks for install, type-check, API tests and build. | Exploratory complete |
| DG-BM-005 | Kubernetes / runtime probes | Add Kubernetes manifests with API, web and database runtime probes. | Exploratory complete |
Protocol
- Both branches start from the same application commit.
- The control branch does not receive FORGE-generated files unless the benchmark says so.
- Every inspected file, edited file, retry and failed command is counted.
- Verification commands are identical across both branches where the task allows it.
- FORGE-only metrics are published as internal estimates, not billing data.
Result format
Benchmark results are recorded in a JSON shape under forge/evidence/results/ and validated against the public schema in forge/evidence/result.schema.json. The schema keeps measured run data, FORGE estimates and provider billing data in separate fields.