MILESTONE 2
Controlled multi-task benchmarking
The next FORGE evidence milestone compares plain AI and FORGE-assisted AI across security, database, CI/CD and infrastructure tasks from the same application baseline.
Why this exists
DG-BM-001 proved FORGE works on an external project, but it was a pilot. Controlled benchmarking tightens the method: same base commit, same task brief, same acceptance criteria, same verification commands and visible limitations.
Claim boundary: FORGE can report estimated context reduction and observed workflow differences. It will not claim provider-token or credit savings unless direct provider billing telemetry is captured.
Planned benchmark matrix
| ID | Area | Goal | Status |
|---|---|---|---|
| DG-BM-002 | Authentication / security | Protect mutating API behavior and verify rejected/allowed paths. | Planned |
| DG-BM-003 | Database migration | Add schema-changing behavior with migration evidence. | Planned |
| DG-BM-004 | GitHub Actions CI/CD | Add pull-request checks for install, type-check, test and build. | Planned |
| DG-BM-005 | Kubernetes / runtime probes | Add deployment assets with liveness/readiness verification. | Planned |
Protocol
- Both branches start from the same application commit.
- The control branch does not receive FORGE-generated files unless the benchmark says so.
- Every inspected file, edited file, retry and failed command is counted.
- Verification commands are identical across both branches where the task allows it.
- FORGE-only metrics are published as internal estimates, not billing data.
Result format
Benchmark results are recorded in a JSON shape under forge/evidence/results/ and validated against the public schema in forge/evidence/result.schema.json. The schema keeps measured run data, FORGE estimates and provider billing data in separate fields.