Skip to content

chore(bench): add analyzer benchmark suite and PR regression CI#116

Open
veksen wants to merge 1 commit intomainfrom
veksen/add-benchmarks-v3
Open

chore(bench): add analyzer benchmark suite and PR regression CI#116
veksen wants to merge 1 commit intomainfrom
veksen/add-benchmarks-v3

Conversation

@veksen
Copy link
Copy Markdown
Member

@veksen veksen commented Apr 18, 2026

Summary

  • Adds src/remote/optimizer.bench.ts — a vitest bench suite that measures QueryOptimizer.start() on small (3t/5q), medium (20t/100q), and large (300t/1000q) shapes against a single testcontainer Postgres.
  • Adds .github/workflows/benchmark.yaml — runs the bench on both PR HEAD and the merge-base, uses scripts/compare-bench.mjs to diff the two JSON reports, posts/updates a single PR comment with a regression table, and fails the check when any bench's mean regresses by >20%.
  • Wires up npm run bench and a benchmark.include pattern in vitest.config.ts.

Test plan

  • First PR run emits a "no baseline" comment (base commit predates the bench file) — confirms graceful fallback.
  • Subsequent PRs show a diff table with 🔴/🟢/⚪/🆕 verdicts and RME deltas.
  • npm run bench works locally.

🤖 Generated with Claude Code

Adds a vitest bench suite that measures QueryOptimizer.start() across
small/medium/large DB shapes via a single testcontainer Postgres, plus
a PR workflow that runs the bench on both HEAD and base, posts a
comparison comment, and fails the check on >20% regression.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Query Doctor Analysis

View full run details

83 queries analyzed

2 pre-existing issues

Using assumed statistics (10000000 rows/table). For better results, sync production stats.

@github-actions
Copy link
Copy Markdown

Benchmark comparison

Threshold: ±20% on mean. 🔴 regression · 🟢 improvement · ⚪ within noise · 🆕 new/removed.

Benchmark Base mean PR mean Δ RME (base → PR)
src/remote/optimizer.bench.ts > query optimizer > large (300 tables, 1000 queries) 141173ms 140295ms -0.6% ±1.6% → ±1.5%
src/remote/optimizer.bench.ts > query optimizer > medium (20 tables, 100 queries) 2190ms 2206ms +0.7% ±4.1% → ±2.3%
src/remote/optimizer.bench.ts > query optimizer > small (3 tables, 5 queries) 79.37ms 77.31ms -2.6% ±11.1% → ±9.8%

Benchmarks use testcontainers + wall-time; some noise is expected. Treat single-digit deltas as not-significant.
Base commit: ``

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant