chore(bench): add analyzer benchmark suite and PR regression CI#116
Open
chore(bench): add analyzer benchmark suite and PR regression CI#116
Conversation
Adds a vitest bench suite that measures QueryOptimizer.start() across small/medium/large DB shapes via a single testcontainer Postgres, plus a PR workflow that runs the bench on both HEAD and base, posts a comparison comment, and fails the check on >20% regression. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Query Doctor Analysis
83 queries analyzed
2 pre-existing issues
SELECT "guests"."id", "guests"."session_id", "guests"."username", "guests"."avatar_path", "guests"."color", "guests"."side", "guests"."audio_recording_path", "guests"."audio_recording_public", "gue...
indexassets(event_id, inserted_at desc)
cost 31,003,449 → 1,493 (100% reduction)SELECT * FROM guest_ip_addresses WHERE ip_address = '127.0.0.1';
indexguest_ip_addresses(ip_address)
cost 154,402 → 8 (100% reduction)
Using assumed statistics (10000000 rows/table). For better results, sync production stats.
Benchmark comparisonThreshold: ±20% on mean. 🔴 regression · 🟢 improvement · ⚪ within noise · 🆕 new/removed.
Benchmarks use testcontainers + wall-time; some noise is expected. Treat single-digit deltas as not-significant. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
src/remote/optimizer.bench.ts— a vitest bench suite that measuresQueryOptimizer.start()on small (3t/5q), medium (20t/100q), and large (300t/1000q) shapes against a single testcontainer Postgres..github/workflows/benchmark.yaml— runs the bench on both PR HEAD and the merge-base, usesscripts/compare-bench.mjsto diff the two JSON reports, posts/updates a single PR comment with a regression table, and fails the check when any bench's mean regresses by >20%.npm run benchand abenchmark.includepattern invitest.config.ts.Test plan
npm run benchworks locally.🤖 Generated with Claude Code