feat: gate main on a real Grafana Cloud consistency check#25
Merged
Conversation
…uild/lint Adds the exact verification flow this project's maintainer used by hand: write to a local file first (ground truth), validate the file, then read back from Loki/Prometheus/Tempo and diff each response against that file field-by-field. A tool's own "found it" report was never sufficient proof by itself -- this is what would have caught the recent uuid ESM-crash regression before it shipped. - tools/validation/grafana-cloud/full-consistency-check.sh: orchestrates the existing (but never unified) query-*.ts --compare-with tooling into one command. Accepts --env-file so it works identically for local dev and CI. Includes retry/backoff around the Tempo check specifically -- its search index measurably lags Loki/Prometheus, confirmed empirically. - .github/workflows/ci.yml: new grafana-cloud-consistency job, required for the CI summary to pass. Two-stage: sovdev-selftest first (the exact tool a real customer runs, fails fast before the slower full E2E test), then the full consistency check. Uses dedicated sovdev-ci-ingest/ sovdev-ci-verify credentials (LBAC-scoped to sovdev-ci-company-lookup* only) -- entirely separate from personal dev keys or any customer's. - tools/validation/grafana-cloud/README.md: didn't exist before, documents the whole tools/validation/grafana-cloud/ directory. - contributor/testing/grafana-cloud.md: documents the CI key setup, including a real gotcha hit while creating them -- the Label selectors section is collapsed by default and doesn't appear until scopes are checked, and label selectors only apply to logs/metrics reads, never traces. Verified locally end-to-end with the actual CI credentials before trusting them in the workflow -- both sovdev-selftest and the full consistency check pass cleanly against real Grafana Cloud data. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
grafana-cloud-consistencyCI job: write to a local file (ground truth), validate it, read back from Loki/Prometheus/Tempo, diff each response against the file field-by-field. This is what would have caught the recentuuidESM-crash regression before it shipped — a build/lint pass alone didn't.sovdev-selftestfirst (fail fast, same tool a real customer runs), then the full E2E consistency check only if that passes.sovdev-ci-ingest/sovdev-ci-verifycredentials, LBAC-scoped tosovdev-ci-company-lookup*only — separate from personal dev keys or any customer's.tools/validation/grafana-cloud/full-consistency-check.shorchestrates the existing (but never unified)query-*.ts --compare-withtooling into one command, usable both locally and in CI via--env-file.tools/validation/grafana-cloud/README.md(didn't exist before) and documents the CI key setup incontributor/testing/grafana-cloud.md.Test plan
full-consistency-check.shlocally end-to-end against real Grafana Cloud data, using the actual CI credentials (not just personal dev keys) — Loki 17/17, Prometheus 5/5, Tempo 4/4 (after Tempo's expected indexing delay)sovdev-selftest --backend grafana-cloudlocally with the same CI credentials — all 4 checks passgrafana-cloud-consistencyjob run for real before merging🤖 Generated with Claude Code