Add full dbsync config coverage#3537
Open
ArturWieczorek wants to merge 14 commits into
Open
Conversation
Prepare to extend db-sync config-option coverage: - Add Column members used by upcoming subtests (Redeemer.FEE, Tx.DEPOSIT, TxOut.CONSUMED_BY_TX_ID, StakeRegistration.DEPOSIT, PoolUpdate.DEPOSIT, GovActionProposal.X_EPOCH/EXPIRATION). - Add ColumnCondition.NOT_ZERO and IS_NOT_NULL for positive column checks. - Extract a reusable wait_for_tables_not_empty() retry helper and use it in the governance subtest (behavior preserving).
Add Phase 1 config-option subtests that need no extra on-chain activity: - plutus disable: redeemer / redeemer_data / datum stay empty. - metadata disable: tx_metadata stays empty. - tx_out consumed: tx_in empty, consumed_by_tx_id column added; with force_tx_in=True tx_in is populated again. - tx_out use_address_table: the address table exists and is populated. Group the new scenarios under a per-phase `_subtests_phase1()` generator so `get_subtests` stays within complexity limits as coverage grows. Add `column_exists` / `column_data_type` helpers (and `query_column_data_type`).
Add Phase 2 enable-side subtests (depend on on-chain activity): - plutus enable: script / redeemer / redeemer_data / datum populated. - metadata enable: tx_metadata populated; keys filter keeps only the configured metadata key. - shelley enable: stake_registration / pool_update populated. - shelley disable independence: epoch_stake stays populated (ledger-controlled). Document a db-sync discrepancy found via TDD: legacy ShelleyRegCert stake registrations are inserted regardless of shelley=disable (Certificate.hs insertDelegCert is unguarded, unlike the Conway path). The documented expectation (stake_registration empty) is kept and marked xfail until db-sync gates the legacy path.
Add Phase 3 subtests for ledger-derived data: - ledger enable: reward / epoch_stake / ada_pots / epoch_param populated, redeemer.fee computed, and a positive (ledger-derived) tx deposit recorded. - ledger disable: those tables empty, redeemer.fee null, and no positive tx deposit remains. tx.deposit is not asserted uniformly NULL because db-sync still records 0 for some txs even without ledger state; the meaningful effect is that ledger-derived (positive) deposits are dropped. - ledger ignore: ledger-derived tables empty (state maintained but unused). - pool_stat enable/disable: pool_stat populated only when enabled.
Add Phase 4 off-chain pool metadata subtests: - offchain_pool_data enable: db-sync fetches the cluster pools' registered metadata into off_chain_pool_data with no fetch errors (polled, since the fetch loop is asynchronous). Skipped when db-sync was not started with --allow-private-offchain-urls. - offchain_pool_data disable: no fetch happens (off_chain_pool_data and off_chain_pool_fetch_error empty); the on-chain pool_metadata_ref remains. Add dbsync_utils.allow_private_offchain_urls_enabled() to gate the off-chain tests (detected from the running db-sync process args). Enable --allow-private-offchain-urls for the "dbsync config" CI selection so the off-chain tests can fetch localhost metadata.
Add Phase 5 off-chain vote metadata coverage: - Extend DBSyncConfigBuilder with `offchain_vote_data` (default, builder method and build() output); db-sync defaults it to "disable" since 13.7.0.1. - offchain_vote_data disable: with governance enabled, on-chain anchors (voting_anchor) are recorded but all off_chain_vote_* tables stay empty, proving the fetch is gated independently of the governance flag. - offchain_vote_data enable: fetches anchor metadata (recorded as data or a fetch error). Skipped unless db-sync allows private URLs and a fetchable vote anchor exists; the suite's anchors are non-CIP and unpublished, so the CIP sub-tables are not asserted here. Move the off_chain_vote_* tables out of GOVERNANCE_TABLES into a dedicated OFFCHAIN_VOTE_TABLES group so the governance subtest covers only on-chain governance data and off-chain vote coverage is flag-gated.
Add Phase 6 subtests for the remove_jsonb_from_schema option: - disable: jsonb columns (datum.value, cost_model.costs, gov_action_proposal.description) keep the jsonb type. - enable: those columns no longer use the jsonb type. Assertions use column data types (schema-level, independent of row counts) and are limited to jsonb columns not also governed by json_type, to isolate this option's effect.
Add Phase 7 preset subtests that exercise db-sync's own preset expansion by
emitting a raw `preset` key (DBSyncConfigBuilder.build now emits {"preset": ...}
when a preset is selected):
- full: tx_out / ma_tx_out / tx_metadata / redeemer / drep_registration
populated, tx_cbor empty.
- only_governance: governance data populated, tx_out / ma_tx_out empty.
- disable_all: tx_out / ma_tx_out / redeemer / drep_registration empty.
- only_utxo: metadata / plutus off (tx_metadata, redeemer empty). tx_out /
ma_tx_out are not asserted (bootstrap bulk UTxO load is environment-sensitive
and unreliable on a fast dev cluster).
Document a db-sync discrepancy found via TDD: the only_utxo preset enables
governance (onlyUTxOInsertOptions sioGovernance=True) although the docs describe
it as UTxO-only. The documented expectation (drep_registration empty) is kept
and marked xfail until db-sync / the docs are reconciled.
Mark in offchain_vote_data_enable that full is_valid (TRUE/FALSE/NULL) and off-chain vote sub-table assertions become possible once the anchor test-data vectors land (cardano-node-tests PR #3497) and a governance test registers them. No new data files needed: governance_action_anchor.json already provides a conformant CIP-100 vector, and #3497 adds the non-conformant and invalid-JSON negatives (reusable across anchor types).
Condense subtest docstrings/comments and materialize the tables argument in wait_for_tables_not_empty so a generator is not exhausted by the first failing poll. Simplify with_preset to emit only the `preset` key and let db-sync expand it, dropping the now-unused per-preset config maps and the Column members / NOT_ZERO condition they relied on. Wire the only_utxo preset to real cardano-db-sync blockers: #2150 (bootstrap crash near tip) xfails the restart, #2151 (governance enabled contrary to the docs) replaces the inline xfail. Add skip guards for offchain_pool_data when no pool metadata is on chain, and drop the unguarded shelley_disable_stake_registration subtest.
Replace the opaque _subtests_phase1..7 helpers with helpers named after the db-sync config option each set exercises (_subtests_tx_out, _subtests_plutus, _subtests_ledger, ...), and fold the previously inline subtests into them so each option's enable and disable scenarios live together. Pure refactor: all 31 subtest bodies are unchanged and still reported by scenario name; no test logic changes. Verified on a local node + db-sync 13.7.2.1 cluster (30 passed, 1 xfailed).
Collaborator
Author
|
CI tests: 💚 
Local tests: passed 💚 |
Two failures surfaced only in the full db-sync regression suite: - offchain_pool_data_enable asserted off_chain_pool_fetch_error is empty. In a full run other pools have unreachable metadata and a transient fetch error can precede a successful retry, so drop that assertion and keep the off_chain_pool_data NOT_EMPTY check. - The only_utxo preset crash (db-sync #2150) and benign pool_stat "pool not found" warnings tripped the teardown log-error check. Register ignore rules for those expected lines via add_ignore_rule.
The only_utxo preset triggers the db-sync #2150 bootstrap crash, whose unexpected dbsync exit is logged to supervisord.log. The teardown log check has no ignore path for supervisord.log, so that crash fails the run even though the subtest is otherwise known-broken. Skip it (still visible in the report, tracked by #2150) instead of running and crashing db-sync. Also narrow the db-sync log ignore rule to the benign pool_stat queryPoolHashId warning and drop the ineffective supervisord ignore rule. Fix the GOVERNANCE_TABLES comment: off-chain vote anchors need --allow-private-offchain-urls only for cluster-local URLs; public anchor URLs are fetched regardless.
ArturWieczorek
marked this pull request as ready for review
July 15, 2026 18:32
Add disable_epoch_true / disable_epoch_false subtests: with disable_epoch=true the `epoch` rollup view returns no rows; with false it is populated. Adds a with_disable_epoch builder method, emitted only when explicitly set so the config for all other subtests is unchanged. Verified on a local db-sync 13.7.2.1 cluster. json_type was evaluated but is not covered: in 13.7.2.1 the option is parsed and validated (text/jsonb/disable) yet never referenced by any schema or insert module, so it has no observable effect (text and jsonb produce identical jsonb columns), leaving nothing to assert.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2652