Skip to content

Add validation config and counters output for CDC Wonder#2103

Open
abhishekjaisw wants to merge 3 commits into
datacommonsorg:masterfrom
abhishekjaisw:cdc_wonder_validation_and_counters
Open

Add validation config and counters output for CDC Wonder#2103
abhishekjaisw wants to merge 3 commits into
datacommonsorg:masterfrom
abhishekjaisw:cdc_wonder_validation_and_counters

Conversation

@abhishekjaisw

Copy link
Copy Markdown
Contributor

This PR completes the onboarding of CDC Wonder NNDSS Infectious Weekly data into the Datacommons pipeline by adding validation guardrails.

Key changes:

  1. Added Golden Data Validation: Includes baseline golden_data summaries and observations to run differential tests and safeguard against unexpected regression.
  2. Fixed Missing References: Resolves mapping issues and flags missing references appropriately to pass strict validation rules.
  3. Counters Output Generation: Adjusted the scripts in manifest.json to output processing metrics into the counters/ directory and configured the pipeline to persist them natively into the source_files destination on GCS.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces validation configuration and golden data files for the CDC Wonder NNDSS Infectious Weekly import. Specifically, it adds golden observations and summary reports, updates the import manifest to output counters and run validation, and defines validation rules in validation_config.json. The reviewer's feedback highlights critical path resolution issues in validation_config.json where golden and input file paths must be prefixed with ../../input1/ instead of ../../../../ to execute correctly in the validation environment. Additionally, the reviewer suggests ensuring the counters/ directory exists (e.g., via a .gitkeep file) to prevent runtime file-not-found errors when writing counters.

Comment thread statvar_imports/cdc/CDCWonder_NNDSS_Infectious_Weekly/manifest.json
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant