Add validation config and counters output for CDC Wonder#2103
Add validation config and counters output for CDC Wonder#2103abhishekjaisw wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces validation configuration and golden data files for the CDC Wonder NNDSS Infectious Weekly import. Specifically, it adds golden observations and summary reports, updates the import manifest to output counters and run validation, and defines validation rules in validation_config.json. The reviewer's feedback highlights critical path resolution issues in validation_config.json where golden and input file paths must be prefixed with ../../input1/ instead of ../../../../ to execute correctly in the validation environment. Additionally, the reviewer suggests ensuring the counters/ directory exists (e.g., via a .gitkeep file) to prevent runtime file-not-found errors when writing counters.
This PR completes the onboarding of CDC Wonder NNDSS Infectious Weekly data into the Datacommons pipeline by adding validation guardrails.
Key changes:
golden_datasummaries and observations to run differential tests and safeguard against unexpected regression.manifest.jsonto output processing metrics into thecounters/directory and configured the pipeline to persist them natively into thesource_filesdestination on GCS.