Skip to content

Added Golden Checks for EIA Imports [EIA_Electricity, EIA_Coal, EIA_Petroleum, EIA_NaturalGas, EIA_International, EIA_TotalEnergy, EIA_SEDS, EIA_NuclearOutages] and CensusCountyBusinessPatterns#2101

Open
kartik-s21 wants to merge 37 commits into
datacommonsorg:masterfrom
kartik-s21:addedgoldens

Conversation

@kartik-s21

@kartik-s21 kartik-s21 commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

This PR introduces golden checks for EIA Imports and CensusCountyBusinessPatterns. By adding these golden test files, we ensure that future data imports and processing scripts produce consistent output, preventing unexpected regressions.

Key Changes

Added Golden Files: Generated and added golden test files for EIA imports, and CensusCountyBusinessPatterns data.

Test Results (validation_output.csv)-
EIA_Electricity - link SOP
EIA_Coal - link SOP
EIA_Petroleum - link SOP
EIA_NaturalGas- link SOP
EIA_SEDS - link SOP
EIA_NuclearOutages - link SOP
EIA_TotalEnergy - link SOP
EIA_International - link SOP

NOTE - For EIA_International, cron scheduler is corrected as it was previously not working. Also, a CL is raised to resolve the missing_references issue. link

CensusCountyBusinessPatterns - link SOP

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds validation configurations and golden observation/summary files for several US EIA open data imports and New York education datasets. While many golden files were successfully added, several referenced golden summary report files are missing from the PR, specifically those for EIA Electricity, EIA Petroleum, EIA Total Energy, and New York English and Math education data. These missing files must be generated and included to ensure the validation checks can run successfully.

Comment thread scripts/us_eia/opendata/validation_config_EIA_Electricity.json
Comment thread scripts/us_eia/opendata/validation_config_EIA_Petroleum.json
Comment thread scripts/us_eia/opendata/validation_config_EIA_TotalEnergy.json
Comment thread statvar_imports/us_education/new_york_education/validation_config.json Outdated
Comment thread statvar_imports/us_education/new_york_education/validation_config.json Outdated
@kartik-s21
kartik-s21 requested a review from saanikaaa July 17, 2026 09:41
@kartik-s21 kartik-s21 changed the title Added Golden Checks for EIA Imports Added Golden Checks for EIA Imports [EIA_Electricity, EIA_Coal, EIA_Petroleum, EIA_NaturalGas, EIA_International, EIA_TotalEnergy, EIA_SEDS, EIA_NuclearOutages], NewYork_Education and CensusCountyBusinessPatterns Jul 17, 2026
@pulkeet-wq

Copy link
Copy Markdown
Contributor

Few pointers to take into consideration:

  1. NewYork_educaiton is failing with missing references. Therefore, revert changes for that import and create a new PR for the same
  2. Edit SOP docs for EIA_COAL, EIA_ELECTRICITY and EIA_International as discussed on the call

@kartik-s21 kartik-s21 changed the title Added Golden Checks for EIA Imports [EIA_Electricity, EIA_Coal, EIA_Petroleum, EIA_NaturalGas, EIA_International, EIA_TotalEnergy, EIA_SEDS, EIA_NuclearOutages], NewYork_Education and CensusCountyBusinessPatterns Added Golden Checks for EIA Imports [EIA_Electricity, EIA_Coal, EIA_Petroleum, EIA_NaturalGas, EIA_International, EIA_TotalEnergy, EIA_SEDS, EIA_NuclearOutages] and CensusCountyBusinessPatterns Jul 20, 2026
@kartik-s21

Copy link
Copy Markdown
Contributor Author

Few pointers to take into consideration:

  1. NewYork_educaiton is failing with missing references. Therefore, revert changes for that import and create a new PR for the same
  2. Edit SOP docs for EIA_COAL, EIA_ELECTRICITY and EIA_International as discussed on the call

Updated the PR with -

  1. Reverted all the changes done to NewYork_Education import and creating a separate PR for it (including the validation fix)
  2. Updated the SOP Document for all the imports to make sure the right information gets sent.

@kartik-s21
kartik-s21 requested a review from balit-raibot July 21, 2026 03:07
@balit-raibot

Copy link
Copy Markdown
Contributor

@gemini-code-assist review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces validation configurations and golden data files for Census County Business Patterns and various US EIA OpenData datasets. However, several referenced golden files are missing from the pull request, specifically the golden summary reports and observations for Census County Business Patterns, as well as the golden summary reports for EIA Electricity, EIA Petroleum, and EIA Total Energy.

Comment thread scripts/census_county_business_patterns/validation_config.json
Comment thread scripts/us_eia/opendata/validation_config_EIA_Electricity.json
Comment thread scripts/us_eia/opendata/validation_config_EIA_Petroleum.json
Comment thread scripts/us_eia/opendata/validation_config_EIA_TotalEnergy.json
@pulkeet-wq

Copy link
Copy Markdown
Contributor

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants