adding goldens for BLS imports#2090
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds golden summary reports for the c_cpi_u and cpi_u datasets. The reviewer pointed out that the golden summary report for cpi_w is missing and should be added to ensure complete test coverage for all datasets defined in the manifest.
balit-raibot
changed the title
|
@gemini-code-assist review |
There was a problem hiding this comment.
Code Review
This pull request introduces validation configurations and golden summary reports for the US BLS CPI and CES State data imports, updating their manifests to reference the new validation files. The reviewer identified that the relative paths to the golden summary reports in both validation_config.json files are incorrect, pointing to non-existent directories by going up four levels instead of three. Correcting these paths is necessary to ensure the validation runner can locate the golden files.
…-raibot/data into adding_goldens_bls_imports
|
@gemini-code-assist review |
There was a problem hiding this comment.
Code Review
This pull request adds validation configurations and golden summary reports for several US Bureau of Labor Statistics (BLS) datasets, including CPI, JOLTS, and CES State. Specifically, it introduces validation_config.json files to define validation rules (such as deleted records percentage thresholds and golden data checks) and updates the respective manifest.json files to reference them. It also adds the corresponding golden summary report CSV files. As there are no review comments, I have no feedback to provide.
|
LGTM |
| "input_files": "../../../../output/bls_ces_state.csv" | ||
| } | ||
| } | ||
| ] |
There was a problem hiding this comment.
Please also enable checks for MAX_DATE_LATEST and MAX_DATE_CONSISTENT so that deletions fr a few Svs for latest year can be caught as well.
There was a problem hiding this comment.
Pls also add SQL_VALIDATOR to check max date is previous month.
There was a problem hiding this comment.
added these checks for BLS_CPI_Category as was discussed because of its threshold of 0.6%
There was a problem hiding this comment.
These checks were implemented and executed in test environment.
Seems like we can not add any of these checks as explained below:
- MAX_DATE_LATEST ==> This will make the import fail in the month of January and February every year (because till 21st February, the max_latest_date will give "year-1" instead of current year.
- MAX_DATE_CONSISTENT ==> This will fail every time because while cpi_u and cpi_w have latest date as previous month, c_cpu_u has latest date as "previous month -1". Also we found 213 SVs whose max_latest_date is more than 2 months old (even back to 2019) (https://storage.mtls.cloud.google.com/datcom-import-test/statvar_imports/us_bls/cpi_category/BLS_CPI_Category/2026_07_20T06_58_01_530319_07_00/input0/validation/validation_output.csv)
- SQL_VALIDATOR ==> This will make the import fail every execution before 21st of the month, as source updates data on 21st of every month.
Also we tried to implement the logic that max_latest_date is either previous or previous-1 month but it also failed as 213 SVs have way too old max_dates (validation report attached above). We can manually exclude these SVs in the SQL query but doesn't seem a good practice.
4 participants
This PR adds golden data for BLS imports: BLS_CPI, BLS_CES_State, BLS_JOLTS, BLS_CPI_Category and US_CPI.
NOTE: BLS_CPI and BLS_JOLTS don't have observationAbout column in the observations output file.
The imports have only one observation about which is being used by .tmcf file.