NREL / buildstockbatch

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

CI issues with newer version of pandas and existing parquet files in repo

nmerket opened this issue · comments

Describe the bug

The CI is returning errors on all runs in this test for python > 3.8. The old parquet files store an object datatype while the newer ones have a python[string] datatype.

To Reproduce
Steps to reproduce the behavior:

  1. Happens on any CI run.

Expected behavior

Tests pass

Logs

From the CI logs:

        # results parquet
        test_pq = pd.read_parquet(os.path.join(test_path, 'baseline', 'results_up00.parquet')).sort_values('building_id')\
            .reset_index().drop(columns=['index'])
        reference_pq = pd.read_parquet(os.path.join(reference_path, 'baseline', 'results_up00.parquet'))\
            .sort_values('building_id').reset_index().drop(columns=['index'])
>       pd.testing.assert_frame_equal(test_pq, reference_pq)
E       AssertionError: Attributes of DataFrame.iloc[:, 4] (column name="completed_status") are different
E       
E       Attribute "dtype" are different
E       [left]:  string[python]
E       [right]: object

Platform (please complete the following information):

  • Simulation platform: ubuntu-latest on GitHub Actions
  • BuildStockBatch version, branch, or sha: develop
  • resstock or comstock repo version, branch, or sha: develop
  • Local Desktop OS: [e.g. Windows, Mac, Linux, especially important if running locally]

Additional context

Two ideas for how to address this:

  1. (easy but could break again) Open the testing parquet files in the repo in a newer version of pandas, convert the columns to string and save them back. This should solve the error, but something like this may happen again.
  2. (harder but more maintainable in the long run) Change these tests to instead of comparing an expected parquet to a generated one, use the newer integration test framework where you can actually run ResStock and generate results. Then you'd check that those results have expected columns and such without comparing two dataframes directly.