test failures on presubmit session
tswast opened this issue · comments
Many tests are failing in the presubmit session.
nox > python -m pip freeze
aiohttp==3.9.1
aiosignal==1.3.1
appdirs==1.4.4
argcomplete==3.2.1
asttokens==2.4.1
asyncmock==0.4.2
atpublic==4.0
attrs==23.2.0
bidict==0.22.1
# Editable Git install with no remote (bigframes==0.19.2)
-e /tmpfs/src/github/python-bigquery-dataframes
black==23.12.1
bleach==6.1.0
cachetools==5.3.2
certifi==2023.11.17
cffi==1.16.0
cfgv==3.4.0
charset-normalizer==3.3.2
click==8.0.4
click-plugins==1.1.1
cligj==0.7.2
cloudpickle==3.0.0
colorlog==6.8.0
comm==0.2.1
coverage==7.4.0
cryptography==41.0.7
db-dtypes==1.2.0
decorator==5.1.1
distlib==0.3.8
docstring-inheritance==2.1.2
docutils==0.20.1
et-xmlfile==1.1.0
exceptiongroup==1.2.0
execnet==2.0.2
executing==2.0.1
filelock==3.13.1
fiona==1.9.5
frozenlist==1.4.1
fsspec==2023.12.2
gcp-docuploader==0.6.5
gcp-releasetool==1.17.0
gcsfs==2023.12.2.post1
geopandas==0.14.2
google-api-core==1.34.0
google-auth==2.26.2
google-auth-oauthlib==1.2.0
google-cloud-bigquery @ git+[https://github.com/googleapis/python-bigquery.git@1f96439b3dbd27f11be5e2af84f290ec6094d0a4](https://www.google.com/url?q=https://github.com/googleapis/python-bigquery.git@1f96439b3dbd27f11be5e2af84f290ec6094d0a4&sa=D)
google-cloud-bigquery-connection==1.14.0
-e git+[https://github.com/googleapis/python-bigquery-storage.git@fb0d6f07d4b7582cf01ec98fee3081ec34995603#egg=google_cloud_bigquery_storage](https://www.google.com/url?q=https://github.com/googleapis/python-bigquery-storage.git@fb0d6f07d4b7582cf01ec98fee3081ec34995603%23egg%3Dgoogle_cloud_bigquery_storage&sa=D)
google-cloud-core==2.4.1
google-cloud-functions==1.14.0
google-cloud-iam==2.13.0
google-cloud-resource-manager==1.11.0
google-cloud-storage==2.14.0
google-cloud-testutils==1.4.0
google-crc32c==1.5.0
google-resumable-media==2.7.0
googleapis-common-protos==1.62.0
greenlet==3.0.3
grpc-google-iam-v1==0.13.0
grpcio==1.60.0
grpcio-status==1.48.2
humanize==4.9.0
ibis-framework==7.1.0
identify==2.5.33
idna==3.6
importlib-metadata==7.0.1
importlib-resources==6.1.1
iniconfig==2.0.0
ipython==8.20.0
ipywidgets==8.1.1
jaraco.classes==3.3.0
jedi==0.19.1
jeepney==0.8.0
Jinja2==3.1.3
joblib==1.3.2
jupyterlab-widgets==3.0.9
keyring==24.3.0
markdown-it-py==3.0.0
MarkupSafe==2.1.4
matplotlib-inline==0.1.6
mdurl==0.1.2
mock==5.1.0
more-itertools==10.2.0
multidict==6.0.4
multipledispatch==1.0.0
mypy-extensions==1.0.0
nh3==0.2.15
nodeenv==1.8.0
nox==2023.4.22
numpy==1.26.3
oauthlib==3.2.2
openpyxl==3.1.2
packaging==23.2
pandas==2.2.0
pandas-gbq==0.20.0
parso==0.8.3
parsy==2.1
pathspec==0.12.1
pexpect==4.9.0
pins==0.8.4
pkginfo==1.9.6
platformdirs==4.1.0
pluggy==1.3.0
pooch==1.8.0
pre-commit==3.6.0
prompt-toolkit==3.0.43
proto-plus==1.23.0
protobuf==3.20.3
ptyprocess==0.7.0
pure-eval==0.2.2
pyarrow==14.0.2
pyarrow-hotfix==0.6
pyasn1==0.5.1
pyasn1-modules==0.3.0
pycparser==2.21
pydata-google-auth==1.8.2
Pygments==2.17.2
PyJWT==2.8.0
pyperclip==1.8.2
pyproj==3.6.1
pytest==7.4.4
pytest-asyncio==0.23.3
pytest-cov==4.1.0
pytest-mock==3.12.0
pytest-retry==1.6.1
pytest-timeout==2.2.0
pytest-xdist==3.5.0
python-dateutil==2.8.2
pytz==2023.3.post1
PyYAML==6.0.1
readme-renderer==42.0
requests==2.31.0
requests-oauthlib==1.3.1
requests-toolbelt==1.0.0
rfc3986==2.0.0
rich==13.7.0
rsa==4.9
scikit-learn==1.4.0
scipy==1.12.0
SecretStorage==3.3.3
shapely==2.0.2
six==1.16.0
SQLAlchemy==2.0.25
sqlglot==19.9.0
stack-data==0.6.3
tabulate==0.9.0
threadpoolctl==3.2.0
tomli==2.0.1
toolz==0.12.0
tqdm==4.66.1
traitlets==5.14.1
twine==4.0.2
typing_extensions==4.9.0
tzdata==2023.4
urllib3==2.1.0
virtualenv==20.25.0
wcwidth==0.2.13
webencodings==0.5.1
widgetsnbextension==4.0.9
xarray==2024.1.0
xxhash==3.4.1
yarl==1.9.4
zipp==3.17.0
...
def test_read_gbq_after_session_start_must_comply_with_default_location(
scalars_pandas_df_index,
test_data_tables,
test_data_tables_tokyo,
dataset_id_permanent_tokyo,
read_method,
query_prefix,
):
# Form query as a table name or a SQL depending on the test scenario
query_tokyo = test_data_tables_tokyo["scalars"]
query = test_data_tables["scalars"]
if query_prefix:
query_tokyo = f"{query_prefix} {query_tokyo}"
query = f"{query_prefix} {query}"
# Initially there is no location set in the bigquery options
assert not bpd.options.bigquery.location
# Starting user journey with anything other than read_gbq*, such as
# read_pandas would bind the session to default location US
> df = bpd.read_pandas(scalars_pandas_df_index)
[tests/system/small/test_pandas_options.py:143](https://cs.corp.google.com/piper///depot/google3/tests/system/small/test_pandas_options.py?l=143):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[bigframes/pandas/__init__.py:570](https://cs.corp.google.com/piper///depot/google3/bigframes/pandas/__init__.py?l=570): in read_pandas
return global_session.with_default_session(
[bigframes/core/global_session.py:65](https://cs.corp.google.com/piper///depot/google3/bigframes/core/global_session.py?l=65): in with_default_session
return func(get_global_session(), *args, **kwargs)
[bigframes/session/__init__.py:849](https://cs.corp.google.com/piper///depot/google3/bigframes/session/__init__.py?l=849): in read_pandas
return self._read_pandas(pandas_dataframe, "read_pandas")
[bigframes/session/__init__.py:889](https://cs.corp.google.com/piper///depot/google3/bigframes/session/__init__.py?l=889): in _read_pandas
load_job = self.bqclient.load_table_from_dataframe(
[.nox/system_prerelease/lib/python3.11/site-packages/google/cloud/bigquery/client.py:2713](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/google/cloud/bigquery/client.py?l=2713): in load_table_from_dataframe
dataframe.to_parquet(
[.nox/system_prerelease/lib/python3.11/site-packages/pandas/util/_decorators.py:333](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pandas/util/_decorators.py?l=333): in wrapper
return func(*args, **kwargs)
[.nox/system_prerelease/lib/python3.11/site-packages/pandas/core/frame.py:3101](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pandas/core/frame.py?l=3101): in to_parquet
return to_parquet(
[.nox/system_prerelease/lib/python3.11/site-packages/pandas/io/parquet.py:480](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pandas/io/parquet.py?l=480): in to_parquet
impl.write(
[.nox/system_prerelease/lib/python3.11/site-packages/pandas/io/parquet.py:190](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pandas/io/parquet.py?l=190): in write
table = self.api.Table.from_pandas(df, **from_pandas_kwargs)
[pyarrow/table.pxi:3869](https://cs.corp.google.com/piper///depot/google3/pyarrow/table.pxi?l=3869): in pyarrow.lib.Table.from_pandas
???
[.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py:613](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py?l=613): in dataframe_to_arrays
arrays = [convert_column(c, f)
[.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py:613](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py?l=613): in <listcomp>
arrays = [convert_column(c, f)
[.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py:600](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py?l=600): in convert_column
raise e
[.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py:594](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py?l=594): in convert_column
result = pa.array(col, type=type_, from_pandas=True, safe=safe)
[pyarrow/array.pxi:340](https://cs.corp.google.com/piper///depot/google3/pyarrow/array.pxi?l=340): in pyarrow.lib.array
???
[pyarrow/array.pxi:82](https://cs.corp.google.com/piper///depot/google3/pyarrow/array.pxi?l=82): in pyarrow.lib._ndarray_to_array
???
[pyarrow/array.pxi:70](https://cs.corp.google.com/piper///depot/google3/pyarrow/array.pxi?l=70): in pyarrow.lib._ndarray_to_type
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E pyarrow.lib.ArrowTypeError: ('Did not pass numpy.dtype object', 'Conversion failed for column geography_col with type geometry')
[pyarrow/error.pxi:91](https://cs.corp.google.com/piper///depot/google3/pyarrow/error.pxi?l=91): ArrowTypeError
...
=========================== short test summary info ============================
FAILED tests/system/small/test_dataframe.py::test_dataframe_bool_aggregates[all_axis0]
FAILED tests/system/small/test_dataframe.py::test_dataframe_bool_aggregates[any_axis0]
FAILED tests/system/small/test_dataframe.py::test_df_pivot[values2-int64_too-columns2]
FAILED tests/system/small/test_groupby.py::test_dataframe_groupby_analytic[cumprod]
FAILED tests/system/small/test_pandas_options.py::test_read_gbq_after_session_start_must_comply_with_default_location[read_gbq-on-table-name]
FAILED tests/system/small/test_pandas_options.py::test_read_gbq_after_session_start_must_comply_with_default_location[read_gbq-on-sql]
FAILED tests/system/small/test_pandas_options.py::test_read_gbq_after_session_start_must_comply_with_default_location[read_gbq_table-on-table-name]
FAILED tests/system/small/test_pandas_options.py::test_read_gbq_after_session_start_must_comply_with_default_location[read_gbq_query-on-sql]
FAILED tests/system/small/test_series.py::test_groupby_window_ops[cumprod] - ...
FAILED tests/system/small/test_dataframe.py::test_to_pandas_downsampling_option_override
FAILED tests/system/small/test_series.py::test_series_add_prefix - AssertionE...
FAILED tests/system/small/test_series.py::test_series_add_suffix - AssertionE...
FAILED tests/system/small/test_session.py::test_read_pandas - pyarrow.lib.Arr...
FAILED tests/system/small/test_session.py::test_read_pandas_multi_index - pya...
FAILED tests/system/small/test_session.py::test_read_pandas_rowid_exists_adds_suffix
FAILED tests/system/small/test_series.py::test_string_astype_int - AssertionE...
FAILED tests/system/small/test_session.py::test_read_pandas_tokyo - pyarrow.l...
FAILED tests/system/small/test_session.py::test_read_csv_local_default_engine[default_sep]
FAILED tests/system/small/test_session.py::test_read_csv_gcs_default_engine
FAILED tests/system/small/test_session.py::test_read_csv_local_default_engine[custom_sep]
20 failed, 1599 passed, 21 skipped, 7 xfailed, 804 warnings in 468.88s (0:07:48)
nox > Command py.test --quiet -n=20 --timeout=600 --junitxml=small_prerelease_3.11_sponge_log.xml --cov=bigframes --cov=tests/system/small --cov-append --cov-config=.coveragerc --cov-report=term-missing --cov-fail-under=0 tests/system/small failed with exit code 1
nox > Session system_prerelease failed.
nox > Ran multiple sessions:
nox > * unit_prerelease: success
nox > * system_prerelease: failed
The issue could be solved by a fix in python client change: googleapis/python-bigquery#1796