Skip to content

Fix Python unit tests broken by google-api-core arriving via cloud-sql-python-connector 1.23.0 - #40623

Merged
shunping merged 7 commits into
apache:masterfrom
shunping:fix-python-workflow-1
Oct 16, 2026
Merged

shunping merged 7 commits into
apache:masterfrom
shunping:fix-python-workflow-1

Conversation

@shunping

@shunping shunping commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

cloud-sql-python-connector 1.23.0 was released on 2026-10-15 and is pulled in by the test extra. It adds a new dependency on google-cloud-sql, which requires google-api-core. As a result, google-api-core is now installed in the non-GCP py3xx tox environments, but google-apitools is not.

Some tests treated "google.api_core imports" as meaning "GCP dependencies are installed", or were skipped only because it failed to import. Those tests now run in environments that can't support them. Python Unit Tests on master has been failing on macOS and Windows since this release.

Failure 1: dataframe/io_test.py

  • ReadGbqTransformTests.test_bad_schema_public_api_direct_read fails with ValueError: Please install GCP Dependencies.
  • ReadGbqTransformTests.test_unsupported_callable: the expected "got a callable instead" error never appears. Instead it raises TypeError: isinstance() arg 2 must be a type, a tuple of types, or a union.
  • Cause: the class was previously skipped when google.api_core was missing. Now that we have google.api_core, but we are still missing the apitools-based BigQuery client in the environment.

Failed workflows and examples:

Failure 2: io/requestresponse_it_test.py and transforms/enrichment_handlers/cloudsql_it_test.py

  • TestRedisCache.test_rrio_no_coder_exception fails with
    docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory')).
  • TestExternalSQLServerEnrichment fails with the same reason.
  • Cause: the module skips itself on any ImportError, and apache_beam/io/requestresponse.py unconditionally imports google.api_core.exceptions. Before this release, that import failure was the only thing that skipped the module in non-GCP envs. Now the module loads, TestRedisCache runs, and its Redis testcontainer can't start on runners without a Docker daemon, such as GitHub's macOS runners. It also doesn't work on GitHub's windows runners, because windows runners have a Docker daemon but they only run Windows containers.

Failed workflows and examples:

Failure 3: transforms/enrichment_handlers/cloudsql_it_test.py in PostCommit_Python_Versions

  • TestExternalMySQLEnrichment.* fails during setup with TimeoutError: Container did not emit logs containing '.*: ready for connections.*: ready for connections.*' within 15.000 seconds.
  • Cause: these tests used to be skipped because google.api_core was missing (see Failure 2). They now run on the Linux runners, which exposes a problem with the testcontainers timeout config. (https://exquisite-dusk-036c28.netlify.app/features/wait_strategies/#configuring-wait-behavior)

Failed run:

@shunping

Copy link
Copy Markdown
Collaborator Author

cc'ed @Amar3tto

@shunping
shunping marked this pull request as ready for review October 16, 2026 01:33

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR updates the test skip guard for ReadGbqTransformTests and ReadUsingReadGbqTests from google.api_core.exceptions.GoogleAPICallError to apitools.base.py.exceptions.HttpError. This accurately reflects the actual dependency on google-apitools for the BigQuery client and resolves test failures in non-GCP environments where google-api-core is now transitively installed.

🔍 General Feedback

  • The root cause analysis is clear and accurate.
  • Aligning the skip condition with io/gcp/bigquery_test.py and removing the redundant TableFieldSchema check is clean and appropriate.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@shunping
shunping requested a review from Amar3tto October 16, 2026 01:35
requestresponse_it_test.py used to be skipped in non-GCP envs only
because google-api-core was missing. Now that it comes in through
cloud-sql-python-connector 1.23.0, the tests run on macOS and Windows
runners, which have no Docker daemon.
@shunping shunping changed the title Correctly skip read_gbq tests when apitools is unavailable Fix Python unit tests broken by google-api-core arriving via cloud-sql-python-connector 1.23.0 Oct 16, 2026
@shunping
shunping marked this pull request as draft October 16, 2026 03:02
@github-actions github-actions Bot added the core label Oct 16, 2026
@shunping
shunping marked this pull request as ready for review October 16, 2026 03:44
@codecov

codecov Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.65217% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 79.92%. Comparing base (76f6e6d) to head (f19f3b4).
⚠️ Report is 202 commits behind head on master.

Files with missing lines Patch % Lines
sdks/python/conftest.py 90.90% 1 Missing ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master   #40623       +/-   ##
=============================================
+ Coverage     56.16%   79.92%   +23.75%     
=============================================
  Files          1124      488      -636     
  Lines        178336    82556    -95780     
  Branches       1489        0     -1489     
=============================================
- Hits         100170    65986    -34184     
+ Misses        75645    16570    -59075     
+ Partials       2521        0     -2521     
Flag Coverage Δ
go ?
java ?
python 79.92% <95.65%> (+0.37%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

testcontainers 4.x reads testcontainers_config (max_tries * sleep_time),
not waiting_utils.config, and ignores TC_TIMEOUT/TC_SLEEP_TIME. CI ended
up with a 15s timeout instead of 300s, so MySQL containers timed out.
run_lint.sh requires files with `import unittest` to call
unittest.main(); import SkipTest directly instead.
@shunping
shunping merged commit 1f76c9c into apache:master Oct 16, 2026
91 checks passed
@shunping
shunping deleted the fix-python-workflow-1 branch October 16, 2026 14:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants