[bigquery] 2.69.0: SIGABRT in Conscrypt JNI_OnLoad (could not find method serverCertificateRequested) on macOS arm64/JDK 21; 2.68.0 unaffected

[hayssams]

TL;DR: google-cloud-bigquery 2.69.0 causes a native SIGABRT in Conscrypt's JNI_OnLoad (could not find method serverCertificateRequested) on macOS arm64 / JDK 21, killing the JVM; 2.68.0 with a byte-identical grpc/conscrypt graph is fine. Bisect, crash reports, and thread-dump analysis below.

Correction (supersedes "What hangs" below): this is a crash, not a hang

We initially reported this as an indefinite hang because the observable symptom — from inside our sbt build — is a forked test JVM that goes silent at 0% CPU forever. Capturing thread dumps forced us to actually find the forked JVM's OS process, and it wasn't there: the forked JVM had already terminated, non-gracefully, seconds after printing the fingerprint lines. macOS's crash reporter caught it. See "Thread dump / crash analysis" below for the full evidence — this changes our diagnosis substantially and we believe it now, not the TLS-handshake framing, is the actionable root cause.

Environment

Component	Value
com.google.cloud:google-cloud-bigquery	2.69.0 (hangs) vs 2.68.0 (clean) — adjacent releases, nothing between them on Maven Central
JDK	Temurin 21.0.11 (macOS arm64) — not JDK 17; see correction below
OS	macOS (Darwin 25.5.0), Apple Silicon (arm64)
Build tool	sbt 1.11.5, Scala 2.13.18
io.grpc:* (api/netty-shaded/etc.)	1.82.2 — identical in both the hanging and the passing configuration
org.conscrypt:conscrypt-openjdk-uber	2.6.0 — identical in both the hanging and the passing configuration
com.google.protobuf:protobuf-java	Reproduces with both 4.33.2 and 4.35.1 (dependency override) — not protobuf-version-dependent
Spark on classpath	4.1.3 (see "what we could not reproduce" below — Spark's presence does not appear to matter)

Correction to environment assumptions: earlier internal notes assumed JDK 17. We re-verified: the sbt launcher and the forked test JVM both run on Temurin 21.0.11 (JAVA_HOME resolves to a 21.0.11 install; sbt itself prints Eclipse Adoptium Java 21.0.11 at startup; no Test / javaHome override exists in the build). The failing environment is JDK 21, not 17.

What hangs

A forked JVM launched by sbt testOnly (via Test / fork := true, SubProcess(ForkOptions()...)) to run a single ScalaTest suite. The suite calls into a Starlake Main().run(Array("transform", ...)) code path that ends up submitting a SQL query job to BigQuery via the plain com.google.cloud.bigquery.BigQuery client (BigQueryOptions.getDefaultInstance().getService() → bigQuery.query(QueryJobConfiguration...)). This is the plain Java client's query/job-polling path — not the BigQuery Storage Read/Write API, and not a Spark connector call (Spark is present on the classpath and initializes earlier in the same process, but the BigQuery call itself goes through google-cloud-bigquery directly).

The forked JVM logs two lines and then goes silent forever — 0% CPU, no further output, no exception, no stack trace — right at the point where it should be opening its first network connection to bigquery.googleapis.com:

could not find method serverCertificateRequested
Warning: Unable to read from client, please check on client for further details of the problem.

The sbt parent process then blocks forever in ServerSocket.accept() waiting for the forked test JVM's handshake, which never arrives. We have let it sit for 5+ minutes with zero CPU activity and zero additional output; it never recovers or times out on its own.

These two lines are the JNI method-lookup error emitted by Conscrypt's native layer (common/src/jni/main/include/conscrypt/jniutil.h) when it can't resolve a Java-side callback method (serverCertificateRequested, part of the client-socket TLS handshake callback interface) via GetMethodID at native-library-load time. The lines never appear on a passing run.

Decisive bisect

Methodology: hold every other dependency at a fixed baseline, flip one artifact at a time, run the live suite (SL_REMOTE_TEST=true sbt testOnly *TransformIntegration2Spec*), and classify the result as a ~45–60s pass or a >5-minute 0%-CPU hang.

google-cloud-bigquery version	Result
2.60.0	PASS (43s)
2.65.0	PASS (51s)
2.67.0	PASS (45s)
2.68.0	PASS (47s)
2.69.0	HANGS (reproduces the exact fingerprint above, every time)

2.68.0 and 2.69.0 are adjacent on Maven Central — no intermediate release exists — so this bisect is as precise as it can be: the regression appeared in exactly the 2.69.0 release.

We initially suspected a transitive-dependency clash, since 2.69.0's own POM bumps grpc-netty-shaded 1.81.0→1.82.2 and conscrypt-openjdk-uber 2.5.2→2.6.0 in place. That hypothesis does not survive a second data point: in our final configuration, google-cloud-datacatalog 1.101.0 independently pulls the exact same grpc-netty-shaded 1.82.2 / conscrypt-openjdk-uber 2.6.0 onto the classpath alongside google-cloud-bigquery 2.68.0 (not 2.69.0) — same resolved grpc/conscrypt versions as the hanging configuration — and that combination passes cleanly (48s, verified via sbt evicted showing a single consistent resolution, no version conflicts). Since the grpc/conscrypt versions are byte-identical between one hanging and one passing configuration, the regression is not a transitive-version clash; it appears to be something inside the google-cloud-bigquery 2.69.0 artifact itself (client-side behavior — e.g. a changed connection-setup path, transport configuration, or TLS-provider selection). We were not able to pin down the exact code change: google-cloud-bigquery versions above 2.60.0 are no longer tagged in the googleapis/java-bigquery repository (it was archived and folded into googleapis/google-cloud-java around 2026-03-23), and we could not locate a source tag or changelog entry for 2.69.0 specifically to diff against 2.68.0 — only the resolved POM, which is included below.

POM diff excerpt (2.68.0 → 2.69.0, full diff available on request):

- <version>2.68.0</version>
+ <version>2.69.0</version>
...
- grpc-netty-shaded / grpc-* family: 1.81.0 -> 1.82.2
- conscrypt-openjdk-uber:            2.5.2  -> 2.6.0
- google-auth-library-*:             1.49.0 -> 1.50.0
- protobuf-java-util:                4.33.2 -> 4.33.6
- google-cloud-bigquerystorage (transitive gapic): 3.30.0 -> 3.31.0
+ new exclusions for org.jspecify:jspecify added throughout

What we could NOT reproduce (important, please read before investigating)

We tried to build a minimal standalone repro to hand you something more actionable than "run our whole test suite." None of the three standalone attempts reproduced the hang, each run against the live project with ambient Application Default Credentials, JDK 21.0.11, in under 3 seconds:

1. A bare java -cp <coursier-resolved classpath for google-cloud-bigquery:2.69.0 only> process doing BigQueryOptions.getDefaultInstance().getService().query(QueryJobConfiguration.newBuilder("SELECT 1 AS one").build()) — completed in 1.5s.
2. The same program run against the exact classpath used by our failing sbt-forked test (all ~150 jars, with only the google-cloud-bigquery jar swapped from 2.68.0 to 2.69.0 — every transitive dependency, including grpc-netty-shaded 1.82.2 and conscrypt-openjdk-uber 2.6.0, byte-identical to the failing run) — completed in 1.0s.
3. The same, plus the exact Test / javaOptions JVM flags our build passes to the forked test JVM (--add-opens for java.base/{java.lang,java.lang.invoke,java.lang.reflect,java.io,java.net,java.nio,java.util,java.util.concurrent,java.util.concurrent.atomic,jdk.internal.ref,sun.nio.ch,sun.nio.cs,sun.security.action,sun.util.calendar}=ALL-UNNAMED, -Xmx4g -Xss4m -XX:ReservedCodeCacheSize=128m) — completed in 1.5s.
4. The same as Make authentication simpler #3, but with a local SparkSession (local[2], Spark 4.1.3) fully initialized before the BigQuery call, to mirror what our real failing test does before it reaches the query — completed in ~3s including Spark startup/shutdown.

So we cannot currently hand you a bare-JVM repro that hangs. The hang appears to require something specific to our real failing process that none of the above capture — possibly sbt's ForkMain launcher/IPC mechanism itself, possibly our own Main().run() initialization sequence (which touches HdfsStorageHandler/a custom ProxyLocalFileSystem, Jinja templating, and config loading before the BigQuery call), or possibly a race/timing condition that these short-lived standalone programs don't hit. We flag this openly: this weakens confidence that the defect is purely inside google-cloud-bigquery 2.69.0, but the bisect result itself (2.68.0 clean / 2.69.0 hangs, all else byte-identical) is solid and repeatable, so we're reporting it as-is rather than sitting on it.

Steps to reproduce (as best we can generalize)

We don't have a reduction that reproduces outside our own project yet (see above). What we can say generically:

1. Use google-cloud-bigquery 2.69.0 on JDK 21 (Temurin, macOS arm64 confirmed; other platforms untested) with grpc-netty-shaded/conscrypt-openjdk-uber resolved to 1.82.2/2.6.0.
2. From a forked JVM launched by sbt (Test / fork := true, SubProcess(ForkOptions(...))) — not a plain java invocation — run a JVM process that: initializes a local Spark 4.1.3 session and Hadoop's local filesystem shims, then calls BigQueryOptions.getDefaultInstance().getService() and submits a QueryJobConfiguration via .query(...), using ambient Application Default Credentials.
3. On 2.68.0 this completes in under a minute. On 2.69.0, the process prints the two fingerprint lines above and then hangs indefinitely (0% CPU) before completing the TLS handshake to bigquery.googleapis.com.

What we checked and ruled out

• Not a protobuf-java version issue: reproduces identically with protobuf-java 4.33.2 and 4.35.1.
• Not a grpc/conscrypt transitive-version clash: the passing 2.68.0 configuration resolves the identical grpc-netty-shaded 1.82.2 / conscrypt-openjdk-uber 2.6.0 as the hanging 2.69.0 configuration (confirmed via sbt evicted).
• Not Spark's presence on the classpath by itself: our standalone repro Update gh-pages landing page to look like gcloud-node? #4 above initializes Spark 4.1.3 first and still does not hang.
• Not JDK 17 vs 21 as originally suspected in our internal notes — we corrected this; the actual environment is JDK 21 throughout, so this is not a candidate explanation for a 17-vs-21 discrepancy (there isn't one).

Thread dump / crash analysis

We reproduced the failure again on 2026-08-20 specifically to capture a thread dump of the "hung" forked JVM. We could not get one — because there was no such process to dump. Here is what we found instead, which we believe is strictly better evidence.

Setup. SL_REMOTE_TEST=true sbt -batch "testOnly *TransformIntegration2Spec*" with google-cloud-bigquery bumped to 2.69.0 in the working tree. The forked test JVM logged the same two fingerprint lines as always:

could not find method serverCertificateRequested
Warning: Unable to read from client, please check on client for further details of the problem.

Within the same second, we went looking for the forked JVM's OS process to jstack it (searching by parent PID of the sbt launcher, by ForkMain/ScalaTest process name, and by full ps listing) and found no such process anywhere on the system — not hung, not a zombie, gone. The sbt parent process (PID 96392) itself sat at 0% CPU indefinitely, exactly as previously reported.

The forked JVM didn't hang — it crashed. macOS's crash reporter had written a .ips diagnostic report for a java process at the exact same timestamp as the fingerprint log lines, with "parentPid": 96392 (our sbt process) confirming it's the forked test JVM (PID 99102 in this run):

"exception": {"type": "EXC_BAD_ACCESS", "signal": "SIGABRT",
              "subtype": "KERN_INVALID_ADDRESS at 0x0000000000000028"},
"termination": {"code": 6, "namespace": "SIGNAL", "indicator": "Abort trap: 6"}

The faulting thread's native stack is unambiguous:

Thread 27 "Java: pool-1-thread-1" (crashed)
  libsystem_kernel.dylib      __pthread_kill
  libsystem_pthread.dylib     pthread_kill
  libsystem_c.dylib           abort
  libconscrypt_openjdk_jni-osx-aarch_....dylib  conscrypt::jniutil::init(JavaVM*, JNIEnv*) (.cold.6)
  libconscrypt_openjdk_jni-osx-aarch_....dylib  conscrypt::jniutil::init(JavaVM*, JNIEnv*)
  libconscrypt_openjdk_jni-osx-aarch_....dylib  JNI_OnLoad
  libjava.dylib                Java_jdk_internal_loader_NativeLibraries_load
  ... (JNI library-load trampoline frames, symbols not resolved) ...
  libjvm.dylib                 JavaCalls::call_helper / call_virtual / thread_entry / JavaThread::thread_main_inner

This directly ties the two fingerprint log lines to the crash: Conscrypt's native jniutil::init (called once, the first time the native library is loaded, from JNI_OnLoad) resolves a table of Java-side callback method IDs via GetMethodID — including serverCertificateRequested, part of the client-socket TLS handshake callback interface. When it can't resolve one, it logs could not find method <name> and then falls into a cold/error path (.cold.6) that calls native abort() — a deliberate SIGABRT, not a segfault from memory corruption. The Warning: Unable to read from client... line is emitted by the same code path (it's Conscrypt's generic native-layer error message, printed via its JNI error-reporting helper before aborting). The crash happens on a background thread ("pool-1-thread-1", a java.util.concurrent pool worker — consistent with a gRPC/Conscrypt executor thread lazily loading the native library on first TLS use), but abort() terminates the whole process, not just the thread.

Reproducible every time, not a one-off. We found three more .ips crash reports from earlier attempts earlier the same day (20:30, 20:48, 21:01, all before our final 22:00 run), each with a different crashing PID but the identical faulting-thread name, stack, signal, and termination reason (conscrypt::jniutil::init(JavaVM_*, JNIEnv_*) (.cold.6) → abort → SIGABRT). This is a deterministic crash, not a race.

Why our build reports it as a "hang." jstack on the sbt parent process (PID 96392, still alive and burning 0% CPU, exactly as expected since it has nothing to do but wait) shows the thread responsible for collecting test results blocked exactly where you'd expect if its counterpart never showed up:

"ScalaTestRunner$Skeleton$1" ... java.lang.Thread.State: RUNNABLE
  at java.net.ServerSocket.accept(java.base/ServerSocket.java:543)
  at org.scalatest.tools.Framework$ScalaTestRunner$Skeleton$1$React.tryReact(Framework.scala:906)
  at org.scalatest.tools.Framework$ScalaTestRunner$Skeleton$1.run(Framework.scala:831)

This is ScalaTest's sbt-integration Framework, which opens a ServerSocket and blocks in accept() for the forked JVM to dial back and report results. Since the forked JVM aborted before ever making that connection, accept() blocks forever — there is no timeout anywhere in this path. So "the test suite hangs forever" is real and reproducible, but it is a downstream consequence of an upstream JVM crash, not a network/TLS-handshake stall as we originally described it. (We did not find a deadlock, a stuck Future, or a CountDownLatch anywhere — there was simply no live thread left on the "other side" to make progress.)

Revised understanding of "what we could not reproduce." Our three standalone minimal repros (see below) never triggered this, which now makes sense: they were all short-lived, single-shot processes that likely never happened to have Conscrypt's native library lazily loaded from a background thread pool worker under whatever load-order/timing condition triggers the method-lookup failure — or they simply exited before hitting the code path that loads Conscrypt's JNI library at all (e.g., if TLS/gRPC transport setup is lazy and only triggered by our particular call sequence, which touches Spark/Hadoop initialization, Jinja templating, and our own config loading before the BigQuery call). We were not able to reduce it further, but we no longer think that weakens the report: we now have four independent, byte-for-byte-identical native crash signatures directly implicating google-cloud-bigquery 2.69.0's resolved Conscrypt native library, which is categorically stronger evidence than log lines alone.

Open questions for the google-cloud-bigquery team

• Did 2.69.0 change anything about gRPC channel/TLS provider selection (e.g. explicit Conscrypt provider registration, ALPN negotiation path, connection pooling/keepalive defaults) relative to 2.68.0? We could not find a changelog or source diff for 2.69.0 to check this ourselves (see the archived-repo note above).
• Is there a version skew between the Conscrypt Java-side classes that end up on the classpath under 2.69.0's dependency resolution (interfaces/callback method signatures) and the native conscrypt-openjdk-uber:2.6.0 binary's compiled expectations (the method IDs jniutil::init tries to resolve via GetMethodID)? Our bisect showed conscrypt-openjdk-uber itself resolves to the identical 2.6.0 in both the passing and failing configurations, so if this is the mechanism, the skew would have to be on the Java-class side — e.g. a relocated/shaded copy of a Conscrypt-adjacent interface bundled inside google-cloud-bigquery 2.69.0 or one of its other bumped transitives (google-auth-library-* 1.49.0→1.50.0, protobuf-java-util 4.33.2→4.33.6, google-cloud-bigquerystorage gapic 3.30.0→3.31.0) that shadows or conflicts with the real Conscrypt Java API at classload time.
• Should Conscrypt's native jniutil::init treat a GetMethodID miss for serverCertificateRequested as fatal (abort()/SIGABRT) rather than a recoverable initialization failure? A hard process abort on a background thread, triggered by ordinary classpath composition rather than corrupt/malicious input, seems more severe than the failure mode warrants — but this may be intentional/by-design in Conscrypt and out of google-cloud-bigquery's hands; we're not sure who owns this.
• Separately, and lower priority: sbt/ScalaTest's fork-test reporting channel (org.scalatest.tools.Framework$ScalaTestRunner$Skeleton$1$React.tryReact, blocked in ServerSocket.accept()) has no timeout for a forked JVM that dies before connecting back. That's arguably a robustness gap worth reporting to scalatest/sbt independently of this issue, since any forked-JVM crash (not just this one) would produce the same indefinite hang on the harness side.

Raw evidence

Full .ips crash reports (4 independent repros, byte-identical faulting-thread stacks) and the sbt-parent jstack dump captured during this investigation are available on request; the crash reports live under ~/Library/Logs/DiagnosticReports/java-2026-08-20-*.ips on the machine where this was reproduced (macOS 26.5.2, Apple Silicon, Temurin 21.0.11).

We're happy to run more diagnostics (thread dump of the hung process, -Djavax.net.debug=ssl:handshake, strace/dtruss on the forked JVM) if that would help narrow this down — we didn't want to over-invest before checking whether this was already known.