Repository navigation
Register the SQL surface on every session Spark builds - #75
Merged
estebanzimanyi merged 1 commit intoOct 10, 2026
Merged
estebanzimanyi merged 1 commit into
estebanzimanyi merged 1 commit into
Conversation
MobilitySparkExtensions, named in spark.sql.extensions, registers MobilitySparkSql on every session Spark builds: the one SparkSession.builder() answers, each newSession(), each Spark Connect session and each Thrift Server session. It injects a check rule whose builder calls MobilitySparkSql.registerAll on the SparkSession it is given, and Spark runs that builder once per SparkSession when it builds its analyzer, before any function of it resolves. The rule itself checks nothing. Apache Sedona 1.8.0's SedonaSqlExtensions registers its functions the same way, a check rule whose builder calls SedonaContext.create(spark). A call to MobilitySparkSql.registerAll on a session the extension serves answers alike. The README states the extension as the way to hold the surface. MobilitySparkExtensionsTest checks that a SparkSession built with only the extension named, and one newSession() answers from it, resolve asText(tfloat(1.5, TIMESTAMP '2026-03-01 00:00:00')) to 1.5@2026-03-01 00:00:00+00 and tfloat(...) to the type tfloat, and that a call to registerAll on such a SparkSession answers alike. The floor of the suite is its 41 tests. Witness: with the extension injecting only its optimizer rules, the two session tests fail on "[UNRESOLVED_ROUTINE] Cannot resolve routine `asText`". Measured over MobilityDB 1f41894b35, the catalog of MEOS-API ff5f838173 and the generator of JMEOS 66d285b742: the 41 tests pass with no warning. Over Spark Connect, a server of the Spark 4.0.1 image with Sedona 1.8.0 and Delta 4.0.0 naming the three extensions answers the asText query with 1.5@2026-03-01 00:00:00+00 on two client sessions, beside ST_AsText(ST_Point(1, 2)). Why: Spark Connect and the Thrift Server hand every client a new session, which no caller of registerAll reaches, so a MobilitySpark function resolves there only when the extension registers it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MobilitySparkExtensions, named in spark.sql.extensions, registers MobilitySparkSql on every
session Spark builds: the one SparkSession.builder() answers, each newSession(), each Spark
Connect session and each Thrift Server session. It injects a check rule whose builder calls
MobilitySparkSql.registerAll on the SparkSession it is given, and Spark runs that builder once
per SparkSession when it builds its analyzer, before any function of it resolves.
The rule itself checks nothing. Apache Sedona 1.8.0's SedonaSqlExtensions registers its
functions the same way, a check rule whose builder calls SedonaContext.create(spark). A call to
MobilitySparkSql.registerAll on a session the extension serves answers alike. The README states
the extension as the way to hold the surface.
MobilitySparkExtensionsTest checks that a SparkSession built with only the extension named, and
one newSession() answers from it, resolve asText(tfloat(1.5, TIMESTAMP
'2026-03-01 00:00:00')) to 1.5@2026-03-01 00:00:00+00 and tfloat(...) to the type tfloat, and
that a call to registerAll on such a SparkSession answers alike. The floor of the suite is its 41
tests.
Witness: with the extension injecting only its optimizer rules, the two session tests fail on
"[UNRESOLVED_ROUTINE] Cannot resolve routine
asText".Measured over MobilityDB 1f41894b35, the catalog of MEOS-API ff5f838173 and the generator of
JMEOS 66d285b742: the 41 tests pass with no warning. Over Spark Connect, a server of the Spark
4.0.1 image with Sedona 1.8.0 and Delta 4.0.0 naming the three extensions answers the asText
query with 1.5@2026-03-01 00:00:00+00 on two client sessions, beside ST_AsText(ST_Point(1, 2)).
Why: Spark Connect and the Thrift Server hand every client a new session, which no caller of
registerAll reaches, so a MobilitySpark function resolves there only when the extension
registers it.