Skip to content

Register the SQL surface on every session Spark builds - #75

Merged
estebanzimanyi merged 1 commit into
MobilityDB:mainfrom
estebanzimanyi:feat/register-surface-every-session
Oct 10, 2026
Merged

estebanzimanyi merged 1 commit into
MobilityDB:mainfrom
estebanzimanyi:feat/register-surface-every-session

Conversation

@estebanzimanyi

Copy link
Copy Markdown
Member

MobilitySparkExtensions, named in spark.sql.extensions, registers MobilitySparkSql on every
session Spark builds: the one SparkSession.builder() answers, each newSession(), each Spark
Connect session and each Thrift Server session. It injects a check rule whose builder calls
MobilitySparkSql.registerAll on the SparkSession it is given, and Spark runs that builder once
per SparkSession when it builds its analyzer, before any function of it resolves.
The rule itself checks nothing. Apache Sedona 1.8.0's SedonaSqlExtensions registers its
functions the same way, a check rule whose builder calls SedonaContext.create(spark). A call to
MobilitySparkSql.registerAll on a session the extension serves answers alike. The README states
the extension as the way to hold the surface.

MobilitySparkExtensionsTest checks that a SparkSession built with only the extension named, and
one newSession() answers from it, resolve asText(tfloat(1.5, TIMESTAMP
'2026-03-01 00:00:00')) to 1.5@2026-03-01 00:00:00+00 and tfloat(...) to the type tfloat, and
that a call to registerAll on such a SparkSession answers alike. The floor of the suite is its 41
tests.

Witness: with the extension injecting only its optimizer rules, the two session tests fail on
"[UNRESOLVED_ROUTINE] Cannot resolve routine asText".

Measured over MobilityDB 1f41894b35, the catalog of MEOS-API ff5f838173 and the generator of
JMEOS 66d285b742: the 41 tests pass with no warning. Over Spark Connect, a server of the Spark
4.0.1 image with Sedona 1.8.0 and Delta 4.0.0 naming the three extensions answers the asText
query with 1.5@2026-03-01 00:00:00+00 on two client sessions, beside ST_AsText(ST_Point(1, 2)).

Why: Spark Connect and the Thrift Server hand every client a new session, which no caller of
registerAll reaches, so a MobilitySpark function resolves there only when the extension
registers it.

MobilitySparkExtensions, named in spark.sql.extensions, registers MobilitySparkSql on every
session Spark builds: the one SparkSession.builder() answers, each newSession(), each Spark
Connect session and each Thrift Server session. It injects a check rule whose builder calls
MobilitySparkSql.registerAll on the SparkSession it is given, and Spark runs that builder once
per SparkSession when it builds its analyzer, before any function of it resolves.
The rule itself checks nothing. Apache Sedona 1.8.0's SedonaSqlExtensions registers its
functions the same way, a check rule whose builder calls SedonaContext.create(spark). A call to
MobilitySparkSql.registerAll on a session the extension serves answers alike. The README states
the extension as the way to hold the surface.

MobilitySparkExtensionsTest checks that a SparkSession built with only the extension named, and
one newSession() answers from it, resolve asText(tfloat(1.5, TIMESTAMP
'2026-03-01 00:00:00')) to 1.5@2026-03-01 00:00:00+00 and tfloat(...) to the type tfloat, and
that a call to registerAll on such a SparkSession answers alike. The floor of the suite is its 41
tests.

Witness: with the extension injecting only its optimizer rules, the two session tests fail on
"[UNRESOLVED_ROUTINE] Cannot resolve routine `asText`".

Measured over MobilityDB 1f41894b35, the catalog of MEOS-API ff5f838173 and the generator of
JMEOS 66d285b742: the 41 tests pass with no warning. Over Spark Connect, a server of the Spark
4.0.1 image with Sedona 1.8.0 and Delta 4.0.0 naming the three extensions answers the asText
query with 1.5@2026-03-01 00:00:00+00 on two client sessions, beside ST_AsText(ST_Point(1, 2)).

Why: Spark Connect and the Thrift Server hand every client a new session, which no caller of
registerAll reaches, so a MobilitySpark function resolves there only when the extension
registers it.
@estebanzimanyi
estebanzimanyi merged commit cd3acc9 into MobilityDB:main Oct 10, 2026
2 checks passed
@estebanzimanyi
estebanzimanyi deleted the feat/register-surface-every-session branch October 10, 2026 14:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant