A lightweight Spark SQL task-submission and execution-enhancement framework with parameterized SQL, a typed Catalyst function catalog, processors, and extensible outputs.
- SQL execution — Multi-statement SQL, FreeMarker templates, typed parameters, and quote-aware statement splitting.
- Catalyst catalog — 30 public scalar, aggregate, and table functions for network, date/time, Roaring64, JSON, URL/domain, and vector workloads.
- Processor pipeline — Data masking, time dimensions, data quality, and custom processors. ML training, inference, and model lifecycle are provided by Tributo.
- Extensible output — Spark-native formats plus ServiceLoader-discovered ClickHouse and Doris integrations.
- Catalog integration — Spark catalog configuration for Iceberg, Lance, ClickHouse, and Doris; Iceberg writes use native Spark/Iceberg SQL.
- Materialization — Managed DataFrame materialization with explicit lifecycle and cleanup.
- Observability — Execution reports and Spark SQL, resource, and data-quality metrics.
Structured Streaming is experimental. Connector guarantees depend on the target, transport, table model, and verified deployment conditions; Pista does not claim cross-target atomicity or universal exactly-once delivery.
- JDK 17
- Scala 2.12.18
- Maven 3.8.8 or later
- Apache Spark 3.5.8
mvn clean compile
mvn test
mvn clean package -DskipTestspista/
├── pista-common/ # Errors and shared utilities
├── pista-catalyst/ # Catalyst functions and generated manifest
├── pista-sql/ # Spark SQL configuration, SPI, and materialization
├── pista-test-common/ # Shared test infrastructure
├── pista-batch/ # SparkSQLSubmitter
├── pista-streaming/ # Experimental Structured Streaming entry point
├── pista-connector/ # ClickHouse and Doris integrations
├── pista-metrics/ # Spark and data-quality metrics
├── pista-clickhouse-meta/ # Optional ClickHouse metadata
├── pista-doris-meta/ # Optional Doris metadata
├── pista-assembly/ # spark-submit assembly JAR
├── shell/ # Submission helpers
└── docs/ # Project documentation
| Document | Description |
|---|---|
| Documentation index | Module and reference documentation |
| SQL templates | SQL execution, templates, and parameters |
| Kyuubi batch submission | Experimental submission of the complete Pista runtime through Kyuubi Batch REST API |
| SQL functions | Public function catalog and contracts |
| Processors | Processor usage and extension points |
| Readers and writers | Data-source SPI and target integrations |
| Catalogs | Spark catalog integration |
| Materialization | DataFrame materialization lifecycle |
| Metrics | Metrics collection |
| Streaming | Experimental streaming guide |
| Configuration | Configuration reference |
| Errors | Error reference |
Pista is licensed under the Apache License 2.0. See NOTICE and THIRD_PARTY_NOTICES for attribution and bundled third-party information.