A complete FinTech data platform designed to cover batch and streaming ETL, self service analytics, sensitive data and audit compliance and more for use cases such as lending, transactions and fraud detection.
Modern data platforms aren't built overnight and are not just about transforming data and loading it into a warehouse. Many factors must be taken into consideration: ensuring the data is accurate and modeled in a sustainable way, that the pipelines can be rerun to backfill historical data, the platform can scale to accomodate an order of magnitude of increased growth (i.e. 10x), and much more.
In this project I will be implementing a full fledged end to end data "fintech" (Financial Technology) data platform, supporting stakeholders such as Data Analysts, Data Scientists, Product Engineers and Compliance Analysts.
NOTE: All business scenarios and data in this project are synthetic and are solely for the purpose of experimenting and learning.
Similar to how real world data systems evolve over time, this project will be built gradually with each stage building on the work of the previous stages.
Because this project uses synthetic data, there will need to be tools built out to help generate the data required for this project. There's no data platforms without data!
The platform follows a medallion architecture (Bronze/Silver/Gold) on Delta Lake, processing data via both batch and streaming pipelines:
- Extensible configuration based data ingestion layer
- Staging data layer for raw, immutable data from source systems
- Intermediate data layer for cleansed, deduplicated data
- Business facing data layer with highly optimized tables for analysis and model training
- Self service tooling layer for dashboarding, feature backfills and more
- Observability stack encompassing metrics tracking, data lineage, oncall alerting, notifications and compliance logging
A more detailed design doc can be found here.
This platform supports two deployment modes — Local (Docker Compose, for development) and AWS (managed services, for production), with GCP planned. Files are organized under deployment/ by cloud provider.
NOTE: This setup is running into issues with the Airflow webserver stuck in a constant restart loop due to a conflict with the Airflow database migration. Issue tracked here.
Simplified view. See the detailed architecture diagram and the design doc for the full breakdown.
Requirements:
- 4 CPU minimum (8 CPU or more for more cushion)
- 16GB RAM minimum (24GB RAM or more for more cushion)
If your local machine does not meet these requirements, proceed to the AWS Deployment section below.
# Install brew and related packages
sh scripts/local/mac_quickstart.sh
# Start Colima (used for more memory efficient VM management)
colima start --memory 16 --cpu 4
# Create necessary directories
mkdir -p data logs
# Initialize Databases
sudo docker compose up airflow-init -d
sudo docker compose up metabase-init -d
# Start up rest of the stack
docker compose up -d
# Verify the stack
docker exec -it fintech-data-platform-airflow-scheduler-1 airflow version
docker exec -it fintech-data-platform-spark-worker-1 spark-submit --version
docker exec -it fintech-data-platform-spark-master-1 cat /opt/spark/conf/spark-env.sh
Simplified view. See the detailed architecture diagram for the full component breakdown.
Note: This deployment provisions managed AWS services (MSK, EMR Serverless, Aurora, S3) for a production-grade stack, orchestrated via Terraform.
- Terraform >= 1.5.0
- AWS CLI configured with appropriate credentials
- An S3 bucket for Terraform state (set up once)
All AWS deployment files live under deployment/aws/:
deployment/aws/
├── terraform/
│ ├── main.tf # Main infrastructure definition
│ ├── variables.tf # Input variables
│ ├── outputs.tf # Output values
│ ├── versions.tf # Provider and backend config
│ ├── terraform.tfvars.example
│ ├── user_data.sh # EC2 startup script
│ └── lambda/ # Lambda for cost optimization
│ ├── start_instances.py
│ └── stop_instances.py
├── iam/
│ ├── fintech-data-platform-terraform-policy-1.json
│ ├── fintech-data-platform-terraform-policy-2.json
│ └── s3-bucket-policy.txt
└── scripts/
└── aws_user_data_script.sh # Quick EC2 Docker Compose setup
# 1. Create IAM role/user for terraform deployment
# 2. Add IAM permissions to above role/user (see deployment/aws/iam/)
# 3. Configure AWS credentials
aws configure
# 4. (If using MFA) Get temporary credentials via STS
# Replace placeholders with your account number, MFA device name, and token
aws sts get-session-token \
--serial-number arn:aws:iam::{YOUR-AWS-ACCOUNT-NUMBER}:mfa/{YOUR-MFA-DEVICE-NAME} \
--profile {YOUR-AWS-PROFILE-NAME} \
--token-code {TOKEN-FROM-AUTHENTICATOR-APP} \
--duration-seconds 43200
# 5. Set the returned credentials as environment variables:
export AWS_ACCESS_KEY_ID=<AccessKeyId>
export AWS_SECRET_ACCESS_KEY=<SecretAccessKey>
export AWS_SESSION_TOKEN=<SessionToken>
# 6. Create S3 bucket for Terraform state (one-time), then paste the name in the below command
aws s3 mb s3://your-terraform-state-bucket --region us-east-1
# 8. Deploy infrastructure
cd deployment/aws/terraform
cp terraform.tfvars.example terraform.tfvars # Edit terraform.tfvars with your values
terraform init -backend-config="bucket=${YOUR-TERRAFORM-STATE-BUCKET-NAME-HERE}"
terraform plan
terraform apply -auto-approve
# 9. Get outputs
terraform outputFor a single-VM deployment running the Docker Compose stack on AWS, see deployment/aws/scripts/aws_user_data_script.sh.
Coming soon. A GCP deployment path is planned and will live under
deployment/gcp/. A startup script for a single Compute VM is already available atdeployment/gcp/scripts/gcp_startup_script.sh— a full Terraform-based deployment (using Dataproc, Pub/Sub, Cloud Storage, Cloud SQL) will follow.
Use scripts/local/generate_mock_data.py to generate synthetic fintech data:
# Generate data for 10,000 users (default)
pipenv run python scripts/generate_mock_data.py
# Generate data for 100,000 users
pipenv run python scripts/generate_mock_data.py --users 100000
# Generate streaming events (default: 1,000)
pipenv run python scripts/generate_mock_data.py --stream
# Generate 100,000 streaming events
pipenv run python scripts/generate_mock_data.py --stream --stream-count 100000The AWS deployment includes an EC2 instance specifically designed for mock data generation.
NOTE: The current user data script (at deployment/aws/terraform/user_data_mock.sh) has the EC2 instance shutting down after initial data generation; this will be updated in the future to run indefinitely.
Once this is implemented: to SSH onto the instance and check the status of the data generation, download the session manager plugin via these instructions.
And then run the following commands (can grab instance ID from the AWS console):
aws ssm start-session --target <instance-id> --region us-east-1
(If generating streaming data) Find the Cluster ARN from the MSK page in the AWS console, then run the following command to get the bootstrap brokers of your MSK cluster:
aws kafka get-bootstrap-brokers --cluster-arn {YOUR-CLUSTER-ARN}
Use scripts/local/generate_mock_data.py to generate synthetic data:
# Batch Data Only
pipenv run python scripts/aws/generate_aws_mock_data.py \
--s3-bucket {YOUR-S3-BUCKET-NAME} \
--s3-prefix bronze/ \
--aws-region us-east-1
# Batch and Streaming data (paste entire list of brokers returned from above command)
pipenv run python scripts/aws/generate_aws_mock_data.py \
--s3-bucket {YOUR-S3-BUCKET-NAME} \
--s3-prefix bronze/ \
--kafka-bootstrap {YOUR-ENTIRE-LIST-OF-KAFKA-BROKERS} \
--kafka-topic fintech.events \
--stream-count 5000 \
--aws-region us-east-1
Architecture diagrams are generated using the diagrams Python library:
# Local architecture
pipenv run python docs/diagrams/local/local_architecture_diagram.py
# AWS architecture
pipenv run python docs/diagrams/aws/aws_architecture.py| Capability | Status | Notes |
|---|---|---|
| Infrastructure | ||
| Local Docker Compose | 🟡 Partial | Spark, Airflow, Redpanda, MinIO, Metabase, Prometheus/Grafana; open issue to fix Airflow container needs resolving |
| AWS Terraform | ✅ Complete | VPC, MSK, EMR Serverless, Aurora, EC2 ASG, Lambda cost-optimization |
| GCP Deployment | 🟡 Planned | Full code (Terraform, startup script, etc) to follow once AWS architecture completely built out |
| Data Layers | ||
| Bronze (raw) | ✅ Complete | Mock data generators produce bronze Parquet/CSV |
| Silver (cleansed) | ❌ Not started | dbt transformation models needed |
| Gold (curated) | ❌ Not started | Business-facing fact/aggregate tables needed |
| SCD Type 2 | Data dictionary has schema; no pipeline yet | |
| Ingestion | ||
| Batch (files) | ✅ Complete | Local + AWS mock data generators |
| Streaming (Kafka) | ✅ Complete | Redpanda (local) / MSK (AWS) with IAM auth |
| Processing | ||
| Spark Batch | No pipeline jobs written yet | |
| Spark Streaming (RTM) | No streaming jobs written yet | |
| Orchestration | ✅ Complete | Airflow (local + AWS EC2) |
| Data Quality | ❌ Not started | Great Expectations + dbt tests planned |
| Observability | ||
| Prometheus / Grafana | 🟡 Partial | Deployed; scrape configs need fixing |
| CloudWatch Dashboard | ✅ Complete | AWS metrics dashboard |
| PagerDuty / Slack | ❌ Not configured | |
| OpenLineage | ❌ Not integrated | |
| Self-Service | ||
| Metabase | ✅ Complete | Deployed with PostgreSQL backend |
| Streamlit / Feature Backfills | ❌ Not started | |
| Compliance | ||
| Audit Tables | ❌ Not started | SOX audit logging needed |
| PII Access Logs | ❌ Not started | |
| 7-Year Retention | ✅ Complete | S3 lifecycle policy configured |
| Security | ||
| IAM / KMS | ✅ Complete | Least-privilege policies, encryption at rest/transit |
| Public Access Blocks | ✅ Complete | S3 bucket public access blocked |
| Restricted Admin CIDRs | ✅ Complete | Configurable via admin_cidr_blocks variable |
| Documentation | ✅ Complete | PRD, design doc, data dictionary, runbook, checklist |
| Document | Description |
|---|---|
| Design Doc | Detailed architecture, technology choices, tradeoffs |
| PRD | Product requirements and business context |
| Data Dictionary | Schema definitions, PII classifications, ownership |
| On-Call Runbook | Incident response procedures |
| Production Readiness Checklist | Go-live requirements |