🚀 GCP Data Engineering Ingestion Pod A lightweight, production-ready Infrastructure as Code (IaC) project that provisions a modern Data Engineering ingestion environment on Google Cloud.

This project uses Terraform to automatically deploy a complete working pod, providing data engineering teams with an isolated, secure workspace to write, test, and execute Python-based data pipelines using dlt (data load tool).

🏗️ Architecture Stack This Terraform configuration deploys the following Google Cloud resources:

☁️ Google Cloud Storage (GCS): A temporary Data Lake bucket for raw data landings.

📊 Google BigQuery: A dedicated Data Warehouse dataset (analytics_dev) for analytical queries.

🖥️ Google Compute Engine (VM): An Ubuntu worker node acting as the ingestion server.

⚙️ Cloud-Init: A bootstrap configuration that prepares the server with Python 3, pip, venv, and git.

📂 Project Structure Plaintext . ├── .envrc # Local environment variables (Git-ignored) ├── main.tf # Core GCP infrastructure configuration ├── variables.tf # Input definitions (Project ID, Env, Machine Size) ├── outputs.tf # Infrastructure outputs (VM IP, Dataset name) └── cloud-init.yml # Server bootstrap script for OS dependencies 📋 Prerequisites Before deploying, ensure you have the following on your local machine (or WSL2):

Terraform installed (v1.0+).

A Google Cloud Project with an active billing account.

The Compute Engine API enabled on your GCP project.

A GCP Service Account JSON key with appropriate permissions (stored securely at ~/.gcp/key.json).

A local RSA SSH Key (~/.ssh/id_rsa.pub) to authenticate with the deployed VM.

⚙️ Setup & Authentication Create a .envrc file in the root of the project to securely pass your credentials to Terraform without hardcoding them in the .tf files.

Bash

.envrc

export TF_VAR_ssh_pub_key="$HOME/.ssh/id_rsa.pub" export GOOGLE_APPLICATION_CREDENTIALS="$HOME/.gcp/your-service-account-key.json" export GOOGLE_PROJECT="your-gcp-project-id" Load the variables into your terminal session:

Bash source .envrc 🚀 Deployment Instructions Provision the infrastructure using the standard Terraform workflow:

Initialize the working directory (downloads the GCP provider):

Bash terraform init Review the execution plan:

Bash terraform plan -var="project_id=$GOOGLE_PROJECT" Deploy the infrastructure:

Bash terraform apply -var="project_id=$GOOGLE_PROJECT" Type yes when prompted. The output will display the public IP address of your new ingestion server.

🧪 Testing the Ingestion Pipeline Once deployed, you can access the VM and immediately start building pipelines.

1. SSH into the ingestion server:

Bash

Use the IP address provided by terraform output

ssh ubuntu@<INGESTION_SERVER_IP> 2. Set up the Python Virtual Environment: To prevent system dependency conflicts, create a venv for pipeline development:

Bash cd ~/pipelines python3 -m venv venv source venv/bin/activate 3. Install dlt and run a pipeline:

Bash pip install "dlt[gcp]" dlt init github bigquery (Configure your .dlt/secrets.toml with your GCP project details and execute the generated Python script to watch data load directly into your BigQuery dataset).

🧹 Cleanup To prevent ongoing cloud charges, permanently destroy all resources managed by this project when you are finished testing:

Bash terraform destroy -var="project_id=$GOOGLE_PROJECT"