📚 Complete Documentation: See INDEX.md for full documentation index with learning paths.
Lightstack is built on three core principles:
- API Compatibility First: 100% OpenStack API compatibility to work seamlessly with Horizon and OpenStack CLI/SDK
- Synchronous Operations: No async state machines - operations complete before API returns
- Fail-Fast Design: External dependency failures return immediately (< 1 second timeouts)
┌─────────────────────────────────────────────────────────────┐
│ lightstack Process │
├─────────────────────────────────────────────────────────────┤
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Keystone │ │ Nova │ │ Neutron │ │
│ │ :35357 │ │ :8774 │ │ :9696 │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Cinder │ │ Glance │ │
│ │ :8776 │ │ :9292 │ │
│ └──────────────┘ └──────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ Common Layer │ │
│ │ - Database Connection Pool │ │
│ │ - JWT Auth Service │ │
│ │ - Middleware (Auth, Logging, CORS) │ │
│ └──────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
↓
┌────────────────┼────────────────┐
↓ ↓ ↓
PostgreSQL libvirt (KVM) Ceph (RBD)
(State DB) (Compute) (Storage)
Responsibilities:
- User authentication (password-based)
- JWT token generation and validation
- Service catalog generation
- User/project/role management
Technology:
- JWT tokens (no token database)
- bcrypt password hashing
- PostgreSQL for user/project storage
Key Design Decisions:
- JWT over Fernet: Stateless tokens, no shared secret rotation needed
- No token blacklist: Tokens expire naturally based on TTL
- Middleware-based auth: All services share the same auth middleware
Responsibilities:
- VM lifecycle management (create, delete, reboot, stop)
- Flavor management
- Hypervisor abstraction
- Microversion negotiation
Technology:
github.com/digitalocean/go-libvirt- Pure Go libvirt bindings- Direct libvirt socket communication (no CGO)
- XML-based VM definitions
Key Design Decisions:
- Synchronous VM creation:
DomainDefineXML()+DomainCreate()in single API call - No async state machine: VM status transitions happen immediately
- Hypervisor mocking: Fake hypervisor stats for Horizon compatibility
Responsibilities:
- Virtual network management
- Subnet/CIDR allocation
- Port attachment to VMs
- Security groups (iptables-based)
- DHCP management (dnsmasq)
Technology:
github.com/vishvananda/netlink- Linux networkinggithub.com/vishvananda/netns- Network namespacesgithub.com/coreos/go-iptables- iptables rules
Key Design Decisions:
- Namespace isolation: Per-project network namespaces
- No VXLAN in v1: Single-node deployment uses bridges only
- iptables security groups: Simple, proven, no eBPF complexity in v1
- dnsmasq for DHCP: Standard, well-tested DHCP server
Responsibilities:
- Volume lifecycle management
- Volume attachment to VMs
- Snapshot management
- Volume types
Technology:
github.com/ceph/go-ceph- Ceph RBD integration- Direct RBD operations (create, delete, snapshot)
Key Design Decisions:
- Ceph-only: No local storage backend in v1
- 1-second timeout: Fast failure if Ceph unavailable
- Synchronous operations: Volume creation completes before API returns
Responsibilities:
- Image metadata management
- Image upload/download
- Image storage (Ceph RBD backed)
- Public/private image visibility
Technology:
github.com/ceph/go-ceph- Ceph RBD integration- Streaming uploads/downloads
Key Design Decisions:
- Ceph-backed storage: Images stored as RBD snapshots
- Metadata in PostgreSQL: Fast queries, RBD only for data
- Streaming transfers: No full image buffering in memory
1. User: openstack server create --flavor m1.small --image cirros test-vm
↓
2. Nova API: POST /v2.1/servers
↓
3. Validate token (JWT signature check)
↓
4. Query PostgreSQL: Fetch flavor, image metadata
↓
5. Query Neutron: Create ports, get network config
↓
6. Generate libvirt XML with RBD backing
↓
7. libvirt.DomainDefineXML() + DomainCreate()
↓
8. Insert instance into PostgreSQL
↓
9. Return HTTP 202 with instance details
1. User: curl -X POST /v3/auth/tokens (with credentials)
↓
2. Keystone: Validate username/password against PostgreSQL
↓
3. bcrypt.CompareHashAndPassword()
↓
4. If scoped: Fetch project and roles from PostgreSQL
↓
5. Generate JWT token (HS256 signature)
↓
6. Build service catalog dynamically
↓
7. Return token in X-Subject-Token header + body
Lightstack implements two-tier rate limiting as Gin middleware (see ADR-002):
REQUEST
|
v
Authenticated?
/ \
NO YES
| |
v v
IP-based Project-based
rate.Limiter rate.Limiter
(auth endpoints) (all other endpoints)
| |
v v
sync.Map sync.Map
[IP -> limiter] [project_id -> limiter]
| |
+-- evict stale entries every 10 min
+-- cap: 10,000 entries max
+-- overflow: allow traffic (degrade open)
Technology: golang.org/x/time/rate (Go standard library)
Key Design Decisions:
- Two-tier, not per-endpoint: IP-based for unauthenticated auth endpoints (
POST /v3/auth/tokens), project-based for all authenticated endpoints - In-process, not Redis: Single-binary architecture avoids external dependencies
- Degrade open: When entry cap (10K) is exceeded, new IPs/projects are allowed through rather than blocked
- Configurable: Enable/disable and adjust rates via
rate_limitingblock inlightstack.yaml
Services receive their database through constructors rather than importing a global variable (see ADR-001 and ADR-003):
main.go
|
+-- database.Open(ctx, url, opts) -> db database.DB (defer db.Close())
+-- database.MigrateUp(url, migrationsPath)
|
+-- health.DatabaseCheck(db)
+-- keystone.BootstrapAdmin(ctx, db, cfg.Keystone)
+-- keystone.NewAuthService(db, jwtSecret, ttl, cache)
+-- keystone.NewService(db, authService, cache)
+-- nova.NewService(db, libvirtURI, mode, cache)
+-- neutron.NewService(db, mode, cache)
+-- neutron.NewVXLANCoordinator(db, vxlanManager, nodeRegistry, ...)
+-- compute.NewNodeRegistry(db, nodeID, tunnelIP, heartbeat)
+-- cinder.NewService(db, mode, cephPool, cephConf, nodeName)
| +-- SweepInterruptedReservations(ctx) (before the listeners open)
+-- glance.NewService(db, mode, cephPool, cephConf, s3opts, cache)
+-- metadata.NewService(db, bindAddr)
Key Design Decisions:
- One contract, two backends:
database.DBis an interface over the method set the code calls on pgx —Query,QueryRow,Exec,Begin— with a PostgreSQL implementation (pgxpool, unchanged) and a SQLite implementation (modernc.org/sqlite, pure Go). The URL's scheme picks one at startup; services never ask which. - Same database, all services: DI is for testability and explicit dependencies, not service isolation
- WithTx(): Transaction helper takes the database:
WithTx(ctx, db, fn) - Helpers: Package-level functions that query take the database as a parameter; the cleanup job opens and closes one of its own
| PostgreSQL | SQLite | |
|---|---|---|
database.url |
postgres://… |
sqlite:///absolute/path/lightstack.db |
| Deployment | single or multi-node | single node (VXLAN refused) |
| Driver | pgx / pgxpool | modernc.org/sqlite through database/sql |
| Concurrency | row and advisory locks | one writer; transactions begin IMMEDIATE, 5 s busy wait |
| SQL | as written | translated on the way in (casts, = ANY, FOR UPDATE, …) |
Callers write PostgreSQL syntax once; internal/database's package documentation lists what both
backends run, and TestSQLConstructs fails the build on anything else. Ids, timestamps, JSON and
lists are text on SQLite, bound and scanned through the same Go types as on PostgreSQL.
Migrations: one migrations/ folder serves both. Each backend reads it through a view —
.postgres/.sqlite files are used by that backend only and replace the plain file of their
number, and SQLite starts at a baseline generated from PostgreSQL's schema at 073
(migrations/README.md). TestSchemaParity compares the two schemas.
No transaction around external work: volume and snapshot creates reserve their row
(creating, under the node's name), create the backing store outside any transaction, then
finalise. At startup each node sets the reservations it left behind to error; as upstream Cinder
does, it leaves their backing for the delete of the errored volume or snapshot. On SQLite, where
one transaction holds the whole file, this is what keeps an image copy from stopping every
project's writes.
Full request path through Lightstack middleware stack:
Client
|
v
Gin Router (port-specific: 35357, 8774, 9696, etc.)
|
v
CORS Middleware (configurable origins)
|
v
Rate Limit Middleware (IP or project-based)
|
v
Auth Middleware (JWT validation, extracts user/project/roles)
| (skipped for: metadata service, version endpoints)
v
Handler (service method on struct with injected pool)
|
v
svc.pool.Query(ctx, ...) -> PostgreSQL or SQLite
|
v
Response (structured error format via common.SendError on failure)
- users: User credentials and metadata
- projects: Project (tenant) metadata
- roles: Role definitions (admin, member, reader)
- role_assignments: User-project-role mappings
- instances: VM instances
- flavors: VM flavor definitions
- networks: Virtual networks
- subnets: IP subnets
- ports: Network ports (attached to VMs)
- volumes: Cinder volumes
- images: Glance image metadata
Key indexes for performance:
instances(project_id)- Fast per-project filteringports(device_id)- Fast port lookup by VMrole_assignments(user_id, project_id)- Fast role checks
- Password hashing: bcrypt (cost factor 10)
- Token signing: HS256 (HMAC-SHA256)
- Token TTL: 24 hours default (configurable)
- Middleware-based: All protected endpoints check X-Auth-Token
- Role-based access: RequireRole() middleware for privileged ops
- Project scoping: Automatic filtering by project_id from token
- Namespace isolation: Projects cannot see each other's networks
- Security groups: Per-VM iptables rules
- Default deny: All traffic blocked unless explicitly allowed
- API latency: < 10ms for read operations
- VM creation: < 2 seconds (depends on libvirt)
- Volume creation: < 1 second (fail-fast if Ceph down)
- Token validation: < 1ms (JWT signature check)
- PostgreSQL: 20 connections default (configurable)
- libvirt: Connection pooling with max 10 connections
- Ceph: Connection per operation (stateless)
- Goroutines per request: Each HTTP request runs in separate goroutine
- Database queries: All queries use context for cancellation
- Netlink operations: Mutex per namespace to prevent races
┌────────────────────────────────┐
│ lightstack Host │
│ │
│ ┌──────────────────────────┐ │
│ │ lightstack Binary │ │
│ │ (All services) │ │
│ └──────────────────────────┘ │
│ │
│ ┌──────────────────────────┐ │
│ │ PostgreSQL │ │
│ └──────────────────────────┘ │
│ │
│ ┌──────────────────────────┐ │
│ │ libvirt + KVM │ │
│ └──────────────────────────┘ │
└────────────────────────────────┘
↓
Ceph Cluster
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Control Node │ │ Compute Node 1 │ │ Compute Node 2 │
│ │ │ │ │ │
│ - Keystone │ │ - Nova Agent │ │ - Nova Agent │
│ - Nova API │ │ - libvirt │ │ - libvirt │
│ - Neutron API │ │ - Neutron Agnt │ │ - Neutron Agnt │
│ - Cinder API │ │ - VXLAN │ │ - VXLAN │
│ - Glance API │ │ │ │ │
│ - PostgreSQL │ │ │ │ │
└─────────────────┘ └─────────────────┘ └─────────────────┘
↓ ↓ ↓
Ceph Cluster
- API request latency (p50, p95, p99)
- Database connection pool stats
- libvirt operation latency
- Ceph operation latency
- Active VMs/volumes/networks count
{
"timestamp": "2024-03-06T12:34:56Z",
"level": "info",
"service": "nova",
"method": "POST",
"path": "/v2.1/servers",
"status": 202,
"duration_ms": 1842,
"user_id": "abc123",
"project_id": "def456"
}- Distributed traces across services
- Span per database query
- Span per libvirt/Ceph operation
- Impact: All API operations fail
- Response: HTTP 503 Service Unavailable
- Mitigation: Database replication in production
- Impact: Volume/image operations fail
- Response: HTTP 503 within 1 second (fast fail)
- Mitigation: Ceph cluster redundancy
- Impact: VM operations fail
- Response: HTTP 503
- Mitigation: Multi-node deployment with live migration
- VXLAN overlay networks
- Floating IPs (external network access)
- Live migration
- Placement API (resource scheduling)
- eBPF security groups (kernel-space filtering)
- High availability (multi-master)
- Heat (orchestration)
- Swift (object storage)
- Ceilometer (telemetry)
- Multi-region support