Lightstack supports multiple storage backends for both Cinder (block storage) and Glance (image service), allowing flexible deployment options from development to production.
Paths below assume the default node-local root,
/var/lib/lightstack. It is thestate_pathsetting (Helm:realMode.hostStoragePath); every local path in this guide moves with it.
- stub: Simulated storage (no actual storage operations)
- local: Local qcow2 files on libvirt host (
/var/lib/lightstack/volumes/) - rbd: Ceph RBD volumes (shared storage cluster)
- local,rbd: Hybrid mode (local + RBD for performance + redundancy)
- stub: Simulated storage (no actual storage operations)
- local: Local raw files (
/var/lib/lightstack/images/) - rbd: Ceph RBD images (shared storage cluster)
- s3: S3-compatible object storage (AWS S3, MinIO, Ceph RGW)
- local,rbd: Hybrid mode (local cache + RBD backing)
- local,s3: Hybrid mode (local cache + S3 backing)
rbd,s3: not supported — rejected at startup (Glance has no RBD backend; see below)
Ceph RGW (S3) is validated (SPEC-014). In
s3mode, image transfers stream to completion (no total-op timeout), HTTPS with an internal CA is supported, and a VM can boot from an RGW-backed image on any node — Lightstack lazily fetches + verifies the base on demand. See S3_CONFIGURATION.md for the backend recipe/quirks and OBJECT_STORAGE_S3.md for tenant-facing object storage.
cinder:
port: 8776
ceph_pool: volumes
ceph_conf: /etc/ceph/ceph.conf
storage_mode: local # "stub", "local", "rbd", or "local,rbd"glance:
port: 9292
ceph_pool: images
ceph_conf: /etc/ceph/ceph.conf
storage_mode: local,s3 # "stub", "local", "rbd", "s3", "local,rbd", "local,s3"
s3_bucket: my-glance-images
s3_region: us-east-1
s3_endpoint: "" # Optional: Custom S3 endpoint (for MinIO, Ceph RGW, etc.)Use Case: Development, testing without storage Behavior: All operations are simulated in memory
storage_mode: stubCharacteristics:
- No actual disk I/O
- Data lost on restart
- Zero dependencies
- Instant operations
Use Case: Single-node deployments, development, testing Behavior: Stores files on local filesystem
storage_mode: localStorage Locations:
- Cinder volumes:
/var/lib/lightstack/volumes/volume-{uuid}.qcow2 - Glance images:
/var/lib/lightstack/images/image-{uuid}.raw
Characteristics:
- Fast (local disk I/O)
- No shared storage required
- Data persists across restarts
- No multi-node support
- Uses sparse files for efficiency
Example:
# After creating a volume, check the file:
ls -lh /var/lib/lightstack/volumes/
# -rw-r--r-- 1 user user 10G Jan 15 10:30 volume-abc-123.qcow2Use Case: Production multi-node clusters Behavior: Stores volumes/images in Ceph RBD
storage_mode: rbd
ceph_pool: volumes # or images
ceph_conf: /etc/ceph/ceph.confRequirements:
- Ceph cluster configured
ceph.confand keyrings in place- Network connectivity to Ceph monitors
Characteristics:
- Shared storage across nodes
- High availability
- Network latency (typically < 1ms in datacenter)
- Supports live migration
- Production-grade durability
Example:
# Check RBD images:
rbd ls volumes
# volume-abc-123
# volume-def-456
# Check image info:
rbd info volumes/volume-abc-123
# rbd image 'volume-abc-123':
# size 10 GiB in 2560 objects
# order 22 (4 MiB objects)Use Case: Cloud deployments, cost-effective image storage Behavior: Stores images in S3-compatible object storage
storage_mode: s3
s3_bucket: my-glance-images
s3_region: us-east-1
s3_endpoint: "" # Optional: for MinIO, Ceph RGWS3 Configuration Options:
glance:
storage_mode: s3
s3_bucket: my-glance-images
s3_region: us-west-2
s3_endpoint: "" # Use AWS defaultAuthentication: Uses AWS SDK default credential chain:
- Environment variables (
AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY) - Shared credentials file (
~/.aws/credentials) - IAM instance profile (on EC2)
glance:
storage_mode: s3
s3_bucket: glance-images
s3_region: us-east-1 # MinIO requires region, use any value
s3_endpoint: http://minio.example.com:9000Environment Variables:
export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
export AWS_ENDPOINT_URL=http://localhost:9000glance:
storage_mode: s3
s3_bucket: images
s3_region: default
s3_endpoint: http://rgw.ceph.local:7480Environment Variables:
export AWS_ACCESS_KEY_ID=<rgw-access-key>
export AWS_SECRET_ACCESS_KEY=<rgw-secret-key>Characteristics:
- Cost-effective for large images
- Scalable object storage
- Slower than block storage (network + object overhead)
- No live migration support (images only)
- Ideal for image archives
Object Key Structure:
s3://my-bucket/images/image-{uuid}.raw
Example:
# List images in S3 bucket:
aws s3 ls s3://my-glance-images/images/
# 2024-01-15 10:30:00 1073741824 image-abc-123.raw
# 2024-01-15 11:45:00 536870912 image-def-456.rawUse Case: Performance-critical workloads with shared storage requirement Behavior:
- Upload: Write to local first, then replicate to RBD
- Download: Read from local if available, fallback to RBD
- Delete: Remove from both (best effort)
storage_mode: local,rbd
ceph_pool: volumes # or images
ceph_conf: /etc/ceph/ceph.confCharacteristics:
- Local speed for cached data
- RBD durability and sharing
- Automatic failover on cache miss
- Requires local disk space
Use Case Example:
Node 1: Creates volume → writes to local + RBD
Node 2: Attaches volume → reads from RBD (local cache miss)
Node 1: Re-attaches volume → reads from local (cache hit)
Use Case: Image service with local caching for frequently used images Behavior:
- Upload: Write to local, asynchronously replicate to S3
- Download: Serve from local if available, fallback to S3
- Delete: Remove from both (best effort)
storage_mode: local,s3
s3_bucket: my-glance-images
s3_region: us-east-1Characteristics:
- Fast local access for hot images
- S3 durability for cold storage
- Automatic S3 fallback
- Reduces S3 egress costs
Example:
# Popular image (Ubuntu 22.04) cached locally:
$ time openstack image download ubuntu-22.04 -f value -c size
# Real: 0.5s (local cache)
# Rare image (FreeBSD 13) fetched from S3:
$ time openstack image download freebsd-13 -f value -c size
# Real: 3.2s (S3 download)Glance's RBD backend is a placeholder in every build (unlike Cinder, there is no
-tags ceph variant for images), so this mode could never write the S3 replica that
its own download-fallback reads — an upload would report success with only one backend
written. Lightstack therefore refuses to start with glance.storage_mode: rbd,s3, and the
chart's values.schema.json rejects it at render time.
Use local,s3 instead (local primary + S3 replica), or plain s3.
- Cross-region redundancy
- Automatic failover
| Mode | Latency | Throughput | Notes |
|---|---|---|---|
| stub | < 1ms | N/A | In-memory only |
| local | 50-200ms | 5-10 GB/s | Local NVMe SSD |
| rbd | 100-500ms | 1-5 GB/s | 10GbE network, 3x replication |
| s3 | 1-5s | 100-500 MB/s | Internet or cross-region |
| local,rbd | 50-200ms (hit) | 5-10 GB/s | Cache hit = local speed |
| local,s3 | 50-200ms (hit) | 5-10 GB/s | Cache hit = local speed |
| Mode | Attachment Time | Live Migration |
|---|---|---|
| stub | Instant | No |
| local | < 100ms | No |
| rbd | 200-500ms | Yes |
| local,rbd | 200-500ms | Yes |
Note: RBD attachment slower due to RADOS connection setup
| Mode | Sparse Files | Deduplication | Compression |
|---|---|---|---|
| stub | N/A | N/A | N/A |
| local | Yes (qcow2) | No | qcow2 native |
| rbd | Yes (thin provisioning) | Pool-level | Pool-level |
| s3 | No (raw objects) | No | Bucket-level |
Goal: Fast iteration, no infrastructure dependencies
cinder:
storage_mode: local
glance:
storage_mode: localCharacteristics:
- Zero external dependencies
- Fast local disk I/O
- Data in
/var/lib/lightstack/ - Single-node only
Goal: Production-ready single server
cinder:
storage_mode: local
glance:
storage_mode: local,s3
s3_bucket: prod-images
s3_region: us-east-1Characteristics:
- Cinder: Local volumes (no live migration needed)
- Glance: Local cache + S3 backup
- S3 provides disaster recovery
- Cost-effective
Goal: High-availability cluster with shared storage
cinder:
storage_mode: rbd
ceph_pool: volumes
glance:
storage_mode: rbd
ceph_pool: imagesCharacteristics:
- Live migration support
- Shared storage across all nodes
- High availability
- Requires Ceph cluster
Goal: On-premises Ceph with cloud backup
cinder:
storage_mode: rbd
ceph_pool: volumes
glance:
storage_mode: local,s3 # rbd,s3 is unsupported for Glance (no RBD image backend)
s3_bucket: cloud-backup-images
s3_region: us-east-1Characteristics:
- Primary storage: Ceph RBD (fast, on-prem)
- Secondary: AWS S3 (disaster recovery)
- Automatic failover to S3
- Cross-region redundancy
Goal: Fully cloud-native deployment
cinder:
storage_mode: local # No S3 for block storage (volumes)
glance:
storage_mode: s3
s3_bucket: glance-images
s3_region: us-west-2Characteristics:
- Glance: S3-backed (cost-effective)
- Cinder: Local volumes (S3 unsuitable for block devices)
- No on-premises storage infrastructure
- Cloud-native
Symptom: Ceph cluster not configured errors
Check:
-
Verify
ceph.confexists and is readable:cat /etc/ceph/ceph.conf
-
Test Ceph connectivity:
ceph status rbd ls volumes
-
Check network connectivity to monitors:
ping <mon-host> telnet <mon-host> 6789
Solution:
- Ensure
ceph.confhas correct monitor addresses - Verify keyring exists:
/etc/ceph/ceph.client.admin.keyring - Check firewall rules for Ceph traffic
Symptom: failed to upload to S3 or S3 client not initialized
Check:
-
Verify AWS credentials:
aws s3 ls s3://my-bucket/
-
Test custom endpoint:
aws --endpoint-url http://minio:9000 s3 ls
-
Check bucket permissions:
aws s3api get-bucket-acl --bucket my-bucket
Solution:
- Set
AWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEY - For custom endpoints, set
s3_endpointin config - Verify bucket exists and you have write permissions
Symptom: failed to create volume file or permission denied
Check:
ls -ld /var/lib/lightstack/volumes/
ls -ld /var/lib/lightstack/images/Solution: Lightstack creates these on startup (SPEC-011 / cloud-storage#510); permission denied
means the Lightstack process cannot write /var/lib/lightstack. In real mode Lightstack runs privileged/root and
creates them automatically. For a non-root install, grant ownership of the base directory once:
sudo install -d -o "$USER" -g "$USER" -m 755 /var/lib/lightstackSymptom: uploaded to local but failed to replicate to S3
Behavior:
- Upload succeeds to local storage
- S3 replication fails (non-blocking)
- Image is available locally
- Warning logged
Solution:
- Check S3 connectivity
- Manual sync if needed:
aws s3 sync /var/lib/lightstack/images/ s3://my-bucket/images/ --exclude "*" --include "image-*.raw"
- Use stub or local modes
- No infrastructure required
- Fast iteration
- Use local mode with periodic S3 sync
- Test S3 integration separately
- Mirror production topology if possible
- Single-node:
localfor Cinder,local,s3for Glance - Multi-node:
rbdfor both (requires Ceph) - Cloud:
local+s3for Glance
- Always use
local,s3(ors3) for Glance - Regularly test S3 failover
- Monitor replication lag
- Use
localcache for frequently accessed images - S3 Standard for active images
- S3 Glacier for archival images
- Monitor S3 egress costs
No migration needed - stub data is ephemeral.
Cinder Volumes:
# For each volume:
qemu-img convert -f qcow2 -O raw \
/var/lib/lightstack/volumes/volume-{uuid}.qcow2 \
rbd:volumes/volume-{uuid}Glance Images:
# For each image:
rbd import /var/lib/lightstack/images/image-{uuid}.raw \
images/image-{uuid}Glance Images:
# Bulk upload:
aws s3 sync /var/lib/lightstack/images/ s3://my-bucket/images/ \
--exclude "*" --include "image-*.raw"Glance Images:
# Export from RBD, upload to S3:
for img in $(rbd ls images); do
rbd export images/$img - | \
aws s3 cp - s3://my-bucket/images/$img.raw
done-
Storage Usage:
- Local:
du -sh /var/lib/lightstack/volumes /var/lib/lightstack/images - RBD:
rbd du volumes/rbd du images - S3: AWS CloudWatch metrics
- Local:
-
Performance:
- Upload time (image create + data upload)
- Download time (image download)
- Attachment time (volume attach)
-
Availability:
- Ceph cluster health
- S3 API availability
- Local disk space
-
Costs (S3):
- Storage costs (GB/month)
- Request costs (GET/PUT)
- Egress costs (data transfer out)
-
Local storage:
- Alert if
/var/lib/lightstack/> 80% disk capacity
- Alert if
-
Ceph:
- Alert on Ceph cluster warnings
- Monitor OSD failures
-
S3:
- Alert on S3 API errors
- Monitor replication lag in hybrid modes
- Track egress costs
- Files stored with user permissions (0644)
- No encryption at rest by default
- Use full-disk encryption (LUKS) if needed
- Ceph authentication via keyrings
- In-transit encryption: Cephx protocol
- At-rest encryption: Ceph encrypted OSDs
- In-transit encryption: HTTPS (TLS 1.2+)
- At-rest encryption: S3 SSE (Server-Side Encryption)
- Access control: IAM policies, bucket policies
- Credential management: Never commit credentials to config
Example S3 Bucket Policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {"AWS": "arn:aws:iam::123456789012:user/lightstack"},
"Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
"Resource": "arn:aws:s3:::my-glance-images/*"
}
]
}| Mode | Use Case | Performance | HA | Cost |
|---|---|---|---|---|
| stub | Development | Instant | No | Free |
| local | Single-node | Fast | No | Low |
| rbd | Multi-node | Medium | Yes | Medium |
| s3 | Cloud/archive | Slow | Yes | Variable |
| local,rbd | Performance + HA | Fast (cache hit) | Yes | Medium |
| local,s3 | Cache + cloud | Fast (cache hit) | Yes | Low-Medium |
| unsupported — rejected at startup | — | — | — |
Choose the mode that best fits your deployment requirements, considering performance, availability, and cost constraints.