Skip to content

Latest commit

 

History

History
668 lines (519 loc) · 16.4 KB

STORAGE_MODES.md

File metadata and controls

668 lines (519 loc) · 16.4 KB

Storage Modes Guide

Lightstack supports multiple storage backends for both Cinder (block storage) and Glance (image service), allowing flexible deployment options from development to production.

Paths below assume the default node-local root, /var/lib/lightstack. It is the state_path setting (Helm: realMode.hostStoragePath); every local path in this guide moves with it.

Overview

Cinder Storage Modes

  • stub: Simulated storage (no actual storage operations)
  • local: Local qcow2 files on libvirt host (/var/lib/lightstack/volumes/)
  • rbd: Ceph RBD volumes (shared storage cluster)
  • local,rbd: Hybrid mode (local + RBD for performance + redundancy)

Glance Storage Modes

  • stub: Simulated storage (no actual storage operations)
  • local: Local raw files (/var/lib/lightstack/images/)
  • rbd: Ceph RBD images (shared storage cluster)
  • s3: S3-compatible object storage (AWS S3, MinIO, Ceph RGW)
  • local,rbd: Hybrid mode (local cache + RBD backing)
  • local,s3: Hybrid mode (local cache + S3 backing)
  • rbd,s3: not supported — rejected at startup (Glance has no RBD backend; see below)

Ceph RGW (S3) is validated (SPEC-014). In s3 mode, image transfers stream to completion (no total-op timeout), HTTPS with an internal CA is supported, and a VM can boot from an RGW-backed image on any node — Lightstack lazily fetches + verifies the base on demand. See S3_CONFIGURATION.md for the backend recipe/quirks and OBJECT_STORAGE_S3.md for tenant-facing object storage.

Configuration

Cinder Configuration (config/lightstack.yaml)

cinder:
  port: 8776
  ceph_pool: volumes
  ceph_conf: /etc/ceph/ceph.conf
  storage_mode: local  # "stub", "local", "rbd", or "local,rbd"

Glance Configuration (config/lightstack.yaml)

glance:
  port: 9292
  ceph_pool: images
  ceph_conf: /etc/ceph/ceph.conf
  storage_mode: local,s3  # "stub", "local", "rbd", "s3", "local,rbd", "local,s3"
  s3_bucket: my-glance-images
  s3_region: us-east-1
  s3_endpoint: ""  # Optional: Custom S3 endpoint (for MinIO, Ceph RGW, etc.)

Storage Mode Details

1. Stub Mode

Use Case: Development, testing without storage Behavior: All operations are simulated in memory

storage_mode: stub

Characteristics:

  • No actual disk I/O
  • Data lost on restart
  • Zero dependencies
  • Instant operations

2. Local Mode

Use Case: Single-node deployments, development, testing Behavior: Stores files on local filesystem

storage_mode: local

Storage Locations:

  • Cinder volumes: /var/lib/lightstack/volumes/volume-{uuid}.qcow2
  • Glance images: /var/lib/lightstack/images/image-{uuid}.raw

Characteristics:

  • Fast (local disk I/O)
  • No shared storage required
  • Data persists across restarts
  • No multi-node support
  • Uses sparse files for efficiency

Example:

# After creating a volume, check the file:
ls -lh /var/lib/lightstack/volumes/
# -rw-r--r-- 1 user user 10G Jan 15 10:30 volume-abc-123.qcow2

3. RBD Mode (Ceph)

Use Case: Production multi-node clusters Behavior: Stores volumes/images in Ceph RBD

storage_mode: rbd
ceph_pool: volumes  # or images
ceph_conf: /etc/ceph/ceph.conf

Requirements:

  • Ceph cluster configured
  • ceph.conf and keyrings in place
  • Network connectivity to Ceph monitors

Characteristics:

  • Shared storage across nodes
  • High availability
  • Network latency (typically < 1ms in datacenter)
  • Supports live migration
  • Production-grade durability

Example:

# Check RBD images:
rbd ls volumes
# volume-abc-123
# volume-def-456

# Check image info:
rbd info volumes/volume-abc-123
# rbd image 'volume-abc-123':
#     size 10 GiB in 2560 objects
#     order 22 (4 MiB objects)

4. S3 Mode (Object Storage)

Use Case: Cloud deployments, cost-effective image storage Behavior: Stores images in S3-compatible object storage

storage_mode: s3
s3_bucket: my-glance-images
s3_region: us-east-1
s3_endpoint: ""  # Optional: for MinIO, Ceph RGW

S3 Configuration Options:

AWS S3

glance:
  storage_mode: s3
  s3_bucket: my-glance-images
  s3_region: us-west-2
  s3_endpoint: ""  # Use AWS default

Authentication: Uses AWS SDK default credential chain:

  1. Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
  2. Shared credentials file (~/.aws/credentials)
  3. IAM instance profile (on EC2)

MinIO (Self-Hosted)

glance:
  storage_mode: s3
  s3_bucket: glance-images
  s3_region: us-east-1  # MinIO requires region, use any value
  s3_endpoint: http://minio.example.com:9000

Environment Variables:

export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
export AWS_ENDPOINT_URL=http://localhost:9000

Ceph RGW (Object Gateway)

glance:
  storage_mode: s3
  s3_bucket: images
  s3_region: default
  s3_endpoint: http://rgw.ceph.local:7480

Environment Variables:

export AWS_ACCESS_KEY_ID=<rgw-access-key>
export AWS_SECRET_ACCESS_KEY=<rgw-secret-key>

Characteristics:

  • Cost-effective for large images
  • Scalable object storage
  • Slower than block storage (network + object overhead)
  • No live migration support (images only)
  • Ideal for image archives

Object Key Structure:

s3://my-bucket/images/image-{uuid}.raw

Example:

# List images in S3 bucket:
aws s3 ls s3://my-glance-images/images/
# 2024-01-15 10:30:00 1073741824 image-abc-123.raw
# 2024-01-15 11:45:00  536870912 image-def-456.raw

5. Hybrid Modes

local,rbd (Local Cache + RBD)

Use Case: Performance-critical workloads with shared storage requirement Behavior:

  • Upload: Write to local first, then replicate to RBD
  • Download: Read from local if available, fallback to RBD
  • Delete: Remove from both (best effort)
storage_mode: local,rbd
ceph_pool: volumes  # or images
ceph_conf: /etc/ceph/ceph.conf

Characteristics:

  • Local speed for cached data
  • RBD durability and sharing
  • Automatic failover on cache miss
  • Requires local disk space

Use Case Example:

Node 1: Creates volume → writes to local + RBD
Node 2: Attaches volume → reads from RBD (local cache miss)
Node 1: Re-attaches volume → reads from local (cache hit)

local,s3 (Local Cache + S3)

Use Case: Image service with local caching for frequently used images Behavior:

  • Upload: Write to local, asynchronously replicate to S3
  • Download: Serve from local if available, fallback to S3
  • Delete: Remove from both (best effort)
storage_mode: local,s3
s3_bucket: my-glance-images
s3_region: us-east-1

Characteristics:

  • Fast local access for hot images
  • S3 durability for cold storage
  • Automatic S3 fallback
  • Reduces S3 egress costs

Example:

# Popular image (Ubuntu 22.04) cached locally:
$ time openstack image download ubuntu-22.04 -f value -c size
# Real: 0.5s (local cache)

# Rare image (FreeBSD 13) fetched from S3:
$ time openstack image download freebsd-13 -f value -c size
# Real: 3.2s (S3 download)

rbd,s3 — NOT SUPPORTED (rejected at startup)

Glance's RBD backend is a placeholder in every build (unlike Cinder, there is no -tags ceph variant for images), so this mode could never write the S3 replica that its own download-fallback reads — an upload would report success with only one backend written. Lightstack therefore refuses to start with glance.storage_mode: rbd,s3, and the chart's values.schema.json rejects it at render time.

Use local,s3 instead (local primary + S3 replica), or plain s3.

  • Cross-region redundancy
  • Automatic failover

Performance Comparison

Latency (Single Image Download, 1GB)

Mode Latency Throughput Notes
stub < 1ms N/A In-memory only
local 50-200ms 5-10 GB/s Local NVMe SSD
rbd 100-500ms 1-5 GB/s 10GbE network, 3x replication
s3 1-5s 100-500 MB/s Internet or cross-region
local,rbd 50-200ms (hit) 5-10 GB/s Cache hit = local speed
local,s3 50-200ms (hit) 5-10 GB/s Cache hit = local speed

Cinder Volume Attachment

Mode Attachment Time Live Migration
stub Instant No
local < 100ms No
rbd 200-500ms Yes
local,rbd 200-500ms Yes

Note: RBD attachment slower due to RADOS connection setup

Storage Efficiency

Mode Sparse Files Deduplication Compression
stub N/A N/A N/A
local Yes (qcow2) No qcow2 native
rbd Yes (thin provisioning) Pool-level Pool-level
s3 No (raw objects) No Bucket-level

Deployment Scenarios

1. Development Workstation

Goal: Fast iteration, no infrastructure dependencies

cinder:
  storage_mode: local
glance:
  storage_mode: local

Characteristics:

  • Zero external dependencies
  • Fast local disk I/O
  • Data in /var/lib/lightstack/
  • Single-node only

2. Production Single-Node

Goal: Production-ready single server

cinder:
  storage_mode: local
glance:
  storage_mode: local,s3
  s3_bucket: prod-images
  s3_region: us-east-1

Characteristics:

  • Cinder: Local volumes (no live migration needed)
  • Glance: Local cache + S3 backup
  • S3 provides disaster recovery
  • Cost-effective

3. Production Multi-Node (Ceph)

Goal: High-availability cluster with shared storage

cinder:
  storage_mode: rbd
  ceph_pool: volumes
glance:
  storage_mode: rbd
  ceph_pool: images

Characteristics:

  • Live migration support
  • Shared storage across all nodes
  • High availability
  • Requires Ceph cluster

4. Hybrid Cloud (Ceph + S3)

Goal: On-premises Ceph with cloud backup

cinder:
  storage_mode: rbd
  ceph_pool: volumes
glance:
  storage_mode: local,s3   # rbd,s3 is unsupported for Glance (no RBD image backend)
  s3_bucket: cloud-backup-images
  s3_region: us-east-1

Characteristics:

  • Primary storage: Ceph RBD (fast, on-prem)
  • Secondary: AWS S3 (disaster recovery)
  • Automatic failover to S3
  • Cross-region redundancy

5. Pure Cloud (S3-Only)

Goal: Fully cloud-native deployment

cinder:
  storage_mode: local  # No S3 for block storage (volumes)
glance:
  storage_mode: s3
  s3_bucket: glance-images
  s3_region: us-west-2

Characteristics:

  • Glance: S3-backed (cost-effective)
  • Cinder: Local volumes (S3 unsuitable for block devices)
  • No on-premises storage infrastructure
  • Cloud-native

Troubleshooting

Ceph Connection Issues

Symptom: Ceph cluster not configured errors

Check:

  1. Verify ceph.conf exists and is readable:

    cat /etc/ceph/ceph.conf
  2. Test Ceph connectivity:

    ceph status
    rbd ls volumes
  3. Check network connectivity to monitors:

    ping <mon-host>
    telnet <mon-host> 6789

Solution:

  • Ensure ceph.conf has correct monitor addresses
  • Verify keyring exists: /etc/ceph/ceph.client.admin.keyring
  • Check firewall rules for Ceph traffic

S3 Connection Issues

Symptom: failed to upload to S3 or S3 client not initialized

Check:

  1. Verify AWS credentials:

    aws s3 ls s3://my-bucket/
  2. Test custom endpoint:

    aws --endpoint-url http://minio:9000 s3 ls
  3. Check bucket permissions:

    aws s3api get-bucket-acl --bucket my-bucket

Solution:

  • Set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
  • For custom endpoints, set s3_endpoint in config
  • Verify bucket exists and you have write permissions

Local Storage Permissions

Symptom: failed to create volume file or permission denied

Check:

ls -ld /var/lib/lightstack/volumes/
ls -ld /var/lib/lightstack/images/

Solution: Lightstack creates these on startup (SPEC-011 / cloud-storage#510); permission denied means the Lightstack process cannot write /var/lib/lightstack. In real mode Lightstack runs privileged/root and creates them automatically. For a non-root install, grant ownership of the base directory once:

sudo install -d -o "$USER" -g "$USER" -m 755 /var/lib/lightstack

Hybrid Mode Replication Failures

Symptom: uploaded to local but failed to replicate to S3

Behavior:

  • Upload succeeds to local storage
  • S3 replication fails (non-blocking)
  • Image is available locally
  • Warning logged

Solution:

  • Check S3 connectivity
  • Manual sync if needed:
    aws s3 sync /var/lib/lightstack/images/ s3://my-bucket/images/ --exclude "*" --include "image-*.raw"

Best Practices

1. Development

  • Use stub or local modes
  • No infrastructure required
  • Fast iteration

2. Staging

  • Use local mode with periodic S3 sync
  • Test S3 integration separately
  • Mirror production topology if possible

3. Production

  • Single-node: local for Cinder, local,s3 for Glance
  • Multi-node: rbd for both (requires Ceph)
  • Cloud: local + s3 for Glance

4. Disaster Recovery

  • Always use local,s3 (or s3) for Glance
  • Regularly test S3 failover
  • Monitor replication lag

5. Cost Optimization

  • Use local cache for frequently accessed images
  • S3 Standard for active images
  • S3 Glacier for archival images
  • Monitor S3 egress costs

Migration Between Modes

From stub to local

No migration needed - stub data is ephemeral.

From local to rbd

Cinder Volumes:

# For each volume:
qemu-img convert -f qcow2 -O raw \
  /var/lib/lightstack/volumes/volume-{uuid}.qcow2 \
  rbd:volumes/volume-{uuid}

Glance Images:

# For each image:
rbd import /var/lib/lightstack/images/image-{uuid}.raw \
  images/image-{uuid}

From local to s3

Glance Images:

# Bulk upload:
aws s3 sync /var/lib/lightstack/images/ s3://my-bucket/images/ \
  --exclude "*" --include "image-*.raw"

From rbd to s3

Glance Images:

# Export from RBD, upload to S3:
for img in $(rbd ls images); do
  rbd export images/$img - | \
    aws s3 cp - s3://my-bucket/images/$img.raw
done

Monitoring

Metrics to Track

  1. Storage Usage:

    • Local: du -sh /var/lib/lightstack/volumes /var/lib/lightstack/images
    • RBD: rbd du volumes / rbd du images
    • S3: AWS CloudWatch metrics
  2. Performance:

    • Upload time (image create + data upload)
    • Download time (image download)
    • Attachment time (volume attach)
  3. Availability:

    • Ceph cluster health
    • S3 API availability
    • Local disk space
  4. Costs (S3):

    • Storage costs (GB/month)
    • Request costs (GET/PUT)
    • Egress costs (data transfer out)

Alerting

  1. Local storage:

    • Alert if /var/lib/lightstack/ > 80% disk capacity
  2. Ceph:

    • Alert on Ceph cluster warnings
    • Monitor OSD failures
  3. S3:

    • Alert on S3 API errors
    • Monitor replication lag in hybrid modes
    • Track egress costs

Security Considerations

Local Mode

  • Files stored with user permissions (0644)
  • No encryption at rest by default
  • Use full-disk encryption (LUKS) if needed

RBD Mode

  • Ceph authentication via keyrings
  • In-transit encryption: Cephx protocol
  • At-rest encryption: Ceph encrypted OSDs

S3 Mode

  • In-transit encryption: HTTPS (TLS 1.2+)
  • At-rest encryption: S3 SSE (Server-Side Encryption)
  • Access control: IAM policies, bucket policies
  • Credential management: Never commit credentials to config

Example S3 Bucket Policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::123456789012:user/lightstack"},
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::my-glance-images/*"
    }
  ]
}

Summary

Mode Use Case Performance HA Cost
stub Development Instant No Free
local Single-node Fast No Low
rbd Multi-node Medium Yes Medium
s3 Cloud/archive Slow Yes Variable
local,rbd Performance + HA Fast (cache hit) Yes Medium
local,s3 Cache + cloud Fast (cache hit) Yes Low-Medium
rbd,s3 unsupported — rejected at startup — — —

Choose the mode that best fits your deployment requirements, considering performance, availability, and cost constraints.