Storage Architecture¶

Overview¶

The luna-image-store service provides abstraction for storing images, objects, and folders on two types of storage backends:

Local Storage - Physical filesystem (hard disk)
S3-Compatible Storage - AWS S3, MinIO, or other S3-compatible systems

All stored items (images, objects, folders) share common storage mechanisms including:

Account-based access control
Metadata management
Lifecycle (TTL) configuration
Automatic cleanup processes

This document describes the common storage architecture used across all API endpoints (images, objects, folders).

Storage Types¶

Images¶

Images are automatically converted to JPG format and stored with thumbnails support:

Endpoints: /buckets/{bucketName}/images
ID Format: UUID v4 (generated or user-provided)
Conversion: All images converted to configured format (default: JPG)
Thumbnails: Automatically generated in different sizes
Content-Type: image/jpeg (or configured format)

Objects¶

Generic binary objects stored without modification:

Endpoints: /buckets/{bucketName}/objects
ID Format: UUID v4 (generated or user-provided)
Supported Types: plain/text, application/json, application/zip, application/pdf
No Conversion: Stored as-is with original Content-Type

Folders¶

Hierarchical folder structure for organizing objects:

Endpoints: /buckets/{bucketName}/folders/{folderName}
Structure: Root folders with nested subfolders
Objects: Stored in folder paths (e.g., folder/subfolder/file.ext)
Metadata: Inheritance from root folder to all subfolders

See Working with Folders for detailed folder-specific documentation.

Account-Based Access Control¶

Ownership Model

All stored items can be associated with an account ID for access control:

Header: Luna-Account-Id (UUID format)
Optional for images/objects: If not provided, item is accessible to all accounts
Required for folders: Root folders must have an owner account
Inheritance: Subfolders and contained objects inherit the owner from root folder

Access Rules

Images/Objects without account_id:
    ✓ Anyone can read
    ✓ Anyone can delete

Images/Objects with account_id:
    ✓ Owner account can read/delete
    ✗ Other accounts receive 404 (acts as if item doesn't exist)
    ✓ Requests without account_id parameter can read

Folders:
    ✓ Owner account can create, read, update, delete
    ✗ Other accounts receive 404 even if folder exists

API Usage

When creating items:

PUT /v1/buckets/my-bucket/images/{imageId} HTTP/1.1
Luna-Account-Id: 12345678-1234-1234-1234-123456789abc
Content-Type: image/jpeg

When retrieving items:

GET /v1/buckets/my-bucket/images/{imageId}?account_id=12345678-1234-1234-1234-123456789abc HTTP/1.1

Warning

Providing an account_id parameter that doesn’t match the item’s owner will result in 404 Not Found, even if the item exists. This is intentional for security and isolation between accounts.

Metadata Management¶

User-Defined Metadata

Custom metadata can be attached to all items using HTTP headers:

PUT /v1/buckets/my-bucket/images/{imageId} HTTP/1.1
Luna-Account-Id: 12345678-1234-1234-1234-123456789abc
Luna-Meta-camera: Canon EOS R5
Luna-Meta-location: Hawaii
Luna-Meta-photographer: John Doe
Content-Type: image/jpeg

Metadata Headers

Prefix: Luna-Meta-*
Format: Luna-Meta-{key}: {value}
Retrieval: Use with_meta=1 query parameter to include metadata in response
Storage:
- Local: Stored in .meta.json files
- S3: Stored in object metadata headers (x-amz-meta-*)

Retrieving Metadata

GET /v1/buckets/my-bucket/images/{imageId}?with_meta=1 HTTP/1.1

Response includes custom headers:

HTTP/1.1 200 OK
Luna-Meta-camera: Canon EOS R5
Luna-Meta-location: Hawaii
Content-Type: image/jpeg

Metadata Structure (Local Storage)

Each .meta.json file contains the following fields:

{
    "Content-Type": "image/jpeg",
    "account_id": "12345678-1234-1234-1234-123456789abc",
    "user_meta": {
        "camera": "Canon EOS R5",
        "location": "Hawaii",
        "photographer": "John Doe"
    },
    "expiry": "2026-04-15"
}

Field Descriptions:

Content-Type (string, required) - MIME type of the stored item (e.g., image/jpeg, application/json, text/plain)
account_id (string, optional) - UUID of the owner account; if present, only this account can access the item
user_meta (object, optional) - Custom metadata provided via Luna-Meta-* headers; stored as key-value pairs
expiry (string, optional) - Expiration date in YYYY-MM-DD format; item will be deleted by cleanup process after this date

Lifecycle Management (TTL)¶

Time-To-Live Configuration

Items can be configured with TTL (Time-To-Live) to automatically expire after a specified number of days:

Parameter: ttl (in days)
Scope: Can be set at bucket, folder, or individual item level
Cleanup: Automatic background process removes expired items

TTL Precedence

The service applies TTL in the following order (highest to lowest priority):

Individual Item TTL - Specified when creating/uploading the item
Folder TTL - Specified when creating a folder (applies to all items in folder)
Bucket TTL - Default for the entire bucket (set in bucket configuration)

Setting TTL

PUT /v1/buckets/my-bucket/images/{imageId}?ttl=30 HTTP/1.1
Content-Type: image/jpeg

This sets the item to expire 30 days after creation.

Internal Storage Structure¶

Local Storage Implementation¶

Directory Layout

{bucketsFolder}/
├── my-bucket/
│   ├── meta.json                      # Bucket configuration and TTL
│   ├── XXXX/                          # Image/object storage by hash
│   │   ├── {uuid}.jpg                 # Image file
│   │   ├── {uuid}.meta.json           # Image metadata
│   │   ├── {uuid}.txt                 # Object file
│   │   └── {uuid}.meta.json           # Object metadata
│   ├── luna_folder_{name}/            # Folder with prefix
│   │   ├── file.jpg                   # File in root folder
│   │   └── subfolder/                 # Subfolder (no prefix)
│   │       └── file2.png
│   ├── luna_meta_folder_{name}/       # Metadata for root folder objects
│   │   └── file.jpg.meta.json
│   └── {name}_luna_folder.json        # Folder ownership metadata

Bucket Metadata File (meta.json)

Each bucket has a meta.json file at its root containing:

{
    "ttl": 90
}

Field Descriptions:

ttl (integer, optional) - Default time-to-live in days for all items in the bucket; can be overridden at folder or item level

Key Characteristics

Images/Objects Storage:
- Stored in subdirectories named by first 4 hex digits of UUID
- Example: 12345678-... → stored in 0x1234/ directory
- Hash distribution ensures balanced directory sizes
Folder Storage:
- Root folders: Prefixed with luna_folder_ (12 chars)
- Subfolders: Use original names without prefix
- Parallel _meta directories for metadata
Metadata Files:
- Each item has corresponding .meta.json file
- Contains Content-Type, account_id, user_meta, expiry date
Automatic Cleanup:
- Runs daily at 1:00 AM (local time)
- Uses file modification time and expiry metadata
- Lock mechanism prevents concurrent cleanup from multiple instances

S3-Compatible Storage Implementation¶

Virtual Structure

In S3, everything is stored as objects with keys. “Directories” are simulated using key prefixes:

Bucket: my-bucket
├── {uuid}                                    # Image or object
├── luna_folder_{name}/                       # Folder marker object (empty)
├── luna_folder_{name}/file.jpg               # File in folder
└── luna_folder_{name}/subfolder/file2.png    # File in subfolder

Key Characteristics

Flat Namespace:
- All items are objects with unique keys
- Forward slashes (/) create virtual “folders”
- No physical directory structure
Metadata Storage:
- Standard metadata: Content-Type, custom S3 metadata headers
- User metadata: x-amz-meta-user_meta (JSON string)
- Account ID: x-amz-meta-account_id
Lifecycle/TTL Management:
- Option 1 (TAGGING=1): Uses S3 object tags (vl-expire: {days})
- Option 2 (TAGGING=0): Uses metadata headers or scheduled cleanup
- S3 natively manages expiration via lifecycle rules
Folder Prefixes:
- Root folders: luna_folder_{name}/ (12 chars prefix)
- Subfolders: Part of object key (e.g., luna_folder_photos/vacation/2024/)

S3 Object Metadata Example

Object Key: luna_folder_photos/vacation/beach.jpg

Metadata Headers:
    Content-Type: image/jpeg
    x-amz-meta-account_id: 12345678-1234-1234-1234-123456789abc
    x-amz-meta-user_meta: {"camera":"Canon","location":"Hawaii"}

Tags (if tagging enabled):
    vl-expire: 30

S3 Metadata Field Descriptions:

Content-Type (standard header) - MIME type of the object
x-amz-meta-account_id (custom metadata) - UUID of the owner account (optional)
x-amz-meta-user_meta (custom metadata) - JSON string containing user-defined metadata from Luna-Meta-* headers
vl-expire (tag, if TAGGING=1) - TTL in days; used by S3 lifecycle rules for automatic expiration

Comparison:Local vs S3 Storage¶

Feature	Local Storage	S3-Compatible Storage
Images/Objects	Stored in hash-based subdirs (`XXXX/{uuid}`)	Flat namespace with UUIDs as keys
Folders	Physical directories with `luna_folder_` prefix	Virtual (object key prefixes) with `luna_folder_` prefix
Metadata Storage	Separate `.meta.json` files	Object metadata headers
TTL Mechanism	Daily cleanup process (1 AM) using file modification time	S3 lifecycle rules or scheduled cleanup (no tagging)
Cleanup Behavior	Checks `expiry` in metadata or file modification time + bucket TTL	S3 manages deletion via lifecycle rules; may have delay
Performance	Direct filesystem access	Network latency, request costs
Scalability	Limited by disk size and I/O	Virtually unlimited
Path Length Limits	OS-dependent (typically 255 bytes per component, 4096 total)	Storage-dependent (S3: 1024 bytes, MinIO: 255 chars total)
Empty Items Cleanup	Removes empty `_meta` dirs; empty data dirs may remain	Objects disappear when deleted; virtual “folders” auto-disappear

Naming Conventions and Prefixes¶

Internal Prefixes

The service uses prefixes to separate different types of data:

Item Type	Prefix/Pattern	Example
Images (Local)	Hash-based: `XXXX/`	`bucket/1a2b/{uuid}.jpg`
Images (S3)	Direct UUID	`bucket/{uuid}`
Objects (Local)	Hash-based: `XXXX/`	`bucket/3c4d/{uuid}.pdf`
Objects (S3)	Direct UUID	`bucket/{uuid}`
Root Folders	`luna_folder_{name}`	`bucket/luna_folder_photos/`
Subfolders	No prefix (part of path)	`luna_folder_photos/vacation/`
**Folder Metadata (Local)	`luna_meta_folder_{name}`	`bucket/luna_meta_folder_photos/`
Subfolder Meta (Local)	`{subfolder}_meta`	`luna_folder_photos/vacation_meta`
Folder Ownership (Local only)	`{name}_luna_folder.json`	`bucket/photos_luna_folder.json`

Prefix Impact on Name Length

The luna_folder_ prefix (12 characters) is added to folder names in both local and S3 storage:

Local Storage: Reduces available name length from filesystem limit (typically 255 bytes → 243 usable)
S3 Storage: Reduces available key length from S3 limit (typically 1024 bytes → 1012 usable)
MinIO: If disk-backed, 255 chars total → 243 usable for folder name

Important

When validating names in your application, account for the luna_folder_ prefix (12 characters) that will be added internally.

Path Length Restrictions¶

Service-Level Limits

Folder Names: 1 to 1024 characters matching [\w\s!@$%&*()\.,;:\-+\?]+ (includes Unicode letters like Cyrillic)
Object Names in Folders: Same pattern as folder names plus / for paths
Full Path: Maximum 1024 characters for complete path (folder + subfolders + object name)
UUID: Images and objects use UUID v4 format (36 characters)

System-Level Limits

The actual maximum length is further restricted by the underlying storage system:

Storage System	Limitation
Linux/ext4	255 bytes per filename 4096 bytes total path
Windows/NTFS	255 characters per filename 32,767 characters total path
AWS S3	1024 bytes per object key
MinIO	255 characters total path (when disk-backed)

Warning

Administrators must verify the limitations of their specific storage system. Even if a name matches the service’s regex pattern, creation will fail if it exceeds the storage system’s limits.

Storage Permissions¶

S3 IAM Permissions¶

Required for All Operations with Images/Objects/Folders

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket-name",
                "arn:aws:s3:::your-bucket-name/*"
            ]
        }
    ]
}

Additional Permissions for Lifecycle/TTL (if tagging enabled)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObjectTagging",
                "s3:PutObjectTagging",
                "s3:GetBucketLifecycleConfiguration",
                "s3:PutBucketLifecycleConfiguration"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket-name",
                "arn:aws:s3:::your-bucket-name/*"
            ]
        }
    ]
}

Local Filesystem Permissions¶

Required Permissions

When using local storage (e.g., Docker volume mounts), the service requires:

Read (r): List directories and read files
Write (w): Create and modify files
Execute (x): Access directories and create subdirectories

Docker/Container Setup

# docker-compose.yml
services:
  luna-image-store:
    volumes:
      - /path/to/storage:/app/storage
    user: "1000:1000"  # Match host UID:GID

Host Setup

# Set ownership
sudo chown -R 1000:1000 /path/to/storage

# Set permissions
sudo chmod -R 755 /path/to/storage

Required Operations

The service needs to perform:

Create directories (buckets, hash dirs, folders, _meta directories)
Create and write files (images, objects, .meta.json files)
Read files and list directories
Delete files and directories
Modify file timestamps

Note

SELinux Users: Add :z or :Z suffix to Docker volume mounts:

volumes:
  - /path/to/storage:/app/storage:z

Best Practices¶

Account Management

Use consistent Luna-Account-Id values for each user or tenant
Store account ID mapping in your application database
Don’t expose internal account IDs to end users

Metadata Usage

Keep metadata values reasonable in size (avoid large JSON blobs)
Use meaningful key names in Luna-Meta-* headers
Store only essential metadata; use external database for complex data

Lifecycle Configuration

Set TTL at the most appropriate level (bucket, folder, or item)
Use folder TTL for groups of related items with same retention policy
Set item-specific TTL only when necessary to override defaults

Path Length Management

Validate full paths before making requests
Account for internal prefixes (luna_folder_ = 12 chars)
Apply stricter limits if using MinIO or systems with shorter limits
Avoid excessive nesting (recommend max 5-7 levels)

Storage Selection

Local Storage: Good for development, testing, or single-server deployments
S3 Storage: Recommended for production, distributed systems, and scalability
MinIO: Good compromise between S3 compatibility and local deployment

Luna Image Store v.3.25.1 documentation