Working with Folders

Overview

The folders functionality allows you to organize objects in a hierarchical directory structure within buckets. Folders can contain both files and nested subfolders, providing a familiar file system-like experience on both local storage and S3-compatible storage systems.

Folders use the same storage architecture as images and objects, including account-based access control, metadata management, and lifecycle configuration. See Storage Architecture for detailed information about the underlying storage mechanisms.

Key features:

  • Create root folders with ownership control via Luna-Account-Id

  • Root folders must be created explicitly; subfolders are created automatically when uploading objects

  • Store objects in hierarchical subfolder structures

  • Subfolders inherit account_id from the root folder (no separate metadata for subfolders)

  • Set lifecycle (TTL) policies at the folder level

  • Query and list objects with prefix filtering and depth control

  • Copy folders and objects between locations

See also

Storage Architecture - Storage architecture, account control, metadata, TTL, and permissions

Folder Naming Restrictions

When creating folders, the following rules apply:

Allowed Characters

Folder names must match the regular expression: [\w\s!@$%&*()\.,;:\-+\?]{1,1024}

  • Unicode letters (Latin, Cyrillic, Greek, and other scripts)

  • Digits (0-9)

  • Spaces

  • Special characters: ! @ $ % & * ( ) . , : ; - + ? _

  • Length: 1 to 1024 characters

Note

The regex uses \w which includes all Unicode letters and digits from any script (Latin, Cyrillic, Greek, Chinese, Arabic, etc.), plus underscore. This means folder names like Documents, Photos-2024, Отчеты, Reports_αβγ, or 文档 are all valid.

Full Path Length Restrictions

The complete object path includes folder name, all subfolders, and the object name:

  • Example: folder1/subfolder1/subfolder2/image.jpg

  • Maximum length: 1024 characters for the entire path

  • The service prefix luna_folder_ is added internally and counts toward storage system limits

System Limitations

The actual maximum length may be further restricted by the underlying storage system:

  • Linux systems: 255 bytes per individual folder/file name component

  • S3-compatible storage: Typically 1024 bytes for full object key

  • MinIO: 255 characters for full object path (stored on local disk, limited by filesystem)

  • Other S3-compatible systems: May have different limits

Warning

Folder Prefix Impact on Name Length (Both Local and S3)

The service adds the luna_folder_ prefix (12 characters) to the main folder name in both local and S3 storage.

For Local Storage:

  • If your filesystem limit is 255 bytes: Maximum folder name length = 243 characters (255 - 12)

  • The check uses os.pathconf(path, "PC_NAME_MAX") to determine the filesystem limit

  • Example: my-folder becomes luna_folder_my-folder on disk

  • The service also creates luna_meta_folder_{folderName} and {folderName}_luna_folder.json

  • All these names must fit within the filesystem’s name length limit

For S3 Storage:

  • AWS S3 limit is typically 1024 bytes for object keys: Maximum folder name length = 1012 characters (1024 - 12)

  • Example: Object key becomes luna_folder_my-folder/file.jpg (prefix included in key)

  • MinIO limitation: If storing on local disk, total path (including prefix) is limited to 255 characters

  • For MinIO: Maximum usable folder name = 243 characters (255 - 12 for prefix)

  • The full object key includes: luna_folder_{folderName}/{subfolders}/{objectName}

Warning

Administrator Responsibility

Administrators must independently verify the limitations of their specific storage system, as they may override or be stricter than the service’s defined limits. For example, MinIO limits the full object path to 255 characters because it stores objects on local disks.

Even if a name matches the regular expression, folder and object creation will fail if it exceeds the underlying storage system’s limitations.

Important

Implement Validation Before Requests

To avoid errors in production, implement validation in your application layer:

  • For both local and S3 storage: Account for the luna_folder_ prefix (12 chars) when checking folder names

  • Check the full path length: 'luna_folder_' + folder_name + '/' + subfolder_path + '/' + object_name

  • Ensure the total is within your storage system’s limits

  • Apply stricter limits if needed for your specific infrastructure

  • Local storage example: len('luna_folder_' + folder_name) <= 255 for Linux/ext4

  • S3 storage example: len('luna_folder_' + folder_name + '/' + full_path) <= 1024 for AWS S3

  • MinIO example: len('luna_folder_' + folder_name + '/' + full_path) <= 255 (disk-backed)

Folder Ownership and Permissions

Folders implement strict account-based access control. Unlike images and objects where account_id is optional, folders require an owner account.

See also

For general information about account-based access control, see Storage Architecture.

Folder-Specific Access Rules

  • When creating a folder, the Luna-Account-Id header must be provided

  • This account becomes the owner of the folder

  • All subfolders automatically inherit the same owner (no separate accounts for subfolders)

  • Only the owner account can:

    • Create objects in the folder

    • List objects in the folder

    • Access objects in the folder

    • Delete the folder and its contents

✓ Folder owner can: Create, Read, Update, Delete all objects and subfolders
✗ Other accounts will receive 404 errors even if the folder exists

API Usage

POST /v1/buckets/my-bucket/folders/my-folder HTTP/1.1
Luna-Account-Id: 12345678-1234-1234-1234-123456789abc
Content-Type: application/json

Warning

If you provide an account_id that doesn’t match the folder owner, you will receive a 404 Not Found error, even if the folder exists. The API will behave as if the folder does not exist for security reasons.

Folder Lifecycle Configuration

Folders support TTL (Time-To-Live) configuration that determines how long objects within the folder are retained. This leverages the same lifecycle mechanisms used for images and objects.

See also

For general information about TTL and lifecycle management, see Storage Architecture and Managing your storage lifecycle.

Setting Folder TTL

When creating a folder, you can specify the ttl parameter in the POST folder API operation (in days). If ttl is not specified, the parent bucket’s TTL configuration is inherited.

TTL Precedence for Folder Objects

  1. Object TTL - specified when creating individual objects (highest priority)

  2. Folder TTL - specified when creating the folder

  3. Bucket TTL - the default for the entire bucket (lowest priority)

To check folder’s lifecycle configuration, use GET folder API operation.

Internal Storage Representation

Folders have a unique storage structure compared to images and objects. Understanding this structure is crucial for migration and troubleshooting.

See also

For general storage architecture including images and objects, see Storage Architecture.

Folder Naming Convention

The folder name you specify in the API is not the physical name in storage. The service adds a prefix to distinguish root folders:

  • API name: my-folder

  • Physical main folder name: luna_folder_my-folder (prefix: 12 characters)

  • Subfolders: Use original names without prefix (e.g., subfolder1/, subfolder2/)

Important

Always use the original folder name (without the prefix) in API requests. Using luna_folder_my-folder will result in a 404 error.

Prefix applies ONLY to the main (root) folder, not to subfolders inside it.

Storage Implementation: Local File System

Directory Structure for Folders

On local storage, folders are implemented as physical directories. This is different from images/objects which use hash-based subdirectories (XXXX/).

See also

For images/objects storage structure, see Storage Architecture.

bucket/
├── XXXX/                               # Images/objects in hash-based dirs
│   └── {uuid}.jpg
├── luna_folder_my-folder/              # Main folder directory (WITH prefix)
│   ├── file1.jpg                       # Object in root folder
│   ├── subfolder1/                     # Nested subfolder (WITHOUT prefix!)
│   │   ├── file2.png
│   │   └── subfolder2/                 # Deeper nesting (WITHOUT prefix!)
│   │       └── file3.pdf
│   ├── subfolder1_meta/                # Metadata for subfolder1 (inside main folder)
│   │   ├── file2.png.meta.json        # Object metadata
│   │   └── subfolder2_meta/           # Metadata for deeper subfolder
│   │       └── file3.pdf.meta.json
├── luna_meta_folder_my-folder/         # Metadata for root folder (at bucket level, WITH prefix)
│   └── file1.jpg.meta.json            # Object metadata for files in root folder
└── my-folder_luna_folder.json          # Root folder ownership metadata ONLY

Key Folder-Specific Characteristics

  1. Physical directories: Each folder and subfolder is an actual directory on disk (unlike hash-based storage for images/objects)

  2. Prefix only for main folder: The luna_folder_ prefix is applied ONLY to the main folder name. Subfolders use their original names without any prefix.

  3. Automatic subfolder creation: Subfolders are created automatically when uploading objects with paths

  4. Parallel metadata structure:

    • Root folder: luna_folder_{name}/ → luna_meta_folder_{name}/ (at bucket level)

    • Subfolders: subfolder/ → subfolder_meta/ (inside the main folder)

  5. Object metadata files: Each object has a corresponding .meta.json file in the parallel _meta directory (see Storage Architecture for field descriptions)

  6. Root folder ownership metadata: Stored ONLY in {folderName}_luna_folder.json at bucket root:

    {
    "account_id": "12345678-1234-1234-1234-123456789abc",
    "ttl": 30
}

    Field Descriptions:

    • account_id (string, required) - UUID of the folder owner; inherited by all subfolders and objects

    • ttl (integer, optional) - Time-to-live in days for objects in this folder; overrides bucket TTL

Important

Subfolders do not have their own metadata files. Only the root folder has the ownership file. All subfolders inherit the account_id from the root folder.

Empty Folder Behavior

  • When all objects in a subfolder are deleted, the empty _meta directory is automatically removed during cleanup

  • The subfolder directory itself may remain empty on disk until explicitly deleted

  • Empty subfolders do not affect functionality but take minimal disk space

Storage Implementation: S3-Compatible Storage

Virtual Directory Structure for Folders

In S3, folders don’t physically exist - they are represented through object key prefixes. This is the same concept used for images/objects, but with the luna_folder_ prefix.

See also

For general S3 storage architecture, see Storage Architecture.

Bucket: my-bucket
├── {uuid}                              # Images/objects (flat namespace)
├── luna_folder_my-folder/              # Empty object representing root folder (WITH prefix)
├── luna_folder_my-folder/file1.jpg     # Object in root folder (prefix in key)
├── luna_folder_my-folder/subfolder1/file2.png     # Subfolder as part of key (NO separate prefix)
└── luna_folder_my-folder/subfolder1/subfolder2/file3.pdf  # Deeper nesting (NO separate prefix)

Note

The luna_folder_ prefix is applied to the main folder only. The full object key is: luna_folder_{folderName}/{subfolder_path}/{objectName}. This prefix consumes 12 characters from your storage system’s key length limit.

Key Folder-Specific Characteristics

  1. Virtual folders: Folders are simulated using forward slashes (/) in object keys

  2. Prefix only for main folder: Unlike images/objects which use direct UUIDs, folders use the luna_folder_ prefix

  3. Automatic “subfolder” creation: Subfolders don’t need to be created explicitly - they exist as part of object keys

  4. Metadata in headers: All metadata is stored in S3 object metadata headers (x-amz-meta-*)

  5. Root folder marker object: The root folder is represented aas an empty object with key luna_folder_{name}/

  6. No subfolder markers: Subfolders are purely virtual - they don’t exist as separate objects

Comparison: Local vs S3 Folder Storage

The following table shows folder-specific differences between local and S3 storage.

See also

For general storage comparison including images/objects, see Storage Architecture.

Feature

Local Storage (Folders)

S3 Storage (Folders)

Root Folder

Physical directory with prefix luna_folder_{name}/

Empty marker object with prefix luna_folder_{name}/

Subfolders

Physical directories (no prefix)

Virtual (part of object key) (no prefix)

Object Storage

In folder directory structure

Flat keys with folder prefix

Metadata Location

Parallel _meta directories: - luna_meta_folder_{name}/ - {subfolder}_meta/

In object metadata headers (x-amz-meta-*)

Folder Ownership

{name}_luna_folder.json file at bucket root

In marker object metadata

Empty Subfolders

Can exist as empty directories after object deletion

Cannot exist (no objects = no prefix in listings)

Empty Subfolder Cleanup

_meta dirs removed by cleanup; data dirs may remain

Automatically disappear from listings when no objects remain

Listing Performance

Filesystem directory scan

S3 ListObjectsV2 with prefixes

Working with Subfolders

Automatic Subfolder Creation

Subfolders are created automatically when you upload objects with paths. For example, uploading an object to photos/vacation/2024/beach.jpg automatically creates the hierarchy photos/vacation/2024/ if it doesn’t exist.

Warning

You cannot create empty subfolders directly. Subfolders only exist when they contain at least one object. This is especially true for S3 storage where folders are virtual.

Warning

Path Length Validation

Remember that the full path (folder_name/subfolder1/subfolder2/.../object_name) must not exceed your storage system’s limit. The service adds the luna_folder_ prefix internally, further reducing available length.

Accessing Subfolder Objects

When retrieving objects from subfolders using the GET folder object API operation, the account_id must match the owner of the root folder, not the subfolder.

Listing Objects with Depth Control

Use the GET folder objects API operation to list objects with different depth levels:

  • depth=0 (default): List all objects recursively (e.g., ["file1.jpg", "file2.png", "subfolder1/file3.jpg"])

  • depth=1: List only immediate children including subfolder markers (e.g., ["file1.jpg", "file2.png", "subfolder1/"])

Prefix Filtering

Combine prefix and depth parameters to explore specific parts of the folder hierarchy:

  • With prefix=docs/ and depth=0: Lists all objects under docs/ recursively

  • With prefix=docs/ and depth=1: Lists only immediate children of docs/

Deleting Folders and Subfolders

Root Folder Deletion

Use the DELETE folder API operation to delete a root folder. This removes it and all its contents recursively:

  • All objects in the folder

  • All subfolders and their contents

  • All metadata files (local storage) or objects (S3)

Subfolder and Object Deletion

Use the DELETE folder object API operation to delete specific objects or subtrees. The recursive parameter controls the behavior. When set to 1, all objects under that path are deleted.

Note

Without the recursive parameter, the deletion will fail if the path contains nested objects (local storage) or matches multiple objects (S3).

What Happens to Empty Subfolders?

  • Local Storage: When all objects in a subfolder are deleted, the parallel _meta directory is automatically removed during cleanup. The subfolder directory itself may remain empty on disk until explicitly deleted.

  • S3 Storage: Empty “subfolders” automatically disappear from object listings because they only exist as prefixes. When no objects have a particular prefix, that prefix is no longer listed.

Copying Folders and Objects

The service supports copying folders and individual objects within the same bucket.

When copying:

  • The destination folder must exist

  • If ttl parameter is provided, it will be applied to copied objects

  • If Luna-Account-Id header is provided, it must match the source folder owner

Migrating Existing Objects to Folders

If you have existing objects in buckets and want to migrate them to use the folder structure, follow these guidelines:

Migration Steps

  1. Create the target folder using POST folder with the desired ttl and Luna-Account-Id

  2. Upload objects to the folder using PUT folder object with the desired hierarchy path

  3. Update your application to use the new folder-based paths

  4. Delete old objects once migration is verified

For Local Storage

  • Old objects are stored in: bucket/XXXX/object-id.extension

  • New folder objects: bucket/luna_folder_{name}/path/to/file.extension

  • Metadata location changes from bucket/XXXX/object-id.meta.json to parallel _meta directories

For S3 Storage

  • Old objects: bucket/object-id

  • New folder objects: bucket/luna_folder_{name}/path/to/file

  • Metadata remains in object headers but may need account_id added

Warning

There is no automatic migration tool. You must manually copy or move objects to folders. Take care to preserve metadata including Content-Type, custom metadata, and account_id.

Verify path lengths during migration: Ensure that migrated object paths (including folder name, subfolders, and object name) do not exceed your storage system’s limitations.

Minimal S3 Permissions

Folders use the same S3 IAM permissions as images and objects, with additional requirements for lifecycle management if using TTL.

See also

For complete S3 permissions documentation, see Storage Architecture.

Summary of Required Permissions

  • s3:ListBucket - List objects in folders, check folder existence

  • s3:GetObject - Retrieve objects and folder metadata

  • s3:PutObject - Create folders and upload objects

  • s3:DeleteObject - Delete folders and objects

If using TTL with tagging (TAGGING=1):

  • s3:GetObjectTagging / s3:PutObjectTagging - Manage TTL tags

  • s3:GetBucketLifecycleConfiguration - Read existing lifecycle rules

  • s3:PutBucketLifecycleConfiguration - Create/update TTL-based lifecycle rules

Note

See Storage Architecture for complete IAM policy examples, bucket policies, and permission breakdowns.

Local Storage Permissions

When using local storage with folders, the service requires standard filesystem permissions.

See also

For complete local storage permissions documentation including Docker/container setup, see Storage Architecture.

Required for Folder Operations

  • Read (r): List directories and read files

  • Write (w): Create directories, create/modify files

  • Execute (x): Access directories, create subdirectories

The service needs to:

  • Create directories (luna_folder_*, subfolders, _meta directories)

  • Create and write files (objects, .meta.json files, *_luna_folder.json ownership files)

  • Read files and list directories

  • Delete files and directories

Note

See Storage Architecture for Docker/container setup examples,common permission commands, and SELinux configuration.

Best Practices

Folder-Specific Best Practices

  • Use meaningful, hierarchical folder names (e.g., year/month/day for time-based organization)

  • Keep folder and object names short to stay within storage system limits

  • Avoid excessive nesting (recommend max 5-7 levels) for better performance

  • Calculate full path length including the luna_folder_ prefix (12 chars)

  • For MinIO and similar systems: Keep total path under 255 characters

Folder Design

  • Create folders with appropriate TTL before uploading objects

  • Use folder-level TTL for groups of related objects with same retention policy

  • Set object-specific TTL only when individual objects need different retention

  • Always specify Luna-Account-Id when creating folders for proper access control

See also

For general best practices including account management, lifecycle management, and storage selection, see Storage Architecture.

Common Issues and Troubleshooting

404 Errors When Accessing Folders

  • Cause: account_id parameter doesn’t match the folder owner

  • Solution: Verify you’re using the correct account ID that was used to create the folder

Cannot Create Empty Subfolders

  • Cause: The API design prevents empty subfolder creation

  • Solution: Create subfolders by uploading an object with the desired path

Folder Name “Not Found” Despite Correct Name

  • Cause: Accidentally using the internal name (e.g., luna_folder_myname)

  • Solution: Use the original folder name (myname) without the prefix

Folder or Object Creation Fails with Path Length Errors

  • Cause: Full path exceeds storage system limitations (e.g., 255 characters in MinIO)

  • Solution:

    • Verify your storage system’s actual path length limit

    • Shorten folder names, subfolder names, or object names

    • Reduce nesting depth

    • Remember that luna_folder_ prefix (12 chars) is added to main folder names

    • Example: For MinIO with 255 char limit, usable path is ~243 characters (255 - 12 for prefix)

Objects Not Expiring as Expected

  • Cause (S3): Lifecycle configuration not properly set up

  • Solution: Ensure lifecycle migration has been run (see S3 bucket lifecycle setup)

  • Cause (Local): Cleanup process not running or encountering errors

  • Solution: Check service logs at 1:00 AM for cleanup execution

Permission Denied Errors

  • Solution: See Storage Architecture for complete permission setup guides (S3 IAM and local filesystem)

See Also