S3 bucket lifecycle setup

Note

Some of the rules do not apply to S3-compatible storage systems that do not natively support tagging. For further information, see the corresponding Section.

Lifecycle configuration for S3 is applied through tag-associated filters, see configuration elements for details. This assumes objects have a tag with a limited set of values, and buckets have a set of rules based on the value of that tag.

Warning

Migration is strictly required for the lifecycle configuration to be fully applied, for the following reasons:

• buckets with no rules will not delete objects even if the user specifies a lifetime for an particular object

• objects with no tags will never be deleted even if the user specifies a lifetime for the bucket

Migration routine

Migration could be run from base_scripts/migrate_settings_tags.

Lifecycle tags & rules

• to support TTL for buckets, vl-expire tag with value default should be added for all the existing objects.

• to support particular TTL for objects, a set or TTL-associated lifecycle rules should be added for existing buckets:

    {
        "ID": "vl-expire-<ttl>",
        "Expiration": {
            "Days": <ttl>,
        },
        "Filter": {"Tag": {"Key": "vl-expire", "Value": <ttl>}},
        "Status": "Enabled",
    }

A set of particular object TTL-associated tag values supported: 1, 2, 3, 4, 5, 6, 7, 14, 30, 60, 90, 180, 365.

Script execution pipeline

The script execution pipeline consists of two stages:

1. Bucket lifecycle configuration extended with a set of TTL-associated lifecycle rules.

2. Each object in the bucket will be assigned with the vl-expire tag if it doesn’t already possess one.

[!TIP] If you want to skip tags assignation to objects for some reason, you might set –update-tags=0 on script launch. Note that the default value of --update-tags equals to 1

The script fails on the first error.

Script launching

1. Set up actual storage configurations in the ./config.conf file. (Ordinary Luna Image Store config file format is supported.)

2. Activate the virtual environment for the Luna Image Store service.

3. Launch the script:

    python migrate_ttl_settings.py --bucket=visionlabs-samples --update-tags=1 --workers=10

[!NOTE] Note that the --workers parameter has the default value equals to 10 and is responsible for the number of parallel workers that are used for objects tags update.

4. Deactivate the virtual environment.

Permission issues

By default, all the S3 resources are private, including buckets, objects, and lifecycle configuration. If needed, rules and default tags could be created manually by resource owner using one of the applicable methods. Please refer to the S3 documentation for details.

See also

• managing lifecycle

• put lifecycle configuration

• managing object tags

• expiring objects

Further work

From now on, all the buckets manually created by resource owner should be supplemented with a number of TTL-associated tag rules:

{
    "ID": "vl-expire-<ttl>}",
    "Expiration": {"Days": <ttl>},
    "Filter": {"Tag": {"Key": "vl-expire", "Value": <ttl>}},
    "Status": "Enabled",
}

for ttl in (1, 2, 3, 4, 5, 6, 7, 14, 30, 60, 90, 180, 365)

as well as a default rule for the bucket (if necessary):

{
    "ID": "vl-expire",
    "Expiration": {"Days": <default-ttl>},
    "Filter": {"Tag": {"Key": "vl-expire", "Value": "default"}},
    "Status": "Enabled",
}

For the S3-compatible storages that do not support tagging natively

For S3-compatible storage systems that do not natively support tagging, an internal mechanism is provided to manage the lifecycle of objects.

When a bucket is created, a special file is placed inside it, containing the default TTL value for objects stored within. If no TTL value is provided during bucket creation, it is set to None, meaning objects will not have an automatically assigned expiration time.

To enable this mechanism, the setting TAGGING = 0 must be specified in the S3 section. When enabled, the object deletion date will be stored in the metadata (Metadata key) for objects where the ttl parameter was specified during creation or modification.

If an object does not have an explicitly set ttl, the default bucket TTL will be applied. However, if no default TTL is configured, the object will remain in storage indefinitely. To explicitly prevent an object from being deleted, it should be assigned a ttl value of -1.

The cleanup mechanism runs once a day to iteratively check all buckets and apply deletion rules where applicable.

Object lifecycle rules take precedence over bucket-level rules. Bucket-level rules are applied only when no specific TTL has been set for an individual object.