S3 bucket lifecycle setup

Lifecycle configuration for S3 is applied through tag-associated filters, see configuration elements for details. This assumes objects have a tag with a limited set of values, and buckets have a set of rules based on the value of that tag.

Warning

Migration is strictly required for the lifecycle configuration to be fully applied, for the following reasons:

• buckets with no rules will not delete objects even if the user specifies a lifetime for an particular object

• objects with no tags will never be deleted even if the user specifies a lifetime for the bucket

Migration routine

Migration could be run from base_scripts/migrate_settings_tags.

Lifecycle tags & rules

• to support TTL for buckets, vl-expire tag with value default should be added for all the existing objects.

• to support particular TTL for objects, a set or TTL-associated lifecycle rules should be added for existing buckets:

    {
        "ID": "vl-expire-<ttl>",
        "Expiration": {
            "Days": <ttl>,
        },
        "Filter": {"Tag": {"Key": "vl-expire", "Value": <ttl>}},
        "Status": "Enabled",
    }

A set of particular object TTL-associated tag values supported: 1, 2, 3, 4, 5, 6, 7, 14, 30, 60, 90, 180, 365.

Script execution pipeline

The script execution pipeline consists of two stages:

1. Bucket lifecycle configuration extended with a set of TTL-associated lifecycle rules.

2. Each object in the bucket will be assigned with the vl-expire tag if it doesn’t already possess one.

[!TIP] If you want to skip tags assignation to objects for some reason, you might set –update-tags=0 on script launch. Note that the default value of --update-tags equals to 1

The script fails on the first error.

Script launching

1. Set up actual storage configurations in the ./config.conf file. (Ordinary Luna Image Store config file format is supported.)

2. Activate the virtual environment for the Luna Image Store service.

3. Launch the script:

    python migrate_ttl_settings.py --bucket=visionlabs-samples --update-tags=1 --workers=10

[!NOTE] Note that the --workers parameter has the default value equals to 10 and is responsible for the number of parallel workers that are used for objects tags update.

4. Deactivate the virtual environment.

Permission issues

By default, all the S3 resources are private, including buckets, objects, and lifecycle configuration. If needed, rules and default tags could be created manually by resource owner using one of the applicable methods. Please refer to the S3 documentation for details.

See also

• managing lifecycle

• put lifecycle configuration

• managing object tags

• expiring objects

Further work

From now on, all the buckets manually created by resource owner should be supplemented with a number of TTL-associated tag rules:

{
    "ID": "vl-expire-<ttl>}",
    "Expiration": {"Days": <ttl>},
    "Filter": {"Tag": {"Key": "vl-expire", "Value": <ttl>}},
    "Status": "Enabled",
}

for ttl in (1, 2, 3, 4, 5, 6, 7, 14, 30, 60, 90, 180, 365)

as well as a default rule for the bucket (if necessary):

{
    "ID": "vl-expire",
    "Expiration": {"Days": <default-ttl>},
    "Filter": {"Tag": {"Key": "vl-expire", "Value": "default"}},
    "Status": "Enabled",
}