Tasks lifecycle

Terminology

In the description below, the tasks service is labeled as TS and the tasks worker service as TWS

Redis objects are labeled the following way:

  1. SUBTASKS_QUEUE - Redis sorted set associated with one task type

  2. SUBTASK_SEPARATION_KEY - Redis key associated with one task

  3. SUBTASK_START_LIST - shared Redis list for records about subtasks starting

  4. SUBTASK_RESULT_LIST - shared Redis list for records about subtasks results

  5. RESULT_AGGREGATOR_QUEUE - Redis sorted set for result aggregator tasks for one task type

  6. TASKS_RESULTS_LIST - shared Redis list for tasks results

  7. CANCELLER_STREAM - shared Redis stream for task cancellation records

  8. SUBTASK_CANCEL_LIST - shared Redis list for task cancellation records

Task creation

  1. TS receives the task creation request

  2. TS validates the task content from the request

  3. TS adds task data to SUBTASKS_QUEUE and start waiting for task separation result to appear in SUBTASK_SEPARATION_KEY

  4. TWS waits for task data to appear in SUBTASKS_QUEUE

  5. TWS splits task content into subtasks

  6. TWS sets SUBTASK_SEPARATION_KEY to string value with subtasks data

  7. TS reads SUBTASK_SEPARATION_KEY value with subtasks data from Redis stores each subtask in the database

  8. TS adds split subtask data to SUBTASKS_QUEUE

  9. TS sends data about the task to the database

  10. TS returns task_id to user

Subtasks processing

  1. TWS reads split subtask data from SUBTASKS_QUEUE

  2. TWS starts processing the subtask and adds a record to SUBTASK_START_LIST

  3. TS waits for record about subtask processing to appear in SUBTASK_START_LIST. Any TS instance can do it

  4. TS updates information about task in the database

  5. If the Image Store is enabled:

    1. TWS saves the report about subtask completion to the Image Store service

  6. TWS sends adds a record to SUBTASK_RESULT_LIST that the subtask has been processed or error occurred

  7. TS waits for a record to appear SUBTASK_RESULT_LIST

  8. TS updates information about the task in the database

  9. TS sets the task status to collect results in the database after processing the last subtask

Merging results

  1. After processing the last subtask TS creates an internal task for merging results and adds this task record to RESULT_AGGREGATOR_QUEUE

  2. TWS receives the created record from RESULT_AGGREGATOR_QUEUE

  3. TWS merges the results of all subtasks

  4. TWS adds the merged result id to TASKS_RESULTS_LIST

  5. TS updates the task status in the database

  6. If the Image Store is enabled:

    1. TWS saves the merged result to the Image Store service

    2. TWS saves result id for the task in the database

Task cancelling

  1. TS receives the task cancellation request

  2. TS updates the status of targeted task and its unfinished subtasks in the database

  3. TS removes queued subtasks data from SUBTASKS_QUEUE

  4. If the result aggregator task is running:

    1. TS removes result aggregator from RESULT_AGGREGATOR_QUEUE

  5. If the result aggregator task isn’t running:

    1. TS writes cancellation records to CANCELLER_STREAM for running subtasks

  6. TWS reads cancellation records from CANCELLER_STREAM for each subtask and cancels its execution

Service shutdown of Tasks Worker

  1. TWS stops running subtasks

  2. TWS adds cancellation records to SUBTASK_CANCEL_LIST

  3. Service shuts down

Service shutdown of Tasks Service

  1. Service shuts down

See the general task processing sequence diagram and sequence diagrams for each task type in the ‘Tasks diagrams’ section of the administrator manual.