Tasks lifecycle¶
Terminology¶
In the description below, the tasks service is labeled as TS and the tasks worker service as TWS
Redis objects are labeled the following way:
SUBTASKS_QUEUE - Redis sorted set associated with one task type
SUBTASK_SEPARATION_KEY - Redis key associated with one task
SUBTASK_START_LIST - shared Redis list for records about subtasks starting
SUBTASK_RESULT_LIST - shared Redis list for records about subtasks results
RESULT_AGGREGATOR_QUEUE - Redis sorted set for result aggregator tasks for one task type
TASKS_RESULTS_LIST - shared Redis list for tasks results
CANCELLER_STREAM - shared Redis stream for task cancellation records
SUBTASK_CANCEL_LIST - shared Redis list for task cancellation records
Task creation¶
TS receives the task creation request
TS validates the task content from the request
TS adds task data to SUBTASKS_QUEUE and start waiting for task separation result to appear in SUBTASK_SEPARATION_KEY
TWS waits for task data to appear in SUBTASKS_QUEUE
TWS splits task content into subtasks
TWS sets SUBTASK_SEPARATION_KEY to string value with subtasks data
TS reads SUBTASK_SEPARATION_KEY value with subtasks data from Redis stores each subtask in the database
TS adds split subtask data to SUBTASKS_QUEUE
TS sends data about the task to the database
TS returns task_id to user
Subtasks processing¶
TWS reads split subtask data from SUBTASKS_QUEUE
TWS starts processing the subtask and adds a record to SUBTASK_START_LIST
TS waits for record about subtask processing to appear in SUBTASK_START_LIST. Any TS instance can do it
TS updates information about task in the database
If the Image Store is enabled:
TWS saves the report about subtask completion to the Image Store service
TWS sends adds a record to SUBTASK_RESULT_LIST that the subtask has been processed or error occurred
TS waits for a record to appear SUBTASK_RESULT_LIST
TS updates information about the task in the database
TS sets the task status to collect results in the database after processing the last subtask
Merging results¶
After processing the last subtask TS creates an internal task for merging results and adds this task record to RESULT_AGGREGATOR_QUEUE
TWS receives the created record from RESULT_AGGREGATOR_QUEUE
TWS merges the results of all subtasks
TWS adds the merged result id to TASKS_RESULTS_LIST
TS updates the task status in the database
If the Image Store is enabled:
TWS saves the merged result to the Image Store service
TWS saves result id for the task in the database
Task cancelling¶
TS receives the task cancellation request
TS updates the status of targeted task and its unfinished subtasks in the database
TS removes queued subtasks data from SUBTASKS_QUEUE
If the result aggregator task is running:
TS removes result aggregator from RESULT_AGGREGATOR_QUEUE
If the result aggregator task isn’t running:
TS writes cancellation records to CANCELLER_STREAM for running subtasks
TWS reads cancellation records from CANCELLER_STREAM for each subtask and cancels its execution
Service shutdown of Tasks Worker¶
TWS stops running subtasks
TWS adds cancellation records to SUBTASK_CANCEL_LIST
Service shuts down
Service shutdown of Tasks Service¶
Service shuts down
See the general task processing sequence diagram and sequence diagrams for each task type in the ‘Tasks diagrams’ section of the administrator manual.