...
Luigi does not provide a pre-defined strategy for exception handling which have to be implemented individually
by the user, if possible. However, Luigi offers a mechanism for so called 'idempotent' tasks which are only run
once, given a specific set of parameters. Hereby, Luigi skipps skips complete tasks while running a pipeline
the second time after encounting encountering an unresolvable dependency (i.e. a predecessor task which failed while execution).
...
| ID | Process | Exception Handling |
|---|---|---|
| 1 | None. | None. |
| 2 | Tasks are stored within a database and updated whenever a task terminates.
| While start up Luigi, check and re-run open pipeline tasks. |
...
We do not provide any rollback meachanismmechanism. Inadequate results are not detected nor cleaned up by Luigi. Therefore, consistent results
are crucial. (Advantages: Reduce complexity of workflow engine and hereby the error rate due rollbacks).
...
Given the mechanism of idempotent tasks and the fact as we can verify an algorithm's exit code, Luigi could simply infer if an algorithm has to be
run or was already completed. A problem appears after restarting Luigi which discards all containers and there exit codes.
A recommended solution is to save all container's exit codes and standard streams (stdout/strerrstderr) within Luigi's database. The solution supports
traceability for historical batch processes and provides Luigi a way to identify complete tasks.
...