Workflow engines or frameworks are software applications which help to build complex pipelines of batch jobs.
They handle dependency resolution, workflow management, visualization and handling failures.
We use Luigi as our workflow engine for Flarecast which was developed by Erik Bernhardsson and Elias Freider
at Spotify. The engine is simple yet mature and supports most of the features required. An alternative to mention
is Airflow which offers more features out of the box. However, Airflow is not yet mature and still work in progress.
A detailed review of Luigi and Airflow are given by Marton Trencseni at
http://bytepawn.com/luigi-airflow-pinball.html (as per 2016-05-03).
References: