Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
By chatting or signing in you agree to the Terms and chat-message logging (revocable in History).