luigi
https://github.com/spotify/luigi
Python
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Python not yet supported4 Subscribers
Add a CodeTriage badge to luigi
Help out
- Issues
- Allow external program tasks to run outside the main thread
- Emit BROKEN_TASK when complete() raises, not DEPENDENCY_MISSING
- Fix binary line iteration for S3 targets
- Fix outdated and swapped Kubernetes doc links
- fix(contrib/sge): handle text and bytes output from qstat and qsub commands
- fix(visualiser): remove external Google Fonts @import for offline environments
- fix(contrib/sge): fix bytes pickle serialization in _dump and guard fstatvfs
- fix(task): rename Task.get_param_names parameter to include_insignificant
- fix(parameter): serialize Task instances using task_id in _DictParamEncoder
- TimeDeltaParameter.serialize cannot round trip negative or sub second durations
- Docs
- Python not yet supported