Luigi Alternatives
Luigi
The purpose of Luigi is to address all the plumbing typically associated with long-running batch processes. You want to chain many tasks, automate them, and failures will happen. These tasks can be anything, but are typically long running things like Hadoop jobs, dumping data to/from databases, running machine learning algorithms, or anything else.
There are other software packages that focus on lower level aspects of data processing, like Hive, Pig, or Cascading. Luigi is not a framework to replace these. Instead it helps you stitch many tasks together, where each task can be a Hive query, a Hadoop job in Java, a Spark job in Scala or Python a Python snippet, dumping a table from a database, or anything else. It's easy to build up long-running pipelines that comprise thousands of tasks and take days or weeks to complete. Luigi takes care of a lot of the workflow management so that you can focus on the tasks themselves and their dependencies.
You can build pretty much any task you want, but Luigi also comes with a toolbox of several common task templates that you use. It includes support for running Python mapreduce jobs in Hadoop, as well as Hive, and Pig, jobs. It also comes with file system abstractions for HDFS, and local files that ensures all file system operations are atomic. This is important because it means your data pipeline will not crash in a state containing partial data.
Best Luigi Alternatives
Hate Luigi? Use another browser! Which one should you go with? Try out these other browsers before making your final decision. Want some more context? We've got a great list of alternatives to Luigi here.
Zenaton
FreemiumClever CloudSoftware as a Service (SaaS)HerokuA Workflow Builder for Developers. Build event-driven processes in days instead of months .
Features:
- Workflow
- Container Orchestration
- Error Handling
- Real-time monitoring
- Workflow Triggers
- API Integration
- Php
- Python
- Ruby
- Scheduling
- Task Automation
- Task Scheduling
- Workflow Automation
Apache Airflow
FreeOpen SourceLinuxAirflow is a platform to programmaticaly author, schedule and monitor data pipelines. Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The...
Features:
- Task Management
- Task Scheduling
- Python
- Workflow
ProActive Workflows & Scheduling
FreeOpen SourceMacWindowsLinuxProActive Workflows & Scheduling allows you to easily execute all your company jobs and business applications, monitor activity and quickly access job results.
Features:
- Python
- Scheduling
- Workflow
Shipyard App
FreemiumWebSoftware as a Service (SaaS)Workflow Automation Platform for Data Teams. Build, Monitor, and Share Data Solutions 10x faster without DevOps.
Features:
- Business Intelligence
- Data analytics
- Data-management
- Data science
- Developer Tools
- Workflow
- Workflow Automation
- Workflow Management
StackStorm
FreeOpen SourceLinuxStackStorm is a powerful open-source automation platform that wires together all of your apps, services and workflows. It’s extendable, flexible, and built with love for...
Features:
- Job scheduling
- REST API
- Scheduling
- SSH
- Workflow Automation
Apache Oozie
FreeOpen SourceLinuxOozie is a workflow scheduler system to manage Apache Hadoop jobs. Oozie Workflow jobs are Directed Acyclical Graphs (DAGs) of actions. Oozie Coordinator jobs are...
Azkaban
FreeOpen SourceLinuxAzkaban is a batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban resolves the ordering through job dependencies and provides an easy to use web...
Metaflow
FreeOpen SourceSelf-HostedFramework for real-life data science; build, improve, and operate end-to-end workflows.
Features:
- Data science
- Workflow Automation
Upvote Comparison
Interest Trends
Luigi Reviews
Add your reviews & share your experience when using Luigi to the world. Your opinion will be useful to others who are looking for the best Luigi alternatives.