Simple, Open Source, Multi-Cloud Workflow Engine
Watch the Demo Video
Digdag is a simple tool that helps you to build, run, schedule, and monitor complex pipelines of tasks. It handles dependency resolution so that tasks run in series or in parallel.
Digdag replaces cron, facilitates IT operations automation, orchestrates data engineering tasks, coordinates machine learning pipelines, and more.
-
Easy deployment
Just download the binary and you are good to go. The same binary can run both server and client.
-
Simple Configuration
Digdag configurations are written in YAML, a widely used simple markup language.
-
Dependency resolution
Read and write workflows with dependencies. Say goodbye to “timing” cron jobs to coordinate multiple tasks.
-
Multi-Cloud
Move and join data across Amazon Redshift/S3, Google BigQuery/Cloud Storage and on-premises databases.
-
Multi-lingual
Native support for Python and Ruby. This is where data scientists and web developers collaborate.
-
Error handling
Workflows sometimes fail. Digdag ships with robust error handling. Resume, not restart.
-
Modular
Digdag is designed to be modular so that each workflow can be re-used in different contexts.
-
Extensible
Digdag is designed to be pluggable so that you can hack on your own operators, control flows and more.
-
Admin UI
Not a command line expert? No worries, Digdag has an admin UI in active development.
-
Secure Secrets
Digdag provides a secrets vault so that sensitive information like API keys and passwords are handled securely.
-
Docker support
Digdag lets you run tasks from Docker containers to support cloud native workflows.
-
Version Controlled
Digdag is workflow as code. Revisions and rollbacks supported.
Getting Started
$ curl -o ~/bin/digdag --create-dirs -L "https://dl.digdag.io/digdag-latest" $ chmod +x ~/bin/digdag $ echo 'export PATH="$HOME/bin:$PATH"' >> ~/.bashrc