What’s Digdag?

Digdag is a simple tool that helps you to build, run, schedule, and monitor complex pipelines of tasks. It handles dependency resolution so that tasks run in order or in parallel.

Digdag fits simple replacement of cron, IT operations automation, data analytics batch jobs, machine learning pipelines, and many more by using Directed Acyclic Graphs (DAG) as the infrastructure.

Getting started

> “Digdag in 5 minutes”

Learn more

> “Architecture”

Table of Contents

• Getting started
  • 1. Requirements.
  • 2. Downloading the latest version
  • 3. Running sample workflow
  • Next steps
• Architecture
  • Automating workflow with Digdag
  • Organizing tasks by groups
  • Workflow as code
  • Running with local mode
  • Running on a server
  • Next steps
• Concepts
  • Projects and revisions
  • Sessions and attempts
  • Scheduled execution and session_time
  • Tasks
  • Export and store parameters
  • Operators and plugins
  • Dynamic task generation and _check/_error tasks
  • Task naming and resuming
  • Workspace
  • Next steps
• Workflow definition
  • Workflow definition: *.dig files
  • “+” is a task
  • operators>
  • Using ${variables}
  • Calculating variables
  • Defining variables
  • !include another file
  • Parallel execution
  • Retrying failed tasks automatically
  • Sending error notification
• Scheduling workflow
  • Setting up a schedule
  • Running scheduler
  • Checking scheduling status
  • Setting an alert if a workflow doesn’t finish within expected time
  • Skipping a next workflow session
  • Skipping backfill
  • Set start/end
• Operators
  • Workflow control operators
  • Treasure Data operators
  • Database operators
  • Network operators
  • Amazon Web Services operators
  • Google Cloud Platform operators
  • Scripting operators
  • Param operators
• Command reference
  • Local-mode commands
  • Server-mode commands
  • Client-mode commands
  • Common options
• Language API - Python
  • Programmable workflow in Python
  • Defining variables
  • Method argument mapping
  • Generating child tasks
• Language API - Ruby
  • Programmable workflow in Ruby
  • Defining variables
  • Method argument mapping
  • Generating child tasks
• REST API
• Command Executor
  • ECS
  • Docker
• Internal architecture
  • Task logging
  • Database
  • API server, agent, workflow executor, and schedule executor
  • Task queue
  • Project archive storage
  • Command executor
  • Extension mechanisms
  • Next steps
• Digdag metrics (experimental)
  • Metrics category
  • Setup
  • API metrics
• Community Contributions
  • Operators
  • Extensions
  • Clients / Tools
  • Contents
• Release Notes
  • Release 0.10.5
  • Release 0.10.4
  • Release 0.10.3
  • Release 0.10.2
  • Release 0.10.1
  • Release 0.10.0
  • Release 0.9.42
  • Release 0.9.41
  • Release 0.9.40
  • Release 0.9.39
  • Release 0.9.38
  • Release 0.9.37
  • Release 0.9.36
  • Release 0.9.35
  • Release 0.9.34
  • Release 0.9.33
  • Release 0.9.32
  • Release 0.9.31
  • Release 0.9.30
  • Release 0.9.29
  • Release 0.9.28
  • Release 0.9.27
  • Release 0.9.26
  • Release 0.9.25
  • Release 0.9.24
  • Release 0.9.23
  • Release 0.9.22
  • Release 0.9.21
  • Release 0.9.20
  • Release 0.9.19
  • Release 0.9.18
  • Release 0.9.17
  • Release 0.9.16
  • Release 0.9.15
  • Release 0.9.14
  • Release 0.9.13
  • Release 0.9.12
  • Release 0.9.11
  • Release 0.9.10
  • Release 0.9.9
  • Release 0.9.8
  • Release 0.9.7
  • Release 0.9.6
  • Release 0.9.5
  • Release 0.9.4
  • Release 0.9.3
  • Release 0.9.2
  • Release 0.9.1
  • Release 0.9.0
  • Release 0.8.22
  • Release 0.8.21
  • Release 0.8.20
  • Release 0.8.19
  • Release 0.8.18
  • Release 0.8.17
  • Release 0.8.16
  • Release 0.8.15
  • Release 0.8.14
  • Release 0.8.13
  • Release 0.8.12
  • Release 0.8.11
  • Release 0.8.10
  • Release 0.8.9
  • Release 0.8.8
  • Release 0.8.7
  • Release 0.8.6
  • Release 0.8.5
  • Release 0.8.4
  • Release 0.8.3
  • Release 0.8.2
  • Release 0.8.1
  • Release 0.8.0
  • Release 0.7.1
  • Release 0.7.0
  • Release 0.6.1
  • Release 0.6.0
  • Release 0.5.9
  • Release 0.5.8
  • Release 0.5.7
  • Release 0.5.6
  • Release 0.5.5
  • Release 0.5.4
  • Release 0.5.3
  • Release 0.5.2
  • Release 0.5.1
  • Release 0.5.0
  • Release 0.4.3
  • Release 0.4.2
  • Release 0.4.1
  • Release 0.4.0
  • Release 0.3.6
  • Release 0.3.5
  • Release 0.3.4
  • Release 0.3.3
  • Release 0.3.2
  • Release 0.3.1
  • Release 0.3.0
  • Release 0.2.8
  • Release 0.2.7
  • Release 0.2.6
  • Release 0.2.5
  • Release 0.2.4
  • Release 0.2.3
  • Release 0.2.2
  • Release 0.2.1
  • Release 0.2.0
• Logo
  • Square
  • Horizontal
  • Logo
  • Links

License

Digdag is open-source software licensed under Apache License, Version 2.0.

See NOTICE for details.