td_for_each>: Repeat using Treasure Data queries

td_for_each> operator loops subtasks for each result rows of a Hive or Presto query on Treasure Data.

Subtasks set at _do section can reference results using ${td.each.COLUMN_NAME} syntax where COLUMN_NAME is a name of column.

For example, if you run a query select email, name from users and the query returns 3 rows, this operator runs subtasks 3 times with ${} and ${} parameters.

    database: www_access

  td_for_each>: queries/users.sql
      echo>: found a user ${} email ${}


When you set those parameters, use digdag secrets command.

  • td.apikey: API_KEY

    The Treasure Data API key to use when running Treasure Data queries.


  • td_for_each>: FILE.sql

    Path to a query template file. This file can contain ${...} syntax to embed variables.


    td>: queries/step1.sql
  • _parallel: BOOLEAN

    Runs the repeating tasks in parallel.


    _parallel: true
  • _do: TASKS

    Tasks to run.


    # Runs a single task
      echo>: ${td.each}
    # Runs multiple tasks
        echo>: ${}
        echo>: ${}
  • database: NAME

    Name of a database.


    database: my_db
  • endpoint: ADDRESS

    API endpoint (default:

  • use_ssl: BOOLEAN

    Enable SSL (https) to access to the endpoint (default: true).

  • engine: presto

    Query engine (presto or hive).


    engine: hive
    engine: presto
  • priority: 0

    Set Priority (From -2 (VERY LOW) to 2 (VERY HIGH) , default: 0 (NORMAL)).

  • job_retry: 0

    Set automatic job retry count (default: 0).

    We recommend that you not set retry count over 10. If the job is not succeessful less than 10 times retry, it needs some fix a cause of failure.

  • presto_pool_name: NAME

    Name of a resource pool to run the query in. Applicable only when engine is presto.


    presto_pool_name: poc
  • hive_pool_name: NAME

    Name of a resource pool to run the query in. Applicable only when engine is hive.


    engine: hive
    pool_name: poc
  • engine_version: NAME

    Specify engine version for Hive and Presto.


    engine: hive
    engine_version: stable
  • hive_engine_version: NAME

    Specify engine version for Hive. This option overrides engine_version if engine is hive.


    engine: hive
    hive_engine_version: stable

Output parameters

  • td.last_job_id or

    The job id this task executed.


  • td.last_job.num_records

    The number of records of this job output.