bq_ddl>: Managing Google BigQuery Datasets and Tables

bq_ddl> operator can be used to create, delete and clear Google BigQuery Datasets and Tables.

_export:
  bq:
    dataset: my_dataset

+prepare:
  bq_ddl>:
    create_datasets:
      - my_dataset_${session_date_compact}
    empty_datasets:
      - my_dataset_${session_date_compact}
    delete_datasets:
      - my_dataset_${last_session_date_compact}
    create_tables:
      - my_table_${session_date_compact}
    empty_tables:
      - my_table_${session_date_compact}
    delete_tables:
      - my_table_${last_session_date_compact}

Secrets

Options

  • create_datasets: LIST

    Create new datasets.

    For detailed information about dataset configuration parameters, see the Google BigQuery Datasets Documentation.

    Examples:

    create_datasets:
      - foo
      - other_project:bar
    
    create_datasets:
      - foo_dataset_${session_date_compact}
      - id: bar_dataset_${session_date_compact}
        project: other_project
        friendly_name: Bar dataset ${session_date_compact}
        description: Bar dataset for ${session_date}
        default_table_expiration: 7d
        location: EU
        labels:
          foo: bar
          quux: 17
        access:
          - domain: example.com
            role: READER
          - userByEmail: [email protected]
            role: WRITER
          - groupByEmail: [email protected]
            role: OWNER
    
  • empty_datasets: LIST

    Create new datasets, deleting them first if they already exist. Any tables in the datasets will also be deleted.

    For detailed information about dataset configuration parameters, see the Google BigQuery Datasets Documentation.

    Examples:

    empty_datasets:
      - foo
      - other_project:bar
    
    empty_datasets:
      - foo_dataset_${session_date_compact}
      - id: bar_dataset_${session_date_compact}
        project: other_project
        friendly_name: Bar dataset ${session_date_compact}
        description: Bar dataset for ${session_date}
        default_table_expiration: 7d
        location: EU
        labels:
          foo: bar
          quux: 17
        access:
          - domain: example.com
            role: READER
          - userByEmail: [email protected]
            role: WRITER
          - groupByEmail: [email protected]
            role: OWNER
    
  • delete_datasets: LIST

    Delete datasets, if they exist.

    Examples:

    delete_datasets:
      - foo
      - other_project:bar
    
    delete_datasets:
      - foo_dataset_${last_session_date_compact}
      - other_project:bar_dataset_${last_session_date_compact}
    
  • create_tables: LIST

    Create new tables.

    For detailed information about table configuration parameters, see the Google BigQuery Tables Documentation.

    Examples:

    create_tables:
      - foo
      - other_dataset.bar
      - other_project:yet_another_dataset.baz
    
    create_tables:
      - foo_dataset_${session_date_compact}
      - id: bar_dataset_${session_date_compact}
        project: other_project
        dataset: other_dataset
        friendly_name: Bar dataset ${session_date_compact}
        description: Bar dataset for ${session_date}
        expiration_time: 2016-11-01-T01:02:03Z
        schema:
          fields:
            - {name: foo, type: STRING}
            - {name: bar, type: INTEGER}
        labels:
          foo: bar
          quux: 17
        access:
          - domain: example.com
            role: READER
          - userByEmail: [email protected]
            role: WRITER
          - groupByEmail: [email protected]
            role: OWNER
    
  • empty_tables: LIST Create new tables, deleting them first if they already exist.

    For detailed information about table configuration parameters, see the Google BigQuery Tables Documentation.

    Examples:

    empty_tables:
      - foo
      - other_dataset.bar
      - other_project:yet_another_dataset.baz
    
    empty_tables:
      - foo_table_${session_date_compact}
      - id: bar_table_${session_date_compact}
        project: other_project
        dataset: other_dataset
        friendly_name: Bar dataset ${session_date_compact}
        description: Bar dataset for ${session_date}
        expiration_time: 2016-11-01-T01:02:03Z
        schema:
          fields:
            - {name: foo, type: STRING}
            - {name: bar, type: INTEGER}
        labels:
          foo: bar
          quux: 17
        access:
          - domain: example.com
            role: READER
          - userByEmail: [email protected]
            role: WRITER
          - groupByEmail: [email protected]
            role: OWNER
    
  • delete_tables: LIST Delete tables, if they exist.

    Examples:

    delete_tables:
      - foo
      - other_dataset.bar
      - other_project:yet_another_dataset.baz
    
    delete_tables:
      - foo_table_${last_session_date_compact}
      - bar_table_${last_session_date_compact}