s3_wait>: Wait for a file in Amazon S3

s3_wait> operator waits for file to appear in Amazon S3.

+wait:
  s3_wait>: my-bucket/my-key

Secrets

  • aws.s3.access_key_id, aws.access_key_id

    The AWS Access Key ID to use when accessing S3.

  • aws.s3.secret_access_key, aws.secret_access_key

    The AWS Secret Access Key to use when accessing S3.

  • aws.s3.region, aws.region

    An optional explicit AWS Region in which to access S3.

  • aws.s3.endpoint

    An optional explicit API endpoint to use when accessing S3. This overrides the region secret.

  • aws.s3.sse_c_key

    An optional Customer-Provided Server-Side Encryption (SSE-C) key to use when accessing S3. Must be Base64 encoded.

  • aws.s3.sse_c_key_algorithm

    An optional Customer-Provided Server-Side Encryption (SSE-C) key algorithm to use when accessing S3.

  • aws.s3.sse_c_key_md5

    An optional MD5 digest of the Customer-Provided Server-Side Encryption (SSE-C) key to use when accessing S3. Must be Base64 encoded.

For more information about SSE-C, See the AWS S3 Documentation.

Options

  • s3_wait>: BUCKET/KEY

    Path to the file in Amazon S3 to wait for.

    Examples:

    s3_wait>: my-bucket/my-data.gz
    
    s3_wait>: my-bucket/file/in/a/directory
    
  • region: REGION

    An optional explicit AWS Region in which to access S3. This may also be specified using the aws.s3.region secret.

  • endpoint: ENDPOINT

    An optional explicit AWS Region in which to access S3. This may also be specified using the aws.s3.endpoint secret. Note: This will override the region parameter.

  • bucket: BUCKET

    The S3 bucket where the file is located. Can be used together with the key parameter instead of putting the path on the operator line.

  • key: KEY

    The S3 key of the file. Can be used together with the bucket parameter instead of putting the path on the operator line.

  • version_id: VERSION_ID

    An optional object version to check for.

  • path_style_access: true/false

    An optional flag to control whether to use path-style or virtual hosted-style access when accessing S3. Note: Enabling path_style_access also requires specifying a region.

Output Parameters

  • s3.last_object

    Information about the detected file.

      {
        "metadata": {
          "Accept-Ranges": "bytes",
          "Access-Control-Allow-Origin": "*",
          "Content-Length": 4711,
          "Content-Type": "application/octet-stream",
          "ETag": "5eb63bbbe01eeed093cb22bb8f5acdc3",
          "Last-Modified": 1474360744000,
          "Last-Ranges": "bytes"
        },
        "user_metadata": {
          "foo": "bar",
          "baz": "quux"
        }
      }
    

Note: The s3_wait> operator makes use of polling with exponential backoff. As such there might be some time interval between a file being created and the s3_wait> operator detecting it.