Edit

@4me/post-ops.uploader

Required variables

• PORT (required)

  • Port the application the will run on

  • Example: PORT=4000

• S3_BUCKET (required)

  • S3 target bucket

  • Example: S3_BUCKET=my-bucket

• S3_BUCKET_PREFIX (required)

  • S3 bucket prefix

  • Example: S3_BUCKET_PREFIX=4me_ops

Optional variables

• LOG_PRETTY

  • Enable/disable human readable logs. JSON logs will be produced otherwise.

  • Example: LOG_PRETTY=1

  • Default value: process.env.NODE_ENV === 'development'

• LOG_LEVEL

  • Change the log level. Possible values are fatal, error, warn, info, debug or trace.

  • Example: LOG_LEVEL=trace

  • Default value: process.env.NODE_ENV === 'development' ? 'debug' : 'info'

• CONFIG

  • Location of the configuration file

  • Example: CONFIG=/config/config.yml

  • Default value: /config.yml

• DRY_RUN

  • Will not upload stuff to S3 when this variable is set. Useful for debugging purposes.

  • Example: DRY_RUN=true

• AWS_ACCESS_KEY_ID

  • AWS Credentials (pulled from env, ~/.aws/config file, or instance metadata)

  • Example: AWS_ACCESS_KEY_ID=yyyy

• AWS_SECRET_ACCESS_KEY

  • AWS Credentials (pulled from env, ~/.aws/config file, or instance metadata)

  • Example: AWS_SECRET_ACCESS_KEY=xxxx

Config file

• config.yml.example (YML):

  # Represents a list of jobs
# Each job will periodically backup something to S3
jobs:
  # Each job must have a unique name
  - name: xman-egll
    # An HTTP job will make an http request to a specific URL and backup the content
    type: http
    # Source URL
    source: http://xman-fetcher-egll/raw
    # Number of seconds between backups
    period: 30
    # Output S3 bucket
    s3_bucket: test
    # Output s3 prefix
    output_folder: 4me_ops/xman
    # Output file prefix
    file_prefix: egll
    # Output file suffix
    file_extension: json
  - name: messaging
    # An SQL job, will backup the content of a PostgreSQL table
    # PG* env variables must be set to establish a connection to PGSQL
    # Output format is CSV
    type: sql
    # The target table
    table: messaging
    # Number of seconds between backups
    period: 30
    # Output S3 bucket
    s3_bucket: test
    # Output s3 prefix
    output_folder: 4me_ops/messaging
    # Output file suffix
    file_extension: csv
    # Data filter
    # This allows backups of only recently created items in a SQL table
    # (optional, defaults to no filtering)
    windowing:
      # Select only records created after (now - duration)
      # In this example, only select records from the last 24h
      duration: 86400
      # Name of the timestamp column
      # (optional, defaults to 'created_at')
      column: 'created_at'

  - name: control_room_history
    # A GraphQL job, will backup the result of a graphql query
    type: graphql
    # The GraphQL endpoint
    endpoint: https://foo.bar/graphql
    # Number of seconds between backups
    period: 30
    # Output S3 bucket
    s3_bucket: test
    # Output s3 prefix
    output_folder: 4me_ops/messaging
    # Output file suffix
    file_extension: csv
    # The GraphQL query
    # If windowing is set, a $windowStart variable must be declared in your query
    # If windowing is not set, this variable must not be declared in your query
    query: |-
      query ControlRoomHistory($windowStart: Timestamp!) {
        controlRoomHistory(environment: "LFEE", start: $windowStart) {
          id
          start
          end
          openSectors
          sectors {
            center
            name
          }
        }
      }
    windowing:
      # When a query is executed, a variable $windowStart will be set to now - duration
      # If the query is executed at 12:00Z and the window duration is set to 3600 seconds,
      # $windowStart will be set to 11:00Z
      duration: 86400