Alibaba Cloud OSS

Alibaba Cloud Long Term Storage

Synopsis

Creates a target that writes log messages to Alibaba Cloud Object Storage Service (OSS) with support for various file formats and authentication methods. The target handles large file uploads efficiently with configurable rotation based on size or event count. Alibaba Cloud OSS provides secure, cost-effective, and high-durability object storage with strong presence in Asia-Pacific regions.

Schema

- name: <string>
  description: <string>
  type: alibabas3
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    key: <string>
    secret: <string>
    region: <string>
    endpoint: <string>
    part_size: <numeric>
    bucket: <string>
    buckets:
      - bucket: <string>
        name: <string>
        format: <string>
        compression: <string>
        extension: <string>
        schema: <string>
    name: <string>
    format: <string>
    compression: <string>
    extension: <string>
    schema: <string>
    max_size: <numeric>
    batch_size: <numeric>
    timeout: <numeric>
    field_format: <string>
    interval: <string|numeric>
    cron: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
`name`	Y		Target name
`description`	N	-	Optional description
`type`	Y		Must be `alibabas3`
`pipelines`	N	-	Optional post-processor pipelines
`status`	N	`true`	Enable/disable the target

Alibaba Cloud OSS Credentials

Field	Required	Default	Description
`key`	Y	-	Alibaba Cloud OSS access key ID
`secret`	Y	-	Alibaba Cloud OSS access key secret
`region`	Y	-	Alibaba Cloud region (e.g., `oss-cn-hangzhou`, `oss-us-west-1`, `oss-ap-southeast-1`)
`endpoint`	Y	-	OSS endpoint URL (format: `https://oss-<region>.aliyuncs.com`)

Connection

Field	Required	Default	Description
`part_size`	N	`5`	Multipart upload part size in megabytes (minimum 5MB)
`timeout`	N	`30`	Connection timeout in seconds
`field_format`	N	-	Data normalization format. See applicable Normalization section

Files

Field	Required	Default	Description
`bucket`	N*	-	Default OSS bucket name (used if `buckets` not specified)
`buckets`	N*	-	Array of bucket configurations for file distribution
`buckets.bucket`	Y	-	OSS bucket name
`buckets.name`	Y	-	File name template
`buckets.format`	N	`"json"`	Output format: `json`, `multijson`, `avro`, `parquet`
`buckets.compression`	N	-	Compression algorithm. See Compression below
`buckets.extension`	N	Matches `format`	File extension override
`buckets.schema`	N*	-	Schema definition file path (required for Avro and Parquet formats)
`name`	N	`"vmetric.{{.Timestamp}}.{{.Extension}}"`	Default file name template when `buckets` not used
`format`	N	`"json"`	Default output format when `buckets` not used
`compression`	N	-	Default compression when `buckets` not used
`extension`	N	Matches `format`	Default file extension when `buckets` not used
`schema`	N	-	Default schema path when `buckets` not used
`max_size`	N	`0`	Maximum file size in bytes before rotation
`batch_size`	N	`100000`	Maximum number of messages per file

* = Either bucket or buckets must be specified. When using buckets, schema is conditionally required for Avro and Parquet formats.

note

When max_size is reached, the current file is uploaded to OSS and a new file is created. For unlimited file size, set the field to 0.

Scheduler

Field	Required	Default	Description
`interval`	N	realtime	Execution frequency. See Interval for details
`cron`	N	-	Cron expression for scheduled execution. See Cron for details

Debug Options

Field	Required	Default	Description
`debug.status`	N	`false`	Enable debug logging
`debug.dont_send_logs`	N	`false`	Process logs but don't send to target (testing)

Details

The Alibaba Cloud OSS target provides enterprise-grade cloud storage integration with comprehensive file format support. OSS offers 99.9999999999% (12 nines) data durability and strong regional coverage across Asia-Pacific, making it ideal for applications serving Asian markets.

Authentication

Requires Alibaba Cloud access credentials. Access keys can be created through the Alibaba Cloud Console under AccessKey Management. RAM (Resource Access Management) users can be created with specific OSS permissions for enhanced security.

Endpoint Configuration

The endpoint URL follows the pattern https://oss-<region>.aliyuncs.com where <region> is your chosen Alibaba Cloud region identifier. Internal endpoints are also available for ECS instances in the same region using https://oss-<region>-internal.aliyuncs.com for cost savings.

Available Regions

Alibaba Cloud OSS is available in numerous regions worldwide:

Region Code	Location
`oss-cn-hangzhou`	China (Hangzhou)
`oss-cn-shanghai`	China (Shanghai)
`oss-cn-beijing`	China (Beijing)
`oss-cn-shenzhen`	China (Shenzhen)
`oss-cn-hongkong`	China (Hong Kong)
`oss-us-west-1`	US (Silicon Valley)
`oss-us-east-1`	US (Virginia)
`oss-ap-southeast-1`	Singapore
`oss-ap-southeast-2`	Australia (Sydney)
`oss-ap-southeast-3`	Malaysia (Kuala Lumpur)
`oss-ap-southeast-5`	Indonesia (Jakarta)
`oss-ap-northeast-1`	Japan (Tokyo)
`oss-ap-south-1`	India (Mumbai)
`oss-eu-central-1`	Germany (Frankfurt)
`oss-eu-west-1`	UK (London)
`oss-me-east-1`	UAE (Dubai)

Loading include...

Templates

The following template variables can be used in file names:

Variable	Description	Example
`{{.Year}}`	Current year	`2024`
`{{.Month}}`	Current month	`01`
`{{.Day}}`	Current day	`15`
`{{.Timestamp}}`	Current timestamp in nanoseconds	`1703688533123456789`
`{{.Format}}`	File format	`json`
`{{.Extension}}`	File extension	`json`
`{{.Compression}}`	Compression type	`zstd`
`{{.TargetName}}`	Target name	`my_logs`
`{{.TargetType}}`	Target type	`alibabas3`
`{{.Table}}`	Bucket name	`logs`

Multipart Upload

Large files automatically use multipart upload protocol with configurable part size (part_size parameter). Default 5MB part size balances upload efficiency and memory usage.

Multiple Buckets

Single target can write to multiple OSS buckets with different configurations, enabling data distribution strategies (e.g., raw data to one bucket, processed data to another).

Schema Requirements

Avro and Parquet formats require schema definition files. Schema files must be accessible at the path specified in the schema parameter during target initialization.

Storage Classes

Alibaba Cloud OSS supports multiple storage classes including Standard, Infrequent Access, Archive, and Cold Archive for cost optimization based on access patterns.

Regional Performance

OSS provides excellent performance for applications serving Asian markets with extensive regional presence across China, Southeast Asia, and other Asia-Pacific regions.

Examples

Basic Configuration

The minimum configuration for a JSON OSS target:

targets:
  - name: basic_oss
    type: alibabas3
    properties:
      key: "LTAI5tAbCdEfGhIjKlMnOpQr"
      secret: "aBcDeFgHiJkLmNoPqRsTuVwXyZ0123456789"
      region: "oss-cn-hangzhou"
      endpoint: "https://oss-cn-hangzhou.aliyuncs.com"
      bucket: "datastream-logs"

Multiple Buckets

Configuration for distributing data across multiple OSS buckets with different formats:

targets:
  - name: multi_bucket_export
    type: alibabas3
    properties:
      key: "LTAI5tAbCdEfGhIjKlMnOpQr"
      secret: "aBcDeFgHiJkLmNoPqRsTuVwXyZ0123456789"
      region: "oss-ap-southeast-1"
      endpoint: "https://oss-ap-southeast-1.aliyuncs.com"
      buckets:
        - bucket: "raw-data-archive"
          name: "raw-{{.Year}}-{{.Month}}-{{.Day}}.json"
          format: "multijson"
          compression: "gzip"
        - bucket: "analytics-data"
          name: "analytics-{{.Year}}/{{.Month}}/{{.Day}}/data_{{.Timestamp}}.parquet"
          format: "parquet"
          schema: "<schema definition>"
          compression: "snappy"

Multiple Buckets with Catch-All

Configuration for routing different log types to specific buckets with a catch-all for unmatched logs:

targets:
  - name: multi_bucket_routing
    type: alibabas3
    properties:
      key: "LTAI5tAbCdEfGhIjKlMnOpQr"
      secret: "aBcDeFgHiJkLmNoPqRsTuVwXyZ0123456789"
      region: "oss-cn-shanghai"
      endpoint: "https://oss-cn-shanghai.aliyuncs.com"
      buckets:
        - bucket: "security-logs"
          name: "security-{{.Year}}-{{.Month}}-{{.Day}}.json"
          format: "json"
        - bucket: "application-logs"
          name: "app-{{.Year}}-{{.Month}}-{{.Day}}.json"
          format: "json"
      bucket: "general-logs"
      name: "general-{{.Timestamp}}.json"
      format: "json"

Parquet Format

Configuration for daily partitioned Parquet files:

targets:
  - name: parquet_analytics
    type: alibabas3
    properties:
      key: "LTAI5tAbCdEfGhIjKlMnOpQr"
      secret: "aBcDeFgHiJkLmNoPqRsTuVwXyZ0123456789"
      region: "oss-cn-shanghai"
      endpoint: "https://oss-cn-shanghai.aliyuncs.com"
      bucket: "analytics-lake"
      name: "events/year={{.Year}}/month={{.Month}}/day={{.Day}}/part-{{.Timestamp}}.parquet"
      format: "parquet"
      schema: "<schema definition>"
      compression: "snappy"
      max_size: 536870912

High Reliability

Configuration with enhanced settings:

targets:
  - name: reliable_oss
    type: alibabas3
    pipelines:
      - checkpoint
    properties:
      key: "LTAI5tAbCdEfGhIjKlMnOpQr"
      secret: "aBcDeFgHiJkLmNoPqRsTuVwXyZ0123456789"
      region: "oss-cn-beijing"
      endpoint: "https://oss-cn-beijing.aliyuncs.com"
      bucket: "critical-logs"
      name: "logs-{{.Timestamp}}.json"
      format: "json"
      timeout: 60
      part_size: 10

With Field Normalization

Using field normalization for standard format:

targets:
  - name: normalized_oss
    type: alibabas3
    properties:
      key: "LTAI5tAbCdEfGhIjKlMnOpQr"
      secret: "aBcDeFgHiJkLmNoPqRsTuVwXyZ0123456789"
      region: "oss-ap-northeast-1"
      endpoint: "https://oss-ap-northeast-1.aliyuncs.com"
      bucket: "normalized-logs"
      name: "logs-{{.Timestamp}}.json"
      format: "json"
      field_format: "cim"

Debug Configuration

Configuration with debugging enabled:

targets:
  - name: debug_oss
    type: alibabas3
    properties:
      key: "LTAI5tAbCdEfGhIjKlMnOpQr"
      secret: "aBcDeFgHiJkLmNoPqRsTuVwXyZ0123456789"
      region: "oss-cn-hangzhou"
      endpoint: "https://oss-cn-hangzhou.aliyuncs.com"
      bucket: "test-logs"
      name: "test-{{.Timestamp}}.json"
      format: "json"
      debug:
        status: true
        dont_send_logs: true

Internal Endpoint

Configuration using internal endpoint for ECS instances in the same region:

targets:
  - name: internal_oss
    type: alibabas3
    properties:
      key: "LTAI5tAbCdEfGhIjKlMnOpQr"
      secret: "aBcDeFgHiJkLmNoPqRsTuVwXyZ0123456789"
      region: "oss-cn-shanghai"
      endpoint: "https://oss-cn-shanghai-internal.aliyuncs.com"
      bucket: "application-logs"
      name: "logs/{{.Year}}/{{.Month}}/{{.Day}}/{{.Timestamp}}.json"
      format: "json"
      compression: "zstd"

Synopsis​

Schema​

Configuration​

Alibaba Cloud OSS Credentials​

Connection​

Files​

Scheduler​

Debug Options​

Details​

Authentication​

Endpoint Configuration​

Available Regions​

Templates​

Multipart Upload​

Multiple Buckets​

Schema Requirements​

Storage Classes​

Regional Performance​

Examples​

Basic Configuration​

Multiple Buckets​

Multiple Buckets with Catch-All​

Parquet Format​

High Reliability​

With Field Normalization​

Debug Configuration​

Internal Endpoint​

Synopsis

Schema

Configuration

Alibaba Cloud OSS Credentials

Connection

Files

Scheduler

Debug Options

Details

Authentication

Endpoint Configuration

Available Regions

Templates

Multipart Upload

Multiple Buckets

Schema Requirements

Storage Classes

Regional Performance

Examples

Basic Configuration

Multiple Buckets

Multiple Buckets with Catch-All

Parquet Format

High Reliability

With Field Normalization

Debug Configuration

Internal Endpoint