Concept
Tremor connects to the external systems using connectors.
Connectors that integrate tremor with upstream systems from where tremor is typically
ingesting data are called Onramps.
Connectors that integrate tremor with downstream systems where tremor is typically
publishing or contributing data to are called Offramps.
Onramps and Offramps use codecs to transform the external wire form of connected
system participants into a structured internal value type tremor understands semantically.
Tremor’s internal type system is JSON-like.
Onramps and Offramps support preprocessors and postprocessors. External data ingested into tremor
via Onramps can be pre-processed through multiple transfomers before a code is applied to convert the
data into tremor internal form. Preprocessors are configured as a chain of transformations. Postprocessors
are applied to values leaving tremor after a codec transform them from tremor internal form to wire
form. Postprocessors are configured as a chain of transformations.
Codecs share similar concepts to extractors but differ in their application. Codecs are applied to external data as they are ingested by or egressed from a running tremor process. Extractors, on the other hand, are tremor internal and convert data from and to tremor’s internal value type.
Data Format
Tremor’s internal data representation is JSON-like. The supported value types are:
- String - UTF-8 encoded
- Numeric (float, integer)
- Boolean
- Null
- Array
- Record (string keys)
Codecs
Tremor neither requires nor validates schemas and works with schemaless or unstructured
data. Validation can be asserted with the tremor-script language. Onramps, Offramps
and other components of tremor may however require or expect conformance with schemas.
For specific components their documentation should be consulted for correct usage.
Tremor supports the encoding and decoding of the following formats:
- json
- msgpack
- influx
- binflux - (binary representation of the influx wire protocol)
- statsd
- yaml
- string - any valid UTF-8 string sequence
Pre- and Postprocessors
Tremor supports the following preprocessing transformations in Onramp configurations:
- lines - split by newline
- lines-null - split by null byte
- lines-pipe - split by
| - base64 - base64 decoding
- decompress - auto detecting decompress
- gzip - gzip decompress
- zlib - zlib decompress
- xz - xz decompress
- snappy - snappy decompress
- lz4 - zl4 decompress
- gelf-chunking - GELF chunking support
- remove-empty - remove emtpy (0 len) messages
- length-prefixerd - length prefixed splitting for streams
Tremor supports the following postprocessing transformations in Offramp configurations: