Tremor connects to the external systems using connectors.
Connectors that integrate Tremor with upstream systems from where Tremor is typically ingesting data are called
Connectors that integrate Tremor with downstream systems where Tremor is typically publishing or contributing data to are called
codecs to transform the external wire form of connected system participants into a structured internal value type Tremor understands semantically.
Tremor's internal type system is JSON-like.
postprocessors. External data ingested into Tremor via
Onramps can be pre-processed through multiple transfomers before a code is applied to convert the data into Tremor-internal form. Preprocessors are configured as a chain of transformations. Postprocessors
are applied to values leaving Tremor after a codec transforms them from Tremor internal form to wire form. Postprocessors are configured as a chain of transformations.
Codecs share similar concepts to extractors, but differ in their application. Codecs are applied to external data as they are ingested by or egressed from a running Tremor process. Extractors, on the other hand, are Tremor-internal and convert data from and to Tremor's internal value type.
Tremor's internal data representation is JSON-like. The supported value types are:
- String- UTF-8 encoded
- Numeric (float, integer)
- Record (string keys)
Tremor neither requires nor validates schemas and works with schemaless or unstructured data. Validation can be asserted with the tremor-script language.
Offramps, and other components of Tremor may, however, require or expect conformance with schemas.
For specific components, their documentation should be consulted for correct usage.
Tremor supports the encoding and decoding of the following formats:
- binflux- (binary representation of the influx wire protocol).
- string- any valid UTF-8 string sequence.
Pre- and Postprocessors#
Tremor supports the following preprocessing transformations in
- lines- split by newline.
- lines-null- split by null byte.
- lines-pipe- split by
- base64- base64 decoding.
- decompress- auto detecting decompress.
- gzip- gzip decompress.
- zlib- zlib decompress.
- xz- xz decompress.
- snappy- snappy decompress.
- lz4- zl4 decompress.
- gelf-chunking- GELF chunking support.
- remove-empty- remove emtpy (0 len) messages.
- length-prefixerd- length prefixed splitting for streams.
Tremor supports the following postprocessing transformations in