Tremor
Fork me on GitHub

Understanding Data

De- and encode data from wire formats.

Concept

Tremor connects to the external systems using connectors.

Connectors that integrate tremor with upstream systems from where tremor is typically ingesting data are called Onramps.

Connectors that integrate tremor with downstream systems where tremor is typically publishing or contributing data to are called Offramps.

Onramps and Offramps use codecs to transform the external wire form of connected system participants into a structured internal value type tremor understands semantically.

Tremor’s internal type system is JSON-like.

Onramps and Offramps support preprocessors and postprocessors. External data ingested into tremor via Onramps can be pre-processed through multiple transfomers before a code is applied to convert the data into tremor internal form. Preprocessors are configured as a chain of transformations. Postprocessors are applied to values leaving tremor after a codec transform them from tremor internal form to wire form. Postprocessors are configured as a chain of transformations.

Codecs share similar concepts to extractors but differ in their application. Codecs are applied to external data as they are ingested by or egressed from a running tremor process. Extractors, on the other hand, are tremor internal and convert data from and to tremor’s internal value type.

Data Format

Tremor’s internal data representation is JSON-like. The supported value types are:

  • String - UTF-8 encoded
  • Numeric (float, integer)
  • Boolean
  • Null
  • Array
  • Record (string keys)

Codecs

Tremor neither requires nor validates schemas and works with schemaless or unstructured data. Validation can be asserted with the tremor-script language. Onramps, Offramps and other components of tremor may however require or expect conformance with schemas.

For specific components their documentation should be consulted for correct usage.

Tremor supports the encoding and decoding of the following formats:

Pre- and Postprocessors

Tremor supports the following preprocessing transformations in Onramp configurations:

Tremor supports the following postprocessing transformations in Offramp configurations: