This is a short canned synopsis of tremor configuration.
Tremor supports dynamic reconfiguration since v0.4.
The tremor runtime is internally structured with repositories and registries of configurable elements or artefacts. These artefacts can be one of:
Onramp - Specify to tremor how to connect to the outside world to ingest or consume external data. For example, the Kafka onramp consumes data from Kafka topics.
Offramp - Specify to tremor how to connect to the outside world to publish data to external systems For example, the Elastic offramp pushes data to ElasticSearch via its bulk upload REST/HTTP API endpoint
Pipeline - Specify to tremor what operations to perform on data ingested ( from any connected upstream source ) and what to contribute or publish downstream ( to any connected downstream target ).
Binding - A binding is a specification of how to interconnect Onramps, Offramps and Pipelines. Binding specifications can be thought of a type of wiring harness
Mapping - A mapping instantiated multiple bindings and fills in template variables.
Specifications for onramps, offramps, pipelines and bindings in tremor should be considered templates. They are stored in an internal tremor repository. A tremor repository stores artefacts, much like git repositories code.
Live onramps, offramps and pipelines in tremor are in a runnable state. They consume typically network bandwidth and some compute in the case of onramps and offramps. They consume compute bandwidth in the case of pipelines.
In order to create live instances of onramps, offramps and pipelines, a binding must be deployed. This is done by specifying a
mapping which consists of a mapping from binding instance url to the template values to fill in placeholders in the Binding specification. For each entry in the
mapping a binding instance is created and all referenced connected onramps, pipelines and offramps are instantiated with it.
Live instances of tremor artefacts are stored in a registry. A tremor registry can be thought of similarly to the Domain Name Service or DNS.
All 'live' or 'deployed' instances in tremor ( onramps, offramps, pipelines ) are managed by a finite state machine.
In this section we explore the two basic types of deployment in tremor.
+------------+ "publish" +--------------+ "bind/deploy" +--------------+
| +-------------------->+ +---------------------->+ |
| | | | | |
| Artefact | onramp, pipeline | Artefact | mapping | Instance |
| | offramp, binding | Repository | | Registry |
| | | | | |
| +<--------------------+ +<----------------------+ |
+------------+ "find" +--------------+ "unbind/undeploy" +--------------+
Tremor leverages the registry/repository and publish-find-bind Service Oriented Architecture patterns to drive its configuration model. Onramp, Offramp and Pipeline configurations can be published as template specifications along with how they should be interconnected as binding specifications.
As all artefacts in tremor are named, when a mapping is published, it deploys all the required onramps, offramps and pipelines automatically. This also means that when a mapping is deleted, that the corresponding live instances are undeployed.
All live or running artefacts have a corresponding state machine that manages its deployment lifecycle. The FSM is a simplified version of the POA worker activator lifecycle from CORBA and other Application Server Platforms.
Using the docker image
When using the tremor docker image configuration is loaded from the folder
/etc/tremor this folder should be mounted into the docker container to propagate the data.
The following files are looked for:
/etc/tremor/logger.yamla log4rs configuration file to control logging in tremor.
*.tricklefiles in this directory and its subdirectories will be loaded as trickle pipelines - trickle pipelines are always loaded before yaml configuration! The file name in this case (the part before
.trickle) is the pipeline id which you can use in places like the binding configuration. The
#!config id = "name"preprocessor directive can be used to overwrite the naming.
*.yamlfiles will be loaded as configuration files and evaluated in order (so mappings cannot refer to artefacts in later files!) - NOTE: defining pipelines in yaml is deprecated and trickle pipelines should be used.
By default tremor is looking into
/usr/local/share/tremor for custom
tremor-script modules and libraries that can be included in your scripts and
trickle files via
tremor-script modules and
tremor-query modules. To adapt the places tremor is looking for your modules, append to the
TREMOR_PATH environment variable:
Static or Bootstrap deployments
Static or Bootstrap deployment allows tremor to be configured at startup with its registry and repository pre-populated with out of the box user defined configuration.
For example, in the following example, tremor is started with a registry and repository that runs a micro benchmark on startup
target/debug/tremor server run -f repo.yaml reg.yaml
# File: repo.yaml
# define a blaster that replays data from an archived JSON log file
- id: blaster
# define a blackhole that runs a 40 second benchmark and then stops
- id: blackhole
# define a passthrough pipeline for benchmarking
- id: main
- id: passthrough
# define how blaster, the pipeline under test, and blackhole interconnect
- id: default
Static deployments support multiple configurations. You can specify multiple configurations by using a list in the config instead of specifying one item as illustrated below:
- id: kafka-in
- id: kafka-again
This file 'loads' the repository on startup with various specifications or templates, but it doesn't do anything. For that, we need to define one or many instances for tremor to deploy. This is done in the reg.yaml file:
/binding/default/01: # deployment '01'
instance: "01" # .. deploys, blaster/01, blackhole/01, pipeline/main/01
Interactive or Operational deployments
Tremor's registry and repository can be configured dynamically via tremor's REST API, and via the tremor-cli tool. For example:
tremor api onramp create blaster.yaml
tremor api offramp create offramp.yaml
tremor api pipeline create main.yaml
tremor api binding create benchmark.yaml
Or via curl:
curl --data-binary @deployment.yaml http://localhost:9898/binding/default/01