Skip to main content
Version: 0.11

Recipes

This document provides a few recipes for common patterns in tremor script. Please note however that it neither is exhaustive nor should those patterns considered the 'only way' to perform certain tasks.

Extracting a raw message#

If the event is a unstructured / raw message parsing it can be tricky since we can not match over a string. The following code offers a solution to it:

# event = "John Doe"
let event =  match {"message": event} of  case r = %{ message ~= dissect|%{first} %{last}| } => r.message  default => dropend;# event = {"first" : "John", "last": "Doe"}

Appending to an array#

When appending to an array we can use the array::push function

use std::array;# event = {"key": "value", "tags": ["tag1"]}let event.tags = array::push(event.tags, "tag2");# event = {"key": "value", "tags": ["tag1", "tag2"]}

Validating over extracted data#

Sometimes we want to validate over extracted data without forcing the extraction to be a regular expression. For validations like the one below this pattern can be used.

use std::array;match event of  # ...  case r = %{message ~= dissect|%{log_level} %{log_timestamp}: %{logger}: %{message}|} when array::contains(["ERROR", "WARN", "INFO", "DEBUG"], result.message.log_level) => let event = merge event of r.message end  # ...end

Here we extract the log_level and validate of that the it is one of ERROR, WARN, INFO or DEBUG by moving the check into the when guard we don't need to use a regular expression for this validation instead can use array membership.

Replacing a field with an extraction#

When extracting a field to merge with with the event and wanting to remove the extracted field we can take advantage of the merge expressions behaviour that it will treat null in merged records as a command to delete the data by setting the field to replace to null before merging.

# event = %{"message": "John Doe"}let event = merge event of match event of  case r = %{message ~= dissect|%{first} %{last}| } => let r = r.message, let r.message = null, r  default => {}end;# event = %{"first": "John", "last": "Doe"}

No effect on non matching case#

If we use merge with match we can make the default case to have no effect by using {}. This is possible since merge on {} is a identity function.

# event = %{"message": "John Doe"}let event = merge event of match event of  case r = %{message ~= dissect|%{first} %{midle} %{last}| } => let r = r.message, let r.message = null, r  default => {}end;# event = %{"message": "John Doe"}

Boolean decisions#

To make boolean decisions we can match on true or false.

use std::type;
match type::is_record(event) of  case true => let event_type = "record"  case false => let event_type = "other"end

Diverting an event to a different channel#

By default the tremor-script operator forwards all events that are not dropped to the out channel for further processing. However it is possible to route events to different channels using the emit keyword. This allows, for example, diverting certain events to reserve bandwidth for a more important subset.

match event.importance of  case "high" => emit # this is the same as emit event => "out"  case "medium" => emit event => "divert"  default => drop # deletes the eventend

The 'null default'#

When the result of a match statement isn't needed - as in we use it purely for it's side effects - and we want the default to have no effect we can use null here.

match event of  case %{ tags ~= ["high-priority"]} => let event.importance = "high"  default => nullend

Testing against the type of a field#

Sometimes we want to know if a field has a certain type. The type module provides help here but common types such as record or array can be checked using their patterns.

match event of  case %{field ~= %{}} => emit "event.field is a record"  case %{field ~= %[]} => emit "event.field is a array"  case %{} when type::is_record(event.field) => emit "event.field is a record"  case %{} when type::is_array(event.field) => emit "event.field is a array"  case %{} when type::is_number(event.field) => emit "event.field is a number (float or integer)"  case %{} when type::is_integer(event.field) => emit "event.field is an integer"  case %{} when type::is_float(event.field) => emit "event.field is a float"  case %{} when type::is_null(event.field) => emit "event.field is null (but set)"  case %{absent field} => emit "event.field is not set"  # ...end

Routing messages#

Tremor script can be used to route messages by combining the emit feature and the fact that the tremor runtime operator allows different outputs.

To route to doing a blue / green split based on a field in a record we could use the following code:

match event of  case %{key == "blue"} => emit event => "blue"  case %{key == "green"} => emit event => "green"  default => dropend
define script split_scriptscript  # see aboveend;create script split_script;
select event from split_script/blue into out/blue;select event from split_script/green into out/green;

Percentage drops of events#

To drop a percentage of all events, functions in the random module can be used.

We generate a random number in a range and based on the outcome, we decide whether we want to drop an event or not. Example:

# drop 50% of the eventsmatch random::integer(0, 100) < 50 of  case true => drop  default => nullend

Most of the time, we want to do this only for certain matching events (as opposed to all events).

let random_number = random::integer(0, 100);match event of  case %{key == "blue"} when random_number < 25 => drop   # drop 25% of blue events  case %{key == "yellow"} when random_number < 75 => drop # drop 75% of yellow events  case %{key == "red"} => drop                            # drop 100% of red events  default => null                                         # drop 0% of other eventsend

Check if a variable is present/absent#

To check if a variable is present, we can rely on the present keyword (and inversely, absent).

# matches default casematch present non_existent_var of  case true => "is present"  default => "not present"end;

Note that this is different from the case where a variable is set to null, for which we can do function-based checks as well as pattern-match with match.

Using non-existent variables in contexts other than present or absent will throw an error terminating the script, so this is useful for guarding against that when needed. This is especially useful when working with meta variables as part of tremor runtime, where -- as part of a pipeline node -- we may need to check if a certain meta variable is set or not (eg: from a previous pipeline node) and act accordingly. For such needs, the approach above can be used. Alternatively, we can also rely on record patterns there:

# tests for presence of $keymatch $ of  case %{present key} => "present"  default => "not present"end;

Since $ gives a record with all the meta variable name-value mapping, this works nicely.