Full Grammar

Rule ModuleFile

The ModuleFile rule defines a module in tremor.

A module is a unit of compilation.

rule ModuleFile ::=
    ModuleBody 
  ;

Represents the compiled form of a tremor source file.

Currently this can be troy deployment files, trickle query files, or tremor script files.

This is part of of tremor's compiler and runtime and not user modifiable.

Rule ModuleBody

The ModuleBody rule defines the structure of a valid module in tremor.

Modules begin with optional module comments.

Modules MUST define at least one statement, but may define many.

Statements are ; semi-colon delimited.

rule ModuleBody ::=
    ( ModComment ) ?  ModuleStmts 
  ;

The body of a compiled tremor source file has a module level comment ( lines beginning with 3 hashes ) and the statements that form the logic of that module.

This is part of of tremor's compiler and runtime and not user modifiable.

Rule ModComment

The ModComment rule specifies module comments in tremor.

Documentation comments for modules are optional.

A module documentation comment begins with a ### triple-hash and they are line delimited.

Muliple successive comments are coalesced together to form a complete comment.

The content of a module documentation comment is markdown syntax.

rule ModComment ::=
     '<mod-comment>' 
  | ModComment  '<mod-comment>' 
  ;

Example

Module level comments are used throughout the tremor standard library and used as part of our document generation process.

Here is a modified snippet from the standard library to illustrate

### The tremor language standard library it provides the following modules:
###
### * [array](std/array.md) - functions to deal with arrays (`[]`)
### * [base64](std/base64.md) - functions for base64 en and decoding
### * [binary](std/base64.md) - functions to deal with binary data (`<< 1, 2, 3 >>`)
### * [float](std/float.md) - functions to deal with floating point numbers
### * [integer](std/integer.md) - functions to deal with integer numbers
### * [json](std/json.md) - functions to deal with JSON
...

Rule DocComment

The DocComment rule specifies documentation comments in tremor.

Documentation comments are optional.

A documentation comment begins with a ## double-hash and they are line delimited.

Muliple successive comments are coalesced together to form a complete comment.

The content of a documentation comment is markdown syntax.

rule DocComment ::=
     '<doc-comment>' 
  | DocComment  '<doc-comment>' 
  ;

Example

Documentation level comments are used throughout the tremor standard library and used as part of our document generation process.

Here is a modified snippet from the standard library to illustrate

## Returns the instance of tremor.
##
## Returns a `string`
intrinsic fn instance() as system::instance;
...

This is a builtin function implemented in rust and used in a script as follows:

use tremor::system;

system::instance()

Rule ModuleStmts

The ModuleStmts rule defines a set of module statements.

Module statements are a ; semi-colon delimited set of ModuleStmt rules

rule ModuleStmts ::=
    ModuleStmt  ';' ModuleStmts 
  | ModuleStmt  ';' ?  
  ;

The set of computed statements in a compiled script.

This is part of of tremor's compiler and runtime and not user modifiable.

Rule ModuleStmt

The ModuleStmt rule defines the statement types that are valid in a tremor module.

rule ModuleStmt ::=
    Use 
  | Const 
  | FnDefn 
  | Intrinsic 
  | DefineWindow 
  | DefineOperator 
  | DefineScript 
  | DefinePipeline 
  | DefineConnector 
  | DefineFlow 
  ;

The set of statements that are legal in any tremor supported domain specific language.

This is part of of tremor's compiler and runtime and not user modifiable.

Rule ConfigDirectives

The ConfigDirectives rule allows line delimited compiler, interpreter or runtime hints to be specified.

rule ConfigDirectives ::=
    ConfigDirective ConfigDirectives 
  | ConfigDirective 
  ;

See ConfigDirective for supported directives.

Rule ConfigDirective

A ConfigDirective is a directive to the tremor runtime.

Directives MUST begin on a new line with the #!config shebang config token.

rule ConfigDirective ::=
     '#!config' WithExpr 
  ;

Providing a metrics internal via a config directive

# Enable metrics with a 10 second interval
#!config metrics_interval_s = 10

Rule Use

Imports definitions from an external source for use in the current source file.

The contents of a source file form a module.

Multiple modules that share a common root can be combined into a single use clause by using use some::root::{m1, m2::sub_module::{m3, m4}}

TREMOR_PATH

The TREMOR_PATH environment path variable is a : delimited set of paths.

Each path is an absolute or relative path to a directory.

When using relative paths - these are relative to the working directory where the tremor executable is executed from.

The tremor standard library MUST be added to the path to be accessible to scripts.

rule Use ::=
     'use' UseTarget 
  ;

Modules

Modules can be scripts. Scripts can store function and constant definitions.

Scripts are stored in .tremor files.

Modules can be queries. Queries can store window, pipeline, script and operator definitions.

Scripts are stored in .trickle files.

Modules can be deployments. Deployments can store connector, pipeline and flow definitions.

Deployments are stored in .troy files.

Conditioning

Modules in tremor are resolved via the TREMOR_PATH environment variable. The variable can refer to multiple directory paths, each separated by a : colon. The relative directory structure and base file name of the source file form the relative module path.

Constraints

It is not recommended to have overlapping or shared directories across the set of paths provided in the tremor path.

It is not recommended to have multiple definitions mapping to the same identifier.

Rule UseTarget

rule UseTarget ::=
    MaybeAliased 
  | ModPath  '::' MaybeAliased 
  | ModPath  '::'  '{' TargetList  '}' 
  ;

Rule TargetList

rule TargetList ::=
    TargetElement 
  | TargetElement  ',' TargetList 
  ;

Rule TargetElement

rule TargetElement ::=
    MaybeAliased 
  | ModPath  '::' MaybeAliased 
  | ModPath  '::'  '{' TargetList  '}' 
  ;

Rule MaybeAliased

rule MaybeAliased ::=
    Ident 
  | Ident  'as' Ident 
  ;

Rule ArgsWithEnd

The ArgsWithEnd rule defines an arguments block with an end block.

rule ArgsWithEnd ::=
    ArgsClause ?  WithEndClause 
  | 
  ;

An internal rule that defines an optional args block with and optional end token.

This rule is used and shared in other rules as part of their definitions.

Rule DefinitionArgs

The DefinitionArgs rule defines an arguments block without an end block.

rule DefinitionArgs ::=
    ArgsClause ?  
  ;

An optional argument block

args arg1, arg 2

This is a shared internal rule used in other rules as part of their definition.

Rule ArgsClause

The ArgsClause rule marks the beginning of an arguments block.

A valid clause has one or many argument expressions delimited by a ',' comma.

rule ArgsClause ::=
     'args' ArgsExprs 
  ;

args
  x = y

Rule ArgsExprs

The ArgsExpr rule is a macro rule invocation based on the Sep separator macro rule.

An args expression is a comma delimited set of argument expressions.

rule ArgsExprs ::=
    Sep!(ArgsExprs, ArgsExpr, ",") 
  ;

Rule ArgsExpr

The ArgExpr rule specifies argument lists.

An argument can be an Ident or an assignment of the form <Ident> = <Expr>

rule ArgsExpr ::=
    Ident  '=' ExprImut 
  | Ident 
  ;

As used in deployment rules to set or override arguments specifications. Arguments specifications define interface parameters that must be set by default ( or overridden ) for something to be well defined.

The tremor runtime checks for ommissions and produces an error for attempted instanciations that omit to provide a value for specified arguments.

Rule CreationWithEnd

The CreationWithEnd rule defines a with block of expressions with a terminal end keyword.

rule CreationWithEnd ::=
    WithEndClause 
  | 
  ;

with x = y end

Rule CreationWith

The CreationWith rule defines an optional with block of expressions without a terminal end keyword.

rule CreationWith ::=
    WithClause 
  | 
  ;

Rule WithClause

The WithClause rule defines a with block with a , comma delimited set of WithExpr rules.

rule WithClause ::=
     'with' WithExprs 
  ;

with x = y

Rule WithEndClause

The WithEndClause rule defines a with clause with an end terminal token.

rule WithEndClause ::=
    WithClause  'end' 
  ;

with x = y end

Rule WithExprs

The WithExprs rule defines a , comma delimited set of WithExpr rules.

rule WithExprs ::=
    Sep!(WithExprs, WithExpr, ",") 
  ;

Rule WithExpr

The WithExpr rule defines a name value binding.

rule WithExpr ::=
    Ident  '=' ExprImut 
  ;

Form

name = <value>

Where:

• name is an identifier.
• <value> is any valid immutable expression.

Example

snot = "badger"

Rule ModularTarget

A ModularTarget indexes into tremor's module path.

In tremor a module is a file on the file system.

A module is also a unit of compilation.

A ModularTarget is a :: double-colon delimited set of identifiers.

Leading :: are not supported in a modular target..

Trailing :: are not supported in a modular target.

rule ModularTarget ::=
    Ident 
  | ModPath  '::' Ident 
  ;

Examples

Loading and using a builtin function

# Load the base64 utilities
use std::base64;

# Base64 encode the current `event`.
base64::encode(event)

Loading and using a builtin function with an alias

# Load the base64 utilities
use std::base64 as snot;

# Base64 encode the current `event`.
snot::encode(event)

Rule Deploy

The Deploy rule defines the logical entry point into Tremor's command oriented deployment syntax. The deployment grammar defines units of deployment that the runtime manages on behalf of users.

The grammar embeds the statement oriented query syntax and expression oriented scripting syntax where appropriate.

A legal deployment is composed of:

• An optional set of module comments
• A sequence of top level expressions. There must be at least one defined.
• An optional end of stream token.

At least one deploy command.

rule Deploy ::=
    ConfigDirectives ( ModComment ) ?  DeployStmts 
  | ( ModComment ) ?  DeployStmts 
  ;

define flow test
flow
  use std::time::nanos;
  define connector metronome from metronome
  with
    config = {
      "interval": nanos::from_seconds(1)
    }
  end;
  define connector exit from exit;
  define pipeline identity
  args
    snot = "badger",
  pipeline
    select args.snot from in into out;
  end;
  create connector metronome;
  create connector exit;
  create pipeline identity with
    snot = "snot"
  end;
  connect /connector/metronome to /pipeline/identity;
  connect /pipeline/identity to /connector/exit;
end;

deploy flow test

Rule DeployStmts

The DeployStmts rule defines the statements that are legal in a deployment module.

Statements in a deployment modules are ; semi-colon delimited.

There MUST be at least one.

There MAY be more than one.

rule DeployStmts ::=
    DeployStmt  ';' DeployStmts 
  | DeployStmt  ';' ?  
  ;

Rule DeployStmt

The DeployStmt rule constrains the statements that are legal in a .troy deployment module.

Importing modules via the use clause is allowed.

Flow definitions and deploy commands are allowed.

rule DeployStmt ::=
    DefineFlow 
  | DeployFlowStmt 
  | Use 
  ;

Rule DeployFlowStmt

The DeployFlowStmt rule defines the content of a command to tremor to deploy a flow.

The flows can be re-parameterized through overriding the default parameters set in their originating source definitions via a with clause.

rule DeployFlowStmt ::=
    ( DocComment ) ?   'deploy'  'flow' Ident  'from' ModularTarget CreationWithEnd 
  | ( DocComment ) ?   'deploy'  'flow' Ident CreationWithEnd 
  ;

define flow test
flow
  use std::time::nanos;
  define connector metronome from metronome
  with
    config = {
      "interval": nanos::from_millis(1)
    }
  end;
  define connector exit from exit;
  define pipeline identity
  pipeline
    select event from in into out;
  end;
  create connector metronome;
  create connector exit;
  create pipeline identity;
  connect /connector/metronome to /pipeline/identity;
  connect /pipeline/identity to /connector/exit;
end;

# The `deploy` statements commands tremor to instanciate the flow `test` - the flow in turn
# will result in a metronome, exit connector, pipeline to be started and interconnected as
# per the `test` definition above
deploy flow test

Rule ConnectorKind

The ConnectorKind rule identifies a builtin connector in tremor.

Connectors in tremor are provided by the runtime and builtin. They can be resolved through an identifier.

Examples

The http_server identifies a HTTP server connector.

The metronome identifies a periodic metronome.

rule ConnectorKind ::=
    Ident 
  ;

The name of a builtin connector implemented in rust provided by the tremor runtime.

Rule FlowStmts

The FlowStmts rule defines a mandatory ; semi-colon delimited sequence of FlowStmtInner rules.

rule FlowStmts ::=
    FlowStmts_ 
  ;

Rule FlowStmts_

The FlowStmts_ rule defines a ; semi-colon delimited sequence of FlowStmtInner rules.

rule FlowStmts_ ::=
    Sep!(FlowStmts_, FlowStmtInner, ";") 
  ;

See FlowStmts rule for details.

This rule wraps away a lalrpop macro call for ease of reference in other rules in the grammar source.

Rule CreateKind

The CreateKind rule encapsulates the artefact types that can be created in the tremor deploymant language.

rule CreateKind ::=
     'connector' 
  |  'pipeline' 
  ;

An internal rule that allows connectors and pipelines to be created and used by the Create rule.

Rule FlowStmtInner

The FlowStmtInner rule defines the body of a flow definition.

rule FlowStmtInner ::=
    Define 
  | Create 
  | Connect 
  | Use 
  ;

• A pipeline or connector definition
• A pipeline or connector instance via create
• A use statement to import a definition
• A connect statement to interlink instances

Rule Define

The Define rule allows connectors and pipelines to be specified.

rule Define ::=
    DefinePipeline 
  | DefineConnector 
  ;

Within a flow definition, allows pipeline and connectors to be defined.

Rule Create

The Create rule creates instances of connectors and pipelines in a flow.

rule Create ::=
     'create' CreateKind Ident  'from' ModularTarget CreationWithEnd 
  |  'create' CreateKind Ident CreationWithEnd 
  ;

Create a connector

create connector foo from snot::foo end;

Create a pipeline

create pipeline bar from badger::bar end;

Rule Connect

The Connect rule defines routes between connectors and pipelines running in a flow.

rule Connect ::=
     'connect'  '/' ConnectFromConnector  'to'  '/' ConnectToPipeline 
  |  'connect'  '/' ConnectFromPipeline  'to'  '/' ConnectToConnector 
  |  'connect'  '/' ConnectFromPipeline  'to'  '/' ConnectToPipeline 
  ;

Defines how to interconnect pipeline and connectors

Given

create connector ingest;
create connector egress;
create pipeline logic;

connect /connector/ingress/out to /pipeline/logic/in;
connect /pipeline/logic/out to  /connector/egress/in;

Defines how the ingress, egress and logic runtime instances are interconnected for data to flow through them in a specified order.

Rule ConnectFromConnector

The ConnectFromConnector rule defines a route from a connector instance.

rule ConnectFromConnector ::=
     'connector'  '/' Ident MaybePort 
  ;

connector/console/out

Connection from the console connector definition via the standard out port.

connector/console

The shorthand form where the standard out port is implied can also be used.

Rule ConnectFromPipeline

The ConnectFromPipeline rule defines route from a pipeline instance.

rule ConnectFromPipeline ::=
     'pipeline'  '/' Ident MaybePort 
  ;

pipeline/filter/out

Connection from the filter pipeline definition via the standard out port.

pipeline/console

The shorthand form where the standard out port is implied can also be used.

Rule ConnectToPipeline

The ConnectToPipeline rule defines route to a pipeline instance.

rule ConnectToPipeline ::=
     'pipeline'  '/' Ident MaybePort 
  ;

pipeline/filter/in

Connection to the filter pipeline definition via the standard in port.

pipeline/console

The shorthand form where the standard in port is implied can also be used.

Rule ConnectToConnector

The ConnectToConnector rule defines a route to a connector instance.

rule ConnectToConnector ::=
     'connector'  '/' Ident MaybePort 
  ;

connector/console/in

Connection to the console connector definition via the standard in port.

connector/console

The shorthand form where the standard in port is implied can also be used.

Rule DefineConnector

The DefineConnector rule defines a connector.

A connector is a runtime artefact that allows tremor to connect to the outside world, or for the outside connector to connect to tremor to send and/or receive data.

The named connector can be parameterized and instanciated via the Create rule

rule DefineConnector ::=
    ( DocComment ) ?   'define'  'connector' Ident  'from' ConnectorKind ArgsWithEnd 
  ;

use std::time::nanos;
define connector metronome from metronome
  with
    config = {
      "interval": nanos::from_millis(1)
    }
end;

Define user defind connector metronome from the builtin metronome connector using a 1 second periodicity interval.

define connector exit from exit;

Define user dfeind connector exit from the builtin exit connector with no arguments specified.

Rule DefineFlow

The DefineFlow rule defines a flow.

A flow is a runtime artefact that informs tremor how to interconnect and launch instances of pipelines and connectors.

A flow can define or use multiple in scope and already define pipelines and connectors and interconnect their streams together.

rule DefineFlow ::=
    ( DocComment ) ?   'define'  'flow' Ident DefinitionArgs  'flow' FlowStmts  'end' 
  ;

define flow test
flow
  use std::time::nanos;
  define connector metronome from metronome
  with
    config = {
      "interval": nanos::from_millis(1)
    }
  end;
  define connector exit from exit;
  define pipeline identity
  args
    snot = "badger",
  pipeline
    select args.snot from in into out;
  end;
  create connector metronome;
  create connector exit;
  create pipeline identity;
  connect /connector/metronome to /pipeline/identity;
  connect /pipeline/identity to /connector/exit;
end;

deploy flow test

A flow definition is a runnable or executable streaming program that describes the connectivity, the logic and how they are interconnected. A deploy statement is responsible for the actual deployment.

Rule Query

The Query rule defines the logical entry point into Tremor's statement oriented query grammar. The grammar is embedded into deployments via define pipeline statements.

Pipelines effectively provide a continous streaming abstraction for event processing. The pipeline query is a graph of connected streams and operators. The operators can be builtin or provided by users through the script operator.

Window definitions for use in windowed statements such as those supported by the select operation are also defined and named so that they can be used by multiple operations and to compose tilt frames - chains of successive windows of data in a streaming continuous select operation.

The syntax supports the definition of sub queries through the same define pipeline syntax as in the deployment grammar.

rule Query ::=
    ConfigDirectives Stmts 
  | Stmts 
  ;

Rule Stmts

The Stmts rule defines a ; semi-colon delimited sequence of Stmt rules.

rule Stmts ::=
    Stmt  ';' Stmts 
  | Stmt  ';' ?  
  ;

Example for a query pipeline

use tremor;
create stream snot;
create stream badger;

# ...

Rule Stmt

The Stmt rule defines the legal statements in a query script.

Queries in tremor support:

• Defining named window, operator, script and pipeline definitions.
• Creating node instances of stream, pipeline, operator and script operations.
• Linking nodes togther to form an execution graph via the select operation.

rule Stmt ::=
    Use 
  | DefineWindow 
  | DefineOperator 
  | DefineScript 
  | DefinePipeline 
  | CreateOperator 
  | CreateScript 
  | CreatePipeline 
  | CreateStream 
  | OperatorSelect 
  ;

Rule DefineWindow

The DefineWindow rule defines a temporal window specification.

A window is a mechanism that caches, stores or buffers events for processing over a finite temporal range. The time range can be based on the number of events, the wall clock or other defined parameters.

The named window can be instanciated via operations that support windows such as the select operation.

rule DefineWindow ::=
    ( DocComment ) ?   'define'  'window' Ident  'from' WindowKind CreationWith ( ScriptState ) ?  ( NamedScript ) ?  ( EmbeddedScriptContent ) ?   'end' 
  ;

define window four from tumbling
with
  size = 4
end;

Rule DefineOperator

The DefineOperator rule defines an operator.

An operator is a query operation composed using the builtin operators provided by tremor written in the rust programming language.

The named operator can be parameterized and instanciated via the CreateOperator rule

rule DefineOperator ::=
    ( DocComment ) ?   'define'  'operator' Ident  'from' OperatorKind ArgsWithEnd 
  ;

define pipeline subq
pipeline
  define operator counter from generic::counter;
  create operator counter;
  select event from in into counter;
  select event from counter into out;
end;

create pipeline subq;

select event from in into subq;
select event from subq into out;

Uses the builtin counter sequencing operator to numerate a stream.

Rule DefineScript

The DefineScript rule defines a named operator based on a tremor script.

A script operator is a query operation composed using the scripting language DSL rather than the builtin operators provided by tremor written in the rust programming language.

The named script can be parameterized and instanciated via the CreateScript rule

rule DefineScript ::=
    ( DocComment ) ?   'define'  'script' Ident DefinitionArgs ScriptState ?  NamedScripts 
  ;

define operator bucket from grouper::bucket;

define script categorize
script
  let $rate = 1;
  let $dimensions = event.logger_name;
  let $class = "test";
  event
end;

create operator bucket;
create script categorize;

select event from in into categorize;
select event from categorize into bucket;
select event from bucket into out;

Rule ScriptState

rule ScriptState ::=
     'state' ExprImut 
  ;

Rule NamedScripts

rule NamedScripts ::=
    NamedScript NamedScripts 
  | EmbeddedScript 
  ;

Rule NamedScript

rule NamedScript ::=
     'script'  'from' Ident TopLevelExprs 
  ;

Rule DefinePipeline

The DefinePipeline rule creates a named pipeline.

A pipeline is a query operation composed using the query langauge DSL instead of a builtin operation provided by tremor written in the rust programming language.

The named pipeline can be parameterized and instanciated via the CreatePipeline rule

rule DefinePipeline ::=
    ( DocComment ) ?   'define'  'pipeline' Ident (  'from' Ports ) ?  (  'into' Ports ) ?  DefinitionArgs Pipeline 
  ;

define pipeline identity
pipeline
  select event from in into out;
end;

Rule OperatorSelect

The OperatorSelect rule provides the select statement in streaming queries.

The statement has

• A target expression
• A stream and port from which to consume events
• A stream and port to which synthetic events are produced
• An optional set of iwndow definitions
• An optional where filter
• An optional having filter
• An optional group by

Unlike ANSI-ISO SQL select operations in tremor do not presume tabular or columnar data. The target expression can be any well-formed and legal value supported by tremor.

rule OperatorSelect ::=
     'select' ComplexExprImut  'from' StreamPort ( WindowClause ) ?  ( WhereClause ) ?  ( GroupByClause ) ?   'into' StreamPort ( HavingClause ) ?  
  ;

The builtin select operator in queries.

A simple non-windowed non-grouped select

select event from in into out;

A simple non-windowed grouped select

select event from in into out group by event.key;

A windowed grouped select operation

select event from in[one_sec] by event.key into out;

Multiple windows can be configured in lower resolutions for multi-resolution windowed expressions where lower resolutions are merged into from higher resolution windows

select aggr::stats::hdr(event.count) form in[one_sec, fifteen_sec, one_min, one_hour] into out;

Rule CreateStream

The CreateStream allows users to create user defined streams beyond the basic buitin set of in, out and err provided by the runtime for a pipeline query.

rule CreateStream ::=
     'create'  'stream' Ident 
  ;

create stream ctrl;

Creates a user defined stream by the provided name ctrl

Rule CreateScript

The CreateScript rule creates an operator based on a tremor script.

A script operator is a query operation composed using the scripting language DSL rather than the builtin operators provided by tremor written in the rust programming language.

The rule causes an instance of the referenced script definition to be created an inserted into the query processing execution graph.

rule CreateScript ::=
     'create'  'script' Ident CreationWithEnd 
  |  'create'  'script' Ident  'from' ModularTarget CreationWithEnd 
  ;

define operator bucket from grouper::bucket;

define script categorize
script
  let $class = "test";
  let $dimensions = [event.type, event.application];
  let $rate = 1;
  event;
end;

create operator bucket;

create script categorize;

select event from in into categorize;

select event from categorize into bucket;

select event from bucket into out;

Rule CreateOperator

The CreateOperator rule creates an operator.

An operator is a query operation composed using the builtin operators provided by tremor written in the rust programming language.

The rule causes an instance of the referenced operator definition to be created an inserted into the query processing execution graph.

rule CreateOperator ::=
     'create'  'operator' Ident CreationWithEnd 
  |  'create'  'operator' Ident  'from' ModularTarget CreationWithEnd 
  ;

# Define a round robin operator with 3 slots
define operator roundrobin from qos::roundrobin
with
  outputs = ["one", "two", "three"]
end;

# create an instance of the operator
create operator roundrobin;

# Filter all inbound events into the rond robin
select event from in into roundrobin;

# Union slots inot outbound port
select event from roundrobin/one into out;
select event from roundrobin/two into out;
select event from roundrobin/three into out;

Rule CreatePipeline

The CreatePipeline rule creates a pipeline.

A pipeline is a query operation composed using the query langauge DSL instead of a builtin operation provided by tremor written in the rust programming language.

The rule causes an instance of the referenced pipeline definition to be created an inserted into the query processing execution graph.

rule CreatePipeline ::=
     'create'  'pipeline' Ident CreationWithEnd 
  |  'create'  'pipeline' Ident  'from' ModularTarget CreationWithEnd 
  ;

# Define a pipeline called `identity`
define pipeline identity
pipeline
  select event from in into out;
end;

# Create an instance of the pipeline
create pipeline identity

Rule MaybePort

The MaybePort rule defines an optional Port.

rule MaybePort ::=
    (  '/' Ident ) ?  
  ;

When interconnecting pipelines and connectors in flow definitions default ports can be inferred by the tremor runtime.

When an alternate port is required, the port specification can be used to explicitly select from available inbound or outbound ports.

Rule StreamPort

The StreamPort rule defines a stream by name with an optional named Port.

When the Port is omitted, tremor will internally default the Port to the appropriate in or out port. Where the err or user defined Ports are preferred, the optional Port specification SHOULD be provided.

rule StreamPort ::=
    Ident MaybePort 
  ;

in/snot

Within a query, allows the port of a specific stream to be referenced directly.

Rule WindowKind

Tumbling

A tumbling window defines a wall-clock-bound or data-bound window of non-overlapping time for storing events. The windows can not overlap, and there are no gaps between windows permissible.

Sliding

A sliding window defines a wall-clock-bound or data-bound window of events that captures an intervalic window of events whose extent derives from the size of the window. A sliding window of size 2 captures up to to events. Every subsequent event will evict the oldest and retain the newest event with the previous ( now oldest ) event.

Conditioning

Both kinds of window store events in arrival order

rule WindowKind ::=
     'sliding' 
  |  'tumbling' 
  ;

Currently only tumbling is implemented.

Rule WindowClause

The WindowClause rule defines an optional window definition for a supporting operation.

rule WindowClause ::=
    WindowDefn 
  ;

Rule Windows

The Windows rule defines a sequence of window definitions that are , comma delimited.

rule Windows ::=
    Windows_ 
  ;

Wraps the Windows_ internal rule for other rules to consume in their definitions

Rule Windows_

The Windows_ rule defines a sequence of window definitions that are , comma delimited.

rule Windows_ ::=
    Sep!(Windows_, Window, ",") 
  ;

A comma delimited set of window references.

Windows can be local or modular

win, my_module::one_sec, my_module::five_sec

The identifers refer to a window definition and can be used in operators to define a temporally bound set of events based on the semantics of the window definition.

In a tilt frame - or set of windows, the output of a window can is the input the next window in a sequence. This is a form of temporal window-driven event compaction that allows memory be conserved.

At 1000 events per second, a 1 minute window needs to store 60,000 events per group per second. But 60 1 second windows can be merged with aggregate functions like dds and hdr histograms.

Say, each histogram is 1k of memory per group per frame - that is a cost of 2k bytes per group.

In a streaming system - indefinite aggregation of in memory events is always a tradeoff against available reosurces, and the relative business value.

Often multiple windows in a tilt frame can be more effective than a single very long lived window.

Rule Window

The Window rule defines a modular target to a window definition.

rule Window ::=
    ModularTarget 
  ;

Rule WindowDefn

The WindowDefn defines a temporal basis over which a stream of events is applicable.

rule WindowDefn ::=
     '[' Windows  ']' 
  ;

Operations supporting windowed aggregate functions in tremor such as the select statement can window incoming streams in the from clause:

select aggr::count(event) from in[one_second, ten_second]
...

Here, we stream events from the in stream into a one_second window. Every second, we stream the aggregate result from the one second window into the ten_second window.

So, even if we have 1 million events per second, the one_second and ten_second windows will convert the event firehose into a trickle. Fun fact: this pun is where the query language got its name from.

Rule WhereClause

The WhereClause defines a predicate expression used to filter ( forward or discard ) events in an operation.

The where clause is executed before a operation processes an event.

rule WhereClause ::=
     'where' ComplexExprImut 
  ;

select event from in
where present event.important
into out

The where filters events before computations occur upon them in operators that support the clause. Any predicate ( boolean ) expression can be used in a where filter.

Rule HavingClause

The HavingClause defines a predicate expression used to filter ( forward or discard ) events in an operation.

The having clause is executed after an operation has processed an event.

rule HavingClause ::=
     'having' ComplexExprImut 
  ;

select event from in
having present event.important
into out

The having filters events after computations has occured within them in operators that support the clause. Any predicate ( boolean ) expression can be used in a having filter.

When appropriate, the where clause should be used in preference over the having clause. It is better to filter early before computation occurs when this is practicable or possible.

Rule GroupByClause

The GroupByClause defines the group by clause of a supporting operation in tremor.

An operator that uses a group by clause maintains the operation for each group captured by the grouping dimensions specified in this clause.

rule GroupByClause ::=
     'group'  'by' GroupDef 
  ;

select event from in[one_second]
having present event.important
group by event.priority
into out

The group by clause groups events based on a group expression. Each computed group effectively has its own memory and computation allocated.

For windowed operations the windows are allocated for each group.

These groups and their windows are independant. This means that opening, closing, filling and recycling of windows is by group.

Rule GroupDef

The GroupDef rule defines the parts of a grouping dimension.

Group segments can be derived from:

• Expressions - for which their serialized values are used.
• Set expressions - which computes a set based on an expression.
• Each expressions - which iterates an expression to compute a set.

rule GroupDef ::=
    ExprImut 
  |  'set'  '(' GroupDefs  ')' 
  |  'each'  '(' ExprImut  ')' 
  ;

Example group definitions

A string value

event.string

The serialization of any legal tremor data value

event

A set based on multiple expressions:

set(event.key, state[key])

An set computed from an interation

let keys = ['snot', 'badger', 'goose'];

...

each(keys)

## Rule GroupDefs

The `GroupDefs` rule defines a `,` comma delimited set of `GroupDef` rules.



![GroupDefs](svg/groupdefs.svg)

```ebnf
rule GroupDefs ::=
    GroupDefs_ 
  ;

Wraps a macro call for use by other productions in the grammar

Rule GroupDefs_

The GroupDefs_ rule defines a , comma delimited set of GroupDef rules.

rule GroupDefs_ ::=
    Sep!(GroupDefs_, GroupDef, ",") 
  ;

A comma delimited set of GroupDef productions

Rule EmbeddedScriptImut

The EmbeddedScriptImut rule defines an optional embedded script.

rule EmbeddedScriptImut ::=
     'script' EmbeddedScriptContentImut 
  ;

Rule EmbeddedScriptContentImut

rule EmbeddedScriptContentImut ::=
    ExprImut 
  ;

Rule Ports

The Ports rule defines a , comma delimited set of stream ports.

rule Ports ::=
    Sep!(Ports, <Ident>, ",") 
  ;

A set of ports exposed in pipeline definitions in their from and into clauses

define pipeline example
  from in, out, err, ctrl
  into in, out, err, ctrl
pipeline
  # A pipeline query implementation
  ...
end

The from and into ports do not need to be the same.

Tremor's compiler and runtime can use these definitions to validate deployments are correct, or discover deployments that are invalid. It is an error to send data to or receive data from a pipeline port that is not specified.

Rule OperatorKind

The OperatorKind rule defines a modular path like reference to a builtin tremor operator.

Operators are programmed in rust native code and referenced via a virtual module path.

rule OperatorKind ::=
    Ident  '::' Ident 
  ;

A modular path identifying a builtin operator.

define operator roundrobin from qos::roundrobin;

Rule EmbeddedScript

The EmbeddedScript rule defines a script using the Script DSL [ Full ].

The script is enclosed in script .. end blocks.

rule EmbeddedScript ::=
    EmbeddedScriptContent  'end' 
  ;

script
  event
end

Rule EmbeddedScriptContent

The EmbeddedScriptContent rule defines an embedded script expression.

rule EmbeddedScriptContent ::=
     'script' TopLevelExprs 
  ;

A single expression embedded within an embedded script

Rule Pipeline

The Pipeline rule defines a block of statements in a pipeline .. end block.

The block MAY begin with an optional set of ConfigDirectives.

rule Pipeline ::=
     'pipeline' ConfigDirectives ?  PipelineCreateInner  'end' 
  ;

An internal rule to the DefinePipeline rule where the pipeline logic is provided.

Rule PipelineCreateInner

The PipelineCreateInner is an internal rule of the Pipeline rule.

The rule defines a ; semi-colon delimited set of one or many Stmts.

rule PipelineCreateInner ::=
    Stmt  ';' Stmts 
  | Stmt  ';' ?  
  ;

This rule allows queries to be defined in the context of a pipeline definition.

Rule Script

The Script rule defines the logical entry point into Tremor's expression oriented scripting language. The scripting langauge can be embedded into queries via the script operator. The scripting language is also used to specify configuration of connectors, pipelines, flows, and operators in the query language.

A legal script is composed of:

• An optional set of module comments
• A sequence of top level expressions. There must be at least one defined.
• An optional end of stream token

Scripting Language Entrypoint

This is the top level of the tremor scripting language tremor

rule Script ::=
    ( ModComment ) ?  TopLevelExprs 
  ;

Type system

Tremor supports a data oriented or value based type system with a syntax that is backwards compatible with JSON.

Any well-formed and legal JSON document is a valid literal in tremor.

Tremor literals for null, boolean, string ( utf-8 ), integer ( 64-bit unsigned ), float ( 64-bit ieee ), arrays, and records are equivalent to their JSON counterparts.

Tremor also supports a binary literal for transporting and processing opaque binary data.

Asymmetric

JSON literals are valid tremor value literals.

Tremor literals MAY NOT always be valid JSON literal.

# The following literal is valid JSON and valid Tremor
[1, "snot", {}];

# The following literal is valid in tremor only
[1, "snot", {}, << data/binary >>, ];

Tremor supports comments, JSON does not. Tremor supports trailing commas in arrays and records, JSON does not. Tremor supports binary literal data, JSON does not.

Note: By default, most connectors in tremor serialize to and from json via a codec. The type system in tremor however is agnostic to the wire format of data that flows through tremor. So data originate as json, as msgpack.

Computations

Tremor also supports a rich expression language with the same support for additive, mutliplicate, comparitive, and logical unary and binary expressions as languages like rust and java.

As most of the data that flows through tremor is heirarchically structured or JSON-like tremor also has rich primitives for structural pattern matching, structural comprehension or iterating over data structures.

Loops

Tremor does not support while loop or other primitives that can loop, recurse or iterate indefinitely.

In an event based system, events are streaming continuously - so infinite loops that can block streams from making forward progress are considered harmful.

There are no loops.

We do support iteration over finite arrays.

We do support depth-limited tail recursive functional programming.

Expression oriented

The script processing is expression oriented. This is to say that every structural form supported by tremor returns a data structure as a result.

Event oriented

Scripts in tremor can emit or drop an `event that is being processed.

The event keyword is the subject. It identifies the value currently being processed.

The emit keyword halts processing succesfully with a value.

The drop keyword halts processing by discarding the current event.

Illustrative example

# Propagate events marked as important and convert them to system alerts
match event of
  case %{ present important } => { "alert": event.message }
  case _ => drop
end;

Rule TopLevelExprs

The TopLevelExprs rule defines semi-colon separated sequence of top level tremor expressions with an optional terminating semi-colon

rule TopLevelExprs ::=
    TopLevelExpr  ';' TopLevelExprs 
  | TopLevelExpr  ';' ?  
  ;

The ToplEvelExprs specifies the expressions that are legal at the top level of a script expression.

script
  event.sum * 2
end

A sequence of ';' semi-colon delimited expressions of the following form are permissible:

• Constants
• Function definitions
• Intrinsic function definitions in the standard library
  • Provided by the runtime to document builtins - not user modifiable without a pull request or feature enhancement
• Arbitrary complex expressions
• Use definitions

Rule InnerExprs

The InnerExprs rule defines the expression forms permissible within another containing scope. Like TopLevelExprs, inner expressions are separated by semi-colons. The semi-colon is optional for the last expression in a set of expressions.

At least one expression MUST be provided.

rule InnerExprs ::=
    Expr  ';' InnerExprs 
  | Expr  ';' ?  
  ;

Example

The following basic forms are permissible

for event.object of case (k, v) => v end;
for event.list of case (i, e) => e end;
match event of case %{} => "record" case _ => "not a record" end;
let list = event.list;
drop;
"any literal or basic expression";
1 + 2 * 3;
emit {"snot": "badger" }
# ...

Rule TopLevelExpr

The TopLevelExpr rule specifies the expression forms that are legal at the outer most scope of a tremor script definition.

The legal forms are:

• Use declarations - these allow external modules to be referenced.
• Constant expressions - these are immutable compile time constants.
• Function definitions - these are user defined functions.
• Intrinsic function definitions - these are builtin funtions provided by the runtime.

rule TopLevelExpr ::=
    Const 
  | FnDefn 
  | Intrinsic 
  | Expr 
  | Use 
  ;

Example

In the tremor standard library many of the top level expressions are use definitions importing sub modules from the module path.

use std::array;     # Import the std array utilities
use std::base64     # Import the std base64 utilities;
use std::binary;    # ...
use std::float;
use std::integer;
use std::json;

Rule Const

The Const rule defines a rule that binds an immutable expression to an identifier.

As the value cannot be changed at runtime.

danger

Constants, by convention are ALL_UPPERCASE to distinguish them from variables. While it is allowed to deviate from this convention, it is not recommended and you will be yelled at by the compiler.

rule Const ::=
    ( DocComment ) ?   'const' Ident  '=' ComplexExprImut 
  ;

How do I create new immutable constant variable in tremor?

use std::base64;
const SNOT = "snot";
const BADGER = "badger";
const SNOT_BADGER = { "#{snot}": "#{base64::encode(badger)}" };

Rule Expr

The Expr rule aliases the SimpleExpr rule.

The alias allows higher levels of the DSL such as the rules in the deployment or query language to avoid some of the internal complexity in the scripting language.

Within the scripting DSLs grammar the different forms and variations of expression are significant.

However, in the higher level we limit exposure to a subset of these forms. This is done for convenience, and for consistency of usage, and ease of learning the language.

rule Expr ::=
    SimpleExpr 
  ;

The effective root of the subset of the expression langauge applicable in most processing context in tremor is captured by this rule.

Rule SimpleExpr

The SimpleExpr rule defines all the structural and simple expressions and literals in tremor.

rule SimpleExpr ::=
    Match 
  | For 
  | Let 
  | Drop 
  | Emit 
  | ExprImut 
  ;

Structural expressions

• Match
• For

Assignment

• Let

Flow

• Emit
• Drop

Basic expressions

Binary, unary and literal expressions

Rule AlwaysImutExpr

The AlwaysImutExpr defines the immutable expression forms in tremor.

Immutable expressions can be reduced at compile time and folded into literals.

rule AlwaysImutExpr ::=
    Patch 
  | Merge 
  | Invoke 
  | Literal 
  | Path 
  | Record 
  | List 
  | StringLiteral 
  | BytesLiteral 
  | Recur 
  ;

Rule Recur

The Recur rule defines stack-depth-limited tail-recursion in tremor functions.

rule Recur ::=
     'recur'  '('  ')' 
  |  'recur'  '(' InvokeArgs  ')' 
  ;

fn fib_(a, b, n) of
  case (a, b, n) when n > 0 => recur(b, a + b, n - 1)
  case _ => a
end;

fn fib(n) with
  fib_(0, 1, n)
end;

Tremor's functional programming langauge supports tail recursion via the recur keyword. Tail recursion in tremor is limited to a fixed stack depth - infinite recursion is not permissible.

Rule ExprImut

The ExprImut is the root of immutable expressions in tremor.

rule ExprImut ::=
    OrExprImut 
  ;

The effective root of the subset of the expression langauge applicable in most immutable processing context in tremor is captured by this rule.

Rule OrExprImut

The OrExprImut rule supports logical or expressions in tremor.

Binary logical or expressions take precedence over logical exclusive or expressions.

rule OrExprImut ::=
    BooleanBinOp!(BinOr, ExprImut, XorExprImut) 
  | XorExprImut 
  ;

let example = a or (b & 0);

Rule XorExprImut

The XorExprImut rule supports logical exclusive or expressions in tremor.

Binary logical exclusive or expressions take precedence over logical and expressions.

rule XorExprImut ::=
    BooleanBinOp!(BinXor, XorExprImut, AndExprImut) 
  | AndExprImut 
  ;

let example = a ^ (b & 0);

Rule AndExprImut

The AndExprImut rule supports logical and expressions in tremor.

Binary logical and expressions take precedence over bitwise or expressions.

rule AndExprImut ::=
    BooleanBinOp!(BinAnd, AndExprImut, BitOrExprImut) 
  | BitOrExprImut 
  ;

x + y

Rule BitOrExprImut

The BitOrExprImut rule supports bitwise or expressions in tremor.

Binary bitwise or expressions take precedence over bitwise exclusive or expressions.

rule BitOrExprImut ::=
    BitXorExprImut 
  ;

let example = a | (b & 0);

Rule BitXorExprImut

The BitXorExprImut rule supports bitwise exclusive or expressions in tremor.

Binary bitwise exclusive or expressions take precedence over bitwise and expressions.

rule BitXorExprImut ::=
    BinOp!(BinBitXor, BitXorExprImut, BitAndExprImut) 
  | BitAndExprImut 
  ;

let example = a ^ (b & 0);

Rule BitAndExprImut

The BitAndExprImut rule supports bitwise and expressions in tremor.

Binary bitwise and expressions take precedence over equality expressions.

rule BitAndExprImut ::=
    BinOp!(BinBitAnd, BitAndExprImut, EqExprImut) 
  | EqExprImut 
  ;

let example = a & (b & 0);

Rule EqExprImut

The EqExprImut rule supports equality expressions in tremor.

Binary equality expressions take precedence over comparitive expressions.

rule EqExprImut ::=
    BinOp!(BinEq, EqExprImut, CmpExprImut) 
  | CmpExprImut 
  ;

let example = a == (b & 0);

Rule CmpExprImut

The CmpExprImut rule supports comparative expressions in tremor.

Binary comparative expressions take precedence over bit shift expressions.

rule CmpExprImut ::=
    BinOp!(BinCmp, CmpExprImut, BitShiftExprImut) 
  | BitShiftExprImut 
  ;

x > y

Rule BitShiftExprImut

The BitShiftExprImut rule supports bit shift expressions in tremor.

Binary bit shift expressions take precedence over bitwise additive expressions.

rule BitShiftExprImut ::=
    BinOp!(BinBitShift, BitShiftExprImut, AddExprImut) 
  | AddExprImut 
  ;

let example = a >>> (b & 0);

Rule AddExprImut

The AddExprImut rule supports additive expressions in tremor.

Binary additive expressions take precedence over multiplicative expressions.

rule AddExprImut ::=
    BinOp!(BinAdd, AddExprImut, MulExprImut) 
  | MulExprImut 
  ;

x - y

Rule MulExprImut

The MulExprImut rule supports multiplicative expressions in tremor.

Binary multiplicative expressions take precedence over unary expressions.

rule MulExprImut ::=
    BinOp!(BinMul, MulExprImut, UnaryExprImut) 
  | UnaryExprImut 
  ;

x * y

Rule UnaryExprImut

The UnaryExprImut rule specifies unary expression operations.

Expressions can be marked as + positive, - negative explicitly when needed.

Otherwise, the expression reduces to a simple unary expression.

The simple unary expression has lower precedence.

rule UnaryExprImut ::=
     '+' UnaryExprImut 
  |  '-' UnaryExprImut 
  | UnarySimpleExprImut 
  ;

Is the root expression for unary expressions in the language.

-(1)

Rule UnarySimpleExprImut

The UnarySimpleExprImut rule specifies predicate unary expression operations.

Expressions can be marked explicitly with not or ! to negate the target simple presence expression.

Otherwise, the expression reduces to a simple presence expression.

The simple presence expression has lower precedence.

rule UnarySimpleExprImut ::=
     'not' UnarySimpleExprImut 
  |  '!' UnarySimpleExprImut 
  | PresenceSimplExprImut 
  ;

not (a and b)

Rule PresenceSimplExprImut

The PresenceSimplExprImut rule specifies presence and simple expressions

Expressions path predicate tests based on the present and absent predicate test expressions, or a simple expression.

Otherwise, the expression reduces to a simple expression.

The simple expression has lower precedence.

rule PresenceSimplExprImut ::=
     'present' Path 
  |  'absent' Path 
  | SimpleExprImut 
  ;

present $.request.method

absent state.cache

Rule ComplexExprImut

The ComplexExprImut rule defines complex immutable expression in tremor.

rule ComplexExprImut ::=
    MatchImut 
  | ForImut 
  | ExprImut 
  ;

Rule Intrinsic

The intrinsic rule defines intrinsic function signatures.

This rule allows tremor maintainers to document the builtin functions implemented as native rust code. The facility also allows document generation tools to document builtin intrinsic functions in the same way as user defined functions.

In short, these can be thought of as runtime provided.

For information on how to define user defined functions see the function rule.

rule Intrinsic ::=
    ( DocComment ) ?   'intrinsic'  'fn' Ident  '('  ')'  'as' ModularTarget 
  | ( DocComment ) ?   'intrinsic'  'fn' Ident  '(' FnArgs  ')'  'as' ModularTarget 
  | ( DocComment ) ?   'intrinsic'  'fn' Ident  '(' FnArgs  ','  '.'  '.'  '.'  ')'  'as' ModularTarget 
  | ( DocComment ) ?   'intrinsic'  'fn' Ident  '('  '.'  '.'  '.'  ')'  'as' ModularTarget 
  ;

Example

From our standard library generated documentation, we can see that the base64 encode function is an intrinsic function.

## Encodes a `binary` as a base64 encoded string
##
## Returns a `string`
intrinsic fn encode(input) as base64::encode;

Rule FnDefn

The FnDefn rule allows user defined functions to be defined.

This rule allows tremor users to create functions for reuse in one or many tremor applications.

rule FnDefn ::=
    ( DocComment ) ?   'fn' Ident  '('  '.'  '.'  '.'  ')'  'with' InnerExprs  'end' 
  | ( DocComment ) ?   'fn' Ident  '(' FnArgs  ','  '.'  '.'  '.'  ')'  'with' InnerExprs  'end' 
  | ( DocComment ) ?   'fn' Ident  '('  ')'  'with' InnerExprs  'end' 
  | ( DocComment ) ?   'fn' Ident  '(' FnArgs  ')'  'with' InnerExprs  'end' 
  | ( DocComment ) ?   'fn' Ident  '('  ')'  'of' FnCases  'end' 
  | ( DocComment ) ?   'fn' Ident  '(' FnArgs  ')'  'of' FnCases  'end' 
  ;

Pattern match based function arguments

Functions defined with an of keyword in their signature use pattern matching against arguments

fn fib_(a, b, n) of
  case (a, b, n) when n > 0 => recur(b, a + b, n - 1)
  case _ => a
end;

Ordinary functions

Functions defined with a with keyword in their signature use ordinary arity based matching.

fn fib(n) with
  fib_(0, 1, n)
end;

Function documentation

In modular functions, it is customary to provide user level documentation for the intended users of a function. Here is an example from the tremor standard library

### Trace Identifiers
###
###

use std::type;
use std::binary;
use std::array;
use std::string;

## Is the `trace_id` valid
##
## Checks the `trace_id` argument to see if it is a valid
## trace id. A legal trace id is one of:
##
## * An array of integers in the range of [0..=255] of length 8
## * A binary 16 byte value
## * A 32-byte hex-encoded string
## * An array of 16 int values
## * Regardless of representation, the value must not be all zeroes
##
## Returns a record when the representation is well-formed of the form:
##
## ```tremor
## {
##    "kind": "string"|"binary"|"array", # Depends on input
##    "valid": true|false,               # True if well-formed and valid
##    "value": "<trace_id>"              # Representation depends on `kind`
## }
## ```
##
## Returns an empty record `{}` when the representation not well-formed
##
fn is_valid(trace_id) of
    # String representation
    case(trace_id) when type::is_string(trace_id) =>
      { "kind": "string", "valid": trace_id != "00000000000000000000000000000000" and string::bytes(trace_id) == 32, "value": trace_id }
    # Binary representation
    case(trace_id) when type::is_binary(trace_id) =>
      let arr = binary::into_bytes(trace_id);
      { "kind": "binary", "valid": binary::len(arr) == 16 and trace_id != << 0:64, 0:64 >>, "value": trace_id }
    # Array representation
    case(trace_id) when type::is_array(trace_id) =>
      { "kind": "array", "valid":  array::len(arr) == 16 and trace_id != [ 0, 0, 0, 0, 0, 0, 0, 0], "value": trace_id }
    case _ =>
      false
end

Rule FnCases

The FnCases rule defines a sequence of cases for structural pattern matching in tremor pattern functions.

rule FnCases ::=
    FnCaseClauses FnCaseDefault 
  | FnCaseDefault 
  ;

connector/console/in

The connector console via its standard in port

Rule FnCaseDefault

The FnCaseDefines rule defines a default match clause for use in pattern match function signatures in tremor.

rule FnCaseDefault ::=
     'case'  '_' Effectors 
  ;

Rule FnCase

The FnCase rule defines an array predicate pattern supporting match clause for use in pattern match function signatures in tremor.

rule FnCase ::=
     'case'  '(' ArrayPredicatePatterns  ')' WhenClause Effectors 
  ;

A case definition in a pattern match function definition

use std::type;

fn snottify(s) of
  # Matches the literal string "badger"
  case ("badger") => "snot badger, hell yea!"
  # Matches any well formed json argument
  case (~ json||) => let s.snot = true; s
  # Matches any literal string
  case (s) when type::is_string(s) => "snot #{s}"
  # Matches, everything else
  case _ => "snot caller, you can't snottify that!"
end;

Rule FnCaseClauses

The FnCaseClauses defines the case syntax to structurally matched function signatures in tremor.

rule FnCaseClauses ::=
    FnCase 
  | FnCaseClauses FnCase 
  ;

Rule FnArgs

The FnArgs rule defines , comma delimited arguments to a tremor function.

rule FnArgs ::=
    Ident 
  | FnArgs  ',' Ident 
  ;

Rule SimpleExprImut

The SimpleExprImut rule defines optionally parenthesized simple immutable expressions in tremor.

rule SimpleExprImut ::=
     '(' ComplexExprImut  ')' 
  | AlwaysImutExpr 
  ;

A parenthetic or simple non-parenthetic expression

( 1 + 2 ) * 3;

Rule Literal

The Literal rule defines the set of primitive literals supported in tremor.

rule Literal ::=
    Nil 
  | Bool 
  | Int 
  | Float 
  ;

Primitive literal types in tremor are integer, floating point, boolean and the null reference.

String literals are UTF-8 encoded values and support interpolation with single line ( regular ) and multi line ( heredoc style ) variant forms.

Rule Nil

The Nil rule defines the syntax of the nil literal in tremor.

rule Nil ::=
     'nil' 
  ;

Example

null # The `null` literal value

Rule Bool

The Bool rule defines the syntax of boolean literal in tremor.

rule Bool ::=
     'bool' 
  ;

Example

true # The boolean `true` literal

false # The boolean `false` literal

Rule Int

The Int rule literal specifes the syntax of integer literals in tremor.

rule Int ::=
     'int' 
  ;

Integer literals

Integer literals in tremor are 64 bit signed values of the form

1234

Values can be separated by an _ ( underscore ) for easy reading

let million = 1_000_000

Rule Float

The Float rule literal specifes the syntax of IEEE float literals in tremor.

rule Float ::=
     'float' 
  ;

Floating point literals

Floating point literals in tremor are 64 bit signed IEEE floating point values of the form

1234.0e10

Values can be separated by an _ ( underscore ) for easy reading

let million = 1_000_000.1234e-5

Rule StringLiteral

The StringLiteral rule defines a string literal in tremor.

Strings are " single-quote or """ triple-quote delimited blocks of UTF-8 text.

A single-quote string is a single line string, supporting sting interpolation.

A triple-quote string is a multi-line string, supporting sting interpolation.

rule StringLiteral ::=
     'heredoc_start' StrLitElements  'heredoc_end' 
  |  '\\' StrLitElements  '\\' 
  |  '\\'  '\\' 
  ;

How do I define single line string literals?

A single line string MUST be on a single line with raw newline characters ( unless escaped ).

"I am a literal string"

How do i define multi line string literals?

A multi line string MUST span multiple lines with raw newline characters.

Multi line strings

"""
I am a
multi
line
string
"""

The following example is a malformed multi line string:

""" snot """

Which when executed will result in a compile time error:

Error:
    1 | """ snot """
      | ^^^ It looks like you have characters tailing the here doc opening, it needs to be followed by a newline

Simple and nested Interpolation

Strings in tremor can be interpolated with internal scripts

"""

I am an #{interpolated} #{event.sum / event.count} string

Interpolations can be simple, as above, or #{
merge event of
  { "#{snot}": """

    #{badger * 1000 + crazy_snake }

    """ }
}
"""

This will result in the output:

"\nI am an interpolated 5.0 string\n\nInterpolations can be simple, as above, or {\"sum\":10,\"count\":2,\"snot\":\"\\n    20001\\n\\n    \"}\n"

Note that the merge operation merges an event { "sum": 10, "count": 10 } with in scope values of snot that evaluates to the literal string "snot" and the numerics badger ( 20 ) and crazy_snake ( 1 ). Interpolations are nestable and field names or any other string literal in tremor can be interpolated.

However, we do not recommend complex nested interpolated strings. Defining a function and calling it may be a better alternative for most applications and uses.

Rule StrLitElements

The StrLitElements rule defines the internal structure of a string literal in tremor.

String literal in tremor support string interpolation via the #{ and } escape sequence. Content within the escape sequence can be any legal and valid tremor expression.

rule StrLitElements ::=
    StringPart StrLitElements 
  |  '\\\\#' StrLitElements 
  |  '#{' ExprImut  '}' StrLitElements 
  | StringPart 
  |  '\\\\#' 
  |  '#{' ExprImut  '}' 
  ;

Inside a literal tremor string

A string literal in tremor is a composition of multiple segments or parts.

These can be composed of:

• One or many single line string parts
• One or many multi line string parts
• A blackslash escaped \\# to escape interpolated syntax, optinally followed by more string literal parts
• Or, a #{ .. } delimited interpolated section
  • Within an interpolated section there are no constraints on raw newline usage
  • For complex interpolated sections, prefer good indentation!

Rule StringPart

The StringPart rule defines a simple or heredoc style string part.

rule StringPart ::=
     'string' 
  |  'heredoc' 
  ;

The " delimited string is single line

The """ delimited string is multi line

The forms can be mixed to construct a string literal

Interpolation is supported by both forms

Rule List

The List rule defines a [ and ] square bracket delimited sequence of zero or many ',' delimited expressions.

rule List ::=
     '[' ListElements  ']' 
  |  '['  ']' 
  ;

How are literal array or list or vectorc data collections defined in tremor?

We use [ .. ] square brackets to delimit list-like data.

Are multi-dimensional arrays supported?

Multi-dimensional arrays are compositional and can be nested

Does tremor support typed lists?

No. A list in tremor can have elements from any supported primitive or structural type.

Validation that a list is for a single type - such as a list of boolean values can be defined as follows:

use std::type;

let bad = [true, false, "snot"];
let good = [true, false ];

fn list_of_bool(l) with
  let valid = true;
  for l of
    case (i,e) when type::is_bool(e) and valid == true => let valid = true
    case (i, otherwise) => let valid = false
  end;
  valid # return true if list is all bool, false otherwise
end;

list_of_bool(bad); # should fail
list_of_bool(good); # should succeed

This user defined function can then be used in guard clauses like when type::is_bool(e) ... in the example code.

Rule ListElements

The ListElements rule defines a , comma delimited sequence of expression elements.

rule ListElements ::=
    ListElements_ 
  ;

List literals

Unlike JSON, trailing commas are supported

[foo,] # A non empty list, a trailing comma is optionally permissible

Except in empty lists, where the idiomatic form is preferred:

[] # An empty list - no trailing comma here!

Rule ListElements_

The ListElements_ rule is internal to the ListElements rule.

The rule defines a sequence of , comma delimited expression elements using the Sep macro rule.

rule ListElements_ ::=
    Sep!(ListElements_, ComplexExprImut, ",") 
  ;

See ListElements rule for details.

This rule wraps away a lalrpop macro call for ease of reference in other rules in the grammar source.

Rule Record

The Record rule defines a set of name-value pairs delimited by , a comma.

Records are enclosed in { and } curly braces.

The record structure in tremor is backwards compatible with JSON.

All JSON records can be read by tremor.

Not all tremor records can be read by a JSON reader as tremor supports computations, comments and trailiing , commas in its record and array structures.

rule Record ::=
     '{' Fields  '}' 
  |  '{'  '}' 
  ;

How are literal records or map types or key-value collections defined in tremor?

We use { .. } squigly braces to delimit record -like data.

Can records be nested?

Record values can be any valid primitive of structural type supported by tremor, including other records.

Does tremor support typed records?

No. A record in tremor can have element values from any supported primitive or structural type in tremor.

Validation that a record conforms to a shape, schema or structure can be achieved through match expressions.

use std::type;

let bad = { "list-of-bool": [true, false, "snot"] };
let good = { "list-of-bool": [true, false ], "flag": true };

fn list_of_bool(l) with
  let valid = true;
  for l of
    case (i,e) when type::is_bool(e) and valid == true => let valid = true
    case (i, otherwise) => let valid = false
  end;
  valid # return true if list is all bool, false otherwise
end;

fn is_good_record(r) with
  let valid = false;
  match r of
    case extract=%{ present flag, list-of-bool ~= %[] } 
      when list_of_bool(extract["list-of-bool"]) => 
       let valid = true
    case _ => 
      let valid = false
  end;
  valid
end;

is_good_record(bad); # should fail
is_good_record(good); # should succeed

This user defined function can then be used in guard clauses like when type::is_bool(e) ... in the example code.

Rule Field

The Field rule defines a : colon delimited name value pair for a record literal.

The name is a string literal.

The value is an expression.

rule Field ::=
    StringLiteral  ':' ComplexExprImut 
  ;

How do i define record fields?

The syntax is similar to JSON:

{ "field": "value" }

With the exception that fields may have an optional ',' terminal trailing comma

Interpolated field names

As literal strings in tremor support string interpolation the following variants are equivalent:

{ "snot": "badger" }

let snot = "snot";
{ "#{snot}": "badger" };

But, not all legal variations are recommended:

let snot = """
snot"""; # This may result in tears
{ "#{snot}": "badger" };

Another legal but likely not useful variation:

let snot = { "snot": "badger" };
{ "#{snot}": "badger" };

Will result in a stringifield json being encoded as the field name:

{"{\"snot\":\"badger\"}":"badger"}

Rule Path

The Path rule defines path operations over expressions.

Path operations structures to be tersely indexed in a path like structure.

Path operations are supported on

• A subset of expressions ( record, array, function )
• Meta keywords like $, args, state, event, group, window

rule Path ::=
    MetaPath 
  | EventPath 
  | StatePath 
  | LocalPath 
  | ConstPath 
  | AggrPath 
  | ArgsPath 
  | ExprPath 
  ;

Path expressions are how events and in memory state in tremor are referenced in scripts.

How to reference metadata

The tremor runtime can provide and consume metadata with the events being processed by the runtime.

Metadata is distinguished by a $ dollar sign

let foo = $request # The local path `foo` is a copy of the metadata `$request`

Connectors such as kafka and http can generate metadata that scripts, queries and pipelines can manipulate and process to tune tremor's runtime behaviour.

How to reference the current streaming event

The current event streaming through tremor in the current pipeline will be available to queries, logics and scripts via the event keyword.

The event keyword can be further dereferenced via path statements

# Where event is a record
let foo = event.snot; # The local path `foo` is a copy of the `snot` field from the current event

# Where event is an array
let foo = event[10]; The local path `foo` is a copy of the 10th element of the current event.

How to reference pipeline state

Scripts in tremor can store state that is available for the lifetime of a pipeline via the state keyword.

The state keyword can be further dereferenced via path statements

# Where state is a record
let foo = state.snot; # The local path `foo` is a copy of the `snot` field from the state record 

# Where state is an array
let foo = state[10]; The local path `foo` is a copy of the 10th element of the state array

### How to reference arguments

For operators and structures that support arguments the `args` keyword can be
used to dereference values via path statements.

```tremor
# Where state is a record
let foo = args.snot; # The local path `foo` is a copy of the `snot` field from the args record 

Args are nominal and always record values in tremor.

How can window state be referenced

Operations supporting windows and groups can dereference the cached state via the window and group keywords which both support path operations.

Rule ExprPathRoot

The ExprPathRoot rule defines a subset of expressions where path operations are supported.

These are:

• Record literals or references to records.
• Array literals or references to arrays.
• The result of function invocations.
• The result of Parenthetic expressions.

rule ExprPathRoot ::=
     '(' ComplexExprImut  ')' 
  | Invoke 
  | Record 
  | List 
  ;

An internal rule to the ExprPath rule that defines legal roots for a dynamic path dereference.

Rule ExprPath

The ExprPath rule defines path operations for expressions.

rule ExprPath ::=
    ExprPathRoot PathSegments 
  ;

Allows dereferencing literal values vi path expressions

{"snot": 0, "badger": 1, "goose": 2}["badger"];
{"snot": 0, "badger": 1, "goose": 2}.badger;

...

some_record_fn().record_field

Rule MetaPath

The MetaPath rule defines path operations for event metadata references.

In the context of a streaming event, allows metadata generated by the runtime to be accessed via path operations.

It is also possible to write to metadata to hint at the runtime to perform certain functions on the event data being forwarded. Tremor operators and connectors can read and write metadata.

rule MetaPath ::=
     '$' Ident PathSegments 
  |  '$' Ident 
  |  '$' 
  ;

How do i reference event metadata?

Events in tremor encapsulate data from other systems sent and received from tremor via configured connectors. Information about that data or metadata can also be provided by the runtime, and used in some operators and connectors to control tremor's runtime behaviour.

Meta-data is accessed via the $ dollar symbol.

let metadata = $;

Metadata can be any legal tremor value, but it is typically a record structure

let metastring = "snot" + $;

Meta-data can be written through via a let operation

let $command = { "do-things": "with-this-meta-request" }

Rule AggrPath

The AggrPath rule defines path operations for group and window references.

In the context of a windowed operation, enables the group and window meta keywords to partipcate in path operations.

rule AggrPath ::=
     'group' PathSegments 
  |  'group' 
  |  'window' PathSegments 
  |  'window' 
  ;

How do I reference the computed group dimension?

use std::record;
define window by_2 from tumbling
with
  size = 2
end;

select {
  "g": group[0], # Extract current group dimension
  "c": aggr::stats::sum(event.c),
}
from in[by_2]
group by set(each(record::keys(event.g))) into out;

Rule ArgsPath

The ArgsPath rule defines path operations for args references.

rule ArgsPath ::=
     'args' PathSegments 
  |  'args' 
  ;

How do i reference arguments?

Arguments are encapsualted via the args keyword symbol.

let what = args;

Arguments are always record structured

    1 | let args = 1;
      |     ^^^^^^^^ Can't assign to a constant expression

Arguments cannot be assigned to or overridden in scripts.

Rule LocalPath

The LocalPath rule enables path operations on locally scoped identifiers.

rule LocalPath ::=
    Ident PathSegments 
  | Ident 
  ;

A local path is simply a path structure that is bounded to a locally defined value

let snot = { "snot": "badger" };

snot.snot # The field 'snot', on the local path reference 'snot'

Rule ConstPath

The ConstPath rule enables path operations on module scoped references.

rule ConstPath ::=
    ModPath  '::' LocalPath 
  ;

A fully scoped path that does not include dynamic resolution or runtime type information.

Rule StatePath

The StatePath rule defines path operations for user defined in memory state in tremor.

Allows the state value to be dereferenced via path operations.

rule StatePath ::=
     'state' PathSegments 
  |  'state' 
  ;

How do i reference state in tremor?

Tremor programs can be stateful in many ways - such as through the state managed by operators and windowed operations by the runtime on behalf of the user provided program.

The state keyword allows an arbitrary value controlled by a users program to be maintained and managed by the user program.

let my_state = state;

State can be written through via a let operation

let state = match state of
  case null => { "count": 1 }
  case _ => { "count"": state.count + 1 }
end;

Rule EventPath

The EventPath rule defines path operations for streaming events in tremor.

Allows the current streaming event to be dereferenced via path operations.

rule EventPath ::=
     'event' PathSegments 
  |  'event' 
  ;

How do i dereference event data?

The current event is accessed via the event keyword.

The event can be any legal tremor value. If it is a record or an array, then it can be dereferenced via the path language in the usual way.

event.snot; # Event record, field 'snot'

let badger = event[0]; Event array, first element

Events can be mutated and manipulated and used as an output

select { "wrapped-event": event } from in into out;

Rule PathSegments

The PathSegments rule specifies the continuation of a path rule.

Form Variation	Description
.<Ident>	A terminal segment dereferencing a record field
<Ident><PathSegments>	A non-terminal segment dereferencing a record field
[<Selector>]	A range or index segment dereferencing an array
[<Selector>]	A terminal range or index segment dereferencing an array
[<Selector>]<PathSegments>	A non-terminal range or index segment dereferencing an array

rule PathSegments ::=
     '.' Ident PathSegments 
  |  '[' Selector  ']' PathSegments 
  |  '[' Selector  ']' 
  |  '.' Ident 
  ;

``tremor a.b a["b"] a[0..5]

Defines how structural values in tremor can be dereferenced to get at internal parts such as
an element of an array, or a field of a record.



## Rule Selector

The `Selector` rule specifies an index or range of an array.

A range is a `:` colon separated pair of expressions.

An index is a single expression.



![Selector](svg/selector.svg)

```ebnf
rule Selector ::=
    ComplexExprImut  ':' ComplexExprImut 
  | ComplexExprImut 
  ;

Field selection for records

Selecing a record field using array notation

let snot = badger["snot"]

Select the field 'snot' from the record 'badger'

Ordinal selection for arrays

let e = badger[0];

Select the 0th ( first ) element of the array 'badger'

Range selection for arrays

let e = badger[0:5];

Select the 0th up to but no including the 5th element of the array 'badger'

Rule Invoke

The Invoke rule specifies the syntax of a function invocation.

rule Invoke ::=
    FunctionName  '(' InvokeArgs  ')' 
  | FunctionName  '('  ')' 
  ;

use std::string;

string::len("snot")

Rule FunctionName

The FunctionName rule defines a path to a function in tremor.

It can be an Ident for functions defined in local scope.

It can be a ModPath for functions in a modular scope.

rule FunctionName ::=
    Ident 
  | ModPath  '::' Ident 
  ;

Rule ModPath

The ModPath rule defines a modular path.

A modular path is a sequence of Idents separated by a :: double-colon.

rule ModPath ::=
    ModPath  '::' Ident 
  | Ident 
  ;

How do i reference something from the standard library?

The standard library contains reusable constants, functions and other definitions that can be used in scripts via the Use and ModPath rules.

For example, if you have a file called foo.tremor in a src folder you can append this to your TREMOR_PATH environment variable

export TREMOR_PATH=/path/to/src

Assuming foo.tremor contains the following code:

fn meaning_of_life() of
  42
end;

We can use this in another script as follows:

use foo;

let meaning = foo::meaning_of_life();

Rule InvokeArgs

The InvokeArgs rule defines a sequence of expression statements.

rule InvokeArgs ::=
    InvokeArgs_ 
  ;

A comma delimited sequence of complex expressions

Rule InvokeArgs_

The InvokeArgs_ rule is an internal rule of the InvokeArgs rule.

The rule specifies a ; semi-colon delimited sequence of expression statements.

rule InvokeArgs_ ::=
    Sep!(InvokeArgs_, ComplexExprImut, ",") 
  ;

See InvokeArgs rule for details.

This rule wraps away a lalrpop macro call for ease of reference in other rules in the grammar source.

Rule Drop

Drop halts event processing for the current event being processed returning control to the tremor runtime, dropping the event.

Constraints

The drop operation should be used with care as the in-flight event is discarded by the runtime. Where circuit breakers, guaranteed delivery and quality of service operations are being managed by the engine downstream these should be carefully programmed so that drop operations have no side-effects on non-functional behaviours of the tremor runtime.

Here be dragons!

rule Drop ::=
     'drop' 
  ;

define script boring
script
  drop
end;
create script boring;
select event from in into boring;
select event from boring into out;

Drop signals to the tremor runtime that an event is not interesting and can be dropped without any further handling by the engine. Drop statements in a script or query result in the processing of the current event halting without any further action bye the tremor runtime.

The dropped event is discarded by the engine.

Rule Emit

Emit halts event processing for the current event being processed returning control to the tremor runtime, emitting a synthetic event as output.

By default, the emit operation will emit events to the standard output port out.

The operation can be redirected to an alternate output port.

rule Emit ::=
     'emit' ComplexExprImut  '=>' StringLiteral 
  |  'emit' ComplexExprImut 
  |  'emit'  '=>' StringLiteral 
  |  'emit' 
  ;

define script route
script
  emit => "not_out"
end;

create script route;
select event from in into route;
select event from route/not_out into out;

Emit signals to the tremor runtime that an event has been processed fully and processing can stop at the point emit is invoked and a synthetic value returned without any further processing.

The emitted event is forwarded by the engine.

Rule Let

The Let rule allows an expression to be bound to a Path.

The Path references the subject of the assignment based on tremor's Path rules.

The bound Path is mutable.

rule Let ::=
     'let' Assignment 
  ;

How do i create a local mutable variable in tremor?

let my_var = "this is a string";

Rule Assignment

The Assignment rule allows an expression to be bound to a Path.

The Path references the subject of the assignment based on tremor's Path rules.

rule Assignment ::=
    Path  '=' SimpleExpr 
  ;

The rule assigns an expression to a path by reference.

x = 2

Assigns x to the value literal 2

x.y = 2

Assigns the field y on record x to the value literal 2

Assignments expressions can be constant such as via the Const rule or mutable such as via the Let rule.

Rule Patch

The Patch rule defines the patch statement in tremor.

rule Patch ::=
     'patch' ComplexExprImut  'of' PatchOperations  'end' 
  ;

Patch insert a field in a record

let a = patch event of
  insert "test" => 1
end;

Default patch templates

patch event of
  default => {"snot": {"badger": "goose"}}
end



## Rule PatchOperations

The `PatchOperations` rule defines a sequence of semi-colon delimited patch operations.



![PatchOperations](svg/patchoperations.svg)

```ebnf
rule PatchOperations ::=
    PatchOperationClause 
  | PatchOperations  ';' PatchOperationClause 
  ;

A sequence of patch operations

Rule PatchField

The PatchField is a string literal identifying a the field of a record to which a PatchOperationClause is being applied.

rule PatchField ::=
    StringLiteral 
  ;

Rule PatchOperationClause

The PatchOperationClause rule defines operations of a patch statement.

A patch operation can:

• Insert, update, copy ( clone ), move ( rename ), merge or erase fields in a record.
• Apply a default operation on a field or on the whole input record.

rule PatchOperationClause ::=
     'insert' PatchField  '=>' ComplexExprImut 
  |  'upsert' PatchField  '=>' ComplexExprImut 
  |  'update' PatchField  '=>' ComplexExprImut 
  |  'erase' PatchField 
  |  'move' PatchField  '=>' PatchField 
  |  'copy' PatchField  '=>' PatchField 
  |  'merge' PatchField  '=>' ComplexExprImut 
  |  'merge'  '=>' ComplexExprImut 
  |  'default' PatchField  '=>' ComplexExprImut 
  |  'default'  '=>' ComplexExprImut 
  ;

How do I insert a new value into a record?

patch {"a": 1, "b": 2, "c": 3 } of
  insert "d" => "delta"
end;

It is a semantic error to insert a value if the field already exists:

Error in foo.tremor:2:3
    1 | patch {"a": 1, "b": 2, "c": 3 } of
    2 |   insert "b" => "bravo"
      |   ^^^^^^^^^^^^^^^^^^^^^ The key that is supposed to be written to already exists: b
    3 | end;

How do I update an existing value in a record?

patch {"a": 1, "b": 2, "c": 3 } of
  update "b" => "bravo"
end;

It is a semantic error to update a value if the field does not already exist:

    1 | patch {"a": 1, "b": 2, "c": 3 } of
    2 |   update "d" => "delta"
      |   ^^^^^^^^^^^^^^^^^^^^^ The key that is supposed to be updated does not exists: d
    3 | end;

How do I insert or update a value in a record?

If the distinction between an insert and an update is not significant the upsert operation will insert a new field, or update an existing field. This operation is more flexible, but does not offer compile time errors to protect against invalid usage. Where possible, use insert or update in preference to upsert when a new field or replacing an existing fields value would be an error given the business logic at hand.

How do I erase a field from a record?

patch {"a": 1, "b": 2, "c": 3 } of
  erase "d"
end;

The field c is removed from our record

How do I rename a field?

patch {"a": 1, "b": 2, "c": 3 } of
  move "c" => "d" # The value MUST be a string literal as it represents a field name
end;

The c field is removed and a d field is added with the value from c

How do I duplicate a field?

Similar to move, the copy operation copies the value of one field to a new field

patch {"a": 1, "b": 2, "c": 3 } of
  copy "c" => "d" # The value MUST be a string literal as it represents a field name
end;

The c field is preserved, and the d field is added with a copy of the value from c

Can I use patch and merge together?

The merge operation in a patch expression can be applied to the patch target record or to a specified field.

patch {"a": 1, "b": 2, "c": 3 } of
  merge "d" => {}
end;