Full Grammar
Rule ModuleFile
The ModuleFile
rule defines a module in tremor.
A module is a unit of compilation.
rule ModuleFile ::=
ModuleBody
;
Represents the compiled form of a tremor source file.
Currently this can be troy
deployment files, trickle
query files, or tremor script
files.
This is part of of tremor's compiler and runtime and not user modifiable.
Rule ModuleBody
The ModuleBody
rule defines the structure of a valid module in tremor.
Modules begin with optional module comments.
Modules MUST define at least one statement, but may define many.
Statements are ;
semi-colon delimited.
rule ModuleBody ::=
( ModComment ) ? ModuleStmts
;
The body of a compiled tremor source file has a module level comment ( lines beginning with 3 hashes ) and the statements that form the logic of that module.
This is part of of tremor's compiler and runtime and not user modifiable.
Rule ModComment
The ModComment
rule specifies module comments in tremor.
Documentation comments for modules are optional.
A module documentation comment begins with a ###
triple-hash and they are line delimited.
Muliple successive comments are coalesced together to form a complete comment.
The content of a module documentation comment is markdown syntax.
rule ModComment ::=
'<mod-comment>'
| ModComment '<mod-comment>'
;
Example
Module level comments are used throughout the tremor standard library and used as part of our document generation process.
Here is a modified snippet from the standard library to illustrate
### The tremor language standard library it provides the following modules:
###
### * [array](std/array.md) - functions to deal with arrays (`[]`)
### * [base64](std/base64.md) - functions for base64 en and decoding
### * [binary](std/base64.md) - functions to deal with binary data (`<< 1, 2, 3 >>`)
### * [float](std/float.md) - functions to deal with floating point numbers
### * [integer](std/integer.md) - functions to deal with integer numbers
### * [json](std/json.md) - functions to deal with JSON
...
Rule DocComment
The DocComment
rule specifies documentation comments in tremor.
Documentation comments are optional.
A documentation comment begins with a ##
double-hash and they are line delimited.
Muliple successive comments are coalesced together to form a complete comment.
The content of a documentation comment is markdown syntax.
rule DocComment ::=
'<doc-comment>'
| DocComment '<doc-comment>'
;
Example
Documentation level comments are used throughout the tremor standard library and used as part of our document generation process.
Here is a modified snippet from the standard library to illustrate
## Returns the instance of tremor.
##
## Returns a `string`
intrinsic fn instance() as system::instance;
...
This is a builtin function implemented in rust and used in a script as follows:
use tremor::system;
system::instance()
Rule ModuleStmts
The ModuleStmts
rule defines a set of module statements.
Module statements are a ;
semi-colon delimited set of ModuleStmt
rules
rule ModuleStmts ::=
ModuleStmt ';' ModuleStmts
| ModuleStmt ';' ?
;
The set of computed statements in a compiled script.
This is part of of tremor's compiler and runtime and not user modifiable.
Rule ModuleStmt
The ModuleStmt
rule defines the statement types that are valid in a tremor module.
rule ModuleStmt ::=
Use
| Const
| FnDefn
| Intrinsic
| DefineWindow
| DefineOperator
| DefineScript
| DefinePipeline
| DefineConnector
| DefineFlow
;
The set of statements that are legal in any tremor supported domain specific language.
This is part of of tremor's compiler and runtime and not user modifiable.
Rule ConfigDirectives
The ConfigDirectives
rule allows line delimited compiler, interpreter or
runtime hints to be specified.
rule ConfigDirectives ::=
ConfigDirective ConfigDirectives
| ConfigDirective
;
See ConfigDirective
for supported directives.
Rule ConfigDirective
A ConfigDirective
is a directive to the tremor runtime.
Directives MUST begin on a new line with the #!config
shebang config token.
rule ConfigDirective ::=
'#!config' WithExpr
;
Providing a metrics internal via a config directive
# Enable metrics with a 10 second interval
#!config metrics_interval_s = 10
Rule Use
Imports definitions from an external source for use in the current source file.
The contents of a source file form a module.
Multiple modules that share a common root can be combined into a single use clause by using use some::root::{m1, m2::sub_module::{m3, m4}}
TREMOR_PATH
The TREMOR_PATH
environment path variable is a :
delimited set of paths.
Each path is an absolute or relative path to a directory.
When using relative paths - these are relative to the working directory where the
tremor
executable is executed from.
The tremor standard library MUST be added to the path to be accessible to scripts.
rule Use ::=
'use' UseTarget
;
Modules
Modules can be scripts. Scripts can store function and constant definitions.
Scripts are stored in .tremor
files.
Modules can be queries. Queries can store window, pipeline, script and operator definitions.
Scripts are stored in .trickle
files.
Modules can be deployments. Deployments can store connector, pipeline and flow definitions.
Deployments are stored in .troy
files.
Conditioning
Modules in tremor are resolved via the TREMOR_PATH
environment variable. The variable can
refer to multiple directory paths, each separated by a :
colon. The relative directory
structure and base file name of the source file form the relative module path.
Constraints
It is not recommended to have overlapping or shared directories across the set of paths provided in the tremor path.
It is not recommended to have multiple definitions mapping to the same identifier.
Rule UseTarget
rule UseTarget ::=
MaybeAliased
| ModPath '::' MaybeAliased
| ModPath '::' '{' TargetList '}'
;
Rule TargetList
rule TargetList ::=
TargetElement
| TargetElement ',' TargetList
;
Rule TargetElement
rule TargetElement ::=
MaybeAliased
| ModPath '::' MaybeAliased
| ModPath '::' '{' TargetList '}'
;
Rule MaybeAliased
rule MaybeAliased ::=
Ident
| Ident 'as' Ident
;
Rule ArgsWithEnd
The ArgsWithEnd
rule defines an arguments block with an end
block.
rule ArgsWithEnd ::=
ArgsClause ? WithEndClause
|
;
An internal rule that defines an optional args
block with and optional end
token.
This rule is used and shared in other rules as part of their definitions.
Rule DefinitionArgs
The DefinitionArgs
rule defines an arguments block without an end
block.
rule DefinitionArgs ::=
ArgsClause ?
;
An optional argument block
args arg1, arg 2
This is a shared internal rule used in other rules as part of their definition.
Rule ArgsClause
The ArgsClause
rule marks the beginning of an arguments block.
A valid clause has one or many argument expressions delimited by a ',' comma.
rule ArgsClause ::=
'args' ArgsExprs
;
args
x = y
Rule ArgsExprs
The ArgsExpr
rule is a macro rule invocation based on the Sep
separator macro rule.
An args expression is a comma delimited set of argument expressions.
rule ArgsExprs ::=
Sep!(ArgsExprs, ArgsExpr, ",")
;
Rule ArgsExpr
The ArgExpr
rule specifies argument lists.
An argument can be an Ident
or an assignment of the form <Ident> = <Expr>
rule ArgsExpr ::=
Ident '=' ExprImut
| Ident
;
As used in deployment rules to set or override arguments specifications. Arguments specifications define interface parameters that must be set by default ( or overridden ) for something to be well defined.
The tremor runtime checks for ommissions and produces an error for attempted instanciations that omit to provide a value for specified arguments.
Rule CreationWithEnd
The CreationWithEnd
rule defines a with
block of expressions with a terminal end
keyword.
rule CreationWithEnd ::=
WithEndClause
|
;
with x = y end
Rule CreationWith
The CreationWith
rule defines an optional with
block of expressions without a terminal end
keyword.
rule CreationWith ::=
WithClause
|
;
Rule WithClause
The WithClause
rule defines a with
block with a ,
comma delimited set of WithExpr
rules.
rule WithClause ::=
'with' WithExprs
;
with x = y
Rule WithEndClause
The WithEndClause
rule defines a with clause with an end
terminal token.
rule WithEndClause ::=
WithClause 'end'
;
with x = y end
Rule WithExprs
The WithExprs
rule defines a ,
comma delimited set of WithExpr
rules.
rule WithExprs ::=
Sep!(WithExprs, WithExpr, ",")
;
Rule WithExpr
The WithExpr
rule defines a name value binding.
rule WithExpr ::=
Ident '=' ExprImut
;
Form
name = <value>
Where:
name
is an identifier.<value>
is any valid immutable expression.
Example
snot = "badger"
Rule ModularTarget
A ModularTarget
indexes into tremor's module path.
In tremor a module
is a file on the file system.
A module
is also a unit of compilation.
A ModularTarget
is a ::
double-colon delimited set of identifiers.
Leading ::
are not supported in a modular target..
Trailing ::
are not supported in a modular target.
rule ModularTarget ::=
Ident
| ModPath '::' Ident
;
Examples
Loading and using a builtin function
# Load the base64 utilities
use std::base64;
# Base64 encode the current `event`.
base64::encode(event)
Loading and using a builtin function with an alias
# Load the base64 utilities
use std::base64 as snot;
# Base64 encode the current `event`.
snot::encode(event)
Rule Deploy
The Deploy
rule defines the logical entry point into Tremor's command
oriented deployment syntax. The deployment grammar defines units of
deployment that the runtime manages on behalf of users.
The grammar embeds the statement oriented query syntax and expression oriented scripting syntax where appropriate.
A legal deployment is composed of:
- An optional set of module comments
- A sequence of top level expressions. There must be at least one defined.
- An optional end of stream token.
At least one deploy command.
rule Deploy ::=
ConfigDirectives ( ModComment ) ? DeployStmts
| ( ModComment ) ? DeployStmts
;
define flow test
flow
use std::time::nanos;
define connector metronome from metronome
with
config = {
"interval": nanos::from_seconds(1)
}
end;
define connector exit from exit;
define pipeline identity
args
snot = "badger",
pipeline
select args.snot from in into out;
end;
create connector metronome;
create connector exit;
create pipeline identity with
snot = "snot"
end;
connect /connector/metronome to /pipeline/identity;
connect /pipeline/identity to /connector/exit;
end;
deploy flow test
Rule DeployStmts
The DeployStmts
rule defines the statements that are legal in a deployment
module.
Statements in a deployment modules are ;
semi-colon delimited.
There MUST be at least one.
There MAY be more than one.
rule DeployStmts ::=
DeployStmt ';' DeployStmts
| DeployStmt ';' ?
;
Rule DeployStmt
The DeployStmt
rule constrains the statements that are legal in a .troy
deployment module.
Importing modules via the use
clause is allowed.
Flow definitions and deploy
commands are allowed.
rule DeployStmt ::=
DefineFlow
| DeployFlowStmt
| Use
;
Rule DeployFlowStmt
The DeployFlowStmt
rule defines the content of a command to tremor
to deploy a flow.
The flows can be re-parameterized through overriding the default
parameters set in their originating source definitions via a with
clause.
rule DeployFlowStmt ::=
( DocComment ) ? 'deploy' 'flow' Ident 'from' ModularTarget CreationWithEnd
| ( DocComment ) ? 'deploy' 'flow' Ident CreationWithEnd
;
define flow test
flow
use std::time::nanos;
define connector metronome from metronome
with
config = {
"interval": nanos::from_millis(1)
}
end;
define connector exit from exit;
define pipeline identity
pipeline
select event from in into out;
end;
create connector metronome;
create connector exit;
create pipeline identity;
connect /connector/metronome to /pipeline/identity;
connect /pipeline/identity to /connector/exit;
end;
# The `deploy` statements commands tremor to instanciate the flow `test` - the flow in turn
# will result in a metronome, exit connector, pipeline to be started and interconnected as
# per the `test` definition above
deploy flow test
Rule ConnectorKind
The ConnectorKind
rule identifies a builtin connector in tremor.
Connectors in tremor are provided by the runtime and builtin. They can be resolved through an identifier.
Examples
The http_server
identifies a HTTP server connector.
The metronome
identifies a periodic metronome.
rule ConnectorKind ::=
Ident
;
The name of a builtin connector implemented in rust provided by the tremor runtime.
Rule FlowStmts
The FlowStmts
rule defines a mandatory ;
semi-colon delimited sequence of FlowStmtInner
rules.
rule FlowStmts ::=
FlowStmts_
;
Rule FlowStmts_
The FlowStmts_
rule defines a ;
semi-colon delimited sequence of FlowStmtInner
rules.
rule FlowStmts_ ::=
Sep!(FlowStmts_, FlowStmtInner, ";")
;
See FlowStmts
rule for details.
This rule wraps away a lalrpop macro call for ease of reference in other rules in the grammar source.
Rule CreateKind
The CreateKind
rule encapsulates the artefact types that can be created in the tremor deploymant language.
rule CreateKind ::=
'connector'
| 'pipeline'
;
An internal rule that allows connectors and pipelines to be created and used by the Create
rule.
Rule FlowStmtInner
The FlowStmtInner
rule defines the body of a flow definition.
rule FlowStmtInner ::=
Define
| Create
| Connect
| Use
;
- A pipeline or connector definition
- A pipeline or connector instance via
create
- A
use
statement to import a definition - A
connect
statement to interlink instances
Rule Define
The Define
rule allows connectors and pipelines to be specified.
rule Define ::=
DefinePipeline
| DefineConnector
;
Within a flow definition, allows pipeline and connectors to be defined.
Rule Create
The Create
rule creates instances of connectors and pipelines in a flow.
rule Create ::=
'create' CreateKind Ident 'from' ModularTarget CreationWithEnd
| 'create' CreateKind Ident CreationWithEnd
;
Create a connector
create connector foo from snot::foo end;
Create a pipeline
create pipeline bar from badger::bar end;
Rule Connect
The Connect
rule defines routes between connectors and pipelines running in a flow.
rule Connect ::=
'connect' '/' ConnectFromConnector 'to' '/' ConnectToPipeline
| 'connect' '/' ConnectFromPipeline 'to' '/' ConnectToConnector
| 'connect' '/' ConnectFromPipeline 'to' '/' ConnectToPipeline
;
Defines how to interconnect pipeline and connectors
Given
create connector ingest;
create connector egress;
create pipeline logic;
connect /connector/ingress/out to /pipeline/logic/in;
connect /pipeline/logic/out to /connector/egress/in;
Defines how the ingress
, egress
and logic
runtime instances
are interconnected for data to flow through them in a specified
order.
Rule ConnectFromConnector
The ConnectFromConnector
rule defines a route from a connector instance.
rule ConnectFromConnector ::=
'connector' '/' Ident MaybePort
;
connector/console/out
Connection from the console
connector definition via the standard out
port.
connector/console
The shorthand form where the standard out
port is implied can also be used.
Rule ConnectFromPipeline
The ConnectFromPipeline
rule defines route from a pipeline instance.
rule ConnectFromPipeline ::=
'pipeline' '/' Ident MaybePort
;
pipeline/filter/out
Connection from the filter
pipeline definition via the standard out
port.
pipeline/console
The shorthand form where the standard out
port is implied can also be used.
Rule ConnectToPipeline
The ConnectToPipeline
rule defines route to a pipeline instance.
rule ConnectToPipeline ::=
'pipeline' '/' Ident MaybePort
;
pipeline/filter/in
Connection to the filter
pipeline definition via the standard in
port.
pipeline/console
The shorthand form where the standard in
port is implied can also be used.
Rule ConnectToConnector
The ConnectToConnector
rule defines a route to a connector instance.
rule ConnectToConnector ::=
'connector' '/' Ident MaybePort
;
connector/console/in
Connection to the console
connector definition via the standard in
port.
connector/console
The shorthand form where the standard in
port is implied can also be used.
Rule DefineConnector
The DefineConnector
rule defines a connector.
A connector is a runtime artefact that allows tremor to connect to the outside world, or for the outside connector to connect to tremor to send and/or receive data.
The named connector can be parameterized and instanciated via the Create
rule
rule DefineConnector ::=
( DocComment ) ? 'define' 'connector' Ident 'from' ConnectorKind ArgsWithEnd
;
use std::time::nanos;
define connector metronome from metronome
with
config = {
"interval": nanos::from_millis(1)
}
end;
Define user defind connector metronome
from the builtin metronome
connector
using a 1 second periodicity interval.
define connector exit from exit;
Define user dfeind connector exit
from the builtin exit
connector
with no arguments specified.
Rule DefineFlow
The DefineFlow
rule defines a flow.
A flow is a runtime artefact that informs tremor how to interconnect and launch instances of pipelines and connectors.
A flow can define or use multiple in scope and already define pipelines and connectors and interconnect their streams together.
rule DefineFlow ::=
( DocComment ) ? 'define' 'flow' Ident DefinitionArgs 'flow' FlowStmts 'end'
;
define flow test
flow
use std::time::nanos;
define connector metronome from metronome
with
config = {
"interval": nanos::from_millis(1)
}
end;
define connector exit from exit;
define pipeline identity
args
snot = "badger",
pipeline
select args.snot from in into out;
end;
create connector metronome;
create connector exit;
create pipeline identity;
connect /connector/metronome to /pipeline/identity;
connect /pipeline/identity to /connector/exit;
end;
deploy flow test
A flow definition is a runnable or executable streaming program that describes the connectivity, the logic and how they are interconnected. A deploy statement is responsible for the actual deployment.
Rule Query
The Query
rule defines the logical entry point into Tremor's statement
oriented query grammar. The grammar is embedded into deployments via
define pipeline
statements.
Pipelines effectively provide a continous streaming abstraction for
event processing. The pipeline query is a graph of connected streams
and operators. The operators can be builtin or provided by users through
the script
operator.
Window definitions for use in windowed statements such as those supported by
the select
operation are also defined and named so that they can be used
by multiple operations and to compose tilt frames - chains of successive
windows of data in a streaming continuous select operation.
The syntax supports the definition of sub queries through the same define pipeline
syntax as in the deployment grammar.
rule Query ::=
ConfigDirectives Stmts
| Stmts
;
Rule Stmts
The Stmts
rule defines a ;
semi-colon delimited sequence of Stmt
rules.
rule Stmts ::=
Stmt ';' Stmts
| Stmt ';' ?
;
Example for a query pipeline
use tremor;
create stream snot;
create stream badger;
# ...
Rule Stmt
The Stmt
rule defines the legal statements in a query script.
Queries in tremor support:
- Defining named
window
,operator
,script
andpipeline
definitions. - Creating node instances of
stream
,pipeline
,operator
andscript
operations. - Linking nodes togther to form an execution graph via the
select
operation.
rule Stmt ::=
Use
| DefineWindow
| DefineOperator
| DefineScript
| DefinePipeline
| CreateOperator
| CreateScript
| CreatePipeline
| CreateStream
| OperatorSelect
;
Rule DefineWindow
The DefineWindow
rule defines a temporal window specification.
A window is a mechanism that caches, stores or buffers events for processing over a finite temporal range. The time range can be based on the number of events, the wall clock or other defined parameters.
The named window can be instanciated via operations that support windows such
as the select
operation.
rule DefineWindow ::=
( DocComment ) ? 'define' 'window' Ident 'from' WindowKind CreationWith ( ScriptState ) ? ( NamedScript ) ? ( EmbeddedScriptContent ) ? 'end'
;
define window four from tumbling
with
size = 4
end;
Rule DefineOperator
The DefineOperator
rule defines an operator.
An operator is a query operation composed using the builtin operators provided by tremor written in the rust programming language.
The named operator can be parameterized and instanciated via the CreateOperator
rule
rule DefineOperator ::=
( DocComment ) ? 'define' 'operator' Ident 'from' OperatorKind ArgsWithEnd
;
define pipeline subq
pipeline
define operator counter from generic::counter;
create operator counter;
select event from in into counter;
select event from counter into out;
end;
create pipeline subq;
select event from in into subq;
select event from subq into out;
Uses the builtin counter sequencing operator to numerate a stream.
Rule DefineScript
The DefineScript
rule defines a named operator based on a tremor script.
A script operator is a query operation composed using the scripting language DSL rather than the builtin operators provided by tremor written in the rust programming language.
The named script can be parameterized and instanciated via the CreateScript
rule
rule DefineScript ::=
( DocComment ) ? 'define' 'script' Ident DefinitionArgs ScriptState ? NamedScripts
;
define operator bucket from grouper::bucket;
define script categorize
script
let $rate = 1;
let $dimensions = event.logger_name;
let $class = "test";
event
end;
create operator bucket;
create script categorize;
select event from in into categorize;
select event from categorize into bucket;
select event from bucket into out;
Rule ScriptState
rule ScriptState ::=
'state' ExprImut
;
Rule NamedScripts
rule NamedScripts ::=
NamedScript NamedScripts
| EmbeddedScript
;
Rule NamedScript
rule NamedScript ::=
'script' 'from' Ident TopLevelExprs
;
Rule DefinePipeline
The DefinePipeline
rule creates a named pipeline.
A pipeline is a query operation composed using the query langauge DSL instead of a builtin operation provided by tremor written in the rust programming language.
The named pipeline can be parameterized and instanciated via the CreatePipeline
rule
rule DefinePipeline ::=
( DocComment ) ? 'define' 'pipeline' Ident ( 'from' Ports ) ? ( 'into' Ports ) ? DefinitionArgs Pipeline
;
define pipeline identity
pipeline
select event from in into out;
end;
Rule OperatorSelect
The OperatorSelect
rule provides the select
statement in streaming queries.
The statement has
- A target expression
- A stream and port from which to consume events
- A stream and port to which synthetic events are produced
- An optional set of iwndow definitions
- An optional
where
filter - An optional
having
filter - An optional
group by
Unlike ANSI-ISO SQL select operations in tremor do not presume tabular or columnar data. The target expression can be any well-formed and legal value supported by tremor.
rule OperatorSelect ::=
'select' ComplexExprImut 'from' StreamPort ( WindowClause ) ? ( WhereClause ) ? ( GroupByClause ) ? 'into' StreamPort ( HavingClause ) ?
;
The builtin select
operator in queries.
A simple non-windowed non-grouped select
select event from in into out;
A simple non-windowed grouped select
select event from in into out group by event.key;
A windowed grouped select operation
select event from in[one_sec] by event.key into out;
Multiple windows can be configured in lower resolutions for multi-resolution windowed expressions where lower resolutions are merged into from higher resolution windows
select aggr::stats::hdr(event.count) form in[one_sec, fifteen_sec, one_min, one_hour] into out;
Rule CreateStream
The CreateStream
allows users to create user defined streams beyond the basic
buitin set of in
, out
and err
provided by the runtime for a pipeline query.
rule CreateStream ::=
'create' 'stream' Ident
;
create stream ctrl;
Creates a user defined stream by the provided name ctrl
Rule CreateScript
The CreateScript
rule creates an operator based on a tremor script.
A script operator is a query operation composed using the scripting language DSL rather than the builtin operators provided by tremor written in the rust programming language.
The rule causes an instance of the referenced script definition to be created an inserted into the query processing execution graph.
rule CreateScript ::=
'create' 'script' Ident CreationWithEnd
| 'create' 'script' Ident 'from' ModularTarget CreationWithEnd
;
define operator bucket from grouper::bucket;
define script categorize
script
let $class = "test";
let $dimensions = [event.type, event.application];
let $rate = 1;
event;
end;
create operator bucket;
create script categorize;
select event from in into categorize;
select event from categorize into bucket;
select event from bucket into out;
Rule CreateOperator
The CreateOperator
rule creates an operator.
An operator is a query operation composed using the builtin operators provided by tremor written in the rust programming language.
The rule causes an instance of the referenced operator definition to be created an inserted into the query processing execution graph.
rule CreateOperator ::=
'create' 'operator' Ident CreationWithEnd
| 'create' 'operator' Ident 'from' ModularTarget CreationWithEnd
;
# Define a round robin operator with 3 slots
define operator roundrobin from qos::roundrobin
with
outputs = ["one", "two", "three"]
end;
# create an instance of the operator
create operator roundrobin;
# Filter all inbound events into the rond robin
select event from in into roundrobin;
# Union slots inot outbound port
select event from roundrobin/one into out;
select event from roundrobin/two into out;
select event from roundrobin/three into out;
Rule CreatePipeline
The CreatePipeline
rule creates a pipeline.
A pipeline is a query operation composed using the query langauge DSL instead of a builtin operation provided by tremor written in the rust programming language.
The rule causes an instance of the referenced pipeline definition to be created an inserted into the query processing execution graph.
rule CreatePipeline ::=
'create' 'pipeline' Ident CreationWithEnd
| 'create' 'pipeline' Ident 'from' ModularTarget CreationWithEnd
;
# Define a pipeline called `identity`
define pipeline identity
pipeline
select event from in into out;
end;
# Create an instance of the pipeline
create pipeline identity
Rule MaybePort
The MaybePort
rule defines an optional Port
.
rule MaybePort ::=
( '/' Ident ) ?
;
When interconnecting pipelines and connectors in flow definitions default ports can be inferred by the tremor runtime.
When an alternate port is required, the port specification can be used to explicitly select from available inbound or outbound ports.
Rule StreamPort
The StreamPort
rule defines a stream by name with an optional named Port
.
When the Port
is omitted, tremor will internally default the Port
to the
appropriate in
or out
port. Where the err
or user defined Port
s are
preferred, the optional Port
specification SHOULD be provided.
rule StreamPort ::=
Ident MaybePort
;
in/snot
Within a query, allows the port of a specific stream to be referenced directly.
Rule WindowKind
Tumbling
A tumbling
window defines a wall-clock-bound or data-bound window of non-overlapping
time for storing events. The windows can not overlap, and there are no gaps between
windows permissible.
Sliding
A sliding
window defines a wall-clock-bound or data-bound window of events that captures
an intervalic window of events whose extent derives from the size of the window. A sliding
window of size 2 captures up to to events. Every subsequent event will evict the oldest and
retain the newest event with the previous ( now oldest ) event.
Conditioning
Both kinds of window store events in arrival order
rule WindowKind ::=
'sliding'
| 'tumbling'
;
Currently only tumbling
is implemented.
Rule WindowClause
The WindowClause
rule defines an optional window definition for a supporting operation.
rule WindowClause ::=
WindowDefn
;
Rule Windows
The Windows
rule defines a sequence of window definitions that are ,
comma delimited.
rule Windows ::=
Windows_
;
Wraps the Windows_
internal rule for other rules to consume in their definitions
Rule Windows_
The Windows_
rule defines a sequence of window definitions that are ,
comma delimited.
rule Windows_ ::=
Sep!(Windows_, Window, ",")
;
A comma delimited set of window references.
Windows can be local or modular
win, my_module::one_sec, my_module::five_sec
The identifers refer to a window definition and can be used in operators to define a temporally bound set of events based on the semantics of the window definition.
In a tilt frame - or set of windows, the output of a window can is the input the next window in a sequence. This is a form of temporal window-driven event compaction that allows memory be conserved.
At 1000 events per second, a 1 minute window needs to store 60,000
events per group per second. But 60 1 second windows can be merged
with aggregate functions like dds
and hdr
histograms.
Say, each histogram is 1k of memory per group per frame - that is a cost of 2k bytes per group.
In a streaming system - indefinite aggregation of in memory events is always a tradeoff against available reosurces, and the relative business value.
Often multiple windows in a tilt frame can be more effective than a single very long lived window.
Rule Window
The Window
rule defines a modular target to a window definition.
rule Window ::=
ModularTarget
;
Rule WindowDefn
The WindowDefn
defines a temporal basis over which a stream of events is applicable.
rule WindowDefn ::=
'[' Windows ']'
;
Operations supporting windowed aggregate functions in tremor such as the select
statement can window incoming streams in the from
clause:
select aggr::count(event) from in[one_second, ten_second]
...
Here, we stream events from the in
stream into a one_second
window.
Every second, we stream the aggregate result from the one second window
into the ten_second
window.
So, even if we have 1 million events per second, the one_second
and ten_second
windows will convert the event firehose into a trickle
. Fun fact: this pun is where the
query language got its name from.
Rule WhereClause
The WhereClause
defines a predicate expression used to filter ( forward or discard ) events in an operation.
The where
clause is executed before a operation processes an event.
rule WhereClause ::=
'where' ComplexExprImut
;
select event from in
where present event.important
into out
The where
filters events before computations occur upon them in operators
that support the clause. Any predicate ( boolean ) expression can be used
in a where
filter.
Rule HavingClause
The HavingClause
defines a predicate expression used to filter ( forward or discard ) events in an operation.
The having
clause is executed after an operation has processed an event.
rule HavingClause ::=
'having' ComplexExprImut
;
select event from in
having present event.important
into out
The having
filters events after computations has occured within them in operators
that support the clause. Any predicate ( boolean ) expression can be used
in a having
filter.
When appropriate, the where
clause should be used in preference over the having
clause.
It is better to filter early before computation occurs when this is practicable or possible.
Rule GroupByClause
The GroupByClause
defines the group by clause of a supporting operation in tremor.
An operator that uses a group by clause maintains the operation for each group captured by the grouping dimensions specified in this clause.
rule GroupByClause ::=
'group' 'by' GroupDef
;
select event from in[one_second]
having present event.important
group by event.priority
into out
The group by
clause groups events based on a group expression. Each computed group effectively
has its own memory and computation allocated.
For windowed operations the windows are allocated for each group.
These groups and their windows are independant. This means that opening, closing, filling and recycling of windows is by group.
Rule GroupDef
The GroupDef
rule defines the parts of a grouping dimension.
Group segments can be derived from:
- Expressions - for which their serialized values are used.
- Set expressions - which computes a set based on an expression.
- Each expressions - which iterates an expression to compute a set.
rule GroupDef ::=
ExprImut
| 'set' '(' GroupDefs ')'
| 'each' '(' ExprImut ')'
;
Example group definitions
A string value
event.string
The serialization of any legal tremor data value
event
A set based on multiple expressions:
set(event.key, state[key])
An set computed from an interation
let keys = ['snot', 'badger', 'goose'];
...
each(keys)
## Rule GroupDefs
The `GroupDefs` rule defines a `,` comma delimited set of `GroupDef` rules.
![GroupDefs](svg/groupdefs.svg)
```ebnf
rule GroupDefs ::=
GroupDefs_
;
Wraps a macro call for use by other productions in the grammar
Rule GroupDefs_
The GroupDefs_
rule defines a ,
comma delimited set of GroupDef
rules.
rule GroupDefs_ ::=
Sep!(GroupDefs_, GroupDef, ",")
;
A comma delimited set of GroupDef
productions
Rule EmbeddedScriptImut
The EmbeddedScriptImut
rule defines an optional embedded script
.
rule EmbeddedScriptImut ::=
'script' EmbeddedScriptContentImut
;
Rule EmbeddedScriptContentImut
rule EmbeddedScriptContentImut ::=
ExprImut
;
Rule Ports
The Ports
rule defines a ,
comma delimited set of stream ports.
rule Ports ::=
Sep!(Ports, <Ident>, ",")
;
A set of ports
exposed in pipeline definitions in their from
and into
clauses
define pipeline example
from in, out, err, ctrl
into in, out, err, ctrl
pipeline
# A pipeline query implementation
...
end
The from
and into
ports do not need to be the same.
Tremor's compiler and runtime can use these definitions to validate deployments are correct, or discover deployments that are invalid. It is an error to send data to or receive data from a pipeline port that is not specified.
Rule OperatorKind
The OperatorKind
rule defines a modular path like reference to a builtin tremor operator.
Operators are programmed in rust native code and referenced via a virtual module path.
rule OperatorKind ::=
Ident '::' Ident
;
A modular path identifying a builtin operator.
define operator roundrobin from qos::roundrobin;
Rule EmbeddedScript
The EmbeddedScript
rule defines a script using the Script DSL [ Full ].
The script is enclosed in script
.. end
blocks.
rule EmbeddedScript ::=
EmbeddedScriptContent 'end'
;
script
event
end
Rule EmbeddedScriptContent
The EmbeddedScriptContent
rule defines an embedded script expression.
rule EmbeddedScriptContent ::=
'script' TopLevelExprs
;
A single expression embedded within an embedded script
Rule Pipeline
The Pipeline
rule defines a block of statements in a pipeline
.. end
block.
The block MAY begin with an optional set of ConfigDirectives
.
rule Pipeline ::=
'pipeline' ConfigDirectives ? PipelineCreateInner 'end'
;
An internal rule to the DefinePipeline
rule where the pipeline logic is provided.
Rule PipelineCreateInner
The PipelineCreateInner
is an internal rule of the Pipeline
rule.
The rule defines a ;
semi-colon delimited set of one or many Stmt
s.
rule PipelineCreateInner ::=
Stmt ';' Stmts
| Stmt ';' ?
;
This rule allows queries to be defined in the context of a pipeline definition.
Rule Script
The Script
rule defines the logical entry point into Tremor's expression
oriented scripting language. The scripting langauge can be embedded into
queries via the script
operator. The scripting language is also used to
specify configuration of connectors, pipelines, flows, and operators in
the query language.
A legal script is composed of:
- An optional set of module comments
- A sequence of top level expressions. There must be at least one defined.
- An optional end of stream token
Scripting Language Entrypoint
This is the top level of the tremor scripting language tremor
rule Script ::=
( ModComment ) ? TopLevelExprs
;
Type system
Tremor supports a data oriented or value based type system with a syntax that is backwards compatible with JSON.
Any well-formed and legal JSON document is a valid literal in tremor.
Tremor literals for null
, boolean
, string
( utf-8 ), integer ( 64-bit unsigned ),
float ( 64-bit ieee ), arrays, and records are equivalent to their JSON counterparts.
Tremor also supports a binary literal for transporting and processing opaque binary data.
Asymmetric
JSON literals are valid tremor value literals.
Tremor literals MAY NOT always be valid JSON literal.
# The following literal is valid JSON and valid Tremor
[1, "snot", {}];
# The following literal is valid in tremor only
[1, "snot", {}, << data/binary >>, ];
Tremor supports comments, JSON does not. Tremor supports trailing commas in arrays and records, JSON does not. Tremor supports binary literal data, JSON does not.
Note: By default, most connectors in tremor serialize to and from json
via a codec. The
type system in tremor however is agnostic to the wire format of data that flows through
tremor. So data originate as json
, as msgpack
.
Computations
Tremor also supports a rich expression language with the same support for additive, mutliplicate,
comparitive, and logical unary and binary expressions as languages like rust
and java
.
As most of the data that flows through tremor is heirarchically structured or JSON-like tremor also has rich primitives for structural pattern matching, structural comprehension or iterating over data structures.
Loops
Tremor does not support while
loop
or other primitives that can loop, recurse or iterate
indefinitely.
In an event based system, events are streaming continuously - so infinite loops that can block streams from making forward progress are considered harmful.
There are no loops.
We do support iteration over finite arrays.
We do support depth-limited tail recursive functional programming.
Expression oriented
The script processing is expression oriented. This is to say that every structural form supported by tremor returns a data structure as a result.
Event oriented
Scripts in tremor can emit
or drop
an `event that is being processed.
The event
keyword is the subject. It identifies the value currently being processed.
The emit
keyword halts processing succesfully with a value.
The drop
keyword halts processing by discarding the current event.
Illustrative example
# Propagate events marked as important and convert them to system alerts
match event of
case %{ present important } => { "alert": event.message }
case _ => drop
end;
Rule TopLevelExprs
The TopLevelExprs
rule defines semi-colon separated sequence of top level
tremor expressions with an optional terminating semi-colon
rule TopLevelExprs ::=
TopLevelExpr ';' TopLevelExprs
| TopLevelExpr ';' ?
;
The ToplEvelExprs
specifies the expressions that are legal at the top level
of a script
expression.
script
event.sum * 2
end
A sequence of ';' semi-colon delimited expressions of the following form are permissible:
- Constants
- Function definitions
- Intrinsic function definitions in the standard library
- Provided by the runtime to document builtins - not user modifiable without a pull request or feature enhancement
- Arbitrary complex expressions
- Use definitions
Rule InnerExprs
The InnerExprs
rule defines the expression forms permissible within another
containing scope. Like TopLevelExprs
, inner expressions are separated by semi-colons.
The semi-colon is optional for the last expression in a set of expressions.
At least one expression MUST be provided.
rule InnerExprs ::=
Expr ';' InnerExprs
| Expr ';' ?
;
Example
The following basic forms are permissible
for event.object of case (k, v) => v end;
for event.list of case (i, e) => e end;
match event of case %{} => "record" case _ => "not a record" end;
let list = event.list;
drop;
"any literal or basic expression";
1 + 2 * 3;
emit {"snot": "badger" }
# ...
Rule TopLevelExpr
The TopLevelExpr
rule specifies the expression forms that are legal at the
outer most scope of a tremor script definition.
The legal forms are:
- Use declarations - these allow external modules to be referenced.
- Constant expressions - these are immutable compile time constants.
- Function definitions - these are user defined functions.
- Intrinsic function definitions - these are builtin funtions provided by the runtime.
rule TopLevelExpr ::=
Const
| FnDefn
| Intrinsic
| Expr
| Use
;
Example
In the tremor standard library many of the top level expressions
are use
definitions importing sub modules from the module path.
use std::array; # Import the std array utilities
use std::base64 # Import the std base64 utilities;
use std::binary; # ...
use std::float;
use std::integer;
use std::json;
Rule Const
The Const
rule defines a rule that binds an immutable expression to an identifier.
As the value cannot be changed at runtime.
Constants, by convention are ALL_UPPERCASE to distinguish them from variables. While it is allowed to deviate from this convention, it is not recommended and you will be yelled at by the compiler.
rule Const ::=
( DocComment ) ? 'const' Ident '=' ComplexExprImut
;
How do I create new immutable constant variable in tremor?
use std::base64;
const SNOT = "snot";
const BADGER = "badger";
const SNOT_BADGER = { "#{snot}": "#{base64::encode(badger)}" };
Rule Expr
The Expr
rule aliases the SimpleExpr
rule.
The alias allows higher levels of the DSL such as the rules in the deployment or query language to avoid some of the internal complexity in the scripting language.
Within the scripting DSLs grammar the different forms and variations of expression are significant.
However, in the higher level we limit exposure to a subset of these forms. This is done for convenience, and for consistency of usage, and ease of learning the language.
rule Expr ::=
SimpleExpr
;
The effective root of the subset of the expression langauge applicable in most processing context in tremor is captured by this rule.
Rule SimpleExpr
The SimpleExpr
rule defines all the structural and simple expressions and literals in tremor.
rule SimpleExpr ::=
Match
| For
| Let
| Drop
| Emit
| ExprImut
;
Structural expressions
- Match
- For
Assignment
- Let
Flow
- Emit
- Drop
Basic expressions
Binary, unary and literal expressions
Rule AlwaysImutExpr
The AlwaysImutExpr
defines the immutable expression forms in tremor.
Immutable expressions can be reduced at compile time and folded into literals.
rule AlwaysImutExpr ::=
Patch
| Merge
| Invoke
| Literal
| Path
| Record
| List
| StringLiteral
| BytesLiteral
| Recur
;
Rule Recur
The Recur
rule defines stack-depth-limited tail-recursion in tremor functions.
rule Recur ::=
'recur' '(' ')'
| 'recur' '(' InvokeArgs ')'
;
fn fib_(a, b, n) of
case (a, b, n) when n > 0 => recur(b, a + b, n - 1)
case _ => a
end;
fn fib(n) with
fib_(0, 1, n)
end;
Tremor's functional programming langauge supports tail recursion via the
recur
keyword. Tail recursion in tremor is limited to a fixed stack
depth - infinite recursion is not permissible.
Rule ExprImut
The ExprImut
is the root of immutable expressions in tremor.
rule ExprImut ::=
OrExprImut
;
The effective root of the subset of the expression langauge applicable in most immutable processing context in tremor is captured by this rule.
Rule OrExprImut
The OrExprImut
rule supports logical or expressions in tremor.
Binary logical or expressions take precedence over logical exclusive or expressions.
rule OrExprImut ::=
BooleanBinOp!(BinOr, ExprImut, XorExprImut)
| XorExprImut
;
let example = a or (b & 0);
Rule XorExprImut
The XorExprImut
rule supports logical exclusive or expressions in tremor.
Binary logical exclusive or expressions take precedence over logical and expressions.
rule XorExprImut ::=
BooleanBinOp!(BinXor, XorExprImut, AndExprImut)
| AndExprImut
;
let example = a ^ (b & 0);
Rule AndExprImut
The AndExprImut
rule supports logical and expressions in tremor.
Binary logical and expressions take precedence over bitwise or expressions.
rule AndExprImut ::=
BooleanBinOp!(BinAnd, AndExprImut, BitOrExprImut)
| BitOrExprImut
;
x + y
Rule BitOrExprImut
The BitOrExprImut
rule supports bitwise or expressions in tremor.
Binary bitwise or expressions take precedence over bitwise exclusive or expressions.
rule BitOrExprImut ::=
BitXorExprImut
;
let example = a | (b & 0);
Rule BitXorExprImut
The BitXorExprImut
rule supports bitwise exclusive or expressions in tremor.
Binary bitwise exclusive or expressions take precedence over bitwise and expressions.
rule BitXorExprImut ::=
BinOp!(BinBitXor, BitXorExprImut, BitAndExprImut)
| BitAndExprImut
;
let example = a ^ (b & 0);
Rule BitAndExprImut
The BitAndExprImut
rule supports bitwise and expressions in tremor.
Binary bitwise and expressions take precedence over equality expressions.
rule BitAndExprImut ::=
BinOp!(BinBitAnd, BitAndExprImut, EqExprImut)
| EqExprImut
;
let example = a & (b & 0);
Rule EqExprImut
The EqExprImut
rule supports equality expressions in tremor.
Binary equality expressions take precedence over comparitive expressions.
rule EqExprImut ::=
BinOp!(BinEq, EqExprImut, CmpExprImut)
| CmpExprImut
;
let example = a == (b & 0);
Rule CmpExprImut
The CmpExprImut
rule supports comparative expressions in tremor.
Binary comparative expressions take precedence over bit shift expressions.
rule CmpExprImut ::=
BinOp!(BinCmp, CmpExprImut, BitShiftExprImut)
| BitShiftExprImut
;
x > y
Rule BitShiftExprImut
The BitShiftExprImut
rule supports bit shift expressions in tremor.
Binary bit shift expressions take precedence over bitwise additive expressions.
rule BitShiftExprImut ::=
BinOp!(BinBitShift, BitShiftExprImut, AddExprImut)
| AddExprImut
;
let example = a >>> (b & 0);
Rule AddExprImut
The AddExprImut
rule supports additive expressions in tremor.
Binary additive expressions take precedence over multiplicative expressions.
rule AddExprImut ::=
BinOp!(BinAdd, AddExprImut, MulExprImut)
| MulExprImut
;
x - y
Rule MulExprImut
The MulExprImut
rule supports multiplicative expressions in tremor.
Binary multiplicative expressions take precedence over unary expressions.
rule MulExprImut ::=
BinOp!(BinMul, MulExprImut, UnaryExprImut)
| UnaryExprImut
;
x * y
Rule UnaryExprImut
The UnaryExprImut
rule specifies unary expression operations.
Expressions can be marked as +
positive, -
negative explicitly when needed.
Otherwise, the expression reduces to a simple unary expression.
The simple unary expression has lower precedence.
rule UnaryExprImut ::=
'+' UnaryExprImut
| '-' UnaryExprImut
| UnarySimpleExprImut
;
Is the root expression for unary expressions in the language.
-(1)
Rule UnarySimpleExprImut
The UnarySimpleExprImut
rule specifies predicate unary expression operations.
Expressions can be marked explicitly with not
or !
to negate the target simple presence expression.
Otherwise, the expression reduces to a simple presence expression.
The simple presence expression has lower precedence.
rule UnarySimpleExprImut ::=
'not' UnarySimpleExprImut
| '!' UnarySimpleExprImut
| PresenceSimplExprImut
;
not (a and b)
Rule PresenceSimplExprImut
The PresenceSimplExprImut
rule specifies presence and simple expressions
Expressions path predicate tests based on the present
and absent
predicate test
expressions, or a simple expression.
Otherwise, the expression reduces to a simple expression.
The simple expression has lower precedence.
rule PresenceSimplExprImut ::=
'present' Path
| 'absent' Path
| SimpleExprImut
;
present $.request.method
absent state.cache
Rule ComplexExprImut
The ComplexExprImut
rule defines complex immutable expression in tremor.
rule ComplexExprImut ::=
MatchImut
| ForImut
| ExprImut
;
Rule Intrinsic
The intrinsic
rule defines intrinsic function signatures.
This rule allows tremor maintainers to document the builtin functions implemented as native rust code. The facility also allows document generation tools to document builtin intrinsic functions in the same way as user defined functions.
In short, these can be thought of as runtime provided.
For information on how to define user defined functions see the function rule.
rule Intrinsic ::=
( DocComment ) ? 'intrinsic' 'fn' Ident '(' ')' 'as' ModularTarget
| ( DocComment ) ? 'intrinsic' 'fn' Ident '(' FnArgs ')' 'as' ModularTarget
| ( DocComment ) ? 'intrinsic' 'fn' Ident '(' FnArgs ',' '.' '.' '.' ')' 'as' ModularTarget
| ( DocComment ) ? 'intrinsic' 'fn' Ident '(' '.' '.' '.' ')' 'as' ModularTarget
;
Example
From our standard library generated documentation, we can see that the base64 encode function is an intrinsic function.
## Encodes a `binary` as a base64 encoded string
##
## Returns a `string`
intrinsic fn encode(input) as base64::encode;
Rule FnDefn
The FnDefn
rule allows user defined functions to be defined.
This rule allows tremor users to create functions for reuse in one or many tremor applications.
rule FnDefn ::=
( DocComment ) ? 'fn' Ident '(' '.' '.' '.' ')' 'with' InnerExprs 'end'
| ( DocComment ) ? 'fn' Ident '(' FnArgs ',' '.' '.' '.' ')' 'with' InnerExprs 'end'
| ( DocComment ) ? 'fn' Ident '(' ')' 'with' InnerExprs 'end'
| ( DocComment ) ? 'fn' Ident '(' FnArgs ')' 'with' InnerExprs 'end'
| ( DocComment ) ? 'fn' Ident '(' ')' 'of' FnCases 'end'
| ( DocComment ) ? 'fn' Ident '(' FnArgs ')' 'of' FnCases 'end'
;
Pattern match based function arguments
Functions defined with an of
keyword in their signature use pattern matching against arguments
fn fib_(a, b, n) of
case (a, b, n) when n > 0 => recur(b, a + b, n - 1)
case _ => a
end;
Ordinary functions
Functions defined with a with
keyword in their signature use ordinary
arity based matching.
fn fib(n) with
fib_(0, 1, n)
end;
Function documentation
In modular functions, it is customary to provide user level documentation for the intended users of a function. Here is an example from the tremor standard library
### Trace Identifiers
###
###
use std::type;
use std::binary;
use std::array;
use std::string;
## Is the `trace_id` valid
##
## Checks the `trace_id` argument to see if it is a valid
## trace id. A legal trace id is one of:
##
## * An array of integers in the range of [0..=255] of length 8
## * A binary 16 byte value
## * A 32-byte hex-encoded string
## * An array of 16 int values
## * Regardless of representation, the value must not be all zeroes
##
## Returns a record when the representation is well-formed of the form:
##
## ```tremor
## {
## "kind": "string"|"binary"|"array", # Depends on input
## "valid": true|false, # True if well-formed and valid
## "value": "<trace_id>" # Representation depends on `kind`
## }
## ```
##
## Returns an empty record `{}` when the representation not well-formed
##
fn is_valid(trace_id) of
# String representation
case(trace_id) when type::is_string(trace_id) =>
{ "kind": "string", "valid": trace_id != "00000000000000000000000000000000" and string::bytes(trace_id) == 32, "value": trace_id }
# Binary representation
case(trace_id) when type::is_binary(trace_id) =>
let arr = binary::into_bytes(trace_id);
{ "kind": "binary", "valid": binary::len(arr) == 16 and trace_id != << 0:64, 0:64 >>, "value": trace_id }
# Array representation
case(trace_id) when type::is_array(trace_id) =>
{ "kind": "array", "valid": array::len(arr) == 16 and trace_id != [ 0, 0, 0, 0, 0, 0, 0, 0], "value": trace_id }
case _ =>
false
end
Rule FnCases
The FnCases
rule defines a sequence of cases for structural pattern matching in tremor pattern functions.
rule FnCases ::=
FnCaseClauses FnCaseDefault
| FnCaseDefault
;
connector/console/in
The connector console
via its standard in
port
Rule FnCaseDefault
The FnCaseDefines
rule defines a default match clause for use in pattern match function signatures in tremor.
rule FnCaseDefault ::=
'case' '_' Effectors
;
Rule FnCase
The FnCase
rule defines an array predicate pattern supporting match clause for use in pattern match function signatures in tremor.
rule FnCase ::=
'case' '(' ArrayPredicatePatterns ')' WhenClause Effectors
;
A case definition in a pattern match function definition
use std::type;
fn snottify(s) of
# Matches the literal string "badger"
case ("badger") => "snot badger, hell yea!"
# Matches any well formed json argument
case (~ json||) => let s.snot = true; s
# Matches any literal string
case (s) when type::is_string(s) => "snot #{s}"
# Matches, everything else
case _ => "snot caller, you can't snottify that!"
end;
Rule FnCaseClauses
The FnCaseClauses
defines the case syntax to structurally matched function signatures in tremor.
rule FnCaseClauses ::=
FnCase
| FnCaseClauses FnCase
;
Rule FnArgs
The FnArgs
rule defines ,
comma delimited arguments to a tremor function.
rule FnArgs ::=
Ident
| FnArgs ',' Ident
;
Rule SimpleExprImut
The SimpleExprImut
rule defines optionally parenthesized simple immutable expressions in tremor.
rule SimpleExprImut ::=
'(' ComplexExprImut ')'
| AlwaysImutExpr
;
A parenthetic or simple non-parenthetic expression
( 1 + 2 ) * 3;
Rule Literal
The Literal
rule defines the set of primitive literals supported in tremor.
rule Literal ::=
Nil
| Bool
| Int
| Float
;
Primitive literal types in tremor are integer, floating point, boolean and the null
reference.
String literals are UTF-8 encoded values and support interpolation with single line ( regular ) and multi line ( heredoc style ) variant forms.
Rule Nil
The Nil
rule defines the syntax of the nil literal in tremor.
rule Nil ::=
'nil'
;
Example
null # The `null` literal value
Rule Bool
The Bool
rule defines the syntax of boolean literal in tremor.
rule Bool ::=
'bool'
;
Example
true # The boolean `true` literal
false # The boolean `false` literal
Rule Int
The Int
rule literal specifes the syntax of integer literals in tremor.
rule Int ::=
'int'
;
Integer literals
Integer literals in tremor are 64 bit signed values of the form
1234
Values can be separated by an _
( underscore ) for easy reading
let million = 1_000_000
Rule Float
The Float
rule literal specifes the syntax of IEEE float literals in tremor.
rule Float ::=
'float'
;
Floating point literals
Floating point literals in tremor are 64 bit signed IEEE floating point values of the form
1234.0e10
Values can be separated by an _
( underscore ) for easy reading
let million = 1_000_000.1234e-5
Rule StringLiteral
The StringLiteral
rule defines a string literal in tremor.
Strings are "
single-quote or """
triple-quote delimited blocks of UTF-8 text.
A single-quote string is a single line string, supporting sting interpolation.
A triple-quote string is a multi-line string, supporting sting interpolation.
rule StringLiteral ::=
'heredoc_start' StrLitElements 'heredoc_end'
| '\\' StrLitElements '\\'
| '\\' '\\'
;
How do I define single line string literals?
A single line string MUST be on a single line with raw newline characters ( unless escaped ).
"I am a literal string"
How do i define multi line string literals?
A multi line string MUST span multiple lines with raw newline characters.
Multi line strings
"""
I am a
multi
line
string
"""
The following example is a malformed multi line string:
""" snot """
Which when executed will result in a compile time error:
Error:
1 | """ snot """
| ^^^ It looks like you have characters tailing the here doc opening, it needs to be followed by a newline
Simple and nested Interpolation
Strings in tremor can be interpolated with internal scripts
"""
I am an #{interpolated} #{event.sum / event.count} string
Interpolations can be simple, as above, or #{
merge event of
{ "#{snot}": """
#{badger * 1000 + crazy_snake }
""" }
}
"""
This will result in the output:
"\nI am an interpolated 5.0 string\n\nInterpolations can be simple, as above, or {\"sum\":10,\"count\":2,\"snot\":\"\\n 20001\\n\\n \"}\n"
Note that the merge operation merges an event { "sum": 10, "count": 10 }
with in scope values of snot
that evaluates to the literal string "snot"
and the numerics badger
( 20
) and crazy_snake ( 1
). Interpolations are nestable and field names or any other string literal in tremor can be interpolated.
However, we do not recommend complex nested interpolated strings. Defining a function and calling it may be a better alternative for most applications and uses.
Rule StrLitElements
The StrLitElements
rule defines the internal structure of a string literal in tremor.
String literal in tremor support string interpolation via the #{
and }
escape
sequence. Content within the escape sequence can be any legal and valid tremor
expression.
rule StrLitElements ::=
StringPart StrLitElements
| '\\\\#' StrLitElements
| '#{' ExprImut '}' StrLitElements
| StringPart
| '\\\\#'
| '#{' ExprImut '}'
;
Inside a literal tremor string
A string literal in tremor is a composition of multiple segments or parts.
These can be composed of:
- One or many single line string parts
- One or many multi line string parts
- A blackslash escaped
\\#
to escape interpolated syntax, optinally followed by more string literal parts - Or, a
#{
..}
delimited interpolated section- Within an interpolated section there are no constraints on raw newline usage
- For complex interpolated sections, prefer good indentation!
Rule StringPart
The StringPart
rule defines a simple or heredoc style string part.
rule StringPart ::=
'string'
| 'heredoc'
;
The "
delimited string is single line
The """
delimited string is multi line
The forms can be mixed to construct a string literal
Interpolation is supported by both forms
Rule List
The List
rule defines a [
and ]
square bracket delimited sequence of zero or many ',' delimited expressions.
rule List ::=
'[' ListElements ']'
| '[' ']'
;
How are literal array or list or vectorc data collections defined in tremor?
We use [
.. ]
square brackets to delimit list-like data.
Are multi-dimensional arrays supported?
Multi-dimensional arrays are compositional and can be nested
Does tremor support typed lists?
No. A list in tremor can have elements from any supported primitive or structural type.
Validation that a list is for a single type - such as a list of boolean values can be defined as follows:
use std::type;
let bad = [true, false, "snot"];
let good = [true, false ];
fn list_of_bool(l) with
let valid = true;
for l of
case (i,e) when type::is_bool(e) and valid == true => let valid = true
case (i, otherwise) => let valid = false
end;
valid # return true if list is all bool, false otherwise
end;
list_of_bool(bad); # should fail
list_of_bool(good); # should succeed
This user defined function can then be used in guard clauses like when type::is_bool(e) ...
in the
example code.
Rule ListElements
The ListElements
rule defines a ,
comma delimited sequence of expression elements.
rule ListElements ::=
ListElements_
;
List literals
Unlike JSON, trailing commas are supported
[foo,] # A non empty list, a trailing comma is optionally permissible
Except in empty lists, where the idiomatic form is preferred:
[] # An empty list - no trailing comma here!
Rule ListElements_
The ListElements_
rule is internal to the ListElements
rule.
The rule defines a sequence of ,
comma delimited expression elements using the Sep
macro rule.
rule ListElements_ ::=
Sep!(ListElements_, ComplexExprImut, ",")
;
See ListElements
rule for details.
This rule wraps away a lalrpop macro call for ease of reference in other rules in the grammar source.
Rule Record
The Record
rule defines a set of name-value pairs delimited by ,
a comma.
Records are enclosed in {
and }
curly braces.
The record structure in tremor is backwards compatible with JSON.
All JSON records can be read by tremor.
Not all tremor records can be read by a JSON reader as tremor supports computations, comments and trailiing ,
commas
in its record and array structures.
rule Record ::=
'{' Fields '}'
| '{' '}'
;
How are literal records or map types or key-value collections defined in tremor?
We use {
.. }
squigly braces to delimit record -like data.
Can records be nested?
Record values can be any valid primitive of structural type supported by tremor, including other records.
Does tremor support typed records?
No. A record in tremor can have element values from any supported primitive or structural type in tremor.
Validation that a record conforms to a shape, schema or structure can be achieved through match expressions.
use std::type;
let bad = { "list-of-bool": [true, false, "snot"] };
let good = { "list-of-bool": [true, false ], "flag": true };
fn list_of_bool(l) with
let valid = true;
for l of
case (i,e) when type::is_bool(e) and valid == true => let valid = true
case (i, otherwise) => let valid = false
end;
valid # return true if list is all bool, false otherwise
end;
fn is_good_record(r) with
let valid = false;
match r of
case extract=%{ present flag, list-of-bool ~= %[] }
when list_of_bool(extract["list-of-bool"]) =>
let valid = true
case _ =>
let valid = false
end;
valid
end;
is_good_record(bad); # should fail
is_good_record(good); # should succeed
This user defined function can then be used in guard clauses like when type::is_bool(e) ...
in the
example code.
Rule Field
The Field
rule defines a :
colon delimited name value pair for a record literal.
The name is a string literal.
The value is an expression.
rule Field ::=
StringLiteral ':' ComplexExprImut
;
How do i define record fields?
The syntax is similar to JSON:
{ "field": "value" }
With the exception that fields may have an optional ',' terminal trailing comma
Interpolated field names
As literal strings in tremor support string interpolation the following variants are equivalent:
{ "snot": "badger" }
let snot = "snot";
{ "#{snot}": "badger" };
But, not all legal variations are recommended:
let snot = """
snot"""; # This may result in tears
{ "#{snot}": "badger" };
Another legal but likely not useful variation:
let snot = { "snot": "badger" };
{ "#{snot}": "badger" };
Will result in a stringifield json being encoded as the field name:
{"{\"snot\":\"badger\"}":"badger"}
Rule Path
The Path
rule defines path operations over expressions.
Path operations structures to be tersely indexed in a path like structure.
Path operations are supported on
- A subset of expressions ( record, array, function )
- Meta keywords like
$
,args
,state
,event
,group
,window
rule Path ::=
MetaPath
| EventPath
| StatePath
| LocalPath
| ConstPath
| AggrPath
| ArgsPath
| ExprPath
;
Path expressions are how events and in memory state in tremor are referenced in scripts.
How to reference metadata
The tremor runtime can provide and consume metadata with the events being processed by the runtime.
Metadata is distinguished by a $
dollar sign
let foo = $request # The local path `foo` is a copy of the metadata `$request`
Connectors such as kafka
and http
can generate metadata that scripts, queries
and pipelines can manipulate and process to tune tremor's runtime behaviour.
How to reference the current streaming event
The current event streaming through tremor in the current pipeline will
be available to queries, logics and scripts via the event
keyword.
The event
keyword can be further dereferenced via path statements
# Where event is a record
let foo = event.snot; # The local path `foo` is a copy of the `snot` field from the current event
# Where event is an array
let foo = event[10]; The local path `foo` is a copy of the 10th element of the current event.
How to reference pipeline state
Scripts in tremor can store state that is available for the lifetime of a pipeline
via the state
keyword.
The state
keyword can be further dereferenced via path statements
# Where state is a record
let foo = state.snot; # The local path `foo` is a copy of the `snot` field from the state record
# Where state is an array
let foo = state[10]; The local path `foo` is a copy of the 10th element of the state array
### How to reference arguments
For operators and structures that support arguments the `args` keyword can be
used to dereference values via path statements.
```tremor
# Where state is a record
let foo = args.snot; # The local path `foo` is a copy of the `snot` field from the args record
Args are nominal and always record values in tremor.
How can window state be referenced
Operations supporting windows and groups can dereference the cached state via the
window
and group
keywords which both support path operations.
Rule ExprPathRoot
The ExprPathRoot
rule defines a subset of expressions where path operations are supported.
These are:
- Record literals or references to records.
- Array literals or references to arrays.
- The result of function invocations.
- The result of Parenthetic expressions.
rule ExprPathRoot ::=
'(' ComplexExprImut ')'
| Invoke
| Record
| List
;
An internal rule to the ExprPath
rule that defines legal roots for a dynamic path dereference.
Rule ExprPath
The ExprPath
rule defines path operations for expressions.
rule ExprPath ::=
ExprPathRoot PathSegments
;
Allows dereferencing literal values vi path expressions
{"snot": 0, "badger": 1, "goose": 2}["badger"];
{"snot": 0, "badger": 1, "goose": 2}.badger;
...
some_record_fn().record_field
Rule MetaPath
The MetaPath
rule defines path operations for event metadata references.
In the context of a streaming event, allows metadata generated by the runtime to be accessed via path operations.
It is also possible to write to metadata to hint at the runtime to perform certain functions on the event data being forwarded. Tremor operators and connectors can read and write metadata.
rule MetaPath ::=
'$' Ident PathSegments
| '$' Ident
| '$'
;
How do i reference event metadata?
Events in tremor encapsulate data from other systems sent and received from tremor via configured connectors. Information about that data or metadata can also be provided by the runtime, and used in some operators and connectors to control tremor's runtime behaviour.
Meta-data is accessed via the $
dollar symbol.
let metadata = $;
Metadata can be any legal tremor value, but it is typically a record structure
let metastring = "snot" + $;
Meta-data can be written through via a let
operation
let $command = { "do-things": "with-this-meta-request" }
Rule AggrPath
The AggrPath
rule defines path operations for group
and window
references.
In the context of a windowed operation, enables the group
and window
meta
keywords to partipcate in path operations.
rule AggrPath ::=
'group' PathSegments
| 'group'
| 'window' PathSegments
| 'window'
;
How do I reference the computed group dimension?
use std::record;
define window by_2 from tumbling
with
size = 2
end;
select {
"g": group[0], # Extract current group dimension
"c": aggr::stats::sum(event.c),
}
from in[by_2]
group by set(each(record::keys(event.g))) into out;
Rule ArgsPath
The ArgsPath
rule defines path operations for args
references.
rule ArgsPath ::=
'args' PathSegments
| 'args'
;
How do i reference arguments?
Arguments are encapsualted via the args
keyword symbol.
let what = args;
Arguments are always record structured
1 | let args = 1;
| ^^^^^^^^ Can't assign to a constant expression
Arguments cannot be assigned to or overridden in scripts.
Rule LocalPath
The LocalPath
rule enables path operations on locally scoped identifiers.
rule LocalPath ::=
Ident PathSegments
| Ident
;
A local path is simply a path structure that is bounded to a locally defined value
let snot = { "snot": "badger" };
snot.snot # The field 'snot', on the local path reference 'snot'
Rule ConstPath
The ConstPath
rule enables path operations on module scoped references.
rule ConstPath ::=
ModPath '::' LocalPath
;
A fully scoped path that does not include dynamic resolution or runtime type information.
Rule StatePath
The StatePath
rule defines path operations for user defined in memory state in tremor.
Allows the state
value to be dereferenced via path operations.
rule StatePath ::=
'state' PathSegments
| 'state'
;
How do i reference state in tremor?
Tremor programs can be stateful in many ways - such as through the state managed by operators and windowed operations by the runtime on behalf of the user provided program.
The state
keyword allows an arbitrary value controlled by a users program to be
maintained and managed by the user program.
let my_state = state;
State can be written through via a let
operation
let state = match state of
case null => { "count": 1 }
case _ => { "count"": state.count + 1 }
end;
Rule EventPath
The EventPath
rule defines path operations for streaming events in tremor.
Allows the current streaming event
to be dereferenced via path operations.
rule EventPath ::=
'event' PathSegments
| 'event'
;
How do i dereference event data?
The current event is accessed via the event
keyword.
The event can be any legal tremor value. If it is a record or an array, then it can be dereferenced via the path language in the usual way.
event.snot; # Event record, field 'snot'
let badger = event[0]; Event array, first element
Events can be mutated and manipulated and used as an output
select { "wrapped-event": event } from in into out;
Rule PathSegments
The PathSegments
rule specifies the continuation of a path rule.
Form Variation | Description |
---|---|
.<Ident> | A terminal segment dereferencing a record field |
<Ident><PathSegments> | A non-terminal segment dereferencing a record field |
[<Selector>] | A range or index segment dereferencing an array |
[<Selector>] | A terminal range or index segment dereferencing an array |
[<Selector>]<PathSegments> | A non-terminal range or index segment dereferencing an array |
rule PathSegments ::=
'.' Ident PathSegments
| '[' Selector ']' PathSegments
| '[' Selector ']'
| '.' Ident
;
``tremor a.b a["b"] a[0..5]
Defines how structural values in tremor can be dereferenced to get at internal parts such as
an element of an array, or a field of a record.
## Rule Selector
The `Selector` rule specifies an index or range of an array.
A range is a `:` colon separated pair of expressions.
An index is a single expression.
![Selector](svg/selector.svg)
```ebnf
rule Selector ::=
ComplexExprImut ':' ComplexExprImut
| ComplexExprImut
;
Field selection for records
Selecing a record field using array notation
let snot = badger["snot"]
Select the field 'snot' from the record 'badger'
Ordinal selection for arrays
let e = badger[0];
Select the 0th ( first ) element of the array 'badger'
Range selection for arrays
let e = badger[0:5];
Select the 0th up to but no including the 5th element of the array 'badger'
Rule Invoke
The Invoke
rule specifies the syntax of a function invocation.
rule Invoke ::=
FunctionName '(' InvokeArgs ')'
| FunctionName '(' ')'
;
use std::string;
string::len("snot")
Rule FunctionName
The FunctionName
rule defines a path to a function in tremor.
It can be an Ident
for functions defined in local scope.
It can be a ModPath
for functions in a modular scope.
rule FunctionName ::=
Ident
| ModPath '::' Ident
;
Rule ModPath
The ModPath
rule defines a modular path.
A modular path is a sequence of Ident
s separated by a ::
double-colon.
rule ModPath ::=
ModPath '::' Ident
| Ident
;
How do i reference something from the standard library?
The standard library contains reusable constants, functions
and other definitions that can be used in scripts via the
Use
and ModPath
rules.
For example, if you have a file called foo.tremor
in a src
folder you can append this to your TREMOR_PATH
environment
variable
export TREMOR_PATH=/path/to/src
Assuming foo.tremor
contains the following code:
fn meaning_of_life() of
42
end;
We can use this in another script as follows:
use foo;
let meaning = foo::meaning_of_life();
Rule InvokeArgs
The InvokeArgs
rule defines a sequence of expression statements.
rule InvokeArgs ::=
InvokeArgs_
;
A comma delimited sequence of complex expressions
Rule InvokeArgs_
The InvokeArgs_
rule is an internal rule of the InvokeArgs
rule.
The rule specifies a ;
semi-colon delimited sequence of expression statements.
rule InvokeArgs_ ::=
Sep!(InvokeArgs_, ComplexExprImut, ",")
;
See InvokeArgs
rule for details.
This rule wraps away a lalrpop macro call for ease of reference in other rules in the grammar source.
Rule Drop
Drop halts event processing for the current event being processed returning control to the tremor runtime, dropping the event.
Constraints
The drop
operation should be used with care as the in-flight event is
discarded by the runtime. Where circuit breakers, guaranteed delivery and
quality of service operations are being managed by the engine downstream
these should be carefully programmed so that drop
operations have no
side-effects on non-functional behaviours of the tremor runtime.
Here be dragons!
rule Drop ::=
'drop'
;
define script boring
script
drop
end;
create script boring;
select event from in into boring;
select event from boring into out;
Drop signals to the tremor runtime that an event is not interesting and can be dropped without any further handling by the engine. Drop statements in a script or query result in the processing of the current event halting without any further action bye the tremor runtime.
The dropped event is discarded by the engine.
Rule Emit
Emit halts event processing for the current event being processed returning control to the tremor runtime, emitting a synthetic event as output.
By default, the emit operation will emit events to the standard output port out
.
The operation can be redirected to an alternate output port.
rule Emit ::=
'emit' ComplexExprImut '=>' StringLiteral
| 'emit' ComplexExprImut
| 'emit' '=>' StringLiteral
| 'emit'
;
define script route
script
emit => "not_out"
end;
create script route;
select event from in into route;
select event from route/not_out into out;
Emit signals to the tremor runtime that an event has been processed fully and processing can stop at the point emit is invoked and a synthetic value returned without any further processing.
The emitted event is forwarded by the engine.
Rule Let
The Let
rule allows an expression to be bound to a Path
.
The Path
references the subject of the assignment based on tremor's Path
rules.
The bound Path
is mutable.
rule Let ::=
'let' Assignment
;
How do i create a local mutable variable in tremor?
let my_var = "this is a string";
Rule Assignment
The Assignment
rule allows an expression to be bound to a Path
.
The Path
references the subject of the assignment based on tremor's Path
rules.
rule Assignment ::=
Path '=' SimpleExpr
;
The rule assigns an expression to a path by reference.
x = 2
Assigns x
to the value literal 2
x.y = 2
Assigns the field y on record x to the value literal 2
Assignments expressions can be constant such as via the Const
rule or mutable such as via the Let
rule.
Rule Patch
The Patch
rule defines the patch
statement in tremor.
rule Patch ::=
'patch' ComplexExprImut 'of' PatchOperations 'end'
;
Patch insert a field in a record
let a = patch event of
insert "test" => 1
end;
Default patch templates
patch event of
default => {"snot": {"badger": "goose"}}
end
## Rule PatchOperations
The `PatchOperations` rule defines a sequence of semi-colon delimited patch operations.
![PatchOperations](svg/patchoperations.svg)
```ebnf
rule PatchOperations ::=
PatchOperationClause
| PatchOperations ';' PatchOperationClause
;
A sequence of patch operations
Rule PatchField
The PatchField
is a string literal identifying a the field of a record to which a PatchOperationClause
is being applied.
rule PatchField ::=
StringLiteral
;
Rule PatchOperationClause
The PatchOperationClause
rule defines operations of a patch
statement.
A patch operation can:
- Insert, update, copy ( clone ), move ( rename ), merge or erase fields in a record.
- Apply a default operation on a field or on the whole input record.
rule PatchOperationClause ::=
'insert' PatchField '=>' ComplexExprImut
| 'upsert' PatchField '=>' ComplexExprImut
| 'update' PatchField '=>' ComplexExprImut
| 'erase' PatchField
| 'move' PatchField '=>' PatchField
| 'copy' PatchField '=>' PatchField
| 'merge' PatchField '=>' ComplexExprImut
| 'merge' '=>' ComplexExprImut
| 'default' PatchField '=>' ComplexExprImut
| 'default' '=>' ComplexExprImut
;
How do I insert a new value into a record?
patch {"a": 1, "b": 2, "c": 3 } of
insert "d" => "delta"
end;
It is a semantic error to insert a value if the field already exists:
Error in foo.tremor:2:3
1 | patch {"a": 1, "b": 2, "c": 3 } of
2 | insert "b" => "bravo"
| ^^^^^^^^^^^^^^^^^^^^^ The key that is supposed to be written to already exists: b
3 | end;
How do I update an existing value in a record?
patch {"a": 1, "b": 2, "c": 3 } of
update "b" => "bravo"
end;
It is a semantic error to update a value if the field does not already exist:
1 | patch {"a": 1, "b": 2, "c": 3 } of
2 | update "d" => "delta"
| ^^^^^^^^^^^^^^^^^^^^^ The key that is supposed to be updated does not exists: d
3 | end;
How do I insert or update a value in a record?
If the distinction between an insert
and an update
is not significant the
upsert
operation will insert a new field, or update an existing field. This
operation is more flexible, but does not offer compile time errors to protect
against invalid usage. Where possible, use insert
or update
in preference
to upsert
when a new field or replacing an existing fields value would be
an error given the business logic at hand.
How do I erase a field from a record?
patch {"a": 1, "b": 2, "c": 3 } of
erase "d"
end;
The field c
is removed from our record
How do I rename a field?
patch {"a": 1, "b": 2, "c": 3 } of
move "c" => "d" # The value MUST be a string literal as it represents a field name
end;
The c
field is removed and a d
field is added with the value from c
How do I duplicate a field?
Similar to move
, the copy
operation copies the value of one field to a new field
patch {"a": 1, "b": 2, "c": 3 } of
copy "c" => "d" # The value MUST be a string literal as it represents a field name
end;
The c
field is preserved, and the d
field is added with a copy of the value from c
Can I use patch
and merge
together?
The merge
operation in a patch
expression can be applied to the patch
target record or to a specified field.
patch {"a": 1, "b": 2, "c": 3 } of
merge "d" => {}
end;