Skip to main content
Version: edge

separate

Splits the input into events, using a given separator, the default being \n (newline).

The default can be overwritten using the separator option.

Buffers any fragment that may be present (after the last separator), till more data arrives. This makes it ideal for use with streaming onramps like tcp, to break down incoming data into distinct events.

Additional options are:

OptionDescriptionRequiredDefault Value
separator The separator to split the incoming bytes atno\n
max_lengththe maximum length in bytes to keep buffering before giving up finding a separator character. This prevents consuming huge ammounts of memory if no separator ever arrives.no8192
bufferedbuffer multiple fragments to find a seperator, if this is set to false each fragment will be considered to be followed by a separator so "hello\nworld" would turn into two events "hello" and "world". With buffered true "hello\nworld" would turn into one event "hello" and "world" will be buffered until a next event includes a \n or the stream ends.notrue

If this preprocessor is only configured by name, it will split on \n, does not enforce a maximum length of 8192 and buffer incoming byte fragments until a separator is found or the max_length is hit, at which point the fragment is discarded.

Example configuration:

define connector foo from ws_client
with
preprocessors = ["separate"],
postprocessors = ["separate"]
codec = "json",
config = {
"url": "ws://localhost:12345"
}
end;

define connector snot from ws_server
with
preprocessors = [
{
"name": "separate",
"config": {
"separator": "|",
"max_length": 100000,
"buffered": false
}
}
],
...
end;