Skip to main content

· 4 min read

Introduction

Hey folks, I am Nupur Agrawal, a third year student at Indian Institute of Technology Roorkee. This blog describes my experience of contributing to Tremor, CNCF sandbox project in the 2021 spring chapter of LFX Mentorship Program, under the mentorship of Matthias Wahl, Anup Dhamala and Heinz Gies.

Project Abstract

Tremor is an event processing system originally designed for the needs of platform engineering and infrastructure. It is built for the users that have a high message volume to deal with and want to build pipelines to process, route, or limit this event stream.

At the beginning of the program, I was given walkthrough of the project by Matthias and he patiently explained me the components and working of tremor. Tremor is nicely documented and the docs can be very useful for referring many things.

My Project

My project's aim was to enable tremor to receive and send Syslog Protocol Messages, a standard protocol used to send system log or event messages. It was desired to support both the standard IETF format and the old BSD format via UDP and TCP/TLS. More detailed description can be found here.

· 13 min read

Introduction

Hello folks! I'm Jigyasa, a final-year computer science engineering student at Indira Gandhi Delhi Technical University for Women pursuing my bachelor's in Technology. This blog is about my experience contributing to Tremor as part of the LFX Mentorship program. i

Learning about Tremor

Tremor is an event processing system for unstructured data with rich support for structural pattern matching, filtering, and transformation. It is built for users that have a high message volume to deal with and want to build pipelines to process, route, or limit this event stream. It has a scripting language called tremor-script and a query language as well called tremor-query or trickle.

I had never worked on an event processing system before this internship. In fact, my first major contribution to open-source was through this mentorship program. To get started with it, my mentor Darach Ennis, suggested me some documents that helped me learn more about it:

/docs/overview/

/docs/course

Apart from that, learning more about the tremor-query, tremor-script, and going through the workshops in the docs can be really helpful.

The codebase of Tremor is in Rust, and since I had no prior experience with Rust, I started learning the language.

· 5 min read

v0.10

Summary

This release is the last minor release before we start turning Tremor into a truly distributed event processing and data distribution engine. We focused on small things that improve usability and ironed out some rough edges here and there.

TL;DR

  • We extended our type system and codecs by the binary type.
  • Elasticsearch offramp now supports Linked Transport.
  • String interpolation done right, now with #{} instead of just {}
  • Release now includes prebuilt binaries, DEB packages and RPMs

New Release Artefacts

To make your life easier installing and Tremor more pleasant, we added new release artefacts. We've now got prebuilt binaries wrapped up in a tar.gz for amd64 linux, DEB, and RPM packages. And, lets not forget our well-known Docker Image.

install prebuilt binary

We are going to explore more channels for release artefacts in the future to get you up and running with Tremor in no time.

In addition to those new release channels, we enabled thin-lto for all builds, which should improve compile-time while providing all the benefits of full link time optimization (LTO). We also link OpenSSL, now statically, to avoid incompatibilities with OpenSSL versions provided by the OS.

· 8 min read

Summary

v0.9

Tremor is an open source event processing and data distribution engine designed to interpose or intercept data in complex systems primarily to inject quality of service, correct data flows and to enrich or support complex traffic shaping, routing and load-balancing based on contextual, situational or content derived dynamic conditions.

Tremor has a powerful ETL language designed for flexibly pattern matching structured data such as JSON efficiently (SIMD-accelerated), and a declarative flow-based SQL language designed for rich JSON-like data to describe data processing and data flow graphs with declarative QoS constraints.

Tremor has been in large scale global production at Wayfair for 2 years now, and is gaining adoption in other organizations.

In this release, we return to Tremor’s roots and focus on quality of service by adding linked-transports to interpose RPC-like and synchronous-blocking protocols. We add circuit breakers to manage upstream and downstream byzantine events, and we provide mechanisms that offer shades of delivery guarantees or intrinsic transactions that, when combined with suitable upstream or downstream systems, can produce lossless data transfers for certain routes whilst remaining best-effort for routes that do not support lossless delivery or recovery in and of themselves with minimal loss.

Lastly, but most importantly, this is the first release of Tremor as a Linux Foundation/CNCF early-stage sandbox project. We would like to thank everyone at Wayfair in Boston and Berlin, the Linux Foundation and the CNCF who facilitated, helped, nurtured, transitioned DNS entries and worked with the legals to make this transition to fully open governance happen. A huge thanks to Chris and Amye at the Linux Foundation.

· 6 min read

With the upcoming Tremor release, 0.9.0, we're moving from threads as a basis for ramps and pipelines to async tasks.

Let's talk about why this is significant, what is changing, and how the architecture is changing.

Note that this is not a comprehensive treatise on threads or async tasks.

The Tremor That Was (threads)

Threads are a basic building block of programs that run multiple pieces of code concurrently. The operating system is responsible for coordinating across competing resource demands.

The OS can preempt, pause, and resume threads. We can leverage infinite or tight loops without the risk of completely blocking the system. These guarantees make concurrent code more accessible, with tools likecrossbeam-channels to build upon.

Threads work especially well in use cases where the system and logical concurrency models are well aligned; or, we can map application threads to logical cores on the system being used. Each thread can happily work away on its part of the logic and pass the result on to the next. The one thread per core model is what tremor 0.8 and earlier used. We had a thread for the onramp, a thread for the pipeline, and a thread for the offramp. As the computational cost of decoding, processing, and encoding was often in the same ballpark, this worked exceptionally well. We managed to push up to 400MB/s of JSON through the system this way (including parsing, tremor-script logic, and serialization).

This design can degenerate badly if there are more ramps and pipelines than cores on the system in use. Throughput degrades rapidly (as in up to 2 orders of magnitude worse at 30:1 ratio). At the time of writing this, the deployment model was one pipeline/ramp group on a four-core system, so it worked well in practice.

However, this places a burden on operators having to think about concurrency and parallelism to tune tremor for optimal performance and capacity.

In SMP systems, we observe other undesireable effects: The moment two communicating threads don't share the same underlying cache, performance plummets. This scenario exists when threads reside on two different CPUs or CCXs (thank you AMD for making me learn so much about CPU caches). As long as two communicating threads share the same cache, data shared between them can avoid trips to main memory and cache coherency protocol overheads. When two threads communicate across different caches, reads/writes may catastrophically collide and introduce overheads that drastically reduce overall performance.

· One min read

We had the great pleasure of spending some time with the crowd of Rust & Tell today. At the meetup, we got the chance to share some of the stories of tremor and Event Processing Cartography. Long story short, if you're curious, the talk was recorded, and we got the slides right here!

· 9 min read

Yesterday, we spent the day on a report that our influx parser was slow, and it turns out it was indeed.

This is an exciting topic as a few days ago, we gave a talk at BoBKonf 2020 on this topic, so this is a great opportunity to show some of the topics and our process in action.

All the topics in this blog are links; the main one above this text is to the pull request, the titles of each section link to the commit that implements the topic discussed. Go ahead, click on some, and take a look!

There are two tools worth introducing here that we used during this performance session.

One is perf, which we used with a minimal setup of perf record and perf report. We use this to get a glance at where code is spending time. This is not perfect, but it is quick for decent results.

The other one is criterion, an excellent Rust framework for microbenchmarks based on the haskell framework with the same name. It is so helpful since it allows us to show changes in performance between changes. That makes it perfect for the kind of incremental improvements our process favors.