Fork me on GitHub



What is Tremor?

Tremor is an event processing engine designed for high density deployment in mission critical 24x7x365 environments.

It started life as a man-in-the-middle proxy retrofitting near real-time traffic shaping to all logging and metrics capture, distribution and processing systems at Wayfair.

The initial release of Tremor supported proactive rate limiting, classification of data streams based on simple rules, and reactive backpressure handling processing in excess of 10TB per day of nested structured data.

Tremor has been extended to support rich extract, transform, load and filtering of data streams with a scripting language designed for expressive ETL on JSON-like data structures with SIMD-accelerated JSON processing.

Tremor has been extended to support aggregate query processing and quartile estimation of metrics data with a query language that builds on the scripting language.

Tremor is currently being extended with clustering support based on the RAFT protocol specification, ring based topologies and V-Nodes inspired by riak-core from Basho Technologies.

Tremor is also a system of plugins, or a framework. Although it is in production for only 50+ production use cases it is designed to be easily extended, bent or stretched to other purposes.

When not to use Tremor?

Tremor displaces, disintermediates or replaces a number of commercial and open source data processing and distribution solutions deployed at scale within Wayfair.

In Wayfair’s case the benefits are:

But, tremor was built with some assumptions:

And, the needs that drove us to build tremor, may not match your needs, if:

If these concerns do not dissuade you, then we would welcome your contributions via our issues list, community chat, via our twitter channel or via contributing and joining our community.

What is the current state of the project?

Tremor is an early stage project that is still under active development. It has been in production at scale at Wayfair since October 2018. Tremor is open source since February 22, 2020.

Tremor is transitioning from closed to fully open community based development.