This example goes into how tremors delivery guarantees work in combination with a sink and source that also support delivery guarantees.
The scenario isn't all-encompassing but looks at the following specific use-case:
- A tremor source that does not hold delivery guarantees.
- A WAL to persist data from the source
- A Kafka API compatible instance where data is sent to
- A secondary tremor instance with a Kafka API compatible source that reads the data and prints it to stdout
With this we can demonstrate the recovery and delivery guarantees using a WAL and a Kafka API compatible set of endpoints.
We can start the example using
docker-compose up. Then with
docker ps we find the Kafka instance and can introduce an artificial error using
docker pause <container>. We will see the messages stopping if we wait for a while we can reenable kafka with
docker unpause <container> and will see the message flow resuming with a number of duplicated but no lost messages.
Kafka compatible products for this scenario
Apache or Confluent Kafka Vectorized Redpanda