Nakadi Concepts

The Nakadi API allows the publishing and consuming of events over HTTP.

A good way to think of events is that they are like messages in a stream processing or queuing system, but have a defined structure that can be understood and validated. The object containing the information describing an event is called an event type.

To publish and consume events, an owning application must first register a new event type with Nakadi. The event type contains information such as its name, the aforementioned owning application, strategies for partitioning and enriching data, and a JSON Schema. Nakadi supports an event type registry API that lists all the available event types.

Once the event type is created, a resource called a stream becomes available for that event type. The stream will accept events for the type from a producer and can be read from by one or more consumers. Nakadi can validate each event that is sent to the stream.

An event type's stream can be divided into one or more partitions. Each event is placed into exactly one partition. Each partition represents an ordered log - once an event is added to a partition its position is never changed, but there is no global ordering across partitions [1].

Consumers can read events and track their position in the stream using a cursor that is given to each partition. Consumers can also use a cursor to read from a stream at a particular position. Multiple consumers can read from the same stream, allowing different applications to read the stream simultaneously.

In summary, applications using Nakadi can be grouped as follows:

  • Event Type Owners: Event type owners interact with Nakadi via the event type registry to define event types based on a schema and create event streams.

  • Event Producers: Producers publish events to the event type's stream, that conform to the event type's schema.

  • Event Consumers: Consumers read events from the event stream. Multiple consumers can read from the same stream.

[1] For more detail on partitions and the design of streams see "The Log" by Jay Kreps.