How to consume and publish Kafka
Apache Kafka is a common message streaming platform used throughout financial markets. The Genesis Platform enables you to consume and/or produce messages seamlessly to Kafka topics.
For example, you can listen to market data ticks directly from your internal Kafka Cluster, publish FIX messages directly to an intermediate topic, or consume messages from an external Kafka source.
Example
We have created an example application that shows how to consume incoming messages and to publish them to a Kafka topic.
The application provides two pipelines, one to consume and one to publish. These have been set up as independent processes. So if a message is published to Kafka when the consumer process is down, the application will process it as soon as the process starts back up.
The examples are within one complete example application, which includes a front end so that you can run and see the data.
You can clone the repo to see the code - which includes comments at the key points to highlight what is being specified - and run the application.
The README file for the application provides set-up instructions for creating a similar application. As well as the pipeline definitions, this includes setting up dependencies, system definitions, the Event Handler (to trigger an event) and the Request Server (to provide data to the front end.)
Publishing to Kafka
In the example application, a pipeline is set up to respond to an event, such as when the user inserts a new price. The pipeline publishes this data to the specified Kafka topic.
Consuming from Kafka
The example contains a second pipeline that runs in the background and polls a Kafka topic. Whenever new records are detected on the topic, it retrieves them and inserts them to the database.
The pipeline only retrieves the next record (or batch of records) once the previous record (or batch) has been added to the database.
Testing
Go to our Testing page for details of our testing tools and the dependencies that you need to declare.
To test the set-up of a Data Pipeline for Kafka on your app:
- Details to follow shortly. Thank you for your patience.
Full technical details
You can find full technical details in our reference documentation: