25. What can you do with Kinesis Data Streams?

Feb 25 2022 • 42 mins

Eoin and Luciano are back with some more AWS messaging services material! This time we talk about Kinesis Data Streams and everything there’s to know about them! In this episode of AWS Bites, we cover the difference between data streaming and queue or message buses. How Kinesis can let you process large batches of messages in near real time. Why you might use Kinesis as a queue or pub/sub bus in some cases. A few different ways to use Kinesis with the focus on avoiding too much complexity. We also provide some insights based on our experience with Kinesis Data Streams including real use cases, monitoring tips and some gotchas to be aware of.

In this episode, we mentioned the following resources:

- Our introductory episode about what services you should use for events: https://www.youtube.com/watch?v=CG7uhkKftoY

- Amazon Kinesis Producer Library (KPL): https://docs.aws.amazon.com/streams/latest/dev/developing-producers-with-kpl.html

- Amazon Kinesis Consumer Library (KCL): https://docs.aws.amazon.com/streams/latest/dev/shared-throughput-kcl-consumers.html

- Kinesis Library wrapper for Node.js: https://github.com/awslabs/amazon-kinesis-client-nodejs

- Kinesis Library wrapper for Python: https://github.com/awslabs/amazon-kinesis-client-python

- Kinesis Data Stream with captures from DynamoDB (for CDC): https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/kds.html

- Kinesis Data Stream with captures from Aurora: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/DBActivityStreams.Overview.html

- Kinesis Data Analytics: https://aws.amazon.com/kinesis/data-analytics/

- Kinesis Firehose: https://aws.amazon.com/kinesis/data-firehose/

- Must-read on Kinesis: Anahit Pogosova’s two-part blog series: https://dev.solita.fi/2020/05/28/kinesis-streams-part-1.html & https://dev.solita.fi/2020/12/21/kinesis-streams-part-2.html

- Cloudonaut Kinesis vs MSK: https://cloudonaut.io/versus/messaging/kinesis-data-streams-vs-msk/