Link copied to clipboard!
Group 9: Real-time Streaming
Event-driven streaming data platform services: Kinesis Data Streams, Kinesis Data Firehose, Kinesis Data Analytics, MSK.
Streaming Principle: Match ingestion patterns (direct vs buffered), processing complexity, and downstream integration to data velocity and transformation requirements.
Services & Roles
Kinesis Data Streams
Real-time data streaming with custom applications and millisecond latency.
- Custom consumers
- Millisecond latency
- Manual scaling
Kinesis Data Firehose
Serverless data delivery to AWS destinations with automatic scaling.
- Zero administration
- Built-in transformations
- Near real-time delivery
Kinesis Data Analytics
SQL-based stream processing for real-time analytics and alerts.
- Standard SQL
- Windowed queries
- Real-time analytics
MSK
Managed Apache Kafka for high-throughput, distributed streaming platform.
- Apache Kafka
- Multi-AZ deployment
- Kafka ecosystem
Key Differences
| Dimension | Kinesis Data Streams | Kinesis Data Firehose | Kinesis Data Analytics | MSK |
|---|---|---|---|---|
| Processing Model | Custom applications | Serverless delivery | SQL analytics | Kafka ecosystem |
| Latency | Milliseconds | 60s+ buffering | Sub-second | Milliseconds |
| Scaling | Manual shards | Automatic | Automatic | Manual brokers |
| Use Case | Custom processing | Data lake ingestion | Real-time dashboards | Event streaming |
| Complexity | High | Low | Medium | High |
Selection Model
0–10 sliders weight streaming requirements, processing complexity, and operational preferences.
Score_DataStreams = 0.30*C_customProcessing + 0.26*C_lowLatency + 0.18*C_flexibleConsumers + 0.12*C_exactlyOnce + 0.08*C_replayability + 0.06*(10 - C_simplicity)
Score_Firehose = 0.32*C_simplicity + 0.28*C_dataLakeIngestion + 0.18*C_serverlessPreference + 0.12*C_builtInTransform + 0.06*C_awsDestinations + 0.04*(10 - C_lowLatency)
Score_Analytics = 0.30*C_sqlAnalytics + 0.26*C_realTimeDashboards + 0.18*C_windowedQueries + 0.14*C_standardSQL + 0.08*C_alerting + 0.04*C_aggregations
Score_MSK = 0.32*C_kafkaEcosystem + 0.24*C_highThroughput + 0.18*C_distributedStreaming + 0.12*C_kafkaCompatibility + 0.08*C_multiConsumer + 0.06*(10 - C_managedService)
Kinesis Data Streams {{vm.scores.datastreams|number:2}}
Kinesis Data Firehose {{vm.scores.firehose|number:2}}
Kinesis Data Analytics {{vm.scores.analytics|number:2}}
MSK {{vm.scores.msk|number:2}}
Primary Emphasis: {{vm.recommended.name}} ({{vm.recommended.score|number:2}})
Heuristics
- High C_customProcessing + C_lowLatency → Kinesis Data Streams for custom apps.
- Strong C_simplicity + C_dataLakeIngestion → Firehose for ETL pipelines.
- High C_sqlAnalytics + C_realTimeDashboards → Analytics for SQL queries.
- C_kafkaEcosystem + C_highThroughput → MSK for Kafka workloads.
Anti-Patterns
- Using Data Streams for simple ETL (Firehose is easier).
- Using Firehose for sub-second processing (high buffering delay).
- Using Analytics for complex transformation logic (use custom apps).
- Using MSK without Kafka expertise (steep learning curve).
Summary
Choose Data Streams for custom processing, Firehose for simple delivery, Analytics for SQL queries, MSK for Kafka ecosystems.