Breaking: Iceberg Topics for Apache Kafka®
The big news this fortnight: Aiven released Iceberg Topics for Apache Kafka®, enabling open-source users to stream data natively as Apache Iceberg™ tables with zero ETL and zero copies. This implementation adds just 3k lines of code to the RemoteStorageManager, requiring no client API or broker changes. The approach eliminates ~60% of Kafka sink connectors by making topics double as Iceberg tables. Read the full announcement
Featured: Apache Iceberg™ & Streaming Data
Perfect timing! We just released Get Kafka-nated Episode 4 featuring Alex Merced from Dremio, co-author of "Iceberg: The Definitive Guide." We explored how Apache Iceberg™ is reshaping streaming data architectures, schema evolution challenges, and how it fits with emerging concepts like diskless Kafka. Listen to the full conversation
KIP Watch - The last two weeks in Kafka improvement proposals
The past two weeks have seen significant progress on several key proposals:
Recently Completed:
KIP-1161: cleanup.policy shouldn't be empty - Approved August 17 with 3 binding votes after extensive community discussion about handling empty configuration lists across Kafka. This prevents configuration errors that could lead to unexpected topic behavior. KIP details
Currently Voting:
KIP-1197: Improve TopicBasedRemoteLogMetadataManager initialization - Voting continues with 1 binding vote. This addresses critical tiered storage initialization failures affecting production deployments. KIP details
KIP-1147: Improve consistency of command-line arguments - Active vote for standardizing CLI tool arguments across Kafka tools. Recent updates address feedback about parameter precedence and consumer/producer property consistency. KIP details
Active Discussions:
KIP-1159: Large message reference-based Serializer - Intensive review continues with Greg Harris raising important questions about improving Kafka's in-band large message handling versus reference-based approaches. This fundamental architectural discussion could shape how Kafka handles oversized messages. KIP details
KIP-1190: Add controller thread idleness metric - Nearing completion after addressing feedback on metric naming and calculation methods. This operational visibility improvement helps debug controller performance issues. KIP details
Release Radar
Kafka 4.1.0 Release Candidate 2 completed voting with positive community feedback. This major release includes Queues for Kafka in preview status and represents a significant platform advancement. Release notes
Articles exploring Kafka and Iceberg worth a read
No more excuses for Stream/Table Duality - Yaroslav Tkachenko, founder at Irontools and expert in all things Flink shares his thoughts on how Iceberg sync supports streaming first architectures. Read it here.
Streams, Tables, and The Illusion of Duality - This is a fantastic exploration of how Kafka and Iceberg work and asks the question “Do Streams and Tables Need to Be in the Same System?”. Go check out Giannis Polyzos’ answer to that question here.
From Stream to Table: Real-Time Analytics Made Simple with Kafka and Iceberg - Not ready to make the move to Iceberg topics but still want to try your hand at getting data from Kafka into Iceberg? Check out this awesome guide by Douglas Young.
Community Discussions
The conversation about Aiven’s implementation of Iceberg Topics is ongoing over on the Aiven Kafka 2.0 repo. Join the discussion here.
Tom Scott, founder at StreamBased, asks over on Reddit Can Kafka → Iceberg pipelines reduce connector complexity?
Next shot brewing: We'll be back in two weeks with more community updates, KIP voting results, and the technical deep dives worth your time.
Bonus: Tiered Storage Deep Dive
Check out the latest episode of Get Kafka-Nated! I had a fantastic conversation with Satish D from Uber, who was a key contributor to KIP 405 that brought tiered storage to Kafka.
We explored what tiered storage actually is, the specific storage challenges at Uber that led to KIP 405, and the biggest technical hurdles in implementing such a fundamental change to Kafka. Satish shared insights on working with the Apache community process including why it can take so long to get major KIPs into open-source Kafka. Watch now
That’s all for this time, check back in 2 weeks from now for more Kafka and Iceberg news, and an episode of the show all about Apache Flink!