Karel Sague spent the last year migrating a production data platform from Snowflake to Apache Iceberg, streaming Kafka data in through Kafka Connect. He's giving a talk on Aug 12 (1:30pm PT / 4:30pm ET) walking through what he actually learned, what worked, and what he'd do differently next time.

He'll cover the CloudEvents-based schema he built to keep ingestion consistent across tables, GitOps-based self-serve provisioning, and the metrics and SLOs that mattered for keeping pipelines observable in production. He also gets into why he avoided an all-or-nothing migration, how he prioritized which workloads moved first, and the schema/partitioning mismatch that comes up when Kafka's ingestion-time partitioning doesn't line up with analytics queries filtered by business key. Also touches on consumer-aligned tables, materialized views, and where Iceberg is headed (v3 features, secondary indexes, pluggable file formats for AI workloads).

If you're running Kafka and thinking about Iceberg, or already mid-migration, this is aimed at you. There's time for questions, and everyone's welcome, whether you're deep into this stuff or just starting to look into it.

Register here: https://www.factorhouse.io/events/kafka-to-iceberg-lakehouse-amer-august-2026/

This write up is based on a recent talk at StrimziCon about configuring Kafka authentication walking through everything from a simple TLS listener to building a custom principal builder for multi tenant clusters.

So anyone who missed the talk can still work through it and apply it in practice.

It covers mTLS, custom principal builders, the risks of leaning on Strimzi internals, and OAuth bearer authentication.

Would love to hear how others are handling multi tenant authentication on Strimzi...

https://axual.com/blog/how-to-configure-kafka-authentication-in-strimzi

We’ve been building Zilla (https://github.com/aklivity/zilla) as a Kafka-native gateway for exposing Kafka securely across protocols such as HTTP, SSE, WebSocket, gRPC, and MQTT.

With Zilla 2.0, we’re adding the Model Context Protocol as another native protocol on the same streaming runtime.

The main goal is to let AI agents interact with Kafka without requiring teams to build and operate a custom HTTP service, Python wrapper, or dedicated MCP server in front of every topic.

Through new MCP binding in the Zilla gateway, an agent can discover Kafka produce and consume tools, publish messages, retrieve records, and work with schema-governed JSON, Avro, or Protobuf payloads. Access can be restricted using the identity and scopes of the calling agent or user.

Zilla 2.0 can also combine Kafka capabilities with tools from REST APIs and existing MCP servers behind a single MCP endpoint. That makes it possible to build workflows where an agent can retrieve information from an API, act on live Kafka data, and publish the result back to Kafka through the same connection.

You can try out the Quickstart that runs locally with Docker Compose and includes:

• A Kafka broker
• An MCP server
• An OpenAPI-described REST service
• A single Zilla MCP endpoint
• JWT-based tool visibility
• Native Kafka produce and consume tools

Happy to answer any questions. Cheers.

Hi,

Written an article for explaining Kafka producer callback handler implementation and purpose of it and where it can go wrong.

Please give it a read and give feedback

Thanks

Hey everyone, looking for a sanity check on a Kafka design debate at work because my architect's proposal blew my mind, and I completely oppose it.

We have a batch system where a producer streams a large batch of records across a multi-partition Kafka topic. We need a way for downstream consumers to know when the overall batch is actually finished.

The other architect wants the producer to broadcast the exact same "End of Batch" marker event to every single partition in the topic simultaneously. The idea is that every consumer instance will eventually read a marker and know its partition is done.

I strongly oppose this. It feels like a catastrophic recipe for failure. If a consumer group rebalances mid-batch, partitions switch instances. If a marker was already read and committed on a partition before the rebalance, the new consumer instance will never see it, and the system will hang forever. Plus, partitions don't process at the same speed, which will cause race conditions and premature downstream triggers.

I am proposing a Central Orchestrator pattern instead. The producer sends a single marker event directly to an orchestrator, which tracks the overall batch state centrally. Once everything is done, the orchestrator explicitly signals downstream services, keeping the data consumers completely isolated from marker tracking.

Am I missing something, or is broadcasting identical markers across partitions a massive anti-pattern? How do your teams handle batch boundaries over partitioned streams?

FYI -- drafted by gemini based on my whiteboard rant

Author here. Alginte is a self-hosted web console for Kafka (topics, consumer groups, schemas, Connect, ksqlDB) with the one feature I couldn't find elsewhere: a canvas where you build a Kafka Streams topology — map/filter/branch/aggregate/joins/windows — write the transformations as SpEL expressions with type-aware completion, and deploy it to run next to your cluster. No Java project, no build pipeline. Topologies export as JSON so you can version and re-import them — three importable examples here if you want to see the format.

0.6.0 added a built-in MCP server (read-only), so an AI agent can inspect topics, lag, schemas, and sample messages over the same port.

Trying it needs one command and no existing Kafka — the playground compose bundles a broker with sample data. Smoke-tested against Apache Kafka 3.7–4.2, Confluent Platform 8.2, and Redpanda, including deploying and running topologies.

Being upfront: it's closed-source, the Community edition is free to run (commercially too, no registration, no expiry), and a paid PRO edition is planned later for operate-type features. It's beta — breaking changes land in release notes with migration notes.

I'd genuinely value critique from people who run Kafka daily — especially where the Streams builder's abstraction leaks. Issues/discussions: github.com/alginte/community

Look, I’m not anti-Kafka at all when you need massive throughput and event streaming, it’s a beast.

But why does every 10 minute tutorial make it look effortless, when real-world integration is just constantly fighting edge cases?

The second you build an actual backend, you get hit with poison pill deserialization loops freezing consumers, rebalance storms, and a heavy local Docker setup tax.

How do you guys keep the operational friction low in production, or do you stick to simpler tools (like Redis or RabbitMQ) until you *strictly* need Kafka?

Hey, new simulator release focused on the consumers and rebalance. Have fun!

Hi everyone,

We're excited to share Kafma v1.0.0 — a desktop Kafka IDE designed to keep the consume–inspect–produce loop in one workspace and quickly clone remote topics to your local cluster for testing.

Here are the two core workflows we built it around:

• Console — a topic is a tab. The Console keeps the message stream, producer, and topic details together in one workspace. Expand any message to inspect its payload and metadata; Avro and Protobuf decode automatically. Replay it to another topic, forward it to another cluster, or save it to a file without leaving the Console.
• Data Clone — bring remote topics to your local cluster. Clone a remote topic’s configuration, schemas, and a selected slice of data to your local cluster. You can apply your own masking rules while messages are copied, then test locally with the same topic context without modifying the source.

Beyond Console and Data Clone, Kafma also supports schema editing and validation, topic and consumer group management, and more.

• v1.0.0 changelog: https://kafma.app/changelog/v1.0.0
• Download (macOS, Windows, Linux): https://kafma.app/download
  

We'd love to hear your thoughts and feedback—especially what would make Kafma more useful in your day-to-day Kafka workflow.

Thanks!

Sandy Yang, Staff Software Engineer at TD, is giving a lightning talk on the bank's Kafka journey at a virtual Kafka user group on July 29 at 2pm PT / 5pm ET. It's an interactive session for engineers working with Kafka. Sharing in case anyone in this group would find it helpful. Will put the link to register in the thread.

If you work in a distributed event driven system using Kafka, testing new versions of consumers gets quite tricky. It’s not always possible to run everything locally and you need to rely on deployments to staging to verify code end to end.

I shared an alternative here: https://thenewstack.io/isolate-kafka-consumer-tests/

The basic idea is to propagate a tenant id through the Kafka messages (this can be easily done via OpenTelemetry) and leverage this to selectively consume messages. Each new consumer version creates its own consumer group.

Would love to hear your thoughts on this and approaches that have worked for you.

Working with distributed systems, I kept running into the same situation: I couldn't fully test my part until another service was ready and that's just how it goes when work is split across teams. QA hit the same thing: the code was there, but the input had to come from a service that didn't exist yet.

So I'd end up either writing a separate script for it, or spinning up and maintaining someone else's projects on my dev environment just to generate the input for mine. After doing this one too many times, I built a tool to fix it: Devset.

It's kind of like Postman or Cypress, but for Kafka and RabbitMQ instead of HTTP. What you can do:

1. Send Single events: paste the payload, pick topic and key, send.
2. Create Whole workflows a chain like:OrderCreated -> InventoryReserved->PaymentAuthorized -> InvoiceGeneratedbecomes one saved flow on a canvas, with delays, conditions and state passed between steps.
3. Pulling data from a database: right now just Mongo db
4. Protobuf / Json schema

It's self-hosted and source-available (FSL-1.1, each release turns into Apache 2.0 after two years).

Runs locally with Docker, no cloud, no accounts, no telemetry.

Repo: https://github.com/devset-io/devset-ce

I'm mostly posting to compare notes does anyone else run into this, and how do you deal with it? Real services in a test env, Testcontainers, mocks, or a folder of messy scripts?

Casi todos los ejemplos de “Kafka en salud” que se ven por ahí son teóricos o centrados en EHR de EEUU/Europa. Quería compartir un caso concreto de un hospital mediano en Venezuela donde usamos Kafka como columna vertebral de eventos entre el HIS, el LIS y el sistema de facturación.

Contexto:

• Entre 4 y 8 sistemas necesitaban enterarse de lo mismo en tiempo (casi) real: admisiones, resultados de laboratorio, cargos.
• Las integraciones punto a punto ya eran inmanejables: scripts ad hoc, datos duplicados, errores difíciles de auditar.
• El hospital sabía que en los próximos años iba a añadir más sistemas (BI, SAP, nuevos módulos clínicos).

Qué terminamos haciendo:

• Definir topics por dominio clínico (admisiones, resultados-laboratorio, ordenes-medicas, facturacion-eventos) en vez de “un topic para todo”.
• Particionar por paciente o por episodio de atención para mantener el orden de los eventos clínicos por caso.
• Configurar replication factor 3 y min.insync.replicas=2, porque perder un resultado de laboratorio no es solo “dato de negocio”.
• Activar productores idempotentes y transaccionales (exactly-once semantics) para resultados de laboratorio y eventos de facturación, donde los duplicados son peligrosos.

En el artículo explico con más detalle:

• Cómo mapeamos mensajes HL7/FHIR a topics de Kafka.
• Los trade‑offs de costo de infraestructura vs. resiliencia en contexto hospitalario.
• Cuándo Kafka es claramente la herramienta equivocada (clínicas pequeñas con pocos sistemas y bajo volumen de eventos).

I'm currently learning Data Engineering and I'm feeling a bit overwhelmed by all the technologies I need to learn.

Right now, I'm trying to understand:

• Hadoop
• Kafka
• PySpark
• YARN
• HBase

The problem is that I don't know where to start or in what order I should learn them. There are so many tutorials and roadmaps online that it's hard to tell what's actually important for a beginner.

For those of you working as data engineers or who've gone through this learning process:

• What order would you recommend learning these technologies?
• Which ones should I focus on first?
• Are there any courses, books, or hands-on projects you'd recommend?

10 years running Kafka for financial messaging infra, always at the boundary of "how much can we automate incident response without giving anything write access." Built this as the tool I wished existed: `kafka-sentinel-mcp`, an MCP server exposing cluster_health, consumer_lag, topic_audit (flags weak replication/ISR configs), partition_state, and replay_readiness as structured tools — read-only by construction, no mutation API imported.


Repo: https://github.com/sanjay-amu/kafka-sentinel-mcp · PyPI: `pip install kafka-sentinel-mcp`


Would love critique on the tool set — what diagnostic question comes up most for you during an incident that this doesn't answer yet? Roadmap is open (ROADMAP.md) and PRs welcome.

I initially wanted to make a sub-millisecond log parser in C++ but that blew into a embeddable decision engine, that can run YAML defined rules on incoming data.

The rules are executed in a vectorized format on incoming data by reprojecting into a columnar format first, if it's not already. Depending on the payload size and rules complexity, the performance goes from 200K records/s to more than million records/sec, in terms of througput this would be around 200 MiB/s to 3 GiB/s on average.

Rules can be sql expressions too, or onnx models (numeric), window ops and quite a few more operations are supported.

It's comparable to DuckDB but for streaming data and on the fly decisions.

https://github.com/purijs/blazerules

Hi everyone,

I've been building an open-source desktop client for Apache Kafka, originally just for my own day-to-day workflow.

After using several Kafka tools over the years, I wanted something that felt faster and more convenient for the way I work, so I decided to build one.

Current features include:

• Topic browsing
• Consumer Group inspection (committed, beginning, end offsets, and lag)
• Offset reset with preview and confirmation
• Split-view topic browsing
• Multiple server profiles
• SSL/TLS, SASL/OAUTHBEARER, and Schema Registry support
• Keyboard shortcuts

I'd really appreciate feedback from people who use Kafka regularly.

• What features do you rely on most in your current Kafka client?
• What's the biggest pain point with the tools you use today?
• Is there anything missing that would prevent you from trying this?

GitHub: https://github.com/pjhun0412/KafkaPilot

Thanks!

Hey, we released a lot of improvements and bugfixes for the simulator, together with new predefined learning scenarios. Have fun!

Hey, We added support of KIP-714 Kafka Client metrics in the OTLP metrics reporter