Kafka excels at moving data in-transit, but not so much about protecting data at rest — which is necessary for enterprises. Let’s delve into the current state of affairs exploring a typical scenario you may find in your organization, and our solutions to this challenge. — Kafka: from Technical POC to Serious Business. Picture this: Kafka, an exceptional platform that enables seamless data sharing across an entire enterprise. It all starts innocently enough, with a team tackling a simple use case — perhaps log management or telemetry.

Conduktor Gateway sits between your Kafka clusters and your applications and works with the Apache Protocol. We review different ways to deploy it, pros and cons. — “Cool but how do I deploy this thing?!” This is the first question you ask yourself when you have a new toy to play with. …

Kafka is amazing and easy to start with. It’s rapidly adopted in any organization. Then the associated costs start to grow fast as well! Let’s embark on the path to cost efficiency to avoid any surprises. — Introduction Each company has its own Kafka story, but the pattern is often the same: You start with a few clusters, and then you end up with a lot of them You have a few topics, and then you have thousands of them.

The most interesting feedback we got at Kafka Summit regarding what we do and where we’re going — Here we are, one week after the Kafka Summit conference that took place in London. It’s a wonderful event where the whole Kafka community gathers and discusses the latest evolution in the domain. As a company leading the Kafka tooling space, we were in the place fully present, enthusiastic, motivated…

The Conduktor Engineers team The Conduktor developer team is composed of a group of experienced and curious professionals. Scala features/enables a lot of characteristics we value, so that’s the language we chose. Our DNA is our Kafka expertise and we believe that modern software architectures have to be built around streaming. Functional Programming We believe that…

This post is an excerpt of my full article on my blog: https://www.sderosiaux.com/articles/2019/08/07/kafka-streams-topology-and-optimizations/. It’s quite long but has many insights, explanations, diagrams, and code examples. Why? Optimizations clearly reduce the load on our Kafka Cluster by avoiding to create unnecessary internal topics and simplify the Topology our Kafka Streams applications use…

Or how an interview led to a training seance — Today, someone asked me this question: “What makes you a Kafka expert?” at the beginning of a job interview where I applied to work as a Freelance. The job was interesting because the company was considering moving from a monolith to a Kafka stack and micro-services. A huge step! I…

Or how to deal with errors in a functional way — Working with my team, I had to deal with some business errors which were Exceptions. I was a bit nervous about them, and was having a “not good enough”, “not expressive enough” feeling. Having a Scala background, I decided to show them how we could handle errors differently. Let’s look…

Working in a large company, my team and I are working with JSON data (let’s not talk about CSV and XML please). We had to find out what were the definitions and business rules of the models we received from other services. Trust me, it’s not that easy. …