Kafka idempotent producers

According to the documentation, an idempotent kafka producer maintains a sequence number, ensuring deduplication of events and also, ordering. Does this mean that within a single producer session, an idempotent kafka producer would indef

Continue reading →

Is there a way to consume payloads from kafka without using @Incoming (Quarkus, smallrye, kafka)?

I'm creating a publish method using Emitter And I'm trying to create a unit test for this method. I don't want to use the InMemoryConnector (as described in the quarkus doc) I want to use the actual kafka connector configured in my appli

Continue reading →

How to terminate backround processes using bash script

I am trying to write a script which will book up a local Kafka cluster to speed up my development process. I was able to book up the environment but I am having trouble automating the tear down of the cluster. What the cluster contains:

Continue reading →

No logs after running Zookeeper and kafka serer in Windows

I am also facing the same issue, when I start zookeeper didn't get any error or logs just blank. Could you please anybody help me to resolve. in zookeeper.properties(folder temp I have manually created): dataDir=F:/kafka_2.12-3.4enter im

Continue reading →

Apache Kafka vs Apache Spark

Apache Kafka: Distributed Messaging System Apache Spark: Real-time data processing How can I use both technologies in a real-time data pipeline for data processing? In terms of real-time data streaming, for me, they both do the job. How

Continue reading →

/opt/bitnami/scripts/libkafka.sh: line 364: arr[2]: unbound variable

I am not so experienced in Kafka, I am to deploy it in my Kubernetes but it is throwing an error. I am sending my artifact. Please could you know how to solve it? Error

Continue reading →

Can not set custom parameters(numPartitions,defaultReplicationFactor) for bitnami/kafka

I created an Azure K8s cluster. Then I install the bitnami/kafka by using helm: helm install kafka bitnami/kafka --set replicaCount=3,provisioning.enabled=true,provisioning.replicationFactor=3,provisioning.numPartitions=3,extraConfig="l

Continue reading →

Is ksqlDB suitable for reconsuming messages with a specific key only?

We have the situation where we wish to use ksqlDB for the aggregation of various invoice data. The exact aggregation that should occur here is dependent on user-specific configuration values which, for example, determine when the cut-off

Continue reading →