Apache Kafka producer and consumer with FastAPI and aiokafka (2022)

Apache Kafka producer and consumer with FastAPI and aiokafka

After I finally published flashgeotext last month, it was time for the next item on the to-do list - Apache Kafka. After I somehow avoided the topic completely for the last years, aside from that one friend that wouldn’t shut up about it, I noticed that more often than not Apache Kafka knowledge is a requirement for a lot of positions in data engineering nowadays. So I did what I recommand everybody starting out - check out Tim Berglund’s introduction on Apache Kafka on YouTube and skinny-dip into the rabbit hole from there.

Find the finished project on GitHub: geo-stream-kafka

Always great to read an introduction into a complex topic from somebody who just recently started out with it, right? What could go wrong. So in its core, Apache Kafka is a messaging system with somebody/something producing a message on the one side and a somebody/something consuming the message on the other side, and a lot of magic in between.

Kafka core principles

To zoom in on the magic part, when a producer sends a message, the message is pushed into Kafka topics. When a consumer consumes a message it is pulling the message from a Kafka topic. Kafka topics reside within a so-called broker (eg. Zookeeper). Zookeeper provides synchronization within distributed systems and in the case of Apache Kafka keeps track of the status of Kafka cluster nodes and Kafka topics.

Apache Kafka producer and consumer with FastAPI and aiokafka (1)

You can have multiple producers pushing messages into one topic, or you can have them push to different topics. Messages within topics can be retained indefinitly or be discarded after a certain time, depending on the needs. When a consumer starts consuming a message, it starts from the first message that has been pushed to the topic and continues from thereon. Kafka stores the so-called offset, basically a pointer telling the consumer which messages have been consumed and what is still left to indulge. Messages will stay within the topic, yet when the same consumer pulls messages from the topic it will only receive messages from the offset onwards.

There is actually a lot more to dissect here, but if you want to understand as much Apache Kafka to be dangerous, I’d say we leave it at that and I direct you to more competent, no less handsome, people to tell you more.

Okay, we have the theory let’s build something with it. This time, instead of a niche NLP problem I wanted to go back to my origins as a geospatial engineer. I figured I would build a producer/consumer architecture for geodata. We will have a FastAPI endpoint that will wrap the producer logic and a consumer FastAPI WebSocket endpoint that a leaflet.js map-application then can tap into. So far I have used Leaflet.js to display static data, so why not try and use it with dynamic data instead. What I had in mind was an architecture like that:

Apache Kafka producer and consumer with FastAPI and aiokafka (2)

Setup an Apache Kafka cluster

Hell no, I would never dare to even try to comprehend what’s necessary to build that up from scratch. BUT, we live in different times now and there is that handsome technology called Docker. We also can be thankful that respective members of the interwebs community like wurstmeister (german for Saugagemaster) prepare containers like Kafka-docker that are ready to be used for you local (maybe more?) development entertainment. Now I will say that even that container didn’t come without troubles for me. I just wasn’t able to connect to the Kafka broker from outside the container. There is wurstmeisters kafka connectivity guide that helped me to shed some light on the networking inside, but it didn’t help me to get a connection from the outside. The docs suggest that you add

environment: KAFKA_ADVERTISED_HOST_NAME: hostname KAFKA_CREATE_TOPICS: "geostream:1:1" KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
(Video) Simple project FastAPI Kafka PubSub

to your docker-compose file. There are three things to unpack. We can create the Kafka Topic right here and give it 1 partition and 1 replica. We connect Kafka container to the zookeeper container. Now to the connection problem. So I thought that I wanted to advertise Kafka to localhost. Yet it took me some time to understand why that would not be possible. See, what I learned is that Docker on Mac sets up a VM (yaya old news blabla), so localhost for Kafka inside its container is not your localhost but the IP of the VM your Kafka is running in. So I added this to my .zshenv file:

and was able to advertise the hostname like so:

environment: KAFKA_ADVERTISED_HOST_NAME: ${DOCKER_KAFKA_HOST}

et voila, it’s alive!

Test the Apache Kafka cluster

Wait, how do we know it’s alive and taking messages? We create a minimum viable Kafka producer and a minimum viable Kafka consumer, spin up to terminals and fire the up the producer and the consumer.

Setup a minimum viable producer

from pykafka import KafkaClientimport timeclient = KafkaClient("127.0.0.1:9092")geostream = client.topcis["geostream"]with geostream.get_sync_producer() as producer: i = 0 for _ in range(10): producer.produce(("Kafka is not just an author " + str(i).encode("ascii")) i += 1 time.sleep(1)

Setup a minimum viable consumer

from pykafka import KafkaClientclient = KafkaClient(hosts="127.0.0.1:9092")def get_messages(topicname): def events(): for message in client.topics[topicname].get_simple_consumer(): yield f"i.value.decode()" return events()for x in get_messages("geostream"): print(x)
(Video) Kafka Tutorial | Python Producer and Consumer code | Python with Apache Kafka

If you see your consumer printing out what the producer is pushing, you’re good to go. If you set up your networking incorrectly, then you will notice it at this stage at the latest.

FastAPI Apache Kafka producer

As shown in my sketch I want to wrap the producer into a FastAPI endpoint. This allows for more than one entity at a time to produce messages to a topic, but also enables me to flexibly change topics that I want to produce messages to with FastAPI endpoint path parameters. I opted to use aiokafka instead of pykafka to make use of FastAPIs async capabilities, but also to plague myself and get a better understanding of async programming (still pending).

FastAPIs on_event("startup) and on_event("shutdown") make the use of a aiokafka producer easy.

loop = asyncio.get_event_loop()aioproducer = AIOKafkaProducer( loop=loop, client_id=PROJECT_NAME, bootstrap_servers=KAFKA_INSTANCE)@app.on_event("startup")async def startup_event(): await aioproducer.start()@app.on_event("shutdown")async def shutdown_event(): await aioproducer.stop()

Afterwards we can use the aioproducer in our application.

@app.post("/producer/{topicname}")async def kafka_produce(msg: ProducerMessage, topicname: str): await aioproducer.send(topicname, json.dumps(msg.dict()).encode("ascii")) response = ProducerResponse( name=msg.name, message_id=msg.message_id, topic=topicname ) return response

FastAPI Apache Kafka consumer

The Apache Kafka consumer endpoint with FastAPI turned out to be a completely different beast. I wasn’t able to setup up something similar to the pykafka MVP. In Flask you can do something like and push server sent events to whoever is calling the /consumer/<topicname> endpoint:

@app.route('/consumer/<topicname>')def get_messages(topicname): def events(): for message in client.topics[topicname].get_simple_consumer(): yield f"i.value.decode()" return Response(events(), mimetype="text/event-stream")
(Video) Realtime Maps - Python Kafka Consumer (6)

This way we can in the Leaflet application we can add an EventSource and an EventListener that would take incoming events and do something with it - exactly as we wanted it to be in the first place (see architecture). I was not able to do the same with FastAPI and an async generator. That is not to say it is not possible, but my dumb ass wasn’t able to figure out how and didn’t bother to open an issue with FastAPI.

What I did figure out, however, is that there is a beautiful thing called Websockets and that FastAPI happily supports the likes. Since FastAPI is built on-top of starlette we can use class-basedd endpoints and especially the WebsocketEndpoint to handle incoming Websocket Sessions.

@app.websocket_route("/consumer/{topicname}")class WebsocketConsumer(WebSocketEndpoint): async def on_connect(self, websocket: WebSocket) -> None: # get topicname from path until I have an alternative topicname = websocket["path"].split("/")[2] await websocket.accept() await websocket.send_json({"Message": "connected"}) loop = asyncio.get_event_loop() self.consumer = AIOKafkaConsumer( topicname, loop=loop, client_id=PROJECT_NAME, bootstrap_servers=KAFKA_INSTANCE, enable_auto_commit=False, ) await self.consumer.start() self.consumer_task = asyncio.create_task( self.send_consumer_message(websocket=websocket, topicname=topicname) ) logger.info("connected") async def on_disconnect(self, websocket: WebSocket, close_code: int) -> None: self.consumer_task.cancel() await self.consumer.stop() logger.info(f"counter: {self.counter}") logger.info("disconnected") logger.info("consumer stopped") async def on_receive(self, websocket: WebSocket, data: typing.Any) -> None: await websocket.send_json({"Message": data}) async def send_consumer_message(self, websocket: WebSocket, topicname: str) -> None: self.counter = 0 while True: data = await consume(self.consumer, topicname) response = ConsumerResponse(topic=topicname, **json.loads(data)) logger.info(response) await websocket.send_text(f"{response.json()}") self.counter = self.counter + 1

When an application starts a websocket connection with out websocket endpoint we grab the event loop, use that to build and start the aiokafka consumer, start it and start a consumer task in the loop. Once this is set, everytime the consumer pulls a new message it is forwarded to the application through the websocket. When the leafletjs application either specifically closes the websocket or the browser is closed, we close the websocket, cancel the consumer_task and stop the consumer.

Leaflet application

I have to admit that it’s been some time that I have touched JavaScript at all, so bare with me here.

First we will declare our map and websocket connection to the /consumer/{topicname} endpoint.

var map = new L.Map('map');var linesLayer = new L.LayerGroup();var ws = new WebSocket("ws://127.0.0.1:8003/consumer/geostream");var osmUrl = 'https://{s}.basemaps.cartocdn.com/light_all/{z}/{x}/{y}{r}.png', osmAttribution = '&copy; <a href="https://www.openstreetmap.org/copyright">OpenStreetMap</a> contributors &copy; <a href="https://carto.com/attributions">CARTO</a>', osm = new L.tileLayer(osmUrl, {maxZoom: 18, attribution: osmAttribution});var colors = ["#8be9fd", "#50fa7b", "#ffb86c", "#ff79c6", "#bd93f9", "#ff5555", "#f1fa8c"];lines = {};

When client and backend established the silent agreement to use WebSockets, we can declare what we want to do, whenever the websocket receives a new message. Every message is an event. And every event consists of metadata and event.data that can be parsed with JSON.parse().

(Video) Harshit Prasad - High Performance Data Processing with Python, Kafka and Elasticsearch

ws.onmessage = function(event) { console.log(event.data) obj = JSON.parse(event.data) [...]

One of the requirements was to display more than one entity that pushes messages through the producer, Kafka and the consumer on the map as a live-event. For the leaflet application to associate an event to an entity, I hash events by the name of the entity that is sending them. If there is a new name in an event, it’ll be hashed into a dictionary and added as a new layer on the map. As I wanted every entity to be represented with a different color, the color will be randomly grabbed from the list of colors and hashed alongside the position of the event/entity.

 [...] if(!(obj.name in lines)) { lines[obj.name] = {"latlon": []}; lines[obj.name]["latlon"].push([obj.lat, obj.lon]); lines[obj.name]["config"] = {"color": colors[Math.floor(Math.random()*colors.length)]}; } else { lines[obj.name]["latlon"].push([obj.lat, obj.lon]); } line = L.polyline(lines[obj.name]["latlon"], {color: lines[obj.name]["config"]["color"]}) linesLayer.addLayer(line) map.addLayer(linesLayer);};

On thing to keep in mind is, that when you zoom on the map the bounds will be messed up and the events will not properly draw polylines along the trajectory of the entity. To fix this I added an zoomend trigger:

that will clear the layers until the next event arrives and then continues to draw the trajectories.

Result

Apache Kafka producer and consumer with FastAPI and aiokafka (3)

· python, kafka, FastAPI

FAQs

How do you run producer and consumer in Kafka? ›

Short Answer
  1. Provision your Kafka cluster.
  2. Download and setup the Confluent CLI.
  3. Create the Kafka topic.
  4. Start a console consumer.
  5. Produce events to the Kafka topic.
  6. Produce records with full key-value pairs.
  7. Start a consumer to show full key-value pairs.
  8. Teardown Confluent Cloud resources.

Can a Kafka producer write to multiple partitions? ›

Using the right partitioning strategies allows your application to handle terabytes of data at scale with minimal latency. A Kafka producer can write to different partitions in parallel, which generally means that it can achieve higher levels of throughput.

Can a Kafka producer write to multiple topics? ›

Kafka is able to seamlessly handle multiple producers that are using many topics or the same topic. The consumer subscribes to one or more topics and reads the messages. The consumer keeps track of which messages it has already consumed by keeping track of the offset of messages.

What is Kafka consumer and Kafka producer? ›

The Kafka Producer API allows applications to send streams of data to the Kafka cluster. The Kafka Consumer API allows applications to read streams of data from the cluster.

Does Kafka producer need ZooKeeper? ›

ZooKeeper and Kafka

For now, Kafka services cannot be used in production without first installing ZooKeeper. * This is true even if your use case requires just a single broker, single topic, and single partition. *Starting with v2. 8.0, Kafka can be run without ZooKeeper.

Is Kafka consumer push or pull? ›

Is Kafka Push or Pull? Kafka Consumers pull data from the topic they are listening to. This allows different consumers to consume messages at different rates without the broker (Kafka) having to determine the right data rate for each consumer. It also allows better batch processing mechanics.

Why do we have 3 replications in Kafka? ›

This is all about high availability, the recommended replication factor for Kafka is 3, meaning that you are going to have 3 replicas for every piece of data sent to your cluster.

Can two consumers consume from same partition at the same time in Kafka? ›

A consumer can be assigned to consume multiple partitions. So the rule in Kafka is only one consumer in a consumer group can be assigned to consume messages from a partition in a topic and hence multiple Kafka consumers from a consumer group can not read the same message from a partition.

How do you improve Kafka producer performance? ›

  1. Provision your Kafka cluster.
  2. Initialize the project.
  3. Write the cluster information into a local file.
  4. Download and setup the Confluent CLI.
  5. Create a topic.
  6. Run a baseline producer performance test.
  7. Run a producer performance test with optimized throughput.
  8. Teardown Confluent Cloud resources.

What are the 4 major Kafka Apis? ›

The Admin API for inspecting and managing Kafka objects like topics and brokers. The Producer API for writing (publishing) to topics. The Consumer API for reading (subscribing to) topics. The Kafka Streams API to provide access for applications and microservices to higher-level stream processing functions.

How many Kafka topics is too many? ›

Is There a Limit on the Number of Topics in a Kafka Instance?
FlavorBrokersMaximum Partitions per Broker
kafka.4u8g.cluster3–30500
kafka.8u16g.cluster3–301000
kafka.12u24g.cluster3–301500
kafka.16u32g.cluster3–302000
1 more row
11 Aug 2022

What if leader goes down in Kafka? ›

Kafka does not create a new replica when a broker goes down. If the offline broker was a leader, a new leader is elected from the replicas that are in-sync. If no replicas are in-sync it will only elect an out of sync replica if unclean.

Is Kafka LIFO or FIFO? ›

Kafka supports a publish-subscribe model that handles multiple message streams. These message streams are stored as a first-in-first-out (FIFO) queue in a fault-tolerant manner. Processes can read messages from streams at any time.

Is Kafka a bus or broker? ›

Kafka is a message bus developed for high-ingress data replay and streams. Kafka is a durable message broker that enables applications to process, persist, and re-process streamed data. Kafka has a straightforward routing approach that uses a routing key to send messages to a topic.

What is the difference between Kafka consumer and Kafka stream? ›

Difference between Kafka Streams and Kafka Consumer

Kafka Streams is an easy data processing and transformation library within Kafka used as a messaging service. Whereas, Kafka Consumer API allows applications to process messages from topics.

Why did Kafka remove ZooKeeper? ›

Replacing ZooKeeper with internally managed metadata will improve scalability and management, according to Kafka's developers. Change is coming for users of Apache Kafka, the leading distributed event-streaming platform.

Does Kafka need JDK or JRE? ›

Zookeeper and Kafka are written in Java so you'll need JDK.

How many producers can Kafka have? ›

One producer for all topics and partitions

Probably the optimal choice for most applications. The Kafka producer batches any written message before sending them to the Kafka cluster for up to batch.

Is Kafka using TCP or UDP? ›

Kafka uses a binary protocol over TCP. The protocol defines all APIs as request response message pairs.

How can I make Kafka consumer faster? ›

Increasing the number of partitions and the number of brokers in a cluster will lead to increased parallelism of message consumption, which in turn improves the throughput of a Kafka cluster; however, the time required to replicate data across replica sets will also increase.

How many consumers can Kafka handle? ›

While Kafka allows only one consumer per topic partition, there may be multiple consumer groups reading from the same partition.

Is Kafka replication synchronous or asynchronous? ›

Since the replication is asynchronous, there is no guarantee that a commit message can survive any broker failure.

What is the difference between replica and replicate? ›

Replicates may not be exact copies of the original, in the way that duplicates are. If an art student paints Van Gogh's Starry Night, the new painting is a replicate, not a duplicate, since it would not be exactly identical. Usually, in this context, however, one would call it a replica, not a replicate.

Can we have multiple records with the same key in Kafka? ›

By default, Kafka producer relies on the key of the record to decide to which partition to write the record. For two records with the same key, the producer will always choose the same partition. Why is this important? Sometimes we need to deliver records to consumers in the same order as they were produced.

What happens if there are more partitions than consumers? ›

You can have fewer consumers than partitions (in which case consumers get messages from multiple partitions), but if you have more consumers than partitions some of the consumers will be “starved” and not receive any messages until the number of consumers drops to (or below) the number of partitions.

Is Kafka consumer multithreaded? ›

When implementing a multi-threaded consumer architecture, it is important to note that the Kafka consumer is not thread safe. Multi-threaded access must be properly synchronized, which can be tricky. This is why the single-threaded model is commonly used.

Can a Kafka consumer read from multiple clusters? ›

A consumer group, such as a Kafka Streams-based application, can process data from a single Kafka cluster only. Therefore, multi-topic subscriptions or load balancing across the consumers in a consumer group are possible only within a single Kafka cluster.

Why Kafka writes are fast? ›

Why is Kafka fast? Kafka achieves low latency message delivery through Sequential I/O and Zero Copy Principle. The same techniques are commonly used in many other messaging/streaming platforms. Zero copy is a shortcut to save the multiple data copies between application context and kernel context.

Why is consumer slow Kafka? ›

Kafka Consumers

If there are way too many producers writing data to the same topic when there are a limited number of consumers then then the reading processes will always be slow.

How do I reduce Kafka latency? ›

Apache Kafka® provides very low end-to-end latency for large volumes of data. This means the amount of time it takes for a record that is produced to Kafka to be fetched by the consumer is short. If you're using a dedicated cluster, adding additional CKUs can reduce latency.

Which language is best for Kafka? ›

Kafka is a JVM based platform, so the mainstream programming language of the client is Java. But as the community is growing tremendously, high-quality open-sourced Python clients are also available and being used in production.

What are 3 most common APIs? ›

There are also three common types of API architectures: REST, a collection of guidelines for lightweight, scalable web APIs. SOAP, a stricter protocol for more secure APIs. RPC, a protocol for invoking processes that can be written with XML (XML-RPC) or JSON (JSON-RPC).

What is the difference between API and Kafka? ›

But when you need to build an event streaming platform, you use the Kafka API.
...
Difference Between Kafka APIs and REST API.
Kafka APIsREST API
It provides bidirectional communication.The REST API is unidirectional, i.e., you can only send or receive a response at a time.
4 more rows
3 Nov 2021

What are disadvantages of Kafka? ›

Disadvantages Of Apache Kafka

Do not have complete set of monitoring tools: Apache Kafka does not contain a complete set of monitoring as well as managing tools. Thus, new startups or enterprises fear to work with Kafka. Message tweaking issues: The Kafka broker uses system calls to deliver messages to the consumer.

What if Kafka producer fails? ›

Unable to reach Kafka cluster

The producer may fail to push message to a topic due to a network partition or unavailability of the Kafka cluster, in such cases there are high chances of messages being lost, hence we need a retry mechanism to avoid loss of data.

What happens if Kafka topic is full? ›

cleanup. policy property from topic config which by default is delete , says that "The delete policy will discard old segments when their retention time or size limit has been reached." So, if you send record with producer api and topic got full, it will discard old segments.

How do you prevent duplicates in Kafka? ›

This can be avoided by configuring the Kafka Producer to be idempotent. This article describes how duplicate events can be published and how to make the Producer idempotent. Duplicate messages can occur in the scenario where: A Producer attempts to write a message to a topic partition.

Where Kafka should not be used? ›

It's best to avoid using Kafka as the processing engine for ETL jobs, especially where real-time processing is needed. That said, there are third-party tools you can use that work with Kafka to give you additional robust capabilities – for example, to optimize tables for real-time analytics.

Can Kafka call a REST API? ›

The Kafka REST Proxy is a RESTful web API that allows your application to send and receive messages using HTTP rather than TCP. It can be used to produce data to and consume data from Kafka or for executing queries on cluster configuration.

Is Kafka stateful or stateless? ›

Stateful operations in Kafka Streams are backed by state stores. The default is a persistent state store implemented in RocksDB, but you can also use in-memory stores. State stores are backed up by a changelog topic, making state in Kafka Streams fault-tolerant.

Is Kafka an AP or CP system? ›

According to the Engineers at LinkedIn (where Kafka was initially founded) Kafka is a CA system: All distributed systems must make trade-offs between guaranteeing consistency, availability, and partition tolerance (CAP Theorem).

Is Kafka ETL tool? ›

Companies use Kafka for many applications (real time stream processing, data synchronization, messaging, and more), but one of the most popular applications is ETL pipelines. Kafka is a perfect tool for building data pipelines: it's reliable, scalable, and efficient.

Is Kafka a memory or disk? ›

Low-Latency IO:

Kafka relies on disk for storage and caching. but the problem is, disks are slower than RAM. It achieves low latency as random access memory(RAM) through Sequential IO.

Is Kafka better than MQ? ›

Apache Kafka scales well and may track events but lacks some message simplification and granular security features. It is perhaps an excellent choice for teams that emphasize performance and efficiency. IBM MQ is a powerful conventional message queue system, but Apache Kafka is faster.

Is Kafka a stream or queue? ›

We can use Kafka as a Message Queue or a Messaging System but as a distributed streaming platform Kafka has several other usages for stream processing or storing data. We can use Apache Kafka as: Messaging System: a highly scalable, fault-tolerant and distributed Publish/Subscribe messaging system.

Does Netflix uses Kafka? ›

Apache Kafka is an open-source streaming platform that enables the development of applications that ingest a high volume of real-time data. It was originally built by the geniuses at LinkedIn and is now used at Netflix, Pinterest and Airbnb to name a few.

Is Kafka batch or stream? ›

Batch Processing is an important technique that helps enterprises to understand the data as well as process huge volumes of it efficiently. Kafka is a leading event streaming software that is used by many companies with even fortune 500 enterprises using its features.

How do you allow the producer and consumer process to run? ›

To allow producer and consumer processes to run concurrently, we must have an available buffer of items that can be filled by the producer and emptied by the consumer. This buffer will reside in a region of memory that is shared by the producer and consumer processes.

Where can I run Kafka consumer? ›

Run Kafka Consumer Console

Kafka provides the utility kafka-console-consumer.sh which is located at ~/kafka-training/kafka/bin/kafka-console-producer.sh to receive messages from a topic on the command line. Create the file in ~/kafka-training/lab1/start-consumer-console.sh and run it.

How do I start a Kafka consumer? ›

Step 1: Start the zookeeper as well as the kafka server initially. Step2: Type the command: 'kafka-console-consumer' on the command line. This will help the user to read the data from the Kafka topic and output it to the standard outputs. Video Player is loading.

How do I produce and consume messages from Kafka topic? ›

Complete the following steps to receive messages that are published on a Kafka topic:
  1. Create a message flow containing a KafkaConsumer node and an output node.
  2. Configure the KafkaConsumer node by setting the following properties: On the Basic tab, set the following properties:
11 Aug 2022

Can a producer also be a consumer? ›

A consumer is someone who pays for goods and services. (Sources B and C) • You can be both a producer and a consumer.

How is producer linked with consumer? ›

Producers use energy and inorganic molecules to make food. Consumers take in food by eating producers or other living things.

Can a producer become a consumer? ›

All producers can become avid consumers, not all avid consumers can become producers.

Can one Kafka topic have multiple consumers? ›

So the rule in Kafka is only one consumer in a consumer group can be assigned to consume messages from a partition in a topic and hence multiple Kafka consumers from a consumer group can not read the same message from a partition.

How many consumer groups can Kafka have? ›

By default, the quota for consumer groups is twice the quota for topics. For example, if you can create a maximum of 50 topics on your Message Queue for Apache Kafka instance, you can create a maximum of 100 consumer groups on the instance.

How Kafka producers produce a message? ›

A producer partitioner maps each message to a topic partition, and the producer sends a produce request to the leader of that partition. The partitioners shipped with Kafka guarantee that all messages with the same non-empty key will be sent to the same partition.

Can Kafka push data to consumer? ›

The same message batch can be compressed and sent to Kafka broker/server in one go and written in compressed form into the log partition. You can even configure the compression so that no decompression happens until the Kafka broker delivers the compressed records to the consumer.

Videos

1. Использование Kafka в ваших python-приложениях [ru] / Александр Тарасенко
(fwdays)
2. Что такое Apache Kafka за 200 секунд
(The Art of Development)

Top Articles

Latest Posts

Article information

Author: Aron Pacocha

Last Updated: 11/13/2022

Views: 6423

Rating: 4.8 / 5 (68 voted)

Reviews: 83% of readers found this page helpful

Author information

Name: Aron Pacocha

Birthday: 1999-08-12

Address: 3808 Moen Corner, Gorczanyport, FL 67364-2074

Phone: +393457723392

Job: Retail Consultant

Hobby: Jewelry making, Cooking, Gaming, Reading, Juggling, Cabaret, Origami

Introduction: My name is Aron Pacocha, I am a happy, tasty, innocent, proud, talented, courageous, magnificent person who loves writing and wants to share my knowledge and understanding with you.