Records = self._poll_once(remaining, max_records, update_offsets=update_offsets)įile "/home/PycharmProjects/myPythonLearning/venv/lib/python3.8/site-packages/kafka/consumer/group.py", line 708, in _poll_once Record_map = self.poll(timeout_ms=timeout_ms, update_offsets=False)įile "/home/PycharmProjects/myPythonLearning/venv/lib/python3.8/site-packages/kafka/consumer/group.py", line 655, in poll : UnsupportedCodecError: Libraries for zstd compression codec not foundįile "/home/PycharmProjects/myPythonLearning/kafka/consumer/consumer.py", line 25, inįile "/home/PycharmProjects/myPythonLearning/kafka/consumer/consumer.py", line 14, in consume_from_topicįile "/home/PycharmProjects/myPythonLearning/venv/lib/python3.8/site-packages/kafka/consumer/group.py", line 1193, in nextįile "/home/PycharmProjects/myPythonLearning/venv/lib/python3.8/site-packages/kafka/consumer/group.py", line 1201, in next_v2įile "/home/PycharmProjects/myPythonLearning/venv/lib/python3.8/site-packages/kafka/consumer/group.py", line 1116, in _message_generator_v2 Getting below error when trying to consume from a ZSTD compressed record from kafka. This seems to me to be the trickiest piece, how to do this somewhat generically/thin so it provides flexibility rather than straight-jacketing the user. such as the level of compression, multi-core vs single (this may not be relevant for our use case), etc. Any custom configs that would be passed to the underlying library.allowed values are None/ gzip/ snappy/other broker-supported values Class-level var indicating which protocol flag to set.I'm thinking the consumer/producer can pass in a custom class. However, there is still protocol-level considerations since the producer has to signal to the broker what compression format was used (which it, in turn, passes to downstream consumers). Would be really nice to refactor the codec compression interfaces into something more generic, allowing users to wrap their favorite library - rather than us having to pick a favorite to integrate with directly.Īgreed.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |