IO isssues multiple thread consume different offset can cause inefficiency on IO?

[Ronaldo]

It says, The broker solved the problems that our first design had. We avoided a large number of queues by partitioning the topic. We introduced parallelism using partitions that avoided bottlenecks while consuming the message.

1. For a single machine, different consumer consume the msg in different offset. Isn’t it inefficient for I/O?
2. I don’t get why the graph has anything do with partition?

---

Course: Grokking Modern System Design Interview for Engineers & Managers - Learn Interactively
Lesson: https://www.educative.io/courses/grokking-modern-system-design-interview-for-engineers-managers/design-of-a-pub-sub-system

[Muhammad_Usman_Malik]

Hi Ronaldo

There are many ways to avoid IO penalty when multiple consumers are reading from different places. Usually, there is an auxiliary in-memory data structure that facilitates such reading. Additionally if message sizes are of constant length, seeking to a specific point is easy, and then multiple messages can be cached for subsequent readings.

We have an advanced design of a key-value store in our Advanced System Design course that shows many of such advanced techniques (https://www.educative.io/courses/grokking-the-principles-and-practices-of-advanced-system-design/int…).

Thank you