In the section, the separate buffer will be used for Enqueue and DeQueue. I wanted to know why 2 buffers and how it can be implemented?
We can implement our queues in such a way that they have separate buffers for enqueuing and dequeuing. Enqueue buffer, once filled, will be dumped to the disk, whereas dequeue buffer will keep a cache of URLs that need to be visited; it can periodically read from disk to fill the buffer.
Hi @akaRahim, hope you are doing well.
The reason to use different buffers is due to the nature of the operation. Enqueue means that you are adding things to the queue and want to process it (dump to disc latter) and DeQueue means you are remove items from the Queue (and you don’t want to wait a long time to retrieve the element from the Queue). Hence, two buffers are needed since they will be storing different things with different contexts.
A buffer can be implemented just like a cache is, with same features of a cache. So, we could implement this in a Key-Value database (i.e. Redis) or something like memcached. For Enqueue we can set a default value that when reached, all items are added to the queue, respecting the order they were added. For Dequeueing, we can keep last tree items in this buffer, when we consume the buffer the item is removed and a new item is dequeue from the Queue and added to buffer.
Hope this helps!
-Artur Baruchi
I don’t think I understood what you wrote @Artur_Baruchi - are you implementing a queue using 2 buffers? what is it that you are suggesting with the Enqueue buffer and Deque buffer.
So when a new url comes in does it go straight to the Enqueue buffer? And if this gets filled up, we write the entire buffer to disk? And then the Dequeue buffer reads from the disk the latest urls to crawl through and passes it on to the queue when it is requested?
Course: Grokking the System Design Interview - Learn Interactively
Lesson: Designing a Web Crawler - Grokking the System Design Interview