In a system like Kafka the logs are written to commit log and then tracked using high water mark for consistent exposure to the end-user. Since, the high water mark is done only when the ack is received from all the followers, how this marking will be done if one of the followers go down in the middle of writing a log on its WAL? What mechanism is used to counter such scenarios? Will it be checking the heartbeat of the follower that will help here?
Hi @Nishant_Sharma, Thanks for reaching out to us.
For each data mutation, the leader first appends it to WAL and then sends it to all the followers. Upon receiving the request, the followers append it to their respective WAL and then send an acknowledgment to the leader. The leader keeps track of the indexes of the entries that have been successfully replicated on each follower. The high-water mark index is the highest index, which has been replicated on the quorum of the followers. The leader can propagate the high-water mark index to all followers as part of the regular Heartbeat message. The leader and followers ensure that the client can read data only up to the high-water mark index. This guarantees that even if the current leader fails and another leader is elected, the client will not see any data inconsistencies.
Hope this helps.
Happy Learning 