I have seen many times that the article creates the reverse mapping of the index server like this kind services, in case if any index server is failed so that we know what all words it was having before failure so that we can rebuild it again.
But my question is keeping the reverse index mapping in Index builder (in this case tweet_id : server_id) is not good idea. Imagine you are doing scale in and scale out of the index server and tweet_id gets shifted around based on the consistent hashing.
How can you maintain in this case reverse index mapping in index builder. ?
Usually fault toleant architecture is being handled in terms of failure detection technique and all. If primary fails and use one of the slaves for primary sure there are many algo are being used in this case.
But keeping in index builder memory reverse mapping is very bad since it will add inefficient maintenance process for maintaining reverse mapping, extra storage is needed for all the tweet_id which is a lot.
Just wondering there might be better strategy where the data is being migrated every certain interval duration in cold storage for backup purpose?..
Any clarification on this part. ?
Thanks