educative.io

The Data Partitioning scheme is making way too many shards

Hi @Design_Gurus,

The database partitioning section says the following:

“Partitioning based on UserID:** Let’s assume we partition based on the hash of the UserID so that we can keep all messages of a user on the same database. If one DB shard is 4TB, we will have “3.6PB/4TB ~= 900” shards for five years. For simplicity, let’s assume we keep 1K shards.”

Isn’t a thousand 4TB shards way too expensive ? Considering the actual cost + maintenance cost (those are physical servers if I am not wrong.) + we are not taking the image metadata storage into picture (which is but a standard feature of chat apps nowadays).