In dropbox design, we split large files into chunks so that it is easy to cache as well as less load on servers. Why doesn’t youtube design involve similar concept?
If we split videos to chunks and store chunks on server based on hash of chunk ids, wouldn’t it be better? Would it be wrong if we decide to go with this approach in an interview?
2 Likes
Hi Anup! yes, you are very right. Youtube does use the concept of chunks while uploading and playing a youtube video. It fetches chunks of video files from different servers (preferably from a server locating closest to the system) while playing the video, which they call as “Sliced bread”. Similarly, while uploading the video, Youtube uses the concept of “Codec (Compress and Decompress)” which makes compress the size of of a video into multiple pixel formats (144p, 240p, etc.) and save each video in form of chunks.
So, both of the techniques “Sliced Bread” and “Codec” work on the principle of making chunks of a single video file.
You can see the following videos for your reference:
Does netflix/youtube/ott platform upload videos in chunks or as a whole file?
If chunk then how and who does the chunking at the client-side?
If the whole file, then does resume happen in case of network/transmission failure? if yes then how?
1 Like
chunking a large file is not required to be done on device. Infact we don’t need client side code to chunk large files. On Browser level only, we can use a simple javascript code where we maintain next chunk offset in memory for given file path to upload a large file in small chunks. Later server can check if MD5 matches with what we calculated before uploading the file.
Hi Sonam,
Thank you for your questions. Let me answer them one by one.
-
Netflix – does not support uploads. However, when you stream from Netflix, it understands your client by requesting client-info through its API and provides chunks/segments of the title/video that is best suited according to your device (mobile phone, desktop, TV).
-
YouTube: Google APIs can allow you to resume interrupted uploads through their APIs. There are a number of techniques that allow you to resume upload of a file that was interrupted due to network, client, or server failures. Take a look at our file upload API in the Grokking the API Design Interview course.
If chunk then how and who does the chunking at the client-side?
Mainly, the server-side handles such complexities. However, depending on the technique used for supporting resumable uploads (for example HTTP range requests, multipart uploads), clients can also play a role.
If the whole file, then does resume happen in case of network/transmission failure? if yes then how?
It will be non-sensical to resume the upload of a large file without chunking because technically we are reuploading the whole file. In general, resumption can happen because of client, network, or even the server failure.
1 Like