Since we are distributing URLs based on the hostnames, we can store these data on the same host.
what does it mean by storing data on the host. isn’t data only stored in the object store and index?
Hi @Dewey_Munoz
As we are following the approach of distributing URLs based on the hostnames, therefore we can store the data i.e, set of URLs that are needed to be visited by that host, on each host. By following this approach much less time will be consumed. This concept can be like storing data in the cache of the host.