Duplicate Eliminator: To make sure the same content is not extracted twice unintentionally.
is the purpose of the duplicate eliminator to make sure that the same url doesn’t get recrawled if it is seen again at a new location?
Hey @Dewey_Munoz
Yes, duplicate eliminator makes sure that the same URL doesn’t get recrawled again. It maintains uniqueness.
Shaheryaar Kamal | Technical Content Engineer