The bot also needs to be aware of the period of time since the last change of a web page to ensure that valuable information or updates is not missed and that processing power isn’t wasted on a link that does not change very much.
