Robots Exclusion Protocol: now with even more flexibility

This is the third and last in the series of blog posts about the Robots Exclusion Protocol (REP). In the first post, Dan Crow introduced robots.txt and the robots META tags, giving an overview of when to use them. In the second post, Dan Crow shared some examples of what you can do with the REP. Today, he will introduce two new features that we have recently added to the protocol. (Read more...)