Home > Yahoo Slurp

Wildcard Characters in robots.txt

Posted: Fri 11/03/2006 by ramiro
Your opinion

Yahoo! Slurp now also recognizes wildcard characters in the robots.txt file enabling webmasters to restrict what gets indexed in a more sophisticated manner.

You can use * to match a sequence of characters and $ as an anchor to indicate the end of a URL string. You can find examples in the Yahoo! Search blog.

Syndicate content