Home > Googlebot
Wildcard Characters in robots.txt
Posted: Fri 11/03/2006 by ramiroYahoo! Slurp now also recognizes wildcard characters in the robots.txt file enabling webmasters to restrict what gets indexed in a more sophisticated manner.
You can use * to match a sequence of characters and $ as an anchor to indicate the end of a URL string. You can find examples in the Yahoo! Search blog.
