Home > ramiro's blog > Wildcard Characters in robots.txt
Wildcard Characters in robots.txt
Posted: Fri 11/03/2006 by ramiroYahoo! Slurp now also recognizes wildcard characters in the robots.txt file enabling webmasters to restrict what gets indexed in a more sophisticated manner.
You can use * to match a sequence of characters and $ as an anchor to indicate the end of a URL string. You can find examples in the Yahoo! Search blog.
Googlebot also supports these two wildcards. For more information go to Google's Webmaster Help Center.
Post new comment