Home > ramiro's blog > Wildcard Characters in robots.txt

Wildcard Characters in robots.txt

Posted: Fri 11/03/2006 by ramiro

Yahoo! Slurp now also recognizes wildcard characters in the robots.txt file enabling webmasters to restrict what gets indexed in a more sophisticated manner.

You can use * to match a sequence of characters and $ as an anchor to indicate the end of a URL string. You can find examples in the Yahoo! Search blog.

Googlebot also supports these two wildcards. For more information go to Google's Webmaster Help Center.

SEO Tools to Help Your Rank Higher

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <p> <br> <img> <h2> <h3> <h4> <h5>

More information about formatting options

Type the characters you see in this picture. (verify using audio)
Type the characters you see in the picture above; if you can't read them, submit the form and a new image will be generated. Not case sensitive.