We’ve improved Webmaster Central’s robots.txt analysis tool to recognize sitemap declarations and relative urls. Earlier versions weren’t aware of sitemaps at all, and understood only absolute URLs; anything else was reported as Syntax not understood. The improved version now tells you whether your sitemap’s URL and scope are valid. You can also test against relative URLs with a lot less typing.
Reporting is better, too. You’ll now be told of multiple problems per line if they exist, unlike earlier versions which only reported the first problem encountered. And we’ve made other general improvements to analysis and validation.
Imagine that you’re responsible for the domain www.example.com and you want search engines to index everything on your site, except for your /images folder. You also want to make sure your sitemap gets noticed, so you save the following as your robots.txt file:
disalow images
user-agent: *
Disallow:
sitemap: http://www.example.com/sitemap.xml
You visit Webmaster Central to test your site against the robots.txtanalysis tool using these two test URLs:
http://www.example.com
/archives
Earlier versions of the tool would have reported this:
See for yourself at http://www.google.com/webmasters/tools.
Related posts:
- Advertisers Cloaking AdWords URLs To Get Around Display URL Policy Remember Google announced that they will be strictly enforcing the...
- TOP Seach Engine Optimization Forums Digital Point Forum SearchEngineWatch Forum SEO Guy Forum SitePoint...
- A don’t-leave-home-without SEO tool When you work in, on, or around the web, you...
- A don’t-leave-home-without SEO tool When you work in, on, or around the web, you...
- Ranking Checker by SEOmoz.com This nice SEO tool allows you to check your rankings...
Related posts brought to you by Yet Another Related Posts Plugin.
Recent Comments