How to Use Custom Robots.txt with Your Blog?
Robots.txt is a text file on the server in that you can write which directories, web pages, or links should not be included for search results. It means you can restrict search engine bots to crawl some directories and web pages or links of your website or blog. Now custom robots.txt
is available for Blogger. In Blogger search option is related to
Labels. If you are not using labels wisely per post, you should disallow the crawl of the search link. In Blogger, by default, the search link is disallowed to crawl. In this robots.txt, you can also write the location of your sitemap file. A sitemap is a file located on a server that contains all posts’ permalinks of your website or blog. Mostly
sitemap is found in XML format, i.e., sitemap.xml.
Table of Contents
Robots.txt Type 1
Robots.txt Type 2
Robots.txt Type 3
Manage Blogger custom robots.txt
Robots.txt Type 1
User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Disallow: /b Allow: / Sitemap: https://www.techprevue.blogspot.com/sitemap.xml
Note: You may alter Blogspot’s default robots.txt like above.’
Robots.txt Type 2
User-agent: Media partners-Google
User-agent: Media partners-Google
Disallow: User-agent: * Disallow: /search Disallow: /b Allow: / Sitemap: https://www.techprevue.blogspot.com/feeds/posts/default?orderby=updated
Note: Don’t forget to change the https://www.techprevue.blogspot.com with your blog address or a custom domain. If you want search engine bots to crawl the most recent 500 posts, then you should need to use following robots.txt type 2. If already you have more than 500 posts on your blog, then you can add one more sitemap line highlighted in red. Robots.txt
Robots.txt Type 3
User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Disallow: /b Allow: / Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500 Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=501&max-results=500
Note: Don’t forget to change the https://www.techprevue.blogspot.com with your blog address or a custom domain.
Mathematical expression for Blogger robots.txt sitemap entries:
Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=(m*0)+1&max-results=m Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=(m*1)+1&max-results=m Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=(m*2)+1&max-results=m Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=(m*3)+1&max-results=m
Sitemap: https://www.techprevue.blogspot.com/atom.xml?redirect=false&start-index=(m*n)+1&max-results=m
Where m=500 and n=1, 2, 3, 4,…, n. If you have organized post labels in a good format and good experience of search engine optimization (SEO), then you can remove the following line –
Disallow: /search
Most important: if you don’t want to submit any blogger post or page to search engines then you can add them like that: For Post add a a line like that –
Disallow: /yyyy/mm/post-name.htmlFor Page add a line like that –
Disallow: /p/page-name.html
Manage Blogger custom robots.txt
For this, please follow these steps carefully. Dashboard ›› Blog’s Settings ›› Search Preferences ›› Crawlers and indexing ›› Custom robots.txt ›› Edit ›› Yes To get a better understanding of this you can take reference of the image given below:
I hope you’ll get the benefit of this post and get better search engine presence and ranking.
