Robots.txt to Disallow Subdomains

2552
Bad bots

Do you want to disallow all of your subdomains or one of them? If yes, just follow the following steps one by one.

To keep in mind

All the search engines like Google, Bing and Duckduckgo treat subdomains as an individual entity. Writing Robots.txt for root domain or any subdomain won’t solve your problem. You will have to write separate Robots.txt files to disallow a domain, subdomain, a directory, or wildcards from your site.

Follow the steps below

theproche.com (Don't want to block)
dev.theproche.com (Want to block)
staging.theproche.com (Want to block)
blog.theproche.com (Want to block comments and admin section)

theproche.com (Don’t want to block)

If you want your root domain to allow for all then do nothing with your robots.txt file. If you don’t upload Robots.txt, SEs treat it allowed by default. Or, you can upload the following file.

User-agent: *
Disallow: 

dev.theproche.com (Want to block)

By placing the below Robots.txt file you will block your subdomain from crawling. If you want to be double sure use Robots.txt and No-index meta as described below.

User-agent: *
Disallow: /
<meta name="robots" content="noindex">
<meta name="googlebot" content="noindex">

blog.theproche.com (Only want to block comments and admin section)

If you run a website with an admin section where some sensitive information exists. We recommend you to block all those directories of your website which you don’t want to get crawled and scrapped by web bots.

Bloggers are often worried about spam comments on their website which can degrade their website reputation. To do so, we recommend you to block that directory also. Use the following code to do so. This file blocks the mentioned directories for all the bots not whole website.

User-agent: *
Disallow: /wp-admin/
Disallow: /comments/

If you have custom coded your website, go ask your developer about your website structure.

Read this article: Robots.txt File to Disallow the Whole Website