Skip to content
Grav 2.0 is officially stable. Read the announcement →

Community guidelines

Please keep discussions civil and on-topic. Repeated violations may lead to a temporary ban.

General

Robots.txt and SEO Footprint

first-time

Started by Benjam 7 years ago · 1 replies · 826 views
7 years ago

Hello, i'm wondering why many folders are blocked in robots.txt and its allows just 3 of them. Why just be simply and serve a normal robots.txt file like

User-agent: *
Disallow:
Sitemap: http://www.my-domaine.com/sitemap.xml

GoogleBot and others try to crawl this blocked folders ? Any exemples ? As a SEO, it's a big footprint for Google and mass PBN deployment.

Just wanna know if there is a good reason or if it's just for a prevention for indexation.

7 years ago

If I understand your question correctly,
Yes, this is mainly in folders where a bot doesn't need to get into and if it does, it has to be notified that it isn't ment to be in there, so that the chance of getting indexed is smaller.

The pages on your site that aren't in a folder where the robots.txt folder ignores stuff are mostly the only pages you want to get to viewers anyway 🙂

Hope it explains what you wanted, good luck!

Suggested topics

Topic Participants Replies Views Activity
General · by Jerry Hunt, 4 days ago
2 74 6 hours ago
General · by pamtbaau, 11 hours ago
1 47 11 hours ago
General · by Andy Miller, 23 hours ago
0 43 23 hours ago
General · by Marcel, 12 months ago
6 346 5 days ago
General · by Duc , 5 days ago
3 39 5 days ago