Google May Change How NoIndex In Robots.txt Works

For years Google has been communicating and we’ve been reporting that Google does not support using the noindex direction within your robots.txt file. Well, people still use it and now Gary Illyes from Google is on the case – he may end up making sure it completely doesn’t work.

In short, John was asked about it again and he gave the same answer he has been giving for years:

We don’t officially support it, so I wouldn’t rely on any particular effect.

???? John ???? (@JohnMu) April 17, 2019

So then Gary Illyes stepped in and said this may go away soon as he reviews the code behind it:

This may go away. I’m cleaning up that part of the googlebot codebase nowadays and that thing sticks out like a nun in a mosh pit

Gary “鯨理” Illyes (@methode) April 18, 2019

Why fully remove support for this? Well, (1) Google has been telling people not to use it and (2) Gary said “Technically, robots.txt is for crawling. The meta tags are for indexing. During indexing they’d be applied at the same stage so there’s no good reason to have both of them.”

Then people said but people use it, don’t drop it. So he said he will investigate and if that is true, he will try to make a pitch to keep it:

Alright. I’ll run a study to see how people are using it. If the majority are screwing themselves with it, then I’ll go ahead for approvals

Gary “鯨理” Illyes (@methode) April 18, 2019

He goes on:

We’re working on educating devs on (basic) SEO. There’s a disconnect between the two groups at most companies I’ve talked to and it’s hurting websites most of the time. That doesn’t mean that you should misuse something though. If you have a screw, use a screwdriver, not a hammer

Gary “鯨理” Illyes (@methode) April 18, 2019

I bet the unofficial support does go away at some point – so just don’t use it!

Forum discussion at Twitter.