Google Can Crawl Web Workers That Schedule Work Immediately

Martin Splitt from Google confirmed that Google’s crawler, GoogleBot, can indeed crawl and index web workers but only if they don’t delay the work. He said if you are using web workers and you want it indexed by Google, you should “schedule your work immediately.”

He said this on Twitter when he was asked if it was true that the new GoogleBot was unable to do so. He said as long as you don’t use the setTimeout, you should be fine.

Here are those tweets:

So this is an interesting one. It’s not straight forward: It turns out web workers are supported BUT rendering doesn’t seem to wait on deferred work. (setTimeout for instance). So as long as you schedule your work immediately, you’ll be fine. 1/2

— Martin Splitt @ ???????? #dafed #novisad (@g33konaut) May 30, 2019

He even showed test cases:

2/2 Test code:
Test code: https://t.co/l7cDqdseAR
MFT results: https://t.co/Y8QI25XWv3 (note that only the very first cat made it into the rendered HTML)

— Martin Splitt @ ???????? #dafed #novisad (@g33konaut) May 30, 2019

But Martin added you should be cautious:

Note that setTimeout is just one example of deferred work. So I advise a lot of caution when using web workers to generate content that you want to get indexed. ????

— Martin Splitt @ ???????? #dafed #novisad (@g33konaut) May 30, 2019

Forum discussion at Twitter.