Microsoft Bing and Yandex new search protocol to fast crawl

Microsoft Bing and Russian search engine Yandex on Monday announced a new search protocol designed to fast crawl for updates of websites.

Called IndexNow, the protocol uses an API to allow websites to easily notify search engines whenever content is created, updated, or deleted. Once the search engines are notified of updates they can quickly crawl and reflect the website changes in their index and search results.

“Ensuring timely information is available for searchers is critical,” Microsoft explained in its Bing Blog.

“Yet historically,” it continued, “one of the biggest pain points for website owners has been to have search engines quickly discover and consider their latest website changes. It can take days or even weeks for new URLs to be discovered and indexed in search engines, resulting in loss of potential traffic, customers and even sales.”

Microsoft maintained that IndexNow is an initiative for both a more efficient and open internet.

It explained that by telling search engines whether an URL has been changed, website owners provide a clear signal helping search engines to prioritize their crawl of those URLs. That limits the need for exploratory crawls to test if the content has changed.

Additionally, search is opened up because by notifying one search engine, a website notifies all search engines that have adopted IndexNow.

Lack of Standards

“IndexNow is a good idea because it simplifies the process of getting new content indexed,” observed Greg Sterling, vice president of market insights at Uberall, maker of location marketing solutions based in Berlin.

“It also ensures that new content will be indexed quickly — or immediately”.

Currently there’s no standard for updating search engines, explained Jon Alexander, vice president of product management at Akamai Technologies, a content delivery network service provider in Cambridge, Mass.