Yahoo Search’s Slurp Indexing Sites

Yahoo just got a step closer to adding Inktomi search engine results to its Yahoo Search Engine- which will be seen as a bit of a blow to Google, who now serves Yahoo search results. Yahoo has just announced a new site indexing robot to crawl the web with - Yahoo! Slurp.

Yahoo’s new robot keeps a similar name to the Inktomi Slurp crawler and some features listed on Yahoo include:

• Yahoo! Slurp has the ability to crawl dynamic links or dynamically generated documents.
• The Yahoo! Slurp crawler collects documents from the web to build a searchable index for search services using the Yahoo! search engine (this helps verify a soon addition of Inktomi to the Yahoo search results). These documents are crawled since other documents on the web contain links to these documents.
• Yahoo! Slurp crawls from your site in the Yahoo! search engines immediately. The documents will be indexed and included into the search database in the near future.
• Yahoo! Slurp will offer cache indexing (similar to Google) and obeys the noarchive meta-tag. If you place: META NAME=”robots” CONTENT=”noarchive” in the head of your web document, Yahoo! Slurp will retrieve the document, but it will not cache or archive the document for use in the PageCache system.
• Yahoo! Slurp also obeys the noindex meta-tag. If you place: META NAME=”robots” CONTENT=”noindex” in the head of your website, Yahoo! Slurp will retrieve the document, but it will not index the document or place it in the search engine’s database.

Last year, Yahoo announced that they plan to make the change over to Inktomi results in the first quarter of 2004, which gives them about 40 days to meet that goal. In addition, its nice to see that the Slurp robot and search functions are all branded Yahoo.

 back to top