Andre
12-03-2004, 08:46 AM
Yahoo Spider Slurps!
As the world's second most popular search
tool, Yahoo moves a tremendous amount
of traffïc and is a very credible alternative
to Google. Yahoo receives over 2.76 billion
page views per day from hundreds of millïons
of unique users. It boasts over 157 million
registered users enjoying mail, shopping and
discussion groups and increasingly personalized
search and news services. For the past two
years, Yahoo, Google and MSN have been
embroiled in a hard-fought battle for the
loyalty of search engine users forcing all three
firms into the hyper-evolution we are witnessing
today. Over the next three Wednesdays we
are going to examine how the Big-3 spiders
work, what they look for and how to best
prepare your sites for multiple visits from the
bots that rank them. Today, we are starting
with Yahoo's bot, SLURP.
Getting Found By Slurp
The first thing to know about Slurp is that
like its better known cousin, Google-bot,
Slurp "discovers" sites by following links from
one site to another, reading and recording
nearly everything it finds in its path. The
majority of websites referenced by Yahoo
were originally included in its database because
they were accessed by Slurp following links
from another site.
Yahoo suggests adding an inbound link to all
pages in your site to guarantee those pages
will be discovered by Slurp. They also recommend
an internal sitemap linked to from the Index
(or home) page of the site. To encourage
Slurp to spend more time deep-crawling your
content, Yahoo recommends the addition of
"good authoritative links pointing into your site",
from highly reputable sources such as news
sites, established business partners and other
sites relevant to your business or service.
Manual submission of the site is only recommended
if for some reason or another Slurp does not
find the site on its own. This is increasingly
rare however as server-logs show Slurp is one
of the most active spiders out there. In other
words, if a site Slurp has already indexed links
to your site, Slurp will almost certainly be visiting
very soon. Webmasters should not have to pay
submission fees to get into Yahoo's index since
according to Yahoo's Tim Mayer, 99% of Yahoo's
index is crawled by Slurp for free.
It is still important to make sure your site is
ready to receive a visit from Slurp. To ensure
Slurp is able to travel across your entire site,
provide standard HREF text links as opposed
to forms, Flash or java script navigation tools.
Webmasters are encouraged to avoid tracking
and communication methods that rely on
using cookies across every page of the site.
If you have a database driven site or a site
that creates unique sessions for each user,
avoid embedding session IDs in URL's. Lastly,
use 404 pages to redirect users (and spiders)
to the root (index) page if a page or site URL
becomes invalid. Yahoo also asks webmasters
of sites with shopping carts to use robot.txt
exclusions in the source of the shopping carts.
Where Your Site Has Been Included. Results May Vary...
Yahoo has seen enormous change over the
past seven years. What started as a paid-inclusion,
human edited search directory, has grown into
the second largest database of indexed content.
Yahoo is on the cutting edge of integrating
several forms of media into their search offerings
and will likely soon produce its own entertainment
content like an online HBO. Yahoo is flirting with
the concept of becoming an infotainment portal
again but the core of its offerings remains firmly
rooted in search.
Yahoo search results come in multiple formats
including: Yahoo-Local, Yahoo-products, MY-Yahoo
(personalized results), specific nation-based Yahoo's,
and the standard Yahoo.com One of Yahoo's goals
appears to be presenting individual search-users
with results that best match their personal needs.
For instance, Yahoo would like to present constantly
updated geographic-specific references when a
user searches for daily-use items such as groceries,
repair-workers, real estate and other services one
would normally use a telephone directory to find.
Similarly, Yahoo wants to present the entire global
database of references when a user searches for
international news, trans-national products or
vacation plans. Being certain your website gets served
up for all levels of search, local, regional and global,
will be important if you wish to serve a market larger
than your general region or community.
Getting Rankings
Yahoo's search engine ranks sites based on a
formula that is very similar to the algorithms used
by rivals Google and MSN. Yahoo values many of
the same elements other search engines do including
keyword enriched domain names, titles, meta tags,
and content. Yahoo also values keywords found in
the anchor text of internal links, though the effect
at Yahoo is not as powerful as it is on Google.
According to Yahoo, well optimized pages and sites
will continue to get good results across all versions
of their search engine. By opening your site to Yahoo
Slurp and performing well-planned optimization
services across every page, a good SEO can nearly
always achieve Top placements on Yahoo. The trick
is in offering Slurp the information it needs to read,
record and rank your site. If that information is
included on each page, a set of text-based links is
woven through the site to provide easy passage
for Slurp, and Yahoo is told what your business is,
where your business is located, and who your
business serves, your site should achieve strong
rankings.
Due to the advent of personal, local, regional and
global search results, it is highly recommended to
add full contact and address information on every
page of a site. This information should be as precise
as possible and should include street address, unit
or suite number, zip or postal code, state or province,
county and country, full telephone information
(including area codes), and if possible, the approximate
longitude and latitude of your business location.
When writing for Slurp here are a few basic fundamentals:
Have a descriptive URL.
Use keyword enriched titles on each page of the site.
Place keyword enriched description and keywords meta tags on each page of the site.
Use robot.txt files to keep Slurp out of your shopping cart or log in pages.
Place keyword enriched text in the first paragraphs of your site-copy.
Use HREF links to direct Slurp through each page of the site.
Add a sitemap page and be certain there is a link from the index page to the sitemap.
Be certain that geographic specific information is mentioned on each page of the site. Always have a contact page that also lists geographic specific information.
Write a press release and send it to as many blogs, news-wires and press release sites as possible.
Acquire strong, relevant incoming links from sites with topics similar to yours.
Update your site frequently.
Enjoy and value your placements
As the world's second most popular search
tool, Yahoo moves a tremendous amount
of traffïc and is a very credible alternative
to Google. Yahoo receives over 2.76 billion
page views per day from hundreds of millïons
of unique users. It boasts over 157 million
registered users enjoying mail, shopping and
discussion groups and increasingly personalized
search and news services. For the past two
years, Yahoo, Google and MSN have been
embroiled in a hard-fought battle for the
loyalty of search engine users forcing all three
firms into the hyper-evolution we are witnessing
today. Over the next three Wednesdays we
are going to examine how the Big-3 spiders
work, what they look for and how to best
prepare your sites for multiple visits from the
bots that rank them. Today, we are starting
with Yahoo's bot, SLURP.
Getting Found By Slurp
The first thing to know about Slurp is that
like its better known cousin, Google-bot,
Slurp "discovers" sites by following links from
one site to another, reading and recording
nearly everything it finds in its path. The
majority of websites referenced by Yahoo
were originally included in its database because
they were accessed by Slurp following links
from another site.
Yahoo suggests adding an inbound link to all
pages in your site to guarantee those pages
will be discovered by Slurp. They also recommend
an internal sitemap linked to from the Index
(or home) page of the site. To encourage
Slurp to spend more time deep-crawling your
content, Yahoo recommends the addition of
"good authoritative links pointing into your site",
from highly reputable sources such as news
sites, established business partners and other
sites relevant to your business or service.
Manual submission of the site is only recommended
if for some reason or another Slurp does not
find the site on its own. This is increasingly
rare however as server-logs show Slurp is one
of the most active spiders out there. In other
words, if a site Slurp has already indexed links
to your site, Slurp will almost certainly be visiting
very soon. Webmasters should not have to pay
submission fees to get into Yahoo's index since
according to Yahoo's Tim Mayer, 99% of Yahoo's
index is crawled by Slurp for free.
It is still important to make sure your site is
ready to receive a visit from Slurp. To ensure
Slurp is able to travel across your entire site,
provide standard HREF text links as opposed
to forms, Flash or java script navigation tools.
Webmasters are encouraged to avoid tracking
and communication methods that rely on
using cookies across every page of the site.
If you have a database driven site or a site
that creates unique sessions for each user,
avoid embedding session IDs in URL's. Lastly,
use 404 pages to redirect users (and spiders)
to the root (index) page if a page or site URL
becomes invalid. Yahoo also asks webmasters
of sites with shopping carts to use robot.txt
exclusions in the source of the shopping carts.
Where Your Site Has Been Included. Results May Vary...
Yahoo has seen enormous change over the
past seven years. What started as a paid-inclusion,
human edited search directory, has grown into
the second largest database of indexed content.
Yahoo is on the cutting edge of integrating
several forms of media into their search offerings
and will likely soon produce its own entertainment
content like an online HBO. Yahoo is flirting with
the concept of becoming an infotainment portal
again but the core of its offerings remains firmly
rooted in search.
Yahoo search results come in multiple formats
including: Yahoo-Local, Yahoo-products, MY-Yahoo
(personalized results), specific nation-based Yahoo's,
and the standard Yahoo.com One of Yahoo's goals
appears to be presenting individual search-users
with results that best match their personal needs.
For instance, Yahoo would like to present constantly
updated geographic-specific references when a
user searches for daily-use items such as groceries,
repair-workers, real estate and other services one
would normally use a telephone directory to find.
Similarly, Yahoo wants to present the entire global
database of references when a user searches for
international news, trans-national products or
vacation plans. Being certain your website gets served
up for all levels of search, local, regional and global,
will be important if you wish to serve a market larger
than your general region or community.
Getting Rankings
Yahoo's search engine ranks sites based on a
formula that is very similar to the algorithms used
by rivals Google and MSN. Yahoo values many of
the same elements other search engines do including
keyword enriched domain names, titles, meta tags,
and content. Yahoo also values keywords found in
the anchor text of internal links, though the effect
at Yahoo is not as powerful as it is on Google.
According to Yahoo, well optimized pages and sites
will continue to get good results across all versions
of their search engine. By opening your site to Yahoo
Slurp and performing well-planned optimization
services across every page, a good SEO can nearly
always achieve Top placements on Yahoo. The trick
is in offering Slurp the information it needs to read,
record and rank your site. If that information is
included on each page, a set of text-based links is
woven through the site to provide easy passage
for Slurp, and Yahoo is told what your business is,
where your business is located, and who your
business serves, your site should achieve strong
rankings.
Due to the advent of personal, local, regional and
global search results, it is highly recommended to
add full contact and address information on every
page of a site. This information should be as precise
as possible and should include street address, unit
or suite number, zip or postal code, state or province,
county and country, full telephone information
(including area codes), and if possible, the approximate
longitude and latitude of your business location.
When writing for Slurp here are a few basic fundamentals:
Have a descriptive URL.
Use keyword enriched titles on each page of the site.
Place keyword enriched description and keywords meta tags on each page of the site.
Use robot.txt files to keep Slurp out of your shopping cart or log in pages.
Place keyword enriched text in the first paragraphs of your site-copy.
Use HREF links to direct Slurp through each page of the site.
Add a sitemap page and be certain there is a link from the index page to the sitemap.
Be certain that geographic specific information is mentioned on each page of the site. Always have a contact page that also lists geographic specific information.
Write a press release and send it to as many blogs, news-wires and press release sites as possible.
Acquire strong, relevant incoming links from sites with topics similar to yours.
Update your site frequently.
Enjoy and value your placements