What is a search engine
robot?
Yes, robots are everywhere. Search engine robots, spiders,
crawlers and bots that is. Actually search engine robots are
programs that a search engine will send out to index web pages.
The search engine robots 'spider' through web pages they find
to assist the crawler based engines index pages that are on
the internet. Each search engine has their own robot, sometimes
more than one and they all do things a bit differently. This
article does not focus on any particular robot or search engine
but rather gives you a general overview of search engine robots.
How do search engine robots
work?
In its essence the robot goes out and visits Site A - it reads
the metadata and the entire text on the site. And, I mean
text! Not images, not text rendered as an image - just plain
old text. The robot also reads the URL's of any outbound links
pointing to Site B and Site C. The search engine robot will
then spider Site B and Site C as well. As you can see the
method of having a good linking strategy will prove to be
very important.
The pages of Site A, B and C are then stored in the search
engines database and then indexed. So when you do a search
on Google, you aren't searching all the documents on the Internet.
You are only looking at the pages that Google has in its index
– by way of the search engine robot. You can experiment
by doing the same search in different engines - you'll get
varying results.
Ah, but you may ask yourself - how does the
search engine robot ever find out about my site? As stated
before, there are just too many pages out there. The search
engine robot will only crawl the pages it knows about. So
you need to let the search engine know your site is out there.
This is where link
popularity is crucial.
Search engine robots and
link popularity
As mentioned earlier you need to have a good
link strategy, plenty of quality inbound links and link reputation.
This is how to really get your site indexed: Let the search
engine robot follow links! This is very effective although
it can be a lot of work. You need to get good quality sites
with relevant content to put a link on their site pointing
back to yours (to check your link popularity try Marketleap's
Link
Popularity Tool).
8 Decmber 2006
|
