Web crawler

(redirected from Search engine robot)
Related to Search engine robot: Search engine spiders

Web′ crawl`er


n.
Computers. spider (def. 6).
[1995–2000]
Translations
Webspinne
robot d’indexation
References in periodicals archive ?
The metatags are read by search engine robots when they visit sites.
Wait a few days for the search engine robots to crawl your pages and add your latest content to their full-text search databases.
Specifically, search engine robots which happen to be blind.
As a result, search engine robots are not placing the sites at the top of their lists, costing companies online hits.
tag=st.util.print (Google and Yahoo kicked WhenU.com out of their indexes for allegedly displaying different web pages to searchers and search engine robots, a process called "cloaking").
A THE robot.txt file specifies which search engine robots - they visit to log your site - to allow in.
* Use META tags (ask your programmer) to hide words on your site that will be found by search engine robots which in turn will index your site in search engines based on the META tags you've chosen.
The search engine robots will do their duty, find the site, and I am in business.
Since then I've looked on quite a lot of other search engines and my site has come up so I guess a lot of the search engine robots have found me themselves.
Make sure your site is built using something called 'style sheets' and don't use Flash or some other technology that search engine robots can't understand.