Top > Computers > Internet > Searching > Search Engines > Robots

  See also:

• Computers: Internet: E-mail: Spam: Preventing  (73)
• Computers: Internet: Searching: Search Engines: Sitemaps  (8)
• Computers: Internet: Web Design and Development: Authoring: Online Tools: robots.txt  (4)
• Computers: Software: Internet: Site Management: Log Analysis  (219)

» ACAP - Automated Content Access Protocol - Standard being developed on behalf of content publishers to communicate permissions information more extensively than is the case with robots.txt. Project documents, implementation and background information.

» All About Search Indexing Robots and Spiders - Search Tools Consulting explains how the search engine programs called "robots" or "spiders" work, and reviews related sites.

» Bots vs Browsers - This large database lists user agents in categories and distinguishes between robots and browsers.

» HTTP User Agent Index - An alphabetical list of user agents and the deployer behind them, compiled by Christoph Rüegg.

» List of Robot Agent Strings - A list from PGTS of Web robots with the identifying data they leave in Web site logs.

» Robot IP Address - Brian Dunnintg provides a list of all the major search engine robot IP addresses, by full class C only.

» Robotstxt.org - Information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots.

» Search Engine IP Addresses - Lists IP addresses of search engine spiders. Can be searched by IP address. Also links to resources on spiders.

» Search Engine Robots and Other User Agents - John A. Fotheringham presents data in tabular form on the robots sent by search engines and other sites to read and index Web pages: their origins, names and IP addresses.

» User Agent String - Tool from ASAP Consulting s.r.o. for detailed user agent string analysis using an online form. Includes databases of browsers and robots.

» User-Agents.org - Large list of search engine spiders, similar web robots, and Web browsers: their web-log identification and links to their originators.

Help build the largest human-edited directory on the web. Submit a Site - Open Directory Project - Become an Editor

The content of this directory is based on the Open Directory and may have been modified by clixShare