 |
||||||||||
 |
||||||||||
SnarfbotSnarfbot is a beta web crawler still in the debugging stage.If you want your site blacklisted, enter the base url. You can also use the form for questions if you like. Some facts about Snarfbot Snarfbot only follows HREF links, so it should not put much load on your webserver. Snarfbot makes a point to avoid binary files, further reducing the load on your webserver. Snarfbot does a good job at not fetching the same document over and over again. Sometimes when Snarfbot is restarted it may inadvertly do this. This is not normal behavior and should rarely happen. Snarfbot is a fun little project that is being done for my own education in bot programming. It's use will be limited to short runs each day. In the near future it probably will not run at all. Snarfbot runs on a HP NetServer LXr Pro. This is a quad CPU unit with half a gig of ram and a raid array. |
|
|||||||||
|
Layout created by: SilverOracle.com |
||||||||||
