<html>
<head>
<title>
Ben Wallad - HW Assignment 1
</title>
</head>
<body>
Benjamin Wallad
<br> Web Bots: The Good and the Bad </br>
<br> <a href="#p1"> What is a Web bot? </a> </br>
<br> <a href="#p2"> Where do you find Web bots? </a> </br>
<br> <a href="#p3"> How do you build a Web bot? </a> </br>
<br> <a href="#p4"> Using Web bots for good </a> </br>
<br> <a href="#p5"> Using Web bots for evil </a> </br>
<br> <a href="#ref"> References </a> </br>
<p id="p1">
<br>
A web bot, or web robot is a program that travels throughout the HTML code of the Web gathering extremely large amounts of data on websites.
Once a web bot reaches a page, it has the ability to copy the code on this page and add all possible links on this page to its stack of
further sites to visit. Since these bots are non-human and are created using complex algorithms, they can travel the web at incredible speeds,
documenting data much faster then any human could possibly accomplish. Today, web bots are used on the Internet for both good and malicious
means. Since these bots are capable of retrieving countless information they will continue to be used to create indexes the Web.
</br>
</p>
<p id="p2">
Web bots are found all over the internet. Search engines use powerful web bots to travel throughout the ever-evolving Web to find new
websites that are not recorded in the search engine’s database. Since new sites are being established on the web everyday, these bots are
needed to find early websites who have not yet been included in the engine’s possible search results. Web bots are also used by bold
engineers who are attempting to archive the entire Internet. This is an extremely hard project, due to the costs of maintaining
these powerful bots and controlling the seemingly infinite amount of data of the Internet.
</p>
<p id="p3">
When building a web bot, a program must be created that can communicate with HTTP and search throughout HTML code to find URL links to
additional network objects. These crawler programs can be started almost anywhere on the Web, and will eventually find almost all pages in
existence until otherwise limited. However, in order for this web crawler to be useful, it must store everything it locates on its journey
throughout the Internet. This data should be archived in a database to be further analyzed and deciphered.
<p>
Other parameters must be followed when developing a bot. The proposed Standard for Robot Exclusion or SRE must be strictly obeyed.
This standard allows Web-site owners to control robot behavior on their sites through various mechanisms. Also, the crawler should be
polite and not overburden a Web site’s bandwidth when passing through it.
</p>
</p>
<p id="p4">
As previously mentioned, web bots can be used in many different ways for good. Brewster Kahle is using web bots for the painstaking process
of archiving the entire Internet. He is using web bots as a tool for a greater purpose on the Internet. Powerful search engines such as
Meta Search, which looks through many search engines to yield users their results, use web bots to increase their available results. Since
web bots gather html data at extreme speeds, they are perfect for gathering more and more web content. Without web bots, many web sites
would be left in the back doors of the Internet, having never been linked to appear on users’ search queries.
</p>
<p id="p5">
However, in the hands of malicious hackers, web bots can become a tool to cause havoc on the Internet. Data-mining bots can secretly gather
private information of web users, and in the wrong hands this information can be put to bad use. E-mail addresses are easily picked up for
repeated spamming by advertisers who pay for lists of addresses gathered through bots. Other web bots are sent throughout the web with
certain parameters to only gather data on specific websites. There have been many instances of stolen website design and architecture, for
many web designers send out these web bots to gather information on their competition. Also, through the right programming, bots can be
created to overload networks and servers to cause them to go down temporarily. With a serves down, hackers can eliminate internet sites
temporarily if they see fit.
</p>
<p id ="ref">
<br>
<a href="http://www.webtechniques.com/archives/1997/05/burner/"> Crawling towards Eternity: Building An Archive of The World Wide Web </a>
</br>
<br>
<a href="http://www.hybrid6.com/webgeek/2007/01/cyber-surveillance-and-internet-data.php"> Cyber-Surveillance and Internet Data-Mining </a>
</br>
<br>
<a href="http://www.robotstxt.org/wc/faq.html#what"> The Web Robots FAQ... </a>
</br>
<br>
<a href="http://firstsearch.oclc.org.online.library.marist.edu/WebZ/FSFETCH?fetchtype=fullrecord:sessionid=fsapp11-33002-f6rz5dzi-xhl9qg:entitypagenum=4:0:recno=7:resultset=2:format=FI:next=html/record.html:bad=error/badfetch.html:entitytoprecno=7:entitycurrecno=7:numrecs=1">
Baffling the Bots </a>
<br>
</p>
<br>
<a href="index.html"> Back to Home </a>
</br>
</body>
</html>