ir.webutils
Class ProximityBeamSearchSpider

java.lang.Object
  ir.webutils.Spider
      ir.webutils.BeamSearchSpider
          ir.webutils.ProximityBeamSearchSpider

public class ProximityBeamSearchSpider
extends BeamSearchSpider

A BeamSearchSpider that uses the ProximityLinkHeuristic instead of the default LinkHeuristic.

Field Summary

Fields inherited from class ir.webutils.BeamSearchSpider

beamSize, goal, goalPage, heuristic

Fields inherited from class ir.webutils.Spider

count, linksToVisit, maxCount, saveDir, slow, visited, webpr

Constructor Summary

ProximityBeamSearchSpider()

Method Summary

protected LinkHeuristic	constructLinkHeuristic() Return default LinkHeuristic.
static void	main(java.lang.String[] args) Search the web using beam search according to the following command options, but stay within the initial host site.

Methods inherited from class ir.webutils.BeamSearchSpider

doCrawl, getNewLinks, go, handleBCommandLineOption, handleHCommandLineOption, handleUCommandLineOption, handleWCommandLineOption, processArgs, scoreLinks

Methods inherited from class ir.webutils.Spider

handleCCommandLineOption, handleDCommandLineOption, handleSafeCommandLineOption, handleSlowCommandLineOption, indexPage, linkToHTMLPage

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ProximityBeamSearchSpider

public ProximityBeamSearchSpider()

Method Detail

constructLinkHeuristic

protected LinkHeuristic constructLinkHeuristic()

Description copied from class: BeamSearchSpider

Return default LinkHeuristic. Specializations can override this method to utilize alternate link heuristics.

Overrides:

constructLinkHeuristic in class BeamSearchSpider

main

public static void main(java.lang.String[] args)

Search the web using beam search according to the following command options, but stay within the initial host site.

• -safe : Check for and obey robots.txt and robots META tag directives.
• -c <maxCount> : Download at most <maxCount> pages.
• -u <url> : Start at <url>.
• -w <strings> : <strings> should be a list of "need strings" separated by ";"'s.
• -h <strings> : <strings> should be a list of "help strings" separated by ";"'s.
• -b <size> : Use a beam width of given <size>
• -slow : Pause briefly before getting a page. This can be useful when debugging.