Did I find the right examples for you? yes no      Crawl my project      Python Jobs

All Samples(4)  |  Call(4)  |  Derive(0)  |  Import(0)

src/c/r/crawler-HEAD/Crawler.py   crawler(Download)
    def __init__(self, seed, referer = None):
        super(Crawler, self).__init__()
        self.addHeader("Host", re.match('.+//([^/]+)?/{0,1}.+$', seed).group(1))
        self.addHeader("Accept", "text/html")
        if referer: self.addHeader("Referer", referer)
    def update_referer(self, url, params = None):
        self.addHeader("Referer", url)
 
    def crawl(self, do, **kwargs):
        return len(map(do, self.generate(**kwargs)))