Did I find the right examples for you? yes no      Crawl my project      Python Jobs

All Samples(3)  |  Call(3)  |  Derive(0)  |  Import(0)
Add a path to the fetch queue but makes sure it's not already there.
returns True if the path was not in the list, False if it's a duplicate

        def add_path_to_fetch_queue(url_obj):
    """
     Add a path to the fetch queue but makes sure it's not already there.
     returns True if the path was not in the list, False if it's a duplicate
    """
    url_string = _get_cached_url_string(url_obj)
    if not url_string in database.path_cache:
        database.fetch_queue.put(url_obj)
        database.path_cache.add(url_string)
        return True
    else:
        return False
        


src/t/a/tachyon-HEAD/tachyon.py   tachyon(Download)
    # Fill work queue with fetch list
    for path in database.paths:
        dbutils.add_path_to_fetch_queue(path)
 
    # Consider some file target as potential path
    for file in database.files:
        if not file.get('no_suffix'):
            file_as_path = file.copy()
            file_as_path['url'] = '/' + file_as_path['url']
            dbutils.add_path_to_fetch_queue(file_as_path)
                path = path.copy()
                path['url'] = validpath['url'] + path['url']
                dbutils.add_path_to_fetch_queue(path)
 
    textutils.output_info('Found ' + str(len(database.valid_paths)) + ' valid paths')