concurrent_tree_crawler.multithreaded_crawler.MultithreadedCrawler

It is also responsible for all the ancillary stuff: makes sure that the state of the tree is saved to disk, sets up the logging level etc.

Parameters:

navigators (list of AbstractTreeNavigators) - list of navigators to be used by the crawler. Each navigator will be run in a separate thread, thus the number of the threads is equal to the number of navigators.
sentinel (AbstractNode) - a technical node which will be made parent of the root node.
activity_schedule (AbstractActivitySchedule) - if None, no schedule is used and the program works until it finishes crawling.
log_file_path - path to the log file. If None, no log file will be used.
state_file_path - path to the file where the state of the program will be saved. If None, the state will not be saved.
save_period - time between saving the tree state. If state_file_path is None, this value is ignored.
logging_level - one of the logging level constants from logging

Sleep (stop program execution) until there's a time to wake up.

Returns: number of seconds: activity time, i.e. time until the start of the next sleep period, None if such time point cannot be determined (as in case when the activity time will not stop in future).

Class MultithreadedCrawler