Multithread Web Crawler

• Write a class to handle multithreading website crawling inside the given domain.

• Feature a breath-first search algorithm and a multithread pool to visit all urls asynchronously.

• Handle various status code, time-out and exceptions in a structured manner.

Runhan Yu
Data Engineer

My interests include big data, cloud computing, software development and biochemistry.