* fix: the crawling error when encountering PDF URLs * Added the unit test for the new feature of crawl tool * fix: address the code review problems * fix: address the code review problems
* test: add unit tests of crawler * test: polish the code of crawler unit tests