Add option to pass configuration to simplecrawler directly #29
Description
There are a lot of options in simplecrawler. Might be useful to allow passing options directly to the crawler for really custom setups. Like limiting crawl depth or number of pages to crawl, etc.