Web crawler with very flexible crawling options. Can either use standalone or can be used with resque to perform clustered crawls.
By chatting or signing in you agree to the Terms and chat-message logging (revocable in History).