We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
There are common practices for broad crawls, explained here: https://docs.scrapy.org/en/latest/topics/broad-crawls.html. It involves modifying many settings. It seems we can provide a Scrapy addon to simplify all of that.
It can start as a third-party package, but I think Scrapy itself may also benefit from such addon.
The text was updated successfully, but these errors were encountered:
Are all of those settings safe to enable for all broad crawls?
Sorry, something went wrong.
I'm not 100% sure. It seems we should review them. Also, we should ensure that the users can change the values we set in the addon.
No branches or pull requests
There are common practices for broad crawls, explained here: https://docs.scrapy.org/en/latest/topics/broad-crawls.html. It involves modifying many settings. It seems we can provide a Scrapy addon to simplify all of that.
It can start as a third-party package, but I think Scrapy itself may also benefit from such addon.
The text was updated successfully, but these errors were encountered: