-
Notifications
You must be signed in to change notification settings - Fork 10.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve typing for Spider.parse(). #6274
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6274 +/- ##
==========================================
+ Coverage 84.88% 88.79% +3.90%
==========================================
Files 161 161
Lines 11927 11867 -60
Branches 1810 1919 +109
==========================================
+ Hits 10124 10537 +413
+ Misses 1523 982 -541
- Partials 280 348 +68
|
|
||
from scrapy.crawler import Crawler | ||
from scrapy.settings import BaseSettings | ||
|
||
CallbackT = Callable[Concatenate[Response, ...], Any] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This could be useful in other places as it's a general type for a spider callback. I didn't think about that further though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually we should use it for Request.callback
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like we can't, probably because of wrong covariance when assigning a specific callback to a request, I need to investigate this separately.
Based on https://stackoverflow.com/questions/77160087/correct-typing-of-a-method-without-a-fixed-signature
Should remove most of the mypy errors for parse() in user spiders (if they use older mypy Spider.parse() will appear untyped so no override error either). Doesn't help with scrapy-poet's callbacks that narrow down
Response
toDummyResponse
: scrapinghub/scrapy-poet#179This should probably have typing tests, I'll do that later.