Skip to content

Releases: lorien/grab

v0.6.40

14 May 08:49
Compare
Choose a tag to compare

Fixed

+- Fix #346: spider does not process initial_urls
+- Fix #344: raise GrabInvalidUrl for pycurl error 3

v0.6.39

10 May 09:06
Compare
Choose a tag to compare

Fixed

  • Fix bug: task generator works incorrectly
  • Fix bug: pypi package misses http api html file
  • Fix bug: dictionary changed size during iteration in stat logging
  • Fix bug: multiple errors in urllib3 transport and threaded network service
  • Fix short names of errors in stat logging
  • Improve error handling in urrllib3 transport
  • Fix #299: multi-added errors
  • Fix bug: pypi package misses http api html file
  • Fix #285: pyquery extension parses html incorrectly
  • Fix #267: normalize handling of too many redirect error
  • Fix #268: fix processing of utf cookies
  • Fix #241: form_fields() fails on some HTML forms
  • Fix normalize_unicode issue in debug post method
  • Fix #323: urllib3 transport fails with UnicodeError on some invalid URLs
  • Fix #31: support for multivalue form inputs
  • Fix #328, fix #67: remove hard link between document and grab
  • Fix #284: option headers affects content of common_headers
  • Fix #293: processing non-latin chars in Location header
  • Fix #324: refactor response header processing

Changed

  • Refactor Spider into set of async. services
  • Add certifi dependency into grab[full] setup target
  • Fix #315: use psycopg2-binary package for postgres cache
  • Related to #206: do not use connection_reuse=False for proxy connections in spider

Removed

  • Remove cache timeout option
  • Remove structured extension

v0.6.38

10 May 09:07
Compare
Choose a tag to compare

Fixed

  • Fix "error:None" in spider rps logging
  • Fix race condition bug in task generator

Added

  • Add original_exc attribute to GrabNetworkError (and subclasses) that points to original exception

Changed

  • Remove IOError from the ancestors of GrabNetworkError
  • Add default values to --spider-transport and --grab-transport options of crawl script

v0.6.37

10 May 09:08
Compare
Choose a tag to compare

Added

  • Add --spider-transport and --grab-transport options to crawl script
  • Add SOCKS5 proxy support in urllib3 transport

Fixed

  • Fix #237: urllib3 transport fails without pycurl installed
  • Fix bug: incorrect spider request logging when cache is enabled
  • Fix bug: crawl script fails while trying to process a lock key
  • Fix bug: urllib3 transport fails while trying to throw GrabConnectionError exception
  • Fix bug: Spider add_task method fails while trying to log invalid URL error

Removed

  • Remove obsoleted hammer_mode and hammer_timeout config options

v0.6.36

10 May 09:09
Compare
Choose a tag to compare

Added

  • Add pylint to default test set

Fixed

  • Fix #229: using deprecated response object inside Grab

Removed

  • Remove spider project template and start_project script