This repository has been archived by the owner on Jan 2, 2023. It is now read-only.
Releases: iconara/bigshift
Releases · iconara/bigshift
v0.4.0
v0.3.2
v0.3.1
v0.3.0
This version adds two new command line arguments:
--[no-]compression
makes it possible to skip compression to work around BigQuery's 4 GiB compressed files limit--steps
makes it possible to run only some parts of a transfer, for example to resume a transfer when BigShift accidentally got killed during the transfer step, or to run the load step against multiple BigQuery datasets.
v0.2.0
This release includes many improvements and updates to the documentation, including:
- You can now set the maximum bad records to allow when loading the BigQuery table:
--max-bad-records
- Specifying the BigQuery table ID is now optional, it will default to the same name as the Redshift table
- The transfer uses Redshift's unload manifest to determine which files to transfer, instead of relying on globs
- The unloaded and transferred files will be removed from S3 and Cloud Storage after the BigQuery table has been loaded
- The data will be compressed to minimize transfer costs
- The S3 and Cloud Storage locations specified are now required to be empty or the transfer will fail
- Logging has been improved, for example the number of files and their sizes are logged before the transfer starts