Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bulk download speed-up #19

Open
RRemelgado opened this issue Aug 26, 2020 · 0 comments
Open

Bulk download speed-up #19

RRemelgado opened this issue Aug 26, 2020 · 0 comments

Comments

@RRemelgado
Copy link

I was trying to download thousands of range maps using BIEN_ranges_species(), and it took over a day to reach 10,000 (without parallelization). Therefore, I had a look at your code.

When dealing with multiple species, you are currently iterating through each and returning a single ESRI shapefile. Instead, I would suggest using the output of .BIEN_sql() with st_as_sf() from the sf package. This require a column with WKT data, which is returned by .BIEN_sql(). Doing so translates the full query into a single sf object, greatly limiting the overhead time.

Using this approach, it took me half a day to download the full set of range maps (> 98,000).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant