Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OSM-POI: processing step fails #146

Open
geoHeil opened this issue Aug 3, 2022 · 0 comments
Open

OSM-POI: processing step fails #146

geoHeil opened this issue Aug 3, 2022 · 0 comments

Comments

@geoHeil
Copy link

geoHeil commented Aug 3, 2022

To reproduce the problem:

git clone https://github.com/kuwala-io/kuwala.git
cd kuwala
cd pipelines

docker-compose run osm-poi
# download, EU, austria
# download, EU, lithuania
docker-compose run --rm osm-parquetizer java -jar target/osm-parquetizer-1.0.1-SNAPSHOT.jar --continent=eu --country=aut
docker-compose run --rm osm-parquetizer java -jar target/osm-parquetizer-1.0.1-SNAPSHOT.jar --continent=eu --country=ltu

manually running outside of docker (AT) due to memory issues in docker (even though more than enough memory is available to docker daemon)

docker-compose run osm-poi --continent=eu --country=ltu <<< stuck here
docker-compose run osm-poi --continent=eu --country=aut <<< fails with checksum error

# process, eu, aut
# process, eu, ltu

docker-compose run admin-boundaries --continent=eu --country=aut
docker-compose run admin-boundaries --continent=eu --country=ltu

docker-compose run google-trends --continent=europe --country=austria --keyword=Altlerchenfeld
docker-compose run google-trends --continent=europe --country=austria --keyword=Gänserndorf
docker-compose run google-trends --continent=europe --country=austria --keyword=Breitenfeld
docker-compose run google-trends --continent=europe --country=austria --keyword=Josefstadt
docker-compose run google-trends --continent=europe --country=austria --keyword="Innere Stadt"
docker-compose run google-trends --continent=europe --country=austria --keyword="St. Pölten"
docker-compose run google-trends --continent=europe --country=austria --keyword=Josefstadt
docker-compose run google-trends --continent=europe --country=austria --keyword=Aspern
docker-compose run google-trends --continent=europe --country=austria --keyword=Essling
docker-compose run google-trends --continent=europe --country=austria --keyword=HADERSDORF
docker-compose run google-trends --continent=europe --country=austria --keyword=Hütteldorf
docker-compose run google-trends --continent=europe --country=austria --keyword=Landstraße
docker-compose run google-trends --continent=europe --country=austria --keyword=Leopoldstadt
docker-compose run google-trends --continent=europe --country=austria --keyword=Aspern

Problems:

Downloading the PBF and generating the parquets works (partially): 1) there seem to be some memory issues in docker

  1. when trying to apply the process step this one fails totally for me.

  2. any subsequent steps i.e. google-poi or even ingestion of other steps of OSM are blocked due to (2)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant