Skip to content

Raw Data API is a set of high-performant APIs for transforming and exporting OpenStreetMap (OSM) data in different GIS file formats

License

Notifications You must be signed in to change notification settings

tony-nyagah/raw-data-api

 
 

Repository files navigation

Raw Data API

Unit test Build

Documentation: hotosm.github.io/raw-data-api

Source Code: github.com/hotosm/raw-data-api

API Documentation : https://api-prod.raw-data.hotosm.org/v1/redoc

Raw Data API is a set of high-performant APIs for transforming and exporting OpenStreetMap (OSM) data in different GIS file formats.

Features

  • Fast: Built on top of FastAPI

  • OAUTH 2.0 Authentication with OpenStreetMap(OSM)

  • Multiple GIS formats support via GDAL's ogr2ogr - see table below for currently supported formats. Out of which , GeoJSON Follows Own Raw Data API conversion script

    Formats Status
    Esri Shapefile ✔️
    KML ✔️
    Mbtiles ✔️
    FlatGeoBuf ✔️
    CSV ✔️
    GeoPackage ✔️
    PGDUMP ✔️
    GeoJSON ✔️
    Pmtiles ✔️
    Geoparquet ✔️

Installation

Raw Data API consists of two elements:

  • A backend database, tools, and scripts: used to import OSM data into a specific database structure and keep it updated.
  • An API that is used to serve data from the backend database.

To setup the backend see Backend Installation

Raw Data API can be installed through docker or locally on your computer.

Configurations

Setup the necessary configurations for Raw Data API from configurations. Setup config.txt in project root or include your env variables in .env

Basic Usage

  • Confirm that Authetication works

    1. Hit the /auth/login/ endpoint
    2. Hit the url returned on the response
    3. You will get an access_token
    4. You can use the access_token in all endpoints that requires authentication.
    5. To check token pass token in /auth/me/. It should return your OpenStreetMap (OSM) profile
  • Try extracting some data:

    You can use the /raw-data/current-snapshot/ endpoint with the following input to check both authentication, database connection and download the export:

curl -d '{"geometry":{"type":"Polygon","coordinates":[[[83.96919250488281,28.194446860487773],[83.99751663208006,28.194446860487773],   [83.99751663208006,28.214869548073377],[83.96919250488281,28.214869548073377],[83.96919250488281,28.194446860487773]]]}}' -H 'Content-Type: application/json'   http://127.0.0.1:8000/v2/raw-data/current-snapshot/

Introduction of sozip

We use sozip optimization while we zip the files , SOZip makes it possible to access large compressed files directly from a .zip file without prior decompression. It is not a new file format, but a profile of the existing ZIP format, done in a fully backward compatible way. ZIP readers that are non-SOZip aware can read a SOZip-enabled file normally and ignore the extended features that support efficient seek capability. Learn more about it in attached link

Tests

  • Raw Data API uses pytest for tests, navigate to the root directory and install package in editable mode:
pip install -e .
  • At this point you should have PostgreSQL + PostGIS extension enabled on your computer, now run Pytest:
py.test -v -s
  • Running individual tests
py.test -k test function name

Contribution & Development

Learn about current priorities and work going through Roadmap & see here CONTRIBUTING

Roadmap

see Roadmap

License

see LICENSE

Version Control

For version control, We use commitizen, follow docs

Authors

Created by HOTOSM and Friends

About

Raw Data API is a set of high-performant APIs for transforming and exporting OpenStreetMap (OSM) data in different GIS file formats

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 88.6%
  • HCL 6.4%
  • Lua 2.9%
  • Gherkin 1.3%
  • Dockerfile 0.5%
  • Shell 0.2%
  • Jinja 0.1%