Data API: Need and Design

[rufuspollock]

Main page: https://datahub.io/notes/design-data-api

• What needs to people have
• What features do people want?
• What tool should we for this?

Job Stories

My rough use case is: i want to push a dataset and get an API i can share with others for them to use.

Bonus: i can meter it and charge for it!

Tools to use

• datasette run in a worker
• RoAPI
• SeaFowl

[rufuspollock]

RoAPI looks really cool

🔗 https://github.com/roapi/roapi

ROAPI automatically spins up read-only APIs for static datasets without requiring you to write a single line of code. It builds on top of Apache Arrow and Datafusion. The core of its design can be boiled down to the following.

Comments from @davidgasquez

This is probably the most straightforward way. Point RoAPI to your slowly changing dataset and have a bunch of API ready. Could be served via Cloud Run/Lambda to reduce costs.

[rufuspollock]

@davidgasquez do you have any ideas re e.g.

• how easy this is to deploy
• do you have to deploy per dataset (or data file)
• can you configure the API

[davidgasquez]

Deploying ROAPI shouldn't be hard (e.g: Docker image on Cloud Run).

Regarding deployment, I can think of two modes. Have 1 instance with all the desired datasets or 1 instance per dataset (similar to what Seafowl advocates for).

Configuration wise, what are you looking to tweak?

[rufuspollock]

@davidgasquez and how would that work out pricing wise? I think the charge is per request so it doesn't matter how many deployments you have (IIRC) ...

Would be worth doing a prototype of this i think!

[davidgasquez]

I think the charge is per request so it doesn't matter how many deployments you have (IIRC) ...

Depending where the deployment is made, having multiple ones might be harder to manage and more expensive. Would also have to pay the cold start times for each instance.

Would be worth doing a prototype of this i think!

https://github.com/datopian/datahub-next/issues/96

[rufuspollock]

SeaFowl

🔗 https://seafowl.io/

With Seafowl, datasets could be packaged and deployed on the edge (CF Workers, Fly.io, Serverless platform, ...). They've figured out things like caching and CDNs. They have a great end to end example showcasing some interesting features.

[davidgasquez]

Spotted Radiant Earth's new Source Cooperative initiative.

Source Cooperative is a neutral, non-profit data-sharing utility that allows trusted organizations to share data without purchasing a data portal SaaS subscription or managing infrastructure. Source allows organizations to share data using standard HTTP methods rather than requiring proprietary APIs or SaaS interfaces. It is currently in private beta.

Their demo is worth watching too!

I think this is an awesome approach to distributing datasets. Paired with things like WASM DuckDB, you get beautiful and useful websites all served from static assets (the browser does the HTTP range requests in real time!).

I've been applying the same ideas on a recent project and wanted to surface this approach (Parquet files on S3) as an easy step between no API/ CSV and REST/GraphQL API.

[rufuspollock]

We did experiments with range request on csv back in 2018 or so with datahub.io. I really like the idea.