ckan performance

[AlbertVas]

Hi everyone,

We are planning to introduce ckan as the main application for research data management at our university, but we have some performance issues that need to be solved. We plan to offer the service to >10k users, some of whom will use it for file sizes >100GB, so transfer speed (or more accurately, processing speed) is critical.

We have deployed ckan 2.10 (plus auxiliary containers) on nodes with 8 cores, 8GB RAM and a Ceph storage backend in Docker form.

I tested the throughput of the ckan application by using the API with curl inside the ckan container with the default URL as endpoint and with sample files of 1-2 GB size, but I was not able to get beyond 100 MB/s and 230 MB/s write and read speed respectively. Temporarily I used a tmpfs, both as the source of the test files and as ckan storage, so the read/write speeds to the storage should not impose any limitations. Since I was inside the ckan container, there was no proxy involved. The system load does not indicate a lack of system resources.

What options are there to optimize performance?
What data throughput can be expected for the application with optimal equipment?
What data throughputs are typically achieved?

Any help will be appreciated!

Best,

Albert

[wardi]

Hello Albert,

Just to understand how you've set things up, are you using Ceph as a network filesystem and pointing the ckan storage directory to it?

If so that would mean all your bytes are being passed through nginx, uwsgi, python and the ceph network filesystem abstraction in each direction. @EricSoroos mentioned that getting a change like #7512 merged into ckan would skip the uwsgi and python code (likely the slowest parts). Another option is to use a ckan extension that offloads uploads and downloads like ckanext-cloudstorage so that clients can connect directly to the file storage service for uploads and downloads.

[AlbertVas]

Hi wardi,

thank you for your response! If i got it right, the usage of X-Accel-Redirect header can bypass the uwsgi/python layer when downloading a resource what should lead to an improvement in transfer speed.

Yes, your assumption is correct: CephFS is mounted at the host system and ckan's storage path is pointing to that mountpoint.

For testing purposes, i used a tmpfs and by doing so avoided passing the data through nginx. Still, it is passed through uwsgi and python and i wonder what transfer speed one can expect an that path.

Best regards,

Albert

[EricSoroos]

The underlying serving of the file is flask.send_file, which uses a variety of methods to send the file, depending on configuration. https://flask.palletsprojects.com/en/2.3.x/api/#flask.send_file . That calls down to werkzeug's sendfile, which uses the internal FileWrapper by default.

This is likely going to be limited by the single threading performance in python. I'd suspect that if you've got an appropriate number of uwisgi offload threads/processes setup, you could have multiple large downloads at that speed to the point of saturating processor/network. (by default anyway).

Using the nginx (or other server supporting X-Accel-Redirect) should completely offload that.

It's also possible that uwsgi could be configured to offload static files, I don't know if that would with just the offload-threads setting, or if we'd need to add the X-Sendfile header as well. https://uwsgi-docs.readthedocs.io/en/latest/StaticFiles.html#x-sendfile

[AlbertVas]

Hello Eric,

thanks for the comment and the hint that probably the Python/Flask performance is the limitation in the process.

Just to understand you correctly: By ckan using the X-Accel redirect directive to return not the file itself, but its path to the proxy, the download speed no longer depends on Python.
Would it be possible to use a similar approach for uploading files in ckan?

I understand that transfer speed measurements can hardly be compared across different setups as they depend on different factors, but I wonder if there are some estimates to get an idea of single-threaded performance in Python? Or maybe some tweaks to improve it?

Thanks and best regards

Albert

[EricSoroos]

I haven't really dug into the upload code in the same way as the download code -- I needed the X-Accel-redirect for the efficient range request so I've been through that code in detail. But yes, in that case, the python code is doing auth and then handing the file off to nginx to serve directly through the kernel sendfile() implementation.

I'm not clear on if there's a way to do this in the upload path, It would probably depend on the details in flask of how they do wrappers for upload. It's a harder problem at any rate, because there's no way to just "sendfile", because it's all coming in via the tcp socket and the file itself needs to be parsed.

I'd generally say that most people who are dealing with large uploads are probably not using filesystem storage, but likely going to an offload system like cloudstorage, where the actual file uploads are going directly to the cloud blobstore. I'm obviously not familiar with what you have to leverage, so I don't know if you have something that's s3 compatible on your infra, but that would likely be the most straight forward way to do it.

[EricSoroos]

Had another look here, and I can't get UWSGI to serve using X-sendfile. It's not as simple as just adding a couple of settings, since it looks like it involves a plugin based build, and I can't get that and python working at the same time (with ~1/2 hour of poking).

However, I've run a benchmark of X-accel-redirect on vs off. YMMV, as this is rough, and on my machine. (4 core VM on a 16/32 core AMD with a fast drive and lots of memory. 34MB file, served from cache, using ab (apache-bench), 10 seconds, localhost/loopback but with ssl)

ab - -t 10 -c 1 'https://labs.staging.derilinx.com/dataset/[id]/resource/[rid]/download/filename.pmtiles'

No X-Accel-Redirect:

1 concurrent request:

Requests per second:    14.13 [#/sec] (mean)
Time per request:       70.785 [ms] (mean)
Time per request:       70.785 [ms] (mean, across all concurrent requests)
Transfer rate:          476680.31 [Kbytes/sec] received
 

4 concurrent requests:

Requests per second:    35.34 [#/sec] (mean)
Time per request:       113.183 [ms] (mean)
Time per request:       28.296 [ms] (mean, across all concurrent requests)
Transfer rate:          1199554.99 [Kbytes/sec] received

X-Accel-Redirect:
1 concurrent request:

Requests per second:    14.17 [#/sec] (mean)
Time per request:       70.551 [ms] (mean)
Time per request:       70.551 [ms] (mean, across all concurrent requests)
Transfer rate:          478258.71 [Kbytes/sec] received

4 concurrent request:
Requests per second:    42.02 [#/sec] (mean)
Time per request:       95.198 [ms] (mean)
Time per request:       23.799 [ms] (mean, across all concurrent requests)
Transfer rate:          1420217.02 [Kbytes/sec] received

So, roughly, I'm seeing 3/4 the performance of X-accel-redirect serving through uwsgi, and a total of 1.4GB/sec when run through nginx. There's still idle, so there's still probably some headroom. But even the single threading is coming in at 2x what you were seeing.