-
-
Notifications
You must be signed in to change notification settings - Fork 6.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FTPS upload of large file (800 GB) using TLS 1.3 gets slower and slower after ~4.5h and 360 GB #13097
Comments
Do you have a server we can test this against? |
Unlikely. Such a connection would be closed. But also, why would it not handle rekeying? If that is even used here. Also: is there anything that argues against this instead being an issue in the server end? |
Unfortunately I don't have a public server
I have to admit, that "ftp -s" working was proof to me that the server side works. Just started a new session and captured an iptrace to find that "ftp -s" uses TLS v1.2, which also works for curl... |
Yes, this is internal only. And yes, since we manage the infrastructure as well and the same file between the same systems works with TLS v1.2, I'm certain that no switch, router or firewall causes the problems |
I run several tests from different systems targeting the same server
So in my perception, the general upload using FTPS and TLS 1.3 works fine. Therefore I would rule out a server side issue. |
Could you run the test the other way round - to any of the other systems? |
Unfortunately I don't have the possibility to run it the other way around. But I tested again with AIX and curl 8.6 as client against another server running a more recent IBM i version (7.5 vs. 7.3). Although the upload speed overall was better due to other infrastructure between the systems, the transfer rate again dropped after 362 GB:
So this seems to be related to the amount of data rather than the time. |
Obviously there is no code or condition anywhere in curl that does anything different after some specific amount of bytes transferred or time spent. My wild guess is that something is done on the TLS layer after some specific time and that triggers a different code path or something in OpenSSL that makes it run slower. It would be interesting to know if curl built with another TLS library or even a current OpenSSL version would behave differently. |
Could this be due to TLS session key renegotiation after some byte or time limit? |
Just some ideas here on how to learn more what is going on:
Hope this helps. |
Also interesting: what if you instead download such a big file between these two systems. Does that also turn slow? |
I did this
Uploaded a file of >800 GB to a server
The upload started as expected but after ~4.5h and ~360 GB the transferrate drops from ~23 MB/s to 500 KB/s and keeps getting slower (~200 KB/s after 5h).
Limiting the TLS version to 1.2, the upload completes in ~10 h and constant 23 MB/s
# curl --tlsv1.2 --tls-max 1.2 -T testfile --netrc --insecure --ssl-reqd ftp://1.2.3.4//targetdir/testfile
Using plain ftps, the upload also works fine in ~10 h and constant 23 MB/s
ftp -s 1.2.3.4
The server runs on IBM i 7.3.0 410 . According to the IBM i support, this issue may be caused by the Curl FTP client not responding to TLSv1.3 rekey requests.
Looking through the known issues, it may also be related to https://curl.se/docs/knownbugs.html#FTPS_upload_data_loss_with_TLS_1
I can't say what happens if the slow upload completes since it would take ages to let it run till completion.
I expected the following
The upload should complete without loss in speed using TLS 1.3
curl/libcurl version
curl -V
curl 8.5.0 (powerpc-ibm-aix7.1.5.0) libcurl/8.5.0 OpenSSL/1.1.1v zlib/1.2.13 libssh2/1.10.0 nghttp2/1.58.0 OpenLDAP/2.5.16
Release-Date: 2023-12-06
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz NTLM SPNEGO SSL threadsafe UnixSockets
operating system
AIX 7200-05-07-2346
The text was updated successfully, but these errors were encountered: