backtrace.io not accepting dumps

[janisozaur]

Operating System

Windows

OpenRCT2 build

0.4.11

Base game

RollerCoaster Tycoon 2

Area(s) with this issue?

This is a development issue

Describe the issue

OpenRCT2 backtrace.io instance is not accepting user uploads of dumps, responding with error code 400:

---------------------------
OpenRCT2
---------------------------
There was a problem while uploading the dump. Please upload it manually to GitHub. It should be highlighted for you once you close this message.
It might be because you are using outdated build and we have disabled its access token. Make sure you are running recent version.
Dump file = C:\Users\janisozaur\Documents\OpenRCT2\crash\bf773064-6437-4d97-a9b7-79d27bc4324d(18d2b5e_x86-64).dmp
Please provide following information as well:
Error code = 400
Response = 
---------------------------
OK   
---------------------------

Steps to reproduce

1. Use Windows
2. Enable debugging tools
3. Use dereference in-game command
4. Select 'Yes' to upload the dump

Attachments

No response

[janisozaur]

As per usual, before v0.4.11 release I changed our backtrace.io token: 80ee864

I have verified on backtrace website it is correct and active:

[janisozaur]

There are no rules set to reject submissions:

[janisozaur]

According to https://docs.saucelabs.com/dev/api/error-reporting/, 400 means malformed request.

Perhaps something to do with latest libraries update? #21771

[janisozaur]

The build from #21771 gets 400 response

[janisozaur]

271a22d, #21771's parent submits dump successfully. #21771 is the regression.

[Lastorder-DC]

I tracked backtrace request using fiddler and found interesting result.

Save file I used
HAR of successful attempt
HAR of failed attempt

Request body is valid multipart/form-data before #21771

Request body is still multipart/form-data but now invalid after #21771

[Lastorder-DC]

Found exact reason.

multipart-body := [preamble CRLF]
                  dash-boundary CRLF
                  body-part *encapsulation
                  close-delimiter
                  [CRLF epilogue]

above are multipart-body's spec.

above are part of failed request.

looks like [CRLF epilogue] and [preamble CRLF] is missing from failed request(only if file is binary format), violating multipart-body's spec.

[Lastorder-DC]

google/breakpad@c7acbce

And I finally found reason why.
Before breakpad v2023.01.27, breakpad was appending a CRLF to all uploaded files.

And since OpenRCT2 never modified attached files in any way, this become [CRLF epilogue] and works like normally.
But breakpad decided to remove this as It might break binaries, and since CRLF at end is no longer appended, form-data become invalid.

[janisozaur]

Thank you very much for this investigation, I was not expecting outside help on this and had this planned for today. Would adding an epilogue help in this case? Inspecting the HAR you attached in chrome suggests the failed attempt contains the config.ini in binary only, which is the first file we attach - this leads me to assuming that perhaps just the epilogue is not enough, as there are issues detecting individual files in the payload

[Lastorder-DC]

@janisozaur I originally thought so but problem is - looks like file is loaded by breakpad library and we're only sending array of file path to library.

[Lastorder-DC]

After some research, I found out that pretty few code is using breakpad to upload dumps and more few code is using SendMultipartPostRequest. most results are coming from OpenRCT2, Gecko, FiveM and its forks.

And most of codes are either using outdated version of breakpad library or maintaining their own fork of breakpad.

So possible solutions are

1. Revert to old version(v2022.07.12)
2. Include and use OpenRCT2's fork of breakpad, reverting google/breakpad@c7acbce
3. Use google_breakpad::HTTPUpload::SendRequest instead. (Below are PoC from ChatGPT)

// Define boundary
std::wstring boundary = L"----WebKitFormBoundary7MA4YWxkTrZu0gW";

// Construct the multipart form data manually
std::wstring form_data;
for (const auto& param : parameters) {
    form_data += L"--" + boundary + L"\r\n";
    form_data += L"Content-Disposition: form-data; name=\"" + param.first + L"\"\r\n\r\n";
    form_data += param.second + L"\r\n";
}
for (const auto& file : files) {
    form_data += L"--" + boundary + L"\r\n";
    form_data += L"Content-Disposition: form-data; name=\"" + file.first + L"\"; filename=\"" + file.second + L"\"\r\n";
    form_data += L"Content-Type: application/octet-stream\r\n\r\n";
    // Assume file content is read into a variable file_content
    form_data += L"\r\n"; // file_content should be added here
}
form_data += L"--" + boundary + L"--\r\n";

// Prepare headers
std::map<std::wstring, std::wstring> headers;
headers[L"Content-Type"] = L"multipart/form-data; boundary=" + boundary;

// Send request
bool success = google_breakpad::HTTPUpload::SendRequest(url, headers, form_data, &timeout, &response, &error);

[janisozaur]

There's also #5342, but when I looked into it (tbh the libraries update was motivated primarily to let me use crashpad), the API is very different, due to handling some other cases that breakpad cannot