fetch_survey error; filename issue?

[BeardedJayhawk]

OK, when I attempt to run fetch_survey I get an error. This is the simplest version of the code that's causing issues:

mysurvey <- fetch_survey(surveyID = 'SV_0UFBbXXXXXXXXX')

and this is the error:

Error in `checkarg_file_name()`:
! Error in `file_name`:
* The file given does not exist: C:/Users/<username>/AppData/Local/Temp/RtmpuUO8XD/Survey Name: The Company Survey - 2023-03 March Form.csv
Run `rlang::last_trace()` to see where the error occurred.
> rlang::last_trace(drop = FALSE)
<error/rlang_error>
Error in `checkarg_file_name()`:
! Error in `file_name`:
* The file given does not exist: C:/Users/<username>/AppData/Local/Temp/RtmpuUO8XD/Survey Name: The Company Survey - 2023-03 March Form.csv
---
Backtrace:
    x
 1. \-qualtRics::fetch_survey(surveyID = "SV_0UFBbXXXXXXXXX")
 2.   \-qualtRics::read_survey(...)
 3.     \-qualtRics:::checkarg_file_name(file_name)
 4.       \-rlang::abort(c("Error in `file_name`:", glue::glue("The file given does not exist: {file_name}")))

I of course edited out/changed some details, but one I did not change is the colon in the survey name. when I look at the temp directory there is a temp file named just "Survey Name".

Is the colon somehow causing the issue? This isn't happening with other surveys that don't have a colon in the name.

[juliasilge]

Yes, that's right. You can read more in #195 about the kinds of problems that people run into with special characters in their survey titles, which translates to special characters in a filename which of course is no good. Since #195 was opened, we have taken on a fs dependency so we could consider using fs::path_sanitize() to try to fix these up.

[juliasilge]

Oh wait, fetch_survey() doesn't call read_survey() anymore! @BeardedJayhawk can you try installing from GitHub and seeing if the development version works for you?

remotes::install_github("ropensci/qualtRics")

[BeardedJayhawk]

alright, now I'm getting a different error:

Error in unz(description = zip_path, filename = csv_filename, open = "rb") : 
  cannot open the connection
In addition: Warning message:
In unz(description = zip_path, filename = csv_filename, open = "rb") :
  cannot open zip file 'C:/Users/<username>/AppData/Local/Temp/RtmpEnkXdF/file1a3444523ed2.zip:Survey Name'
> rlang::last_trace(drop = FALSE)
Error: Can't show last error because no error was recorded yet

how do I go about utilizing fs::path_sanitize()?

Also, a follow up question. Why not use the survey ID as the filename? or is that something qualtrics is doing (which would not surprise me at all)?

[juliasilge]

Wow @jmobrien look at the filename we are ending up with: 👀

'C:/Users/<username>/AppData/Local/Temp/RtmpEnkXdF/file1a3444523ed2.zip:Survey Name'

This may be difficult to sanitize, actually.

Yes @BeardedJayhawk the filenames are coming from Qualtrics, not the R package.

[BeardedJayhawk]

is it possible to edit the survey name through the API? you could check for problem characters, change it, fetch the data, then change it back...

[BeardedJayhawk]

OK, so I looked around the API documentation and it appears that it is possible to rename a survey:

https://api.qualtrics.com/70532f4d66766-update-survey

so maybe temporarily change the name of the survey to match the survey ID, pull down the file, then change it back?

not sure how much of a delay that would cause, but that should address any filename issues, right?

[juliasilge]

I don't know that we would want to do that from fetch_survey(); seems like a pretty invasive thing to do (survey will have a different name 😱 for a bit of time). I think we'll check out our options for sanitizing the filename, if it's possible.

In #215 we discuss supporting an update_survey() function that would include renaming. We could consider starting that work back up, and then someone could themselves rename, fetch, then rename again, if needed.

[jmobrien]

Wow, missed this somehow, notice, sorry. Yeah, that's some wild file naming. Let me review the code and see if I can figure something out.

At a glance, I'm not sure I'm tracking what's going on. The zip file name is randomly generated now (e.g., file1a3444523ed2.zip, see here), so that error message doesn't make a lot of sense to me.

It's true that it is possible to rename surveys (the unfinished PR #266 covers this, actually), but I think we should be reluctant to make server-side changes just to fix things on our end.

[jmobrien]

Okay, still haven't checked this deeply, but just wondering--is the second example the same colon issue, or something different?

The zip file should be (randomly) named, say abcd.zip. It contains a file, [survey name].csv.

If we're requesting file [survey name].csv from within abcd.zip, then the error referenced could make sense--it's talking about not finding the compressed file purportedly represented INSIDE the .zip (e.g. abcd.zip:[survey name].csv), not the container file.

But, for a survey with a colon in the name (say a survey cats: allergic?), we'd expect that error to be something like

cannot open zip file 'C:/Users/<username>/AppData/Local/Temp/[randomstuff]/abcd.zip:cats: allergic?.csv'

That's not what we see here--but @BeardedJayhawk you did say you cleaned up some things in the error message for privacy reasons, yes? Just checking--what exactly did the second error look like (within what you are fine disclosing, of course)?

[BeardedJayhawk]

all I changed was "username" and "Survey Name" in the error message. for context, here's that applied to the full survey name:

"Survey Name: The Company Survey - 2023-03 March Form"

everything occurring after the colon in the survey name does not appear in the error message, nor did ".csv"

here's a question, if I go to "C:/Users//AppData/Local/Temp/RtmpEnkXdF/", should I be seeing a .zip file there? Because I'm not.

[jmobrien]

@BeardedJayhawk Yes, it should be there, but only while the function is running. In this most recent version we try to ensure its deletion as soon as the function call ends, to give people better control over where their response data goes. However, if you run debugonce(qualtRics:::export_responses_filedownload) before calling fetch_survey(), you can stop mid-process and look at the file.

@juliasilge I'm seeing the same issue. If I download the responses from a survey called test: survey, the scan of the zip file (using unzip(list = TRUE) lists the internal file as test: survey.csv, but I then get an error from unz() much like the one @BeardedJayhawk reports. If I unzip the file manually (outside of R), I get a file test/ survey.R.

This is a tough one. Maybe there's a way to properly reference the internal file, but if so I'm not sure how yet.

[jmobrien]

So, the issue here is with the foundational C code in R that makes connections, so this is way above my pay grade (unpaid grade?). I do see another way around it that could work on my system, but I'm not sure if it would work across platforms

@BeardedJayhawk if you grab the file out during debugging as described above, then extract the file manually (using the OS), what is the file name you get?

[juliasilge]

the issue here is with the foundational C code in R that makes connections

This doesn't surprise me, given the pretty wacky filename.

I would like to suggest that the best option here is to return to the work on update_survey() and make it easy for folks to rename their surveys when needed. I don't think we should do any automated name changes, but I think we should make it easier for folks to do this from R and then document that this may be needed if you have a particularly problematic survey name. What do you think about that @jmobrien?

[jmobrien]

Yes, that makes sense; we should enable users to handle this themselves.

[BeardedJayhawk]

@jmobrien so when I choose "extract all" from the context menu I get a windows error that says

An unexpected error is keeping you from copying the file. If you continue to receive this error, you can use the error code to search for help with this problem. 

Error 0x80070057: The parameter is incorrect. 

if I choose "skip" on the error, when I look at the extracted file, the file is there with the full filename except the colon in the filename was replaced with an underscore.

If I use 7zip it does the same thing, but without the error.

[BeardedJayhawk]

OK, hopefully this helps.

I found this solution, and applied it here.

this is what I came up with, and it actually pulls the data into R

zip_path = "C:/Users/<username>/AppData/Local/Temp/RtmpKcyBEr/file9e68551b11c6 - Copy.zip"

ziplist <- unzip(zipfile = zip_path, list = TRUE)
csv_file <- ziplist$Name  #I was running this with [1] appended, but seeing as how there's only one row this appears to work.
zipcon <- read.table(csv_file, header = TRUE, skip = 0  quote="\"", sep = ",")

[jmobrien]

This could work, and is similar to what I was considering. Problem is, it effectively moves us back to something that we just moved away from--precisely because it was creating problems with different characters in survey names (and which I'm beginning to realize may actually be related to this...). So, it's complicated.

Question for you--does the name in ziplist$name still have the colon in it? And it works, even though in the extracted file it was replaced with an underscore? Is that right?

[BeardedJayhawk]

it does still have the colon in it, and appears to work. I came across this solution when I was looking for a way to change the filename of the .csv without extracting it because that step appears to be the source of the issue, and this solution bypasses it entirely.