INGV FDSNWS: some optional parameters advertized in WADL but are rejected in (leading to bad request in Mass downloader)

[pieropoli]

Avoid duplicates

• I searched existing issues

Bug Summary

Hello,

I am running massdownloader to get some data from the ingv data center, but i keep on getting this error:

obspy.clients.fdsn.mass_downloader - ERROR: Client 'INGV' - Failed getting availability: Bad request. If you think your request was valid please contact the developers.
Detailed response of server: Error 400: Bad request
Syntax Error in Request
Including of availability information not supported.
Usage details are available from /fdsnws/station/1/
Request:
/fdsnws/station/1/query?starttime=2023-02-07T23%3A51%3A40.000000&endtime=2023-02-09T00%3A08%3A20.000000&network=IV&station=RUFI&level=channel&matchtimeseries=true&format=text
Request Submitted: 2023-03-09T07:40:01.089Z

Now I checked the INGV webservice (https://webservices.ingv.it/fdsnws/station/1/) and it seems that they do not allow the use of 'matchtimeseries', i report what is the description on the website:

Limit to metadata where selection criteria matches time series data availability. Not implemented, fails if passed true value.

Indeed, repeating the query with a url with matchtimeseries=false from a browser worked;

http://webservices.ingv.it/fdsnws/station/1/query?starttime=2023-02-07T23%3A51%3A40.000000&endtime=2023-02-09T00%3A08%3A20.000000&network=IV&station=TRIF&level=channel&matchtimeseries=false&format=text

I was wondering how can this parameter be controlled when using massdownloader

Best

Code to Reproduce

No response

Error Traceback

No response

ObsPy Version?

3.0

Operating System?

MAC OS

Python Version?

3.0

Installation Method?

pip

[megies]

@pieropoli FDSN mass downloader is looking at what the FDSN server advertizes as capabilities in machine readable form through the respective wadl file:

          <param name="includeavailability" type="xsd:boolean" default="False" fixed="False" style="query">
            <doc title="includeavailability" xml:lang="en">Specify if results should include information about time series data availability. <strong>Not implemented, fails if passed true value.</strong> </doc>
          </param>
          <param name="matchtimeseries" type="xsd:boolean" default="False" fixed="False" style="query">
            <doc title="matchtimeseries" xml:lang="en">Limit to metadata where selection criteria matches time series data availability. <strong>Not implemented, fails if passed true value.</strong>
                        </doc>
          </param>

If matchtimeseries is mentioned in the wadl, then we use it for the request. The textual comment that this parameter should not be used can not be automatically interpreted, obviously. The offical FDSNWS docs do not say much about the wadl conventions (see below), but I believe the correct way to handle this would be to simply leave that parameter out of the wadl.

I'm not too intimate with WADL definitions, I guess it looks like you could set fixed="True" as an alternative to just leaving that unsupported parameter out of the WADL, but even then we would still have to make changes in obspy code for your problem to change.

Long story short, for a short term solution your only way to go would probably be

• patch the code of your obspy installation, if you're comfortable with that (quick and dirty)
• subclass some of the mass downloader routines accordingly (cleaner)

Mid to long term solution

• INGV gets rid of the optional and unsupported parameters from the WADL (would work immediately when server side change is made)
• INGV sets default ="False" fixed="True" and we make adjustments to check that part in the WADL (would only work with new obspy release)

CC @petrrr

[megies]

Who should be made aware of this at INGV, @petrrr?

[pieropoli]

Hello, When I found this kind of issue ***@***.*** was always very useful. Not sure if he's still the person in charge of the database. Il giorno gio 30 mar 2023 alle ore 10:49 Tobias Megies < ***@***.***> ha scritto:

…

Who should be made aware of this at INGV, @petrrr <https://github.com/petrrr>? — Reply to this email directly, view it on GitHub <#3279 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVIF5HS3KGN5XA2K3H6QMLW6VCJJANCNFSM6AAAAAAVUX4MRQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Piero Poli *If you're in the penalty area and don't know what to do with the ball, put it in the net and we'll discuss the options later...*

[megies]

@pieropoli the email address got blanked out by github for privacy reasons, would you mind contacting that person yourself and making them aware of this ticket?

[pieropoli]

Sure Il giorno gio 30 mar 2023 alle ore 11:27 Tobias Megies < ***@***.***> ha scritto:

…

@pieropoli <https://github.com/pieropoli> the email address got blanked out by github for privacy reasons, would you mind contacting that person yourself and making them aware of this ticket? — Reply to this email directly, view it on GitHub <#3279 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVIF5CLCTXZJ54V6SA7EGDW6VGZ3ANCNFSM6AAAAAAVUX4MRQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Piero Poli *If you're in the penalty area and don't know what to do with the ball, put it in the net and we'll discuss the options later...*

[megies]

Any news from INGV on this one?

[perfettp]

I'm experiencing the same error.
Also, similar/related issue #2366

[pieropoli]

I wrote to someone at INGV let's see what they say.