Start Experiment automatically

This is only a draft description for the QA service. The other GERBIL APIs work in a different way

This description is object to change!

It may make sense to automate the configuration (and execution) of some experiments. In the following, the API of the GEBRIL QA service is described. This API can be used for such an automation.

Throughout this description, we will assume that the GERBIL instance is hosted at http://gerbil-qa.cs.uni-paderborn.de:8080/gerbil.

1. Upload Dataset(s) (optional)

If an uploaded dataset should be used within the experiment, the following URL should be used:

http://gerbil-qa.cs.uni-paderborn.de:8080/gerbil/file/upload

The request should be a POST request using the multipart/form-data content type. It should contain the following two parts:

name: Test Dataset
files[]: (binary)

The name is the dataset name (we use Test Dataset) in our example. The files[] field should contain the binary content of the file. The multipart for our example (recorded within a browser) looks as follows:

------WebKitFormBoundaryYUPudzHAr9vleq44
Content-Disposition: form-data; name="name"

Test Dataset
------WebKitFormBoundaryYUPudzHAr9vleq44
Content-Disposition: form-data; name="files[]"; filename="dataset.nt"
Content-Type: text/text

The last part contains the binary data of the file.

The answer will be a JSON object containing some information about the file. However, it is mainly the data that has been sent within the request and can be ignored.

{"files":[{
  "name":"dataset.nt",
  "size":"8Kb",
  "url":null,
  "thumbnailUrl":null,
  "deleteUrl":null,
  "deleteType":null,
  "error":null,
  "fileType":"text/text"}]}

Note: files with the same name will override each other!

This part is likely to be changed in the future

2. Upload System Answers (optional)

If one of the systems that should be evaluated is not available online, uploading the system's answers is an option. It works in nearly the same way as the upload of the dataset above. Again, a request should be sent to the following URL:

http://gerbil-qa.cs.uni-paderborn.de:8080/gerbil/file/upload

The request should be a POST request using the multipart/form-data content type. It should contain the following two parts:

name: Test Dataset
files[]: (binary)

The name is the system name (we use Test System) in our example. The multiselect field should contain the name of the chosen dataset. If an uploaded dataset has been chosen (like in our example), the file name of the dataset should be used. The files[] field should contain the binary content of the answer file. The multipart for our example (recorded within a browser) looks as follows:

------WebKitFormBoundary02Hqn0tqsPGSXnly
Content-Disposition: form-data; name="name"

Test System
------WebKitFormBoundary02Hqn0tqsPGSXnly
Content-Disposition: form-data; name="multiselect"

dataset.nt
------WebKitFormBoundary02Hqn0tqsPGSXnly--
Content-Disposition: form-data; name="files[]"; filename="system.nt"
Content-Type: text/text

The last part contains the binary data of the file.

The answer will be a JSON object containing some information about the file. However, it is mainly the data that has been sent within the request and can be ignored.

{"files":[{
  "name":"system.nt",
  "size":"8Kb",
  "url":null,
  "thumbnailUrl":null,
  "deleteUrl":null,
  "deleteType":null,
  "error":null,
  "fileType":"text/text"}]}

Note: files with the same name will override each other!

This part is likely to be changed in the future

2. Send Experiment Configuration

Finally, the configuration of the experiment can be sent. A GET request should be sent to the following URL:

http://gerbil-qa.cs.uni-paderborn.de:8080/gerbil/execute

The request should have a single parameter named experimentData. The parameter should contain the following JSON:

{
  "type":"QA",
  "matching":"STRONG_ENTITY_MATCH",
  "annotator":[
     "AskNow",
     "OKBQA",
     "Platypus",
     "NIFWS_Test+System+2(http://gerbil-qa.cs.uni-paderborn.de/SomeQA)"],
  "dataset":[
     "NIFDS_Test+Dataset(dataset.nt)"],
  "answerFiles":[
     "AF_Test+System(system.nt)(undefined)(dataset.nt)"],
  "questionLanguage":"en"
}

In the following, we describe the single fields in more detail:

• type: the experiment type. This should be QA for the Question Answering experiment.
• matching: this parameter can be ignored and should simply always have the value mentioned above.
• annotator: the available systems that can be benchmarked online. If no system should be benchmarked online and only uploaded answer files should be evaluated, the array should be empty. There are two types of systems:
  1. A predefined system that can be chosen from the drop-down menu. If a system should be chosen, its name should be listed in this array.
  2. A system that is not listed in the drop-down menu but that is available and supports the QALD API. Its String should be created in the following way: "NIFDS_<system name>(<URL of the endpoint>)", where parts in angular brackets should be replaced by their respective value.
• dataset: the datasets that should be used for the experiment. There are two types of datasets:
  1. A predefined dataset that can be chosen from the drop-down menu. If such a dataset is chosen, its name should be listed in the array. (not shown in the example above)
  2. An uploaded dataset should be used. Its String should be created in the following way: "NIFDS_<dataset name>(<file name>)", where parts in angular brackets should be replaced by their respective value. Please make sure to reuse the dataset and file names that have been used during the upload.
• answerFiles: this array should be empty unless a system answer file has been uploaded before and should be part of the experiment. In that case, a String should be created for each uploaded system answer file in the following way: "AF_<system name>(<system file name>)(undefined)(<dataset name>)", where parts in angular brackets should be replaced by their respective value. Note that in case the dataset has been uploaded, the dataset name should be the file name of the uploaded file (as it can be seen in the example).

The response contains only the ID of the experiment

202208090002

3. Query Experiment Status

The status of the experiment can be queried as usual using the following URL with the ID of the experiment at the end.

http://gerbil-qa.cs.uni-paderborn.de:8080/gerbil/experiment?id=202208090002

However, note that this URL will always answer with HTML. Hence, an automation is not well supported for this at the moment.