500 error from kobocat form submission API for json due to mismatched parameter type

[RobertoMaurizzi]

The problem

While writing a script to create 200.000 artificial submissions with randomized data to evaluate server setup performance, I got stuck for a while by a 500 error while trying to post a form.

It turns out that when submitting a form using Python's requests API, the data= (or json=) parameters NEED to be a Python dictionary.
I got fooled by the API docs in the page to think the submission parameter had to be a STRING containing the JSON, and the server response instead of a 400 was a very unhelpful 500. In addition to that, without setting up Sentry I was unable to get a backtrace of any kind from the logs that would have helped a lot as you can see here:

Backtrace

onadata/apps/api/viewsets/xform_submission_api.py in create_instance_from_json at line 70

def create_instance_from_json(username, request):
    request.accepted_renderer = JSONRenderer()
    request.accepted_media_type = JSONRenderer.media_type
    dict_form = request.data
    submission = dict_form.get('submission')

AttributeError: 'str' object has no attribute 'get'

Proposed fixes

I'm not sure what's the best "fix" for this: its true it's a data submission problem in the user data, on the other hand error reporting, logging and examples could be improved quite easily.

On one side it might be possible to change the code in create_instance_from_json to check if both the whole of request.data is a string (to fix the 500) and after that a check if submission is a string, then throw a 400 telling users they need to send objects/dictionaries and not strings (the 400 if you send submission as string is a not very helpful "invalid JSON at line 0 position 0: '")

On the other side it might be enough to add a Python example to the API page at https://kc.yourkoboserver.org/api/v1/submissions like:

import requests

submission_data = {
  "transport": {
      "available_transportation_types_to_referral_facility": ["ambulance", "bicycle"],
      "loop_over_transport_types_frequency": {
          "ambulance": {
              "frequency_to_referral_facility": "daily"
          },
          "bicycle": {
              "frequency_to_referral_facility": "weekly"
          },
          "boat_canoe": None,
          "bus": None,
          "donkey_mule_cart": None,
          "keke_pepe": None,
          "lorry": None,
          "motorbike": None,
          "taxi": None,
          "other": None
      }
  },
  "meta": {
      "instanceID": "uuid:f3d8dc65-91a6-4d0f-9e97-802128083390"
  }
}

posted_json = {"id": '<your_form_deployment_id>', "submission": submission_data}

resp = requests.post(
    url='https://kc.yourkoboserver.org/api/v1/submissions',
    json=posted_json,
    auth=('super_admin', '<your_super_admin_password>')
)

print(resp)