Unmarshalling fails for nullable json with circe

[felixbr]

Hi,

we discovered an edge-case with akka-http-circe the other day. A working reproduction-case and explanation of the problematic mechanism is here: https://github.com/felixbr/akka-circe-nullable-json-repro-case

The short version is: If you have a null json value in a HttpResponse and try to unmarshal it using something like Unmarshal(response).to[Option[String]] you'll get an DecodingFailure instead of the expected None value.

The reason behind this seems to be a fairly hidden feature of akka-http where it interprets the Option as you trying to handle the DecodingFailure and forwards only the inner type (here String) to the implementation (here circe). Circe then tries to decode null as String and correctly fails.

I'm not quite sure what to do about this other than "avoid using Option as root type". The implicit responsible for this is baked into Unmarshal, so it's always in scope. It should also be noted that it only happens when unmarshalling HttpResponse. HttpEntity doesn't seem to have this "feature".

In our case we expected a nullable List, so our workaround was to define a custom Decoder that avoids Option altogether, but this is no general solution.

Workaround for nullable lists:

// Custom decoder which decodes json arrays as lists but falls back to empty List if the json array is null
implicit def decodeNullableList[A: Decoder]: Decoder[List[A]] =
  Decoder.decodeList[A] or Decoder.decodeOption[List[A]].map(_.getOrElse(List.empty))

// Usage
Unmarshal(response).to[List[String]]

I'm not sure if this is something that needs to be addressed with akka-http directly. In any case I hope this helps other poor souls that happen to stumble over this edge-case.

Cheers
~ Felix

[hseeberger]

Thanks a lot for reporting and presenting so many details!

I do not have the time to investigate this issue right now, but from what you have written it rather seems to be an issue (or feature) of Akka HTTP. Hence I'm ccing Johannes from the Akka team.

@jrudolph, any thoughts?