Async Parsers and Serializers

[kennethmyhra]

Has there been any attempt at adding async-support for the parsers/serializers?
If not, is this something that would be considered useful for the SDK, and would there be any potential show-stoppers adding async-support with the underlying XML/Json libraries?

[brianpos]

The xml parser I'm working on does async (along with cancellation token) however it is slower than the non async code. And also had to work around a bug in the xmlreader async code too.

Not sure if streaming directly to the object is better, or going via the memory stream.

[kennethmyhra]

Currently I am not so concerned about speed, but rather deadlocks when using sync IO in an async request pipeline. I have worked around this in Spark by enabling Sync IO (sync IO is off by default in .net core 3.1). Also, I enabled buffering on the request + called DrainAsync before starting parsing of the request body.

Not sure why you would go via MemoryStream unless you think that would speed up your parser. People using your parser could maybe have the option to do that if you expose a Parse method that takes a Stream?

[brianpos]

Using the memory stream you can do async in the formatters, then sync on the stream. Best of both worlds? (except may use twice the memory, but the current code does that anyway as it runs it into a string, and then to the object)

My parser only works with streams too.

[kennethmyhra]

That is more or less the approach I am using now. buffering up the stream asynchronously and then the parser synchronously parses it. By using a MemoryStream I can maybe get around the need to enable Synchronous IO in the request pipeline since there is no network or file IO involved when it gets to the parser.

[brianpos]

Yup, that's what I've updated to, and have the async parser as an option which reduces mem overhead, but don't know if its truly better yet.

[ewoutkramer]

I considered doing that when writing the new deserializers, but then noticed that the System.Text.Json Utf8JsonReader has no async methods on it either. Now, JsonSerializer.DeserializeAsync does sound like it's async, but I could not figure out that black magic sauce that created an async method on top of the non-async reader.

[dennisbrox]

I'm having problems deserializing in .net 6 using c#
Here is the code I use to serialize and deserialize a communication resource 'com'
using Hl7.Fhir.Model;
using Hl7.Fhir.Serialization;
using System.Text.Json;
...
// set up payload
var x = new Communication.PayloadComponent();
cc = new CodeableConcept();
cc.Text = "xyz";
x.Content = cc;
com.Payload.Add(x);
var options = new JsonSerializerOptions().ForFhir(ModelInfo.ModelInspector);
str= JsonSerializer.Serialize(com, options);
com = JsonSerializer.Deserialize(str, options);

I get the error:
Hl7.Fhir.Serialization.DeserializationFailedException
HResult=0x80131500
Message=One or more errors occurred. (Value is of type 'CodeableConcept', which is not an allowed choice. At Communication.payload.content, line 1, position 478.)
Source=Hl7.Fhir.Base

where str is:
"{"resourceType":"Communication","basedOn":[{"reference":"1912","type":"ServiceRequest","display":"Acne"}],"status":"preparation","priority":"routine","medium":[{"coding":[{"code":"6049138449","display":"phone"}]}],"subject":{"reference":"912345678","type":"Person"},"sent":"2023-07-26T10:00:03.000Z","recipient":[{"reference":"912345678","type":"Person","display":"DEXTER BROX"}],"reasonCode":[{"coding":[{"display":"assessment Acne"}]}],"payload":[{"contentCodeableConcept":{"text":"xyz"}}],"note":[{"text":"2023-06-26 10:00 XXANA/P1 This is a note"}]}"

Why doesn't it like 'contentCodeableConcept' ?

Thanks.

[mbaltus]

Maybe you are using the R4 library while looking at the R5 model in the FHIR spec?
In R4, CodeableConcept is not a valid choice and in R5 it is. If you use the R4 library, the serialization will work since the SDK does not validate before serializing. Parsing will fail, since the SDK does basic structural validation for that. Change to the R5 library and it should work fine.

[dennisbrox]

Downloaded nuget HL7.FHIR.R5 (shows as HL7.FHIR.Conformance 5.0.0 in project packages). Same error. Is there any change to the using libraries with the 5.2 version? Can you get the string I provided to deserialize on your system?
Thanks.

[mbaltus]

I am indeed using the 5.2.0 library, but when downgrading to 5.0.0 it works the same - correctly - for the contentCodeableConcept deserializing.
When deserializing your text however, I get a different error about the reasonCode field not being recognized. It seems that you have mixed up your FHIR versions: reasonCode only exists in R4 not R5, and contentCodeableConcept is only valid in R5 not R4.

[dennisbrox]

I tried deleting all the FHIR stuff from my project and then getting all the R5 stuff from nuget. Now Visual Studio can't find a whole bunch of fhir classes. What exactly are the firely nuget libraries to download.
Thanks.

[mbaltus]

You should only need the Hl7.Fhir.Rx package, where x is the FHIR release you want to work with. See also https://github.com/FirelyTeam/firely-net-sdk#getting-started.
But do make sure that your data is consistent with the chosen release, because as mentioned above your data seems to mix two releases and will not validate against either R4 or R5 release.

[dennisbrox]

Thanks for the hint to look at the start up stuff again - - haven't seen that for a long time. What a mess! Compile compatibility is a joke.
'using Hl7.Fhir.Model' is now invalid. Is there a replacement or is it unnecessary? Even "CodeableConcept" doesn't work. Valid FHIR classes seem to now be undefined in the class library? The "JsonSerializerOptions().ForFhir(ModelInfo.ModelInspector)" for .net 5+ doesn't even work because ForFhir is no longer available. Am I missing something or do I just have to chuck Firely and use another library?
Thanks.

[mbaltus]

Hello Dennis, I understand it is very frustrating when things do not work anymore. And if you do not want to use the Firely SDK, feel free not to.
However, based on your previous information, the issues are unrelated to the SDK package itself. You mentioned using the 5.0 version of the library, and there have not been any breaking changes since, so updating to 5.2 cannot cause this. Also, there are numerous developers that work with the libraries on a daily basis and have not reported anything out of the ordinary. I tried helping you yesterday, and also did not get the issues you describe. The using statement still works, all FHIR datatypes are still in the classes, ForFhir is still present (but maybe you need to add 'using Hl7.Fhir.Serialization'), and I was able to run the program fine - the data was faulty but that is not the library's mistake.

I think your system is somehow messed up by moving to .NET 6 or by changing the NuGet packages in your project. My advice: a good old system reboot, then reload your project to see if that helped. If not, second advice: start with a fresh project, add the Hl7.Fhir.Rx package you need, and try out some code. Then revert your original project to its previous working state, and go step by step if you need to change something - don't try it all at once, so you can determine which step fails.

[dennisbrox]

The problem was the v5 stuff needed 3 other dependent packages installed with updated versions before it would install - - just the way github works that I didn't expect. The errors then went down to a high, but acceptable level. Only problem noted so far is communication.payload[n].content has to be explicitly typed to codeableconcept before being assignable to a codeableconcept variable. But the R5 library handles things better than the R4 one and I like the FHIR V5 changes.

Thanks for your help. Much appreciated.