On-the-fly image processing

[jacopoabramo]

I started implementing some quick on-the-fly image processing in the XimeaManager module, where using Pyro I remotely connected to the ImSwitch server in order to hijack the communication channel and doing a quick median filter which is then stored in the detector as a key-value pair in a dictionary.

The annoying thing is that whenever somebody other image processing algorithms like this they have to hard-code them in the detector itself so far. Would be nice to have a general widget which can do this type of operations regardless of the detector.

Also: any kind of active image processing should be suspended while doing recording (these should only be used in live-view to give a glance of what the processing result looks like), and all the information related to relevant image processing steps should be stored as metadata.

[jacopoabramo]

@kasasxav I have to admit I'm not proud of this hacking you see here, so I wanted to ask: would there be any more efficient way to do this?

[beniroquai]

Hey @jacopoabramo, @kasasxav helped me to do realtime Holoreconstruction, perhaps this is inspiring to you https://github.com/openUC2/ImSwitch/blob/596a35c705731010a17737053f6f5a569ebc260a/imswitch/imcontrol/controller/controllers/HoloController.py#L15

@ranranking and I also implemented an online SIM reconstructor that works nicely. Is this what you're looking for?

[jacopoabramo]

@beniroquai now that you mention this it may make sense to simply have dedicated widgets for this stuff... it's just that these are tasks so trivial that I don't know if the effort is worth it. Maybe a general image processing widget would make more sense...

[beniroquai]

Like napari?😁

…

On Tue, Jun 13, 2023, 23:38 Jacopo Abramo ***@***.***> wrote: @beniroquai <https://github.com/beniroquai> now that you mention this it may make sense to simply have dedicated widgets for this stuff... it's just that these are tasks so trivial that I don't know if the effort is worth it. Maybe a general image processing widget would make more sense... — Reply to this email directly, view it on GitHub <#173 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBE5OFCPV7JYG3U42DMYPTXLDMULANCNFSM6AAAAAAZE7YPPI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jacopoabramo]

Like napari?😁

Not gonna lie, I was thinking about that :P but I'm hoping to get a more general widget for quick image processing... I'd have to think about it

[beniroquai]

@kasasxav ensured to have in-ImSwitch Napari Plugin processing available.

[kasasxav]

Hii,

Not really, I mean it is possible to call napari plugins from ImSwitch now. When widgets are a subclass of NapariHybridWidget they have the function callPlugin() for that. But I think this is different from what @jacopoabramo wants, and also definitely at a different speed that he needs. I used the callPlugin() to reconstruct SIM data, maybe that's what you meant @beniroquai, but I think Jacopo needs something that runs faster and maybe that one doesn't necessarily have to develop a plugin in napari for that.

As ImSwitch is done now, it doesn't implement any type of real-time processing. So that's up to you how you want to do it :) I don't know what is the best way, but it would be nice that it's not in a specific manager (for example the Ximea). You could try to have it in the DetectorManager instead, since it's the superclass. Maybe if you manage to do a general function that can run a specific operation (function as an input) on the frames using threading. Similar to the execOn(). And you can add it maybe in some way. Like add processing -> to this camera -> this operation (with a widget).

[jacopoabramo]

@kasasxav that was my original idea. The thing is that the implementation for this image processing on the Ximea is very specific (it's basically the same multi-dimensional acquisition system that you originally implemented from the useq-schema). Technically, we could implement a more general MDA acquisition widget which could prove useful for everybody. And then I could still use the code I have for the Ximea manager in a more general manner by placing everything into the DetectorManager methods. We could then add some action buttons to enable/disable the real-time processing. I've always wanted to implement the useq-schema package into the ImSwitchServer but I never really understood how this works.

[beniroquai]

In our SIM Implementation we grabbed the frames directly from the detector's buffer and put them into a thread queue for further processing and displaying. I think that's the fastest you can get, no? ;)

…

Message ID: ***@***.***>

[kasasxav]

Yess but my suggestion was with the napari plugin, do you remember? And that turned out to be slower and didn't work in real time, so then you went for that other option instead.

This is a bit what I'm proposing to have a general solution in DetectorsManager to put processing algorithms in threads, and then when you call that function you input as a parameter which is your image processing algorithm you want to use.