How to render canvas with original pixel values?

[lez-s]

For example, this is the code from Gallery.
Since my image is float64/float32, I want to keep the same pixel values ​​unchanged after render. Instead of turning it into int8, is there any way to do it? Thanks.

import sys
from vispy import scene
from vispy import app
from vispy.io import load_data_file, read_png
import numpy as np

canvas = scene.SceneCanvas(keys='interactive')
canvas.size = 512, 512
canvas.show()

# Set up a viewbox to display the image with interactive pan/zoom
view = canvas.central_widget.add_view()

# Create the image
img_data = np.random.rand(512, 512,4).astype(np.float64)
interpolation = 'nearest'

image = scene.visuals.Image(img_data, interpolation=interpolation,
                            parent=view.scene, method='subdivide')

canvas.title = 'Spatial Filtering using %s Filter' % interpolation

# Set 2D camera (the camera will scale to the contents in the scene)
view.camera = scene.PanZoomCamera(aspect=1)
# flip y-axis to have correct aligment
view.camera.flip = (0, 1, 0)
view.camera.set_range()
view.camera.zoom(0.1, (250, 200))

# get interpolation functions from Image
names = image.interpolation_functions
names = sorted(names)
act = 17
rendered = canvas.render()  ##HERE!!! after render the canvas, the image becomes int8.

[djhoese]

Your GPU uses 8-bit RGBA values and I don't think there is any way to change that. Additionally, the default Image behavior is to scale the data before it goes to the GPU. You can specify texture_format as np.float32 or whatever and vispy will pass your data directly to the GPU as floats and the GPU will scale it there. This is usually faster overall and can sometimes provide better bit-depth for color mapping.

That said, even if you get your data in the GPU as some floating point representation, the drawn pixels are still 8 or 10 bit integers. @rougier can correct me on this. I would guess that even if you could tell the .render() method to give you floating point numbers they would just be a representation of the integers in the GPU as floats so no extra precision.

If you're only using vispy to perform some type of interpolation on an image then I would recommend using a different library like scipy or scikit-image.

Again, someone please correct me if I'm missing something here.

[lez-s]

Your GPU uses 8-bit RGBA values and I don't think there is any way to change that. Additionally, the default Image behavior is to scale the data before it goes to the GPU. You can specify texture_format as np.float32 or whatever and vispy will pass your data directly to the GPU as floats and the GPU will scale it there. This is usually faster overall and can sometimes provide better bit-depth for color mapping.

That said, even if you get your data in the GPU as some floating point representation, the drawn pixels are still 8 or 10 bit integers. @rougier can correct me on this. I would guess that even if you could tell the .render() method to give you floating point numbers they would just be a representation of the integers in the GPU as floats so no extra precision.

If you're only using vispy to perform some type of interpolation on an image then I would recommend using a different library like scipy or scikit-image.

Again, someone please correct me if I'm missing something here.

Thank you for the explanation. My actual application scenario is more complicated, which is to put the Mesh and its corresponding color on the canvas and render them from different perspectives. This all works fine except the final output is changed to int8, which unfortunately, is a very serious problems for me.

[djhoese]

Not that I need to know every detail of your use case, but now I'm curious: let's say you could get the floating point colors back (normalized RGB/A in the range 0 to 1), what are you doing with them after that?

[lez-s]

Not that I need to know every detail of your use case, but now I'm curious: let's say you could get the floating point colors back (normalized RGB/A in the range 0 to 1), what are you doing with them after that?

Sure, I am glad to share the details if you have any idea of solving the problem. Actually, it is not color, it is more like pixel intensity range from 0.x ~ 7.x (float32). And its 'color' channel is 6, so now I just divide them into 2 to render it. The output is used as one of the input to train the neural network.

[djhoese]

And its 'color' channel is 6

What do you mean by this? You mentioned a mesh. So you're drawing a mesh with a texture or per-vertex "color"? What are you dividing into 2? I don't do any ML myself, but don't you need to give an integer version of your dataset anyway so it can easily classify things as one thing or another?

[lez-s]

Sorry for the divergency, 'color' channel doesn't matter, it is not important, it is just vertex_colors set by visuals.Mesh.set_data. After rendering, the image only has 4 color channels.

The thing is I would like to keep the pixel intensity unchanged after rendering from 0.x to 7.x . It is not about classification task, so value itself matters more. There is a similar application here mesh.py if you are interested. But they are using normal images, so it is not a big deal for them.
I tried scale the value to int8 and scale them back, it doesn't work well.

[djhoese]

I tried scale the value to int8 and scale them back, it doesn't work well.

It didn't work well because of the loss of precision or because you couldn't figure out how to scale the integers to get near the original values?

I was wondering if as an extreme workaround you could encode the values into the 3 RGB channels, even if temporarily for the .render() call, and then extract something that isn't just 8-bits of precision but 24-bits.

[lez-s]

If there is no other possible solution, I think this hack may work. I just hope there is an official solution to render the image with higher precision.

[djhoese]

I think @rougier isn't available, but he'd probably know best.

It mostly comes down to this utility function:

vispy/vispy/gloo/wrappers.py

Line 620 in c366456

def read_pixels(viewport=None, alpha=True, mode='color', out_type='unsigned_byte'):

Which uses the OpenGL function:

https://registry.khronos.org/OpenGL-Refpages/gl4/html/glReadPixels.xhtml

But while this supports other types, I'm not sure that means anything for the precision. It is all just a representation of what's on the GPU...I think 🤷‍♂️

[lez-s]

I think @rougier isn't available, but he'd probably know best.

It mostly comes down to this utility function:

vispy/vispy/gloo/wrappers.py

Line 620 in c366456

def read_pixels(viewport=None, alpha=True, mode='color', out_type='unsigned_byte'):

Which uses the OpenGL function:

https://registry.khronos.org/OpenGL-Refpages/gl4/html/glReadPixels.xhtml

But while this supports other types, I'm not sure that means anything for the precision. It is all just a representation of what's on the GPU...I think 🤷‍♂️

Thank you. Yes, unfortunately, It is precision doesn't change, it is just int8 scale back to float.

[rougier]

Sorry, I was not available. I'm not sure to fully understand but I think your problem is related to the format of the framebuffer (i.e. the screen) that is using byte representation. It might be possible to have float framebuffer but it depends on your GPU and I'm quite sure this is not part of the GL ES 2.0 so not obvious solution. You could however create a offsscreen framebuffer and use this framebuffer to be copied on screen.

[lez-s]

I do not fully understand your solution, but I will search on that direction. Thank you.

[rougier]

Here is a thread that might help: https://groups.google.com/g/vispy/c/QH33cVZAg8U