Apps run through nested wlroots compositors in "direct scan out" mode are not updated properly

[Supreeeme]

Observed in both Anvil and Niri.
Behavior observed with Cage, Sway, and Labwc wlroots compositors.

Steps to reproduce:

1. $ cage -- firefox (make sure no other firefox is running/run it with a different profile, can also test any other arbitrary app)
2. Try to click anything and observe strange rendering artifacts/unresponsiveness

Notes:

• Behavior does not occur in any wlroots or non wlroots (KDE, Gnome) compositors
• Behavior only occurs when wlroots starts doing "Direct scan out" as indicated by its logs - this seems to occur whenever a window is fullscreened in most compositors. Most easily reproducible in Cage since it auto-fullscreens all windows
• Hovering over menu popups seems to force the window to update properly for some reason
• I tried recording this and it doesn't appear in a recording, pipewire seems to capture the frames just fine

[cmeissl]

I am failing to reproduce this. What system does this occur with?
Artifacts are typically caused by some synchronization issues. Are the clients run in the nested session using wayland
or XWayland?
Please provide more details including the GPU(s) and driver version.

[Supreeeme]

I've seen this on both my (Intel) Macbook and my desktop with an AMD RX 6600, both with Mesa 24.1. I know on the laptop it happens with both Xwayland and Wayland clients, but I'd need to verify again on the desktop if it's happens with Wayland clients. I can say for sure it happens with Steam (through Cage) on the desktop.

[Supreeeme]

More interesting notes:

• It actually doesn't happen in firefox on Wayland with the default profile, but it happens if I run any other profile firefox -P <profile> (wtf?)
• It happens 100% of the time on Firefox through Xwayland (cage -- env WAYLAND_DISPLAY= firefox)

Here's a video, had to record my screen since it doesn't show up in a normal screen recording. When I'm wiggling the mouse around and nothing is happening that's me clicking things to show how it's not updating. Notice how it only updates when I click the hamburger menu and the popup appears.

video2.mp4

[cmeissl]

Strange, I am still unable to reproduce this. Can you provide WAYLAND_DEBUG logs for the nested compositor and the client running in the nested compositor?

[Supreeeme]

WAYLAND_DEBUG log spew from cage. I just opened firefox, maximized the window, and then clicked the popup.
WAYLAND_DEBUG from niri