Hey, thank you for the issue.

Starting with the definition of a modal https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/Attributes/aria-modal#description

Modal dialogs are when content is displayed and the user's interaction is limited to only that section until it is dismissed.

This means everything on the page that is not the modal dialog should be hidden from users.

Shadow dom is just an implementation detail of web apps. It shouldn't affect how a user interacts with the application, this extends to modal dialogs. This means that aria-hidden should be applied to everything not directly containing the modal. However, it would appear that because of the shadow dom, we are not detecting that "directly containing" path.
In other words, it should be like this

+-------------------------------------------------+
| Consuming Application (Parent Document)         |
| +---------------------------------------------+ |
| | Our Web Component (Shadow DOM)              | |
| | +----------------------------------------+  | |
| | |  Modal Dialog                           | | |
| | | (should remain visible)                 | | |
| | +----------------------------------------+  | |
| | [Content in Shadow DOM, should be hidden    | |
| |  by aria-hidden]                            | |
| +--------------------------------------------+  |
| [Content outside our component should be        |
|  hidden]                                        |
+-------------------------------------------------+

I'd suggest having a look through our issues, we have a few of them related to ShadowDOM support. I think you'll run into more pretty quickly based on some previous investigations done around iframes as well #1472 (comment)

@snowystinger Thank you for your comprehensive response.

Based on the definition of modals and the necessity to make everything on the page, except the modal dialog, inaccessible to users, I've devised a solution that respects these accessibility requirements while considering the unique challenges Shadow DOM presents.

The solution aims to ensure that the modal dialog within a Shadow DOM remains accessible while effectively hiding all other content from assistive technologies, as illustrated in your diagram. Here’s how the proposed solution addresses the key points:

1. Within the Shadow DOM: We can pass the modal's ref, which is inside the shadow DOM, and the shadow root to ariaHideOutside, which will hide all elements except the modal. Nothing will be hidden above the shadow boundary.
2. Outside the Shadow DOM: For content outside the Shadow DOM, the strategy involves dynamically applying aria-hidden="true" to elements in the light DOM that are not parents of the Shadow DOM host element, effectively making them inaccessible while the modal is active. This prevents the modal's accessibility state from being influenced by the broader document's aria-hidden status.
3. Siblings of the Shadow DOM Host: All sibling elements to the Shadow DOM's host are also marked with aria-hidden="true", ensuring a seamless integration of the modal's accessibility with the rest of the application. This step is crucial for maintaining the intended user experience, where interaction is limited to the modal dialog until dismissed.

This is the only way I had on mind that ensures that the modal stay accessible and not get affected by the inheritance of any parent having aria-hidden="true".

Do you think this is a valid approach and doesn't violate any of the guidelines?

I already have this PR #6046 ready for review for adding shadow DOM support, and given that my proposed solution is a valid one I can open another one to add the solution proposed above which will mainly have changes for ariaHideOutside function only.

Ah, apologies, I did not notice the user names on both. Thank you.
Does this approach cover a Shadow DOM inside another Shadow DOM?
I'll bring it up with the team. Thanks for noting that the other PR is ready for review.