Personal Space Invader with specular material triggers on all instances of that avatar when activated

[Exairnous]

Description
If a material on an avatar is exported with a specular value other than 0.5 (causing the KHR_materials_specular and KHR_materials_ior extensions to be added) the personal space invader will trigger on all instances of that avatar's material in a room if you are using it when you bump into someone.

Note: outside observers do not see the personal space invader, as expected, it is only those who have triggered it themselves that see this effect.

To Reproduce
Steps to reproduce the behavior:

1. Open a room in three tabs.
2. Change to this avatar: https://hubs.mozilla.com/avatars/E6Lzjgb (at least two of the avatars must be the same, the collision one and the away one, otherwise the bug doesn't trigger. The viewer one could be different, though).
3. Move two avatars close to each other to trigger the personal space invader and move the third away from them while still being in their field of view.
4. See the personal space invader also applied to the away avatar from the point of view of the colliding avatars.
   Note: everything looks fine from the point of view of the away avatar.

Expected behavior
The personal space invader should only be applied to the avatars you are directly colliding with.

Screenshots
Specular present:

Without specular:

One material with specular, one material without:

Hardware

• Device: Laptop
• OS: Linux
• Browser: Firefox & Chromium
• Blender version: 3.3+

Additional context
Only materials with specular are affected by this. Any materials without specular in the avatar are not affected when this bug is triggered.
Thanks to Frikohet for initially finding the bug and doing much of the work to track it down, and for providing the "one with/one without" screenshot.

** Blend file**
personal_space_invader_bug.zip