Use unions with manually drop

[Firstyear]

Since 1.50 enough parts are in place to allow us to use union types with manually drop. This would replace a large quantity of unsafe casting throughout the codebase in the hashmap especially.

Once we are able to target 1.50 we should change to use this.