Add this to your java arguments: -Dfml.earlyprogresswindow=false

There are some mods for Wayland support:

Forge: WayFix

Fabric: WayFix Wayland Fix