Add game as a non steam game to steam, enable proton and add this steam parameter: WINEDLLOVERRIDES="OnlineFix64=n;SteamOverlay64=n;winmm=n,b;dnet=n;steam_api64=n" %command%

Add this to your java arguments: -Dfml.earlyprogresswindow=false

There are some mods for Wayland support:

Forge: WayFix

Fabric: WayFix Wayland Fix