Alacritty should support window decorations on wayland. What compositor are you using? Note that alacritty prefers server-side decoration, but some compositors don't implement it like GNOME's mutter[1], in that case it will draw its own decorations.
I'm using Gnome. And alacritty draws it's own decoration. But they are out of place and more importantly they don't behave like normal windows. For example to maximize a window normally I double click the title-bar but alacritty (I think it's winnit's fault) doesn't react to that. The button placements in the titlebar is also weird and needs me to be really accurate to click the close (x) button.
1: https://gitlab.gnome.org/GNOME/mutter/-/issues/217