Anyway, does anyone know how to get XFCE to source the .Xmodmap file?
Goo Truther 
@astra_underscore add a one-line shell script to the Xfce autostart which runs xmodmap yourfile.xmodmap
That's what I've always done.
@thufie so the reason why this wasn't working was because my dumb ass forgot to install xmodmap 
