OK, false alarm... There is no bug in XFCE after all...
What happens is the following:
- the /etc/X11/xinit/.Xmodmap is read correctly
- the xinitrc script also reads ~/.Xmodmap correctly (tried it with several tests)
But then I had my keyboard layout set in settings and this overrules the Xmodmap definitions, effectively ignoring all my work.
So I enabled "Use system defaults" in the layout-tab and now everything works fine