I have the same problem with XGL on OpenSUSE 10.2 using gnome (haven't tried KDE because I heard it uses more resources).
The 3 edge flipping options (flip when mouse on edge, flip only with window, never flip) don't change anything.
Only the flipping while moving a window works, independently of what option I choose.
I know this post is old, but if anybody knows why...
Maybe these options just haven't been implemented yet.