You can check out some reviews of players that may support OGG at
www.anythingbutipod.com
I (actually my son) have the iAudio X5, which works in the manner you have stated - freedom to move your files where you please and supports all the open formats. It is very nice - open source friendly. The Rockbox project has firmware you can load into it to replace or dual boot, but I have not done that.
The X5 works great with Ubuntu/Gnome, which is what I run.