There is a chance that you were trying to play a CSS-encrypted DVD. Xine was not made to play that type of dvds. I know that ogle is able to css with libcss and libdvdread.
Well, I guess that might be the solution, unless xine is using some otherwordly video output. Just try ogle, it might work wonderfully.