Well, I guess I solved it. When I loaded X I was root. Yes I know, you aren't supposed to do that but it was only twm, I was only using to varify the GUI was working.
Finally I made an account and tried it again and it works. So, technically there is still something wrong with root in X, however since root should not be in X I guess it's not a problem.