What service/process on this server do you wish to connect to? What have you tried so far, and why hasn't it worked? I've not known sockets to care about the difference between OSs involved on client and server.
Message Queues as in MQ/JMS? Sure, why not, just like both ends talking to any other persistance layer like a SQL RDBMS or a filesystem.
Java and shared memory, don't think so due to the JRE/VM. Probably going to need JNI/an Extension, or consider IDL/CORBA (can't use RMI if both ends aren't Java unless I'm mistaken).