You can't, Linux has to reconize the modem before you can use it with VMware. Now your modem may or may not be reconized in Linux but since you have not stated which one it is, it is really hard to provide anything other then some general help.
This is a few years old, but should provide you with some help;
http://walbran.org/sean/linux/linmodem-howto.html
What does the following typed command (from the console or xterm session) show, if anything?
/sbin/lspci -v | grep Modem