Read the Modem-HOWTO at
www.tldp.org. You can use tools such as kppp or minicom to see if your modem is a hardware one (if it is, it will be very easy to use, especially with kppp).
Find out which serial port your modem is on from windows (com1=/dev/ttys0, com2=/dev/ttys1...) and test that port using one of the aforementioned tools. If a test results in "Modem is Busy", it's probably a softmodem. Download the Scanmodem tool from
www.linmodems.org to find out if it's supported