For VOIP: google "asterisk".
As for the waveform - the kind of thing you seem to need is an "oscilloscope" program.
xoscope (for eg) takes input from the soundcard (so it is an audio frequency oscilloscope only) or from RadioShack's
ProbeScope ... but there are many others. As usual, google is your freind. While I havn't researched all of them, I'll bet that many will accept RS232 input.