Live cd is a good idea if you have access to a machine with a decent amount of RAM.
I'm not sure how much you have to spend, but shipping can be a killer. If buying from a US source count on $50CDN plus whatever customs and the taxman decide to do. Inside Canada at least $25.
Used computers are not in short supply. A PIII still has some value. PII's in my experience are next to free. I pick them up at Goodwills. Usually $10 - $15 for a working model, $0 nonworking. Sometimes I patch them together and donate them back.

Now this is a worthwhile cause........the Salvation Army and Red Cross could benefit from this effort!!

Me I go around to pawn shops and look for older Dell and Gateway computers and find P3 800mhz and P4 1.5 ghz chips that are in real good condition......

Maybe I should of explained this better. I'm not hoarding a housefull of old hardware. I walked into a charity shop looking for a zipdrive that I needed for a one time only kind of project. Talking to one of the staff I found out most of the donated computer stuff was ending up in the dumpster. The people running the place didn't know, or care to know, about computers.
I take the stuff home as a hobby. I end up giving most of it away.
I originally posted to explain the value of obsolete hardware. The Goodwill story was just an example.

Yeah, someone suggested that to me, but I don't know how to boot from CD!

You normally just pop it in and reboot.

I went to the Knoppix site and it says you're supposed to burn it as an image??? How will it know to boot from an image? o_O

Knoppix can be purchased from places like Cheapbytes if you don't want to bother about the download and burning.

The idea of downloading bootable media is about the same no matter if it is floppy, CD, or DVD.
Since media has special boot sectors you do not just download files and copy the files to the media, you download an image of the bootable media that is formated correctly to boot and create the new media from the image file. There should be an option to create CD from an image file or some such thing in your CD burning program.

Another fun live CD that is smaller in size is Damn Small Linux.
It has a much smaller set of programs but is also easier to download.

I'd recommend reading this guide to Linux, which I wrote for newbies--it's everything I wish someone had told me when I started five months ago.

If you (or anyone else here) wants the Knoppix 3.7 CD-ROM or the Knoppix 4.0 DVD-ROM, I'd be happy to send one out myself. I can burn a few of them.

Please note, I'm not trying to run a service here, so I may call it quits after a few, but I'm happy to get some people into it.

I'm also more than happy to distribute Gentoo install media.

Basically, I'd call around pawn-shops, charity stores, sellers of used office equipment, and (especially) ask the proprietors at the local independent computer-shops. You'd like to find a spare computer that's two or three years old. Explain that you're buying it to learn with.

If it has a network card, it will have a plug in the back that looks like a telephone cable but it's wider.

You don't want to find something that's too old. Also ask relatives, or see if your folks know someone at work or church or what have you...