I've done this in C and it was a bitch. http://alf.hopto.org/misc/wallclock.tgz
is the whole project (it's supposed to read an anologue clock with a webcam and tell you the time, and it's got a crazy client/server structure... school assignment)
It works for the two cameras I tested it with (and there's code specific to each).
The relevant files are c/camera.[ch], c/YUV420PtoRGB.[ch], and c/RGBtoRGBA.[ch] .
There's also an example video4linux program we copied from somewhere in c/capture_image.c.
video4linux is the API you use to get images from a webcam with linux. It gives you different formats depending on the camera and you have to look up some cracked out way to decode them. That's what's in the YUV420PtoRGB and RGBtoRGBA files. It's possible that YUV420PtoRGB should have been named YUV420PtoRGBA; been a while since I did the project but I remember something like that.
Conclusion: for the love of god use somebody's code who's already done it right. Not mine either because it probably won't work for many different cameras. I'm hoping mine *is* slightly simpler so you can get the idea.