Capturing/Editing from DV camcorder

johnsfine · 06-10-2009, 10:17 AM

I finally tried capturing from my DV camera in Linux (Mepis 8.0) for the first time. I was amazed at how easy it was.

For years I have occasionally captured video from DV to Windows NT, 2000 and XP. It is always an unbelievably frustrating process. I have used capture/edit software bundled with a firewire interface I bought, bundled with a Sony DV camera, bundled with a JVC DV camera, bundled with a Micron computer and bundled with a Dell computer. It is always a nightmare to set up the drivers to be compatible with the firewire interface. It is always a nightmare to set up more drivers to be compatible with the camera. It is always a nightmare to dig through the web site to try to download current software because the included CD (in every one of those bundles) was obsolete when shipped. Most of the bundled software is based on some version of ULead, which has about the most counterintuitive GUI on the planet. It is always a nightmare to figure out the actions needed to make ULead actually capture something or to diagnose what I did wrong when it doesn't. Usually I need to press play on the camera at the right moment in coordination with telling ULead to start capturing. Sometimes ULead controls the camera and starts it playing. I never know which to expect. By the time I'm done capturing, I have no patience left for the nightmare GUI of editing in ULead. Then I don't touch video capture for many months and when I try again, I never understand where all that installed software went and why I need to start all over from the CDs and web sites. Obviously other people don't have so much trouble. I guess I'm just jinxed (multiple cameras, multiple computers, multiple Windows versions, multiple capture/edit programs, over a period of years. So I didn't just have some lemon hardware or software).

My wife wanted me to capture a video on short notice from our camera. I assumed I didn't have time to learn how on Linux from scratch. So I searched all our Windows computers for left over installs of video capture software: no luck. So I searched shelves and drawers for CD's of some of that bundled software, found one, installed it. So I searched web sites for updates to that obsolete software. No luck. So I tried to use it. I just couldn't get it to recognize the camera.

So I had to start from scratch with Linux. A quick google search gave me a few keywords including "dvgrab" to put into searches in Synaptic. That identified several packages I might need. I wasn't selective and didn't take notes (sorry) I just marked whatever looked like I might need it.

Then I tried the man pages for dvgrab: too much dry detail, too little explanation, just like most man pages. So I gave up quickly and just typed the example command
dvgrab foo-
to see if the error messages might help me figure out the hard stuff (configuring for the specific firewire interface, configuring for the specific camera, getting the rest of the command syntax right, etc.)
It immediately worked. It started the camera playing and it captured a sequence of .dv files 1GB each.

I transferred one file to Windows where my wife wanted it, and found the .dv format might be a problem. I opened the .dv file in Linux, which brought up kino. Then I failed in my attempts to get kino to export to mpeg. It wants "mjpegtools" and I can't find that in Syntaptic and haven't done a broader search yet.

With one detail I did understand from the dvgrab man page, I re captured the same video as .mov files. That fits even less video time into each 1GB file than the .dv, but it seems a little more compatible after transfer to Windows.

I also tried a little basic editing with kino. I think it is a very confusing and lame UI, but it is much easier than ULead.

I don't know if I'm using the best tools (dvgrab and kino). I'm surprised the package(s) needed for mpeg aren't in the repositories that I've told Synaptic about (very little that ordinary users use is missing). I'd appreciate some advice on making this even easier and finding the missing pieces.

Especially, what format and how do I get to it, to store these videos without a big loss in detail but with a big increase in the number of minutes per GB. I don't care whether I needs loads of disk space for the capture. But I'd like to make it manageable before sending the result to another computer.

But if any beginners have read this far, my main message is this stuff is much easier than I expected after years of frustration with all those bundled capture/edit packages in Windows.

H_TeXMeX_H · 06-11-2009, 02:36 PM

Well, using dvgrab is pretty straightforward, and the man page tells you all the options available, and there's not much to it. The only really important option is '-format', which I usually set to 'dv2' or 'mpeg2' depending on the camera.

Then would be the time to edit the video if you really have to. kino is actually ok, but I think lives is better:
http://lives.sourceforge.net/

Then, what I would do is re-encode the video to compress it with minimal loss of quality. For this you can use avidemux if you prefer GUI, but I usually use ffmpeg (or mencoder). For the absolute best compression I would use variable bitrate (constant quantizer) and 2 pass encoding. Probably the best is to use .mkv format, x264 video codec, and aac audio codec. This really preserves quality the best and manages to get superb compression ratios. But it depends on what you want to do with these, if you wanted to watch them on a divx capable DVD player I would instead use .avi with Xvid and mp3 ... which is good too, but not as good.

johnsfine · 06-11-2009, 03:39 PM

Quote:

Originally Posted by H_TeXMeX_H

The only really important option is '-format', which I usually set to 'dv2' or 'mpeg2' depending on the camera.

I obviously misunderstood the options about when/how to split the output file. But what it did was OK. I assumed other options I didn't understand were important. I don't understand much about the relationship between the various formats.

Quote:

kino is actually ok, but I think lives is better:
http://lives.sourceforge.net/

I'll try it. I just now skimmed through its documentation which is much more confusing than the kino documentation, which was pretty confusing.

I don't know if I'll ever want to do more than the most basic home video edit, which is identify and cut out the bad parts. That's much harder than it ought to be in kino, but in the Windows programs I've tried, it's even harder than that.

That LiVES website, was yet another thing telling me I need to add debian-multimedia to my list of repositories in Synaptic. Nothing I found has quite told me exactly what to add for that. I haven't had time to try yet. I don't think it will be too hard to figure out once I have the repository list in Synaptic open to try it. (I'm not at that computer now).

Quote:

you can use avidemux if you prefer GUI, but I usually use ffmpeg (or mencoder).

More things to install and try.

Quote:

For the absolute best compression I would use variable bitrate (constant quantizer) and 2 pass encoding. Probably the best is to use .mkv format, x264 video codec, and aac audio codec.

I have no clue what you are talking about. This stuff is more complicated than I am likely to have time to learn.

Quote:

But it depends on what you want to do with these

Some I want to send to various people. Others I want to just archive onto DVD media, but to be read back and played by computer, not by DVD player.

H_TeXMeX_H · 06-11-2009, 04:09 PM

Quote:

Originally Posted by johnsfine

I don't know if I'll ever want to do more than the most basic home video edit, which is identify and cut out the bad parts. That's much harder than it ought to be in kino, but in the Windows programs I've tried, it's even harder than that.

If cutting out bad parts is all you need then you do NOT need kino or lives. In this case I would encode to mpeg2 then just view it with a media player (mplayer) and use an encoder to cut it up (both mplayer and ffmpeg have such options), then you can join the resulting file with just 'cat', so concatenate the file directly. That's the good thing about mpeg2. The less good thing is that it takes up more space, but that won't matter if you re-encode it.

Quote:

Originally Posted by johnsfine

I have no clue what you are talking about. This stuff is more complicated than I am likely to have time to learn.

Some I want to send to various people. Others I want to just archive onto DVD media, but to be read back and played by computer, not by DVD player.

Well, you may want to read up a bit on it, it's not that difficult to understand, there's a video file format, which can be things like .avi .mkv .wmv .mp4 .mov and so on, and these formats "contain" codecs, one for video (for example Divx Xvid x264 mpeg4) and one for audio (mp3 aac ac3). Both codecs and format determine level of possible compression, for example .mkv supports variable bitrate for both audio and video, and it supports more codecs than say .avi (which only supports a few and just barely).

Or here's another explanation:
http://www.shallowsky.com/linux/videoformats.html

Anyway, there are many many codecs and formats to choose from, but the one I recommended is used for ripping blu-ray and such, and maintains a lot of quality with a high compression rate. Soon, hopefully they'll come out with players that are Divx 7 compliant and will play .mkv ... that would be very nice.

johnsfine · 06-13-2009, 07:55 AM

"debian-multimedia" is already in the repository list in Mepis. I just needed to enable it (I had been worried I didn't know exactly what to type in for its location).

After doing that, I installed every package that even seemed to be related to things mentioned in this thread. But I'm still missing a little software and a lot of comprehension.

Quote:

Originally Posted by H_TeXMeX_H

you can use avidemux if you prefer GUI

Tried it, but I couldn't figure out how to make it open any movie files. It just pops up a big window full of error messages.

Quote:

I usually use ffmpeg

I forget what went wrong when I tried that.

Quote:

or mencoder

Unbelievably brutal man page, but I'm managing to do a little with it.

Quote:

I would use variable bitrate (constant quantizer) and 2 pass encoding.

Using x264 and two pass, I couldn't figure out from documentation or from experimentation what constant quantizer would mean in two pass mode. The man page says you can't specify the quantizer for the second pass and mist specify the bitrate. In fact it let me specify either and I couldn't tell from any of the results what I was accomplishing.

Quote:

Probably the best is to use .mkv format

I installed everything that looked like it might be needed for .mkv format. But never got memcoder to be aware of .mkv. I'm I supposed to be able to generate .mkv with mencoder?

Quote:

x264 video codec, and aac audio codec.

I trusted your advice and so far experimented just with those. Even without varying the choice of codec, there are far too many choices for me to understand.

I haven't yet tried copying any results to Windows to try to view there. If I send results to people who just have Windows, what do they need to do to use x264 and aac?

Quote:

This really preserves quality the best and manages to get superb compression ratios.

I don't have a good eye for detail, so I can't really tell. Also the initial quality as captured from the camcorder is probably not very good. I tried a high quality (low quantization if I remember correctly) setting and got about 260MB for a 5 minute video (vs. 1GB original capture) and I couldn't see the difference. Then I tried a low (I think) bit rate and got 46MB for 5 minutes and I still couldn't see the difference.

When I really looked, I saw a lot of what I thought were compression defects, but on closer comparison:
A) Most were really interlacing problems and the same in all versions.
B) The rest were just as bad (though I think different) in the original capture as in the compressed versions.

While looking for other things in the documentation, I saw various things about processing to de interlace the video. Of course I didn't understand any of that.

IIUC, my camcorder produces only interlaced videos so alternate lines are half a frame time off. That is needed to view it correctly on a TV, but on a PC it looks wrong. When the camera or the subject moves, the motion in odd scan lines is half a frame time behind the motion in the even. Have I got that roughly right?

I really didn't understand my choices for what to do about interlacing. Change something in dvgrab when I capture the video? Change something in Kino when I slice out the bad parts? Change something in mencoder when I recompress?

H_TeXMeX_H · 06-13-2009, 08:33 AM

Hmm, well interlacing is a problem. If you deinterlace you will loose about 50% quality:
http://www.mplayerhq.hu/DOCS/HTML-si...ing-interlaced
so it may be best not to deinterlace, but you should use the options listed there instead, and take note of the suggestions.

Technically dvgrab can deinterlace, but it will also produce 50% quality reduction or more.

For mencoder see here for better documentation than the man page:
http://www.mplayerhq.hu/DOCS/HTML-si...encoding-guide

For example for tips on encoding with x264:
http://www.mplayerhq.hu/DOCS/HTML-si...menc-feat-x264

avidemux can be buggy, so if you can use a pre-built binary for your distro, it may be a good idea.

not sure if mencoder can produce .mkv (I know ffmpeg can), but you can use also mkvmerge:
http://www.mplayerhq.hu/DOCS/HTML-si...uxing-matroska

For Window$, you can still use mplayer and vlc, but any decent player with the right codecs should be able to play mkv and x264.

If you have problems installing one of the transcoders like avidemux or ffmpeg, maybe post some of the errors, it may be something trivial.

johnsfine · 06-14-2009, 08:57 PM

Quote:

Originally Posted by H_TeXMeX_H

Probably the best is to use .mkv format, x264 video codec, and aac audio codec. This really preserves quality the best and manages to get superb compression ratios.

I'm still experimenting. But regarding aac audio, did I do it wrong or misunderstand you or what?

The video I'm working with has voice (not music nor other audio content that needs a lot of fidelity). I was testing some very high compression to make a long video practical to send over the net, not worrying about pretty low quality.

The "aac" codec I found was "faac". Is that correct or is it the wrong codec? That didn't compress much by default (so once the video was very compressed, the audio was a large fraction), so I experimented with the documented br=<bitrate> parameter of -faacopts. That made little if any difference, even setting it absurdly low. The resulting compressed audio stayed over 90kb/sec. So I tried a low value for quality (documented as 1-1000). Moderately low values didn't help (didn't reduce the result audio size nor noticeably change the sound). When I got down to quality=10, the result audio size finally went down, but still over 40kb/s, but the voices were massively distorted.

After some reading, I tried mp3lame instead. The documented br parameter behaved almost as expected through moderate values, but at very low values, it didn't go all the way to super compressed and distorted. Maybe there is some minimum. I got the compressed result audio size down to 34kb/s without significant distortion with: -oac mp3lame -lameopts abr:br=10:mode=3
I didn't try enough things to know if there are better ways and/or whether 34kb/s is very compressed for voice. It seems like the info content of voice should be a lot less than 34kb/s, especially in comparison to the video compression.

Meanwhile a small amount of blind experimentation (few slow tests on large files) got me a decent result for extreme (I think) video compression with visible but not horrible distortion:
mencoder -ovc x264 -x264encopts bitrate=300:pass=2:nr=200:subq=6:direct_pred=auto -oac mp3lame -lameopts abr:br=5:mode=3 infile.dv -o outfile.avi
That was done after a somewhat compatible first pass. I don't know if a more compatible first pass would squeeze out a bit more quality, but the first pass was a previous experiment as well as a first pass.

The edited file (infile.dv) was still in the same compression as the original capture and was 7.7GB. The compressed output outfile.dvi was 95.1MB

On my son's Windows system (which I used just to test), Windows Medial player could not play the result, but he had VLC media player configured as his default and it had no problem playing this file. If necessary, I think I can tell the people who get this file to use VLC.

Feel free to suggest improvements in any aspect of what I'm doing.

I'm still even more lost on the other goal, which is getting other captures compressed much less (but still a significant amount) with hopefully near zero loss of quality to archive on DVD only partially edited with hopes of editing more later. I have no eye for detail, so I can't judge at all whether moderate compression is changing the result.

H_TeXMeX_H · 06-15-2009, 03:41 AM

That seems like very good compression, I mean 7.7 GB to 95 MB ... wouldn't you say it's pretty good.

Yes, faac is the free aac encoder, and it usually does a pretty good job. If you're encoding to .avi (which you are), then I would recommend using mp3 audio instead (libmp3lame), it's more compatible. Although x264 usually has better compression than Xvid, you can also try using xvid (libxvid).

One thing that would help me make recommendations would be to run the 'infile.dv' in mplayer in a terminal and post the output here, it should have more info on the various parameters, like resolution, bitrates, etc.

johnsfine · 06-15-2009, 11:24 AM

Quote:

Originally Posted by H_TeXMeX_H

That seems like very good compression, I mean 7.7 GB to 95 MB ... wouldn't you say it's pretty good.

I think the video compression is amazing. The result pretends to be 720*480*29.97 = 10,357,632 pixels per second, but uses only 299,800 bits per second for video. So it gets 34.5 pixels out of each bit.

The audio compression is much less amazing. It is decent with mp3lame, but certainly not impressive. With faac it was much much worse (much less compression with much more distortion). So I was wondering whether I misused faac or whether it isn't good for voice, or what?

Quote:

One thing that would help me make recommendations would be to run the 'infile.dv' in mplayer in a terminal and post the output here, it should have more info on the various parameters, like resolution, bitrates, etc.

I hope you meant start playing it and then abort. That seems to give the sort of info you might want:

Code:

MPlayer 1.0rc2-4.3.2-DFSG-free (C) 2000-2007 MPlayer Team
CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 6400+ (Family: 15, Model: 67, Stepping: 3)
CPUflags:  MMX: 1 MMX2: 1 3DNow: 1 3DNow2: 1 SSE: 1 SSE2: 1
Compiled with runtime CPU detection.
mplayer: could not connect to socket
mplayer: No such file or directory
Failed to open LIRC support. You will not be able to use your remote control.

Playing bigger008.dv.
libavformat file format detected.
[lavf] Video stream found, -vid 0
[lavf] Audio stream found, -aid 1
VIDEO:  [dvsd]  720x480  0bpp  29.970 fps  28771.2 kbps (3512.1 kbyte/s)
==========================================================================
Requested video codec family [qdv] (vfm=dshow) not available.
Enable it at compilation.
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
Selected video codec: [ffdv] vfm: ffmpeg (FFmpeg DV decoder)
==========================================================================
==========================================================================
Opening audio decoder: [pcm] Uncompressed PCM audio decoder
AUDIO: 32000 Hz, 2 ch, s16le, 1024.0 kbit/100.00% (ratio: 128000->128000)
Selected audio codec: [pcm] afm: pcm (Uncompressed PCM)
==========================================================================
AO: [alsa] 48000Hz 2ch s16le (2 bytes per sample)
Starting playback...
VDec: vo config request - 720 x 480 (preferred colorspace: Planar 411P)
Could not find matching colorspace - retrying with -vf scale...
Opening video filter: [scale]
VDec: using Planar 411P as output csp (no 0)
Movie-Aspect is 1.36:1 - prescaling to correct movie aspect.
SwScaler: reducing / aligning filtersize 1 -> 4
SwScaler: reducing / aligning filtersize 5 -> 4
SwScaler: reducing / aligning filtersize 1 -> 1
SwScaler: reducing / aligning filtersize 9 -> 8
[swscaler @ 0x7fbe36fe79d0]SwScaler: BICUBIC scaler, from yuv411p to yuv420p using MMX
[swscaler @ 0x7fbe36fe79d0]SwScaler: using 4-tap MMX scaler for horizontal luminance scaling
[swscaler @ 0x7fbe36fe79d0]SwScaler: using 4-tap MMX scaler for horizontal chrominance scaling
[swscaler @ 0x7fbe36fe79d0]SwScaler: using 1-tap MMX "scaler" for vertical scaling (YV12 like)
[swscaler @ 0x7fbe36fe79d0]SwScaler: 720x480 -> 720x480
VO: [xv] 720x480 => 720x528 Planar YV12
A:   2.6 V:   1.8 A-V:  0.774 ct:  0.002  56/ 56 35% 23% 81.2% 50 0

           ************************************************
           **** Your system is too SLOW to play this!  ****
           ************************************************

Possible reasons, problems, workarounds:
- Most common: broken/buggy _audio_ driver
  - Try -ao sdl or use the OSS emulation of ALSA.
  - Experiment with different values for -autosync, 30 is a good start.
- Slow video output
  - Try a different -vo driver (-vo help for a list) or try -framedrop!
- Slow CPU
  - Don't try to play a big DVD/DivX on a slow CPU! Try some of the lavdopts,
    e.g. -vfm ffmpeg -lavdopts lowres=1:fast:skiploopfilter=all.
- Broken file
  - Try various combinations of -nobps -ni -forceidx -mc 0.
- Slow media (NFS/SMB mounts, DVD, VCD etc)
  - Try -cache 8192.
- Are you using -cache to play a non-interleaved AVI file?
  - Try -nocache.
Read DOCS/HTML/en/video.html for tuning/speedup tips.
If none of this helps you, read DOCS/HTML/en/bugreports.html.

A:   5.7 V:   5.7 A-V:  0.000 ct:  0.001 172/172 22% 15% 26.9% 81 0
Exiting... (Quit)

Compare that to the compressed file:

Code:

MPlayer 1.0rc2-4.3.2-DFSG-free (C) 2000-2007 MPlayer Team
CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 6400+ (Family: 15, Model: 67, Stepping: 3)
CPUflags:  MMX: 1 MMX2: 1 3DNow: 1 3DNow2: 1 SSE: 1 SSE2: 1
Compiled with runtime CPU detection.
mplayer: could not connect to socket
mplayer: No such file or directory
Failed to open LIRC support. You will not be able to use your remote control.

Playing shrunk6.avi.
AVI file format detected.
[aviheader] Video stream found, -vid 0
[aviheader] Audio stream found, -aid 1
VIDEO:  [h264]  720x480  24bpp  29.970 fps  299.8 kbps (36.6 kbyte/s)
Clip info:
 Software: MEncoder dev-SVN-r26940
==========================================================================
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
Selected video codec: [ffh264] vfm: ffmpeg (FFmpeg H.264)
==========================================================================
==========================================================================
Opening audio decoder: [mp3lib] MPEG layer-2, layer-3
AUDIO: 32000 Hz, 2 ch, s16le, 48.0 kbit/4.69% (ratio: 6000->128000)
Selected audio codec: [mp3] afm: mp3lib (mp3lib MPEG layer-2, layer-3)
==========================================================================
AO: [alsa] 48000Hz 2ch s16le (2 bytes per sample)
Starting playback...
VDec: vo config request - 720 x 480 (preferred colorspace: Planar YV12)
VDec: using Planar YV12 as output csp (no 0)
Movie-Aspect is 1.36:1 - prescaling to correct movie aspect.
VO: [xv] 720x480 => 720x528 Planar YV12
A:  11.1 V:  11.1 A-V: -0.001 ct: -0.011 335/335 26%  6%  0.9% 0 0
Exiting... (Quit)

As long as you asked to see the above, can you explain some of it:

Code:

Requested video codec family [qdv] (vfm=dshow) not available.
Enable it at compilation.
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
Selected video codec: [ffdv] vfm: ffmpeg (FFmpeg DV decoder)

It doesn't have the codec for the standard dv camcorder compression? But it opens it anyway??

Code:

A:   2.6 V:   1.8 A-V:  0.774 ct:  0.002  56/ 56 35% 23% 81.2% 50 0

There is a lot of stuff that looks like that while playing or recompressing etc. Is all that documented somewhere? Can I learn something from it?

After compression:

Code:

AUDIO: 32000 Hz, 2 ch, s16le, 48.0 kbit/4.69% (ratio: 6000->128000)

The camcorder only has one microphone. I guess some standard forced the data to be "stereo". The mode=3 in the compress command says to make it "mono" (if I understood the doc right). Without mode=3 it doesn't compress as well. Yet it still says "2 ch". Is mode=3 OK (or will it cause problems later)? What does it mean when it says "2 ch" for data that was one microphone to begin with and was compressed "mono".

Code:

48.0 kbit/4.69%

When I see the numbers on the video kbps, they are consistent: What I ask for (as a bitrate) roughly matches what mencoder reports it has done, which matches what MPlayer reports when playing back the result, which matches the size change in the actual file.

For Audio kbps none of those things are consistent. The bitrate I ask for has very indirect effect on the final kbps Mencoder reports, which doesn't match what Mplayer reports when playing the result, and none of it matches the size in the actual file (deduced incrementally when recompressing with no video change and some audio change).

H_TeXMeX_H · 06-16-2009, 07:03 AM

A few things that might help compression:

video:
Use 'ofps=23.976', to change the fps to lower, but acceptable value.
Use 'interlaced' option.
Use 'bframes=2' option, this turns on b-frams, this will allow for even more compression.
Use 'keyint=300' option, this is good thing to set, it's usually 10 x input FPS (in this case around 30).
Use 'crf=' INSTEAD of 'bitrate=', this enables variable bitrate, setting lower values will increase quality and size, setting higher values will decrease quality and size, values between 1.0 and 50.0, usually 5 - 20 works well.
Use 'subq=5'
Possibly try 'trellis=1' option, remove if you notice artifacts.
If the video is noisy try 'nr=' between 100 and 1000, it will denoise the input video, and thus may safe space and improve quality.
Use 'pass=' option for 2-pass encoding, 1 on the first pass, 2 on the second.
You could also try resizing the video, but it may be difficult with interlaced video.

audio:
I would try encoding using variable bitrate, which should reduce size, but I'm not sure the avi format will support it. If it does not, try the .mp4 or .mkv.
Use 'mode=3' for mono output, this should cut the size in half.
Use 'q=' option INSTEAD of 'br=', this invokes variable bitrate, 1 is best quality largest size, 9 is worst but smallest size. This option will likely not work with avi output, instead try mp4 or mkv.
Use 'aq=0' option, this should maintain quality.

Quote:

Requested video codec family [qdv] (vfm=dshow) not available.

probably the directshow codec, it never works properly anyway, so don't worry about it.

The answer to many of your questions is in the mplayer FAQ, too bad it is down right now

http://www.mplayerhq.hu/DOCS/HTML/en/faq.html#id2539071