By Maxwell Adams
Go to file -> capture mode, then device -> screen capture. It will crash if your desktop isn't set to 32-bit color.
Video -> source. Track active window, do not remove unchanged frames.
Video -> set custom video format. Punch in the resolution of the content in the window you are capturing. While tracking the active window, it will automatically crop off title bars, menu bars, and window borders. So you can set your nes emulator to display at 256x224, then set that to your custom format.
You can play with it at this point. The active window will be displayed inside virtualdub. You'll get cool hall of mirror effect while virtual is selected. Check off video -> stretch to window to do that.
Frame rate and audio rates are in the lower right corner. Usually you want to capture at 30 fps unless you think your computer is cool enough to handle 60 fps. Audio should always be 44/16/stereo.
Now go to capture -> timing and make it look like this:
That should make your audio sync up.
Finally, you need compression settings for video and audio. If your computer is awesome, use xvid for video, set up to twopass - 1st pass, full quality first pass that isn't discarded. You can also find the Lame ACM codec and use that to compress the audio in mp3 format.
If you computer isn't so awesome, find the lossless camstudio codec and use that for video, along with uncompressed audio.
I have a Core 2 Duo E6300, and I can capture video from nnnesterj at 256x224, 60 fps, full quality xvid + 160 kbps mp3 audio. The emulator is in a tiny window, but the virtualdub preview is streched out and huge so I have no trouble seeing what I'm doing. It is awesome.
/edit - Oh yeah, after you capture video and then close virtualdub, you'll find that virtualdub.exe is in task manager using %50 cpu. Just end the task.