Heres x264 compression,
http://www.halo3forum.com/montage-vi...-and-xvid.html
Im guessing you capture with studio 9, and by start capture press settings and use custom capture and put the bitrate to around 7900 and that should fix your quality alot. Then in vegas you can enhance it.