if you are using resolume avenue, the Avenue specific codec is the best to use:
get it here:
http://resolume.com/download/
if you are using resolume 2 then your best bet is to buy Picvideo -(
http://resolume.com/shop/codec.php ) - and use the quality settings outline in the resolume manual. it really is the best universal codec for VJing, and is well worth the investment.
if your computer is bogged down, drop the output size to 640x480 or 400x300 even 320x240 if you want to work with lots of layers on older hardware.... run a trial on a couple clips before you start converting all your files.