High res PNG files and/or lots of PNGs in a deck will do it so my advice to you would be to render all your PNGs as 1 frame (or very short) DXV files.
@ HI5, 8x40 clips per deck is a lot especially with high res files so why don't you try splitting your comp into more decks. Keep in mind that you can switch decks without changing your output so you just need to plan your deck changes in your set to allow enough time to load up all the clips. I typically trigger mine using OSC and time the changes for when I have a longer Q. Whenever possible, I also wait for all my transitions/effects to be done before triggering the deck change.
Also, Resolume scales images quite nicely so you must be zooming your masks a lot, right? As a reminder, if you scale your files 150% in both directions, you end up with a file that 2.25 times the size of your composition so each one of your PNG masks ends uses a lot of resources.
@ Boscobenedini, are you using your 2x Fire Pro D700 for outputs?