I just went back to Avenue 3.3.3, and this is how the Auto Mask effect has always worked.
If you like, I can put in a feature request to adjust its behaviour so that toggling Invert will pull the key from white instead of from black. That would alter the current behaviour, so it's not something we can do in a minor point update.
As an alternative, if you drop your png B/W mask directly on the composition/layer/clip to use it as a mask, inverting the mask will knock out the white.
Edit: To invert your mask. you can also achieve what you want by setting your layer to 50 Mask at 100% and using the Invert RGB on or off to switch between the black & white sections of your PNG mask.
you can get the white away by putting an InvertRGB before and one after the automask in the effect chain.
Software developer, Sound Engineer,
Control Your show with ”Enter” - multiple Resolume servers at once - SMPTE/MTC column launch
try for free: http://programs.palffyzoltan.hu
Invert RGB effect before the Auto Mask is what I currently do. Just seems a little redundant having the invert feature on the Auto Mask with it's current behavior. I would say put it in as a feature request.