I now have the same problems with clips that were rendered in previous versions of the DXV codec. But it only appears if the clip has effects applied that themselves generate an alpha channel (e.g. particles).
In the picture below you can see 2 clips playing and one clip on top of them @ "50 Mask". You can clearly see that those portions off the "cubic clip" that stay on top of the video clip are rendered correctly. Those that are outside have this strange look as if you render a straight alpha and play it inside a player that can't handle it e.g. QuickTime.
Info: I updated QuickTime today and it didn't help.
I don't have the DXV 2.1 installer anymore so I can't test that (can you share it?).
It also happens in previous versions of Avenue.
*edit* It affects only soft edges in alpha channel. For example blurs and trails, trails in particles, etc.