I have the following set up:
Imac OS 10.6, Traktor Pro, VMS4 3.11 firmware
When i started i had a very strong signal with the VMS4 in default EQ mode. I tried both out puts and got distortion whenever i tried to use my GAINS OR MASTER. I have a BOSE PA system and it has a light on the powered system to let you know when your green, your good when your red your getting to much signal.
On page 20 of your VMS4 manual it tells you how to switch from EQ default to POST EQ. when i did this my signal dropped so i did not get double the signal and that stopped the distortion. Basically if your using the traktor EQ and the VMS4 in default then your using that EQ as well (double signal strength). Hope this helps out some. The system is frustrating if you don't know what to look for, and would be easier if the firmware, and mappings were correct and available, but don't give up guys!