@W.A.R.Z.: The VMS4 is set up for external mixer control. Internal mixer control won't work because then the analog source cueing and master mixdown take place inside the VMS4 which is the "external mixer". So for example with Internal mixer control, if a USB/Analog channel were set to "analog" there would be no way for the VMS4 to "separate out" the USB audio for that channel.
Before powering on the VMS4, set the VMS4 to 8 out mode. Then in Traktor software, make sure you assign the left/right channels for each of the 4 sources to the 8 channels used by the VMS4. I would suggest you open up the ADJ ASIO driver to make sure the assigned Traktor channels match the channels for the VMS4. The default ADJ ASIO channels should be in order - 1/2 for ch1, 3/4 for ch2, 5/6 for ch3, 7/8 for ch4.
You are probably getting distortion in external mode because you have the gains set way too high either in Traktor software or on the VMS4. If you set these right there is no distortion. I'm not sure what "autogain" does but whatever you do don't push the gains too high or the audio going into the VMS4 will already be distorted by Traktor.
The other thing you need to watch out for is the Post vs. Pre-EQ mode for the channels. If you want to assign the EQ knobs to software please make sure you set the VMS4 to Post-EQ mode so that the VMS4 internal EQs don't double up on the EQs which can cause severe distortion. Otherwise you can just leave the default Pre-EQ setting on the VMS4. Please refer to the VMS4 owners manual which explains this.