Well, the way I read it, it sounds like you should take the unit back to the store and demand your money back as you don't appear qualified to use one of these units.
But, short of that, here is my suggestion:
This unit is not really a mixer, it's more of a dual-source playback unit. Two units crammed into one chassis. My suggestion would be to get a mixer of some sorts to front end the audio outs of each player on the SDJ1 unit. While it's not absolutely necessary, it should make things a bit easier to manager. While it's all at what appears to be -10db via RCA cables, that should be sufficient to drive an amplifier. It would be better to go into a real mixer, even a real DJ mixer, especially one that has balanced XLR outputs, to then front end the amplifier.
Plus, don't forget, you have a "volume/output level" fader for each player as well.
I mean, I don't even know where to begin. Are you using bad speakers? Bad amp? Are we dealing with any voltage issues that need to be addressed? Is the amp turned up? Can you even interface cleanly with the amp?
Sorry, but I don't know where to begin to help you. I'm fairly sure you're doing something wrong. That's all I can say. Without reading the manual, I did download it and looked at the layout and the back and had it 99% figured out as far as basic operations and signal flow are concerned.