If the controller is behaving erratically on different computers with different cables and different software applications, then things do point to the controller being the source of the problems, but just to be sure, let's try some signal troubleshooting...
Download this app for your Mac.
With your SB3 connected and the app running, configure your environment as such:

And open the preferences to change your settings to match:

Next, press these buttons in this order:
- CH1 LOAD
- CH1 HOT CUE
- CH1 FX 1
- CH1 CUE
- CH1 PLAY
You should see these values appear in the log window:

Repeat the process with the same buttons on CH2:
- CH2 LOAD
- CH2 HOT CUE
- CH2 FX 1
- CH2 CUE
- CH2 PLAY
You should see these values:

If your values do not match those shown above, then there's likely to be a hardware problem, but we can try narrowing things down by resetting the MIDI device within the OS. First, quit MIDI Monitor, then open the Audio MIDI Setup app, and open the MIDI Studio window from the menu:

Click the drop-down at the top of the window and select "New Configuration..." - give it a name and click OK.
You should see your DDJ-SB3 there, and nothing else (except maybe the IAC Driver):

Open the MIDI Monitor app again and see if the values match what I posted above. If they do, please test using your DJ software again and let me know how things go.