Sound output from laptop, via 3.5 output to ddj-400

To the DDJ-400? No, as it doesn't have an audio input.

if I use it as a sound card and listen to music through the rca output, but from the headphone output I cannot. in my previous traktor 2 mk3 controller, this was possible, since 2 audio outputs were reflected in the windows menu at once.

rekordbox doesn't allow for 2 audio devices to be selected, but you CAN use "PC Master Output" mode to have the master audio sent to the Windows default audio device. Please check your Windows sound settings first, then enable the PC Master Output in rekordbox.

And if without recordnox, can the controller be used as a sound card so that the sound goes out to the headphone jack?

If you mean the headphone jack on the DDJ, no - since most applications will only address the master output.

It's a shame, then I will have to return the goods to the store and buy a Tractor.