The connector for S/PDIF is a standard phono (RCA) type connection but the cable itself is 75ohm. Are you sure you're using the right cable? Can you maybe take a photo?
As for recording levels, check this video around the 1:20 mark for adjusting the digital output levels.