Hmmm... it's been a while since I've moved a rekordbox database from Windows to Mac without going through the backup / restore process or moving to an external drive, but I think you should be able to do that. You may need to jump through a few hoops though with relocation once it's up and running.
Are you on rekordbox 5 or 6?