Official comment
There's a bit of a quirk in that if you have a song that doesn't have at least one tag, the inverse tag filtering doesn't work properly. Create a tag and call it "In Collection" then select ALL the songs in your library and add that tag. It should then work as expected.