It's possible to get on the incorrect installation track when installing EndNote on a Mac, which will result in a "Product Key is Invalid" error. The following screenshots are from version X7, but it's been possible from previous versions, and will probably be possible in future versions.
When you double-click the downloaded .dmg, this is what the mounted disk image displays:
At this point, you should drag the EndNote icon as indicated, locate EndNote in the Finder on your Mac, and run it. If you ignore the instructions and double-click the EndNote X7 icon, thereby launching EndNote from the mounted disk image, you will see this:
If you ignore that warning, click "Continue," and enter your key, you will see this:
The solution is to cancel out of the application, drag the EndNote icon to the Applications folder, and then launch it from the Finder.