(Shamelessly stolen from (and expanding upon) The Apache SSL FAQ)

The private key contains a series of numbers. Two of those numbers form the "public key", the others are part of your "private key". The "public key" bits are also embedded in your Certificate (we get them from your CSR). To check that the public key in your cert matches the public portion of your private key, you need to view the cert and the key and compare the numbers. To view the Certificate and the key run the commands:

$ openssl x509 -noout -text -in server.crt
$ openssl rsa -noout -text -in server.key

The `modulus' and the `public exponent' portions in the key and the Certificate must match. But since the public exponent is usually 65537 and it's bothering comparing long modulus you can use the following approach:

$ openssl x509 -noout -modulus -in server.crt | openssl md5
$ openssl rsa -noout -modulus -in server.key | openssl md5

And then compare these really shorter numbers. With overwhelming probability they will differ if the keys are different. As a one-liner:

$ openssl x509 -noout -modulus -in server.pem | openssl md5 ;\
  openssl rsa -noout -modulus -in server.key | openssl md5

And with auto-magic comparison (If more than one hash is displayed, they don't match):

$ (openssl x509 -noout -modulus -in server.pem | openssl md5 ;\
   openssl rsa -noout -modulus -in server.key | openssl md5) | uniq

BTW, if I want to check to which key or certificate a particular CSR belongs you can compute

$ openssl req -noout -modulus -in server.csr | openssl md5