Target Framework Mismatch

When adding a reference to a project I sometimes see the message...

"The Target Framework version for the project is higher than the current project Target Framework version. Would you like to add this reference to your project anyway?"

I understand what the message is indicating but can anyone explain what issues might arise from saying 'Yes'?

How do I check my target framework?

To verify the change, on the menu bar, select Project > Properties to open your project Property Pages dialog box. In the dialog box, select the Configuration Properties > General property page. Verify that . NET Target Framework Version shows the new Framework version.

Suppose you build an application where the exe targets 2.0, but some library targets 3.5. Someone with a .NET 2.0 machine looks at the main project and thinks they can run it - after all, the application targets 2.0, right? When the library is first loaded, they'll get a nasty surprise...