Borderless NSButton turns gray when clicked

Question

Make your button type as NSMomentaryChangeButton.

[myBtn setButtonType:NSMomentaryChangeButton];

If you use NSMomentaryPushInButton then you may get gray rectangle over the button when click.

You should be able to adjust this by changing the state mask. To do so, check out the highlightsBy: and showsStateBy: properties of NSButtonCell - they are for setting the press effect and displaying the effect respectively. The applicable constants for these methods are in the NSCell reference page.