Delphi - How do you generate an event when a user clicks outside modal dialog?

Question

Is it possible to fire an event when the user clicks outside a modal dialog?

OK, Windows provides it's own clues when you do this by making the "bonk" sound, or by flashing the app's taskbar button, but I would like to provide some sort of additional clue for situations where sound is not available and/or the user doesn't recognise the reason for the taskbar flashing. Also, I would like to try using this as a way to bring the modal dialog to the front if it has become hidden behind the main form.

Answer 1

First, to answer the question:

You could capture the mouse when it moves outside the dialog, or when is already outside the dialog at showing. Then you can catch WM_CAPTURECHANGED to fire an OnMouseClickOutside event:

type
  TDialog = class(TForm)
  private
    FMouseInDialog: Boolean;
    FOnMouseClickOutside: TNotifyEvent;
    procedure WMCaptureChanged(var Message: TMessage);
      message WM_CAPTURECHANGED;
    procedure CMMouseLeave(var Message: TMessage); message CM_MOUSELEAVE;
    procedure CMMouseEnter(var Message: TMessage); message CM_MOUSEENTER;
  protected
    procedure DoShow; override;
  public
    property OnMouseClickOutside: TNotifyEvent read FOnMouseClickOutside
      write FOnMouseClickOutside;
  end;

...

procedure TDialog.CMMouseLeave(var Message: TMessage);
begin
  // CM_MOUSELEAVE is also send to the dialog when the mouse enters a control that
  // is within the dialog:
  if not PtInRect(BoundsRect, Mouse.CursorPos) then
  begin
    // Now the mouse is really outside the dialog. Start capturing it:
    MouseCapture := True;
    FMouseInDialog := False;
  end;
  inherited;
end;

procedure TDialog.CMMouseEnter(var Message: TMessage);
begin
  FMouseInDialog := True;
  // Only release capture when it had, otherwise it might affect another control:
  if MouseCapture then
    MouseCapture := False;
  inherited;
end;

procedure TDialog.DoShow;
begin
  inherited DoShow;
  // When mouse is outside the dialog when it should become visible, CM_MOUSELEAVE
  // isn't send because the mouse hasn't been inside yet. So also capture mouse
  // when the dialog is shown:
  MouseCapture := True;
end;

procedure TDialog.WMCaptureChanged(var Message: TMessage);
begin
 // When the dialog loses mouse capture and the mouse is outside the dialog, fire:
 if (not FMouseInDialog) and Assigned(FOnMouseClickOutside) then
    FOnMouseClickOutside(Self);
  inherited;
end;

This works. For both visible and obfuscated dialogs. But as David gratefully commented, this has consequences for controls which depend on mouse capture. There are not many that I know of and most controls like a memo or a menu bar will function normally. But take a combo box: when a combo box is dropped down, the list box captures the mouse. When it loses the mouse, the list is wrapped up. So when your users move the mouse outside the dialog (note that the dropped down list may bé outside the dialog), the combo box will exhibit non-default behaviour.

Secondly, to address the real problem a little more:

Furthermore, the question states specifically the need for this event in case of an hidden dialog. Well, the above mouse leaving and entering code depends on the dialog being visible, so let's forget about all of that, get rid of the drawbacks and reduce the code to:

type
  TDialog = class(TForm)
  private
    FOnMouseClickOutside: TNotifyEvent;
    procedure WMCaptureChanged(var Message: TMessage);
      message WM_CAPTURECHANGED;
  protected
    procedure DoShow; override;
  public
    property OnMouseClickOutside: TNotifyEvent read FOnMouseClickOutside
      write FOnMouseClickOutside;
  end;

...

procedure TDialog.DoShow;
begin
  inherited DoShow;
  MouseCapture := True;
end;

procedure TDialog.WMCaptureChanged(var Message: TMessage);
begin
  if Assigned(FOnMouseClickOutside) then
    FOnMouseClickOutside(Self);
  inherited;
end;

Now, what to do if the event fires? The dialog is still hidden, and a call to BringToFront does not work. (Trust me, I have tested it, although is was pretty nasty to reproduce a hidden dialog). What you should do is bring the dialog above all other windows with SetWindowPos:

procedure TAnyForm.MouseClickOutsideDialog(Sender: TObject);
begin
  if Sender is TDialog then
    SetWindowPos(TWinControl(Sender).Handle, HWND_TOPMOST, 0, 0, 0, 0,
      SWP_NOMOVE or SWP_NOSIZE or SWP_NOACTIVATE or SWP_NOOWNERZORDER);
end;

But since a dialog should always be shown on top of all others, you could rather eliminate the event completely and modify the code to:

type
  TDialog = class(TForm)
  private
    procedure CMShowingChanged(var Message: TMessage);
      message CM_SHOWINGCHANGED;
  end;

...

procedure TDialog.CMShowingChanged(var Message: TMessage);
begin
  if Showing then
    SetWindowPos(Handle, HWND_TOPMOST, 0, 0, 0, 0, SWP_NOMOVE or SWP_NOSIZE
      or SWP_NOACTIVATE or SWP_NOOWNERZORDER);
  inherited;
end;

In conclusion:

Now, this still does not work for message or system dialogs (although you could use these nice dialogs which do), and I have to agree with David to find out why the modal dialog becomes obfuscated. If you have forms with FormStyle = fsStayOnTop (or any window with HWND_TOPMOST as Z-order), then you shóuld use the following appropriate application methods to temporary compensate for these windows:

procedure TAnyForm.Button1Click(Sender: TObject);
var
  Dialog: TDialog;
begin
  Application.NormalizeAllTopMosts;
  Dialog := TDialog.Create(Application);
  try
    Dialog.ShowModal;
  finally
    Dialog.Free;
    Application.RestoreTopMosts;
  end;
end;

In all other cases, the disappearance of a modal dialog indicates that you are doing something out of the ordinary that probably cannot be handled by the VCL.

Answer 2

What you ask for is not easy to achieve. I created a simple project with two forms, a main form and a modal form. I then traced the messages (using Spy++) sent to each form when the main form was clicked whilst the modal form was active. Remember that the main form is disabled as part of the protocol for showing modal forms. This means that Windows knows that the main form cannot receive focus and the window manager doesn't forward the click on to either form. The messages that are sent are in order to perform the blinking effect of the modal form.

Modal form messages

S WM_WINDOWPOSCHANGING lpwp:0018EDA8
R WM_WINDOWPOSCHANGING
S WM_NCACTIVATE fActive:False
R WM_NCACTIVATE fDeactivateOK:True
P message:0x0118 [Unknown] wParam:0000FFF8 lParam:001A9ECC
S WM_NCACTIVATE fActive:True
R WM_NCACTIVATE
P message:0x0118 [Unknown] wParam:0000FFF8 lParam:001A9ECC
S WM_NCACTIVATE fActive:False
R WM_NCACTIVATE fDeactivateOK:True
P message:0x0118 [Unknown] wParam:0000FFF8 lParam:001A9ECC
S WM_NCACTIVATE fActive:True
R WM_NCACTIVATE
P message:0x0118 [Unknown] wParam:0000FFF8 lParam:001A9ECC
S WM_NCACTIVATE fActive:False
R WM_NCACTIVATE fDeactivateOK:True
P message:0x0118 [Unknown] wParam:0000FFF8 lParam:001A9ECC
S WM_NCACTIVATE fActive:True
R WM_NCACTIVATE
P message:0x0118 [Unknown] wParam:0000FFF8 lParam:001A9ECC
S WM_NCACTIVATE fActive:False
R WM_NCACTIVATE fDeactivateOK:True
P message:0x0118 [Unknown] wParam:0000FFF8 lParam:001A9ECC
S WM_NCACTIVATE fActive:True
R WM_NCACTIVATE
P message:0x0118 [Unknown] wParam:0000FFF8 lParam:001A9ECC
S WM_NCACTIVATE fActive:False
R WM_NCACTIVATE fDeactivateOK:True
P message:0x0118 [Unknown] wParam:0000FFF8 lParam:001A9ECC
S WM_NCACTIVATE fActive:True
R WM_NCACTIVATE
P message:0x0118 [Unknown] wParam:0000FFF8 lParam:001A9ECC
S WM_NCACTIVATE fActive:False
R WM_NCACTIVATE fDeactivateOK:True
P message:0x0118 [Unknown] wParam:0000FFF8 lParam:001A9ECC
S WM_NCACTIVATE fActive:True
R WM_NCACTIVATE
P message:0x0118 [Unknown] wParam:0000FFF8 lParam:001A9ECC
S WM_NCACTIVATE fActive:False
R WM_NCACTIVATE fDeactivateOK:True
P message:0x0118 [Unknown] wParam:0000FFF8 lParam:001A9ECC
S WM_NCACTIVATE fActive:True
R WM_NCACTIVATE
P message:0x0118 [Unknown] wParam:0000FFF8 lParam:001A9ECC
S WM_NCACTIVATE fActive:True
R WM_NCACTIVATE

Main form messages

nHittest:FFFE wMouseMsg:WM_LBUTTONDOWN
S WM_WINDOWPOSCHANGING lpwp:0018EDA8
R WM_WINDOWPOSCHANGING
R WM_SETCURSOR fHaltProcessing:False
nHittest:FFFE wMouseMsg:WM_LBUTTONUP
R WM_SETCURSOR fHaltProcessing:False

I don't think there's anything here that you can realistically hook onto. The best you could hope for would be to try and detect a repeated stream of WM_NCACTIVATE messages but I really would not attempt this.

In my opinion you need to look more closely at the fundamental problem. You say that the modal form sometimes is beneath the main form. In that case you are doing something wrong with your window ownership. The main form should be the ultimate owner of your modal form and if that were so then it could never be beneath the main form. In my view you simply need to fix your broken window ownership structure and the problems will disappear.