Enabling option-key shortcuts in MATLAB for Mac

Question

Since R2009b, MATLAB has had marvelously customizable keyboard shortcuts through its Keyboard Shortcuts Preferences. This works very well for customizing shortcuts using command and control on a Mac.

Unfortunately, those keybindings seem to be unable to override MATLAB's built-in character map. For example, if I define option-f as cursor-next-word (a la emacs), it accepts the binding. Hitting the key combination does properly move the cursor to the next word, but it additionally prints the ƒ character! I believe this is from the character map (perhaps as opposed to the input map?). Neither EditorMacro nor KeyBindings are able to override this behavior.

I stumbled across this answer from a tangentially related question which gives me hope. In short, he defined a Java class that can handle keyboard events and replace them with other keystroke input. The solution, however, only works as prescribed on Windows. The following modifications were required to run on a Mac:

I needed to change the keycodes to remap to have 'pressed' in the string, like so:

map = {
    '$' '^'
    '#' char(181)  % might be useful for text formatting
};

to:

map = {
    'alt pressed F' '^'
    'alt pressed B' char(181)  % might be useful for text formatting
};

Unfortunately, after running the code, pressing option-f yields cursor-next-word and the ƒ character, just like before. However, if I disable the cursor-next-word binding from the preferences, then I get both ƒ and ^! Indeed, even if I use a simple action like pressed F, the KeyReplacementAction doesn't replace the action but rather augments it. It seems like this behavior is unique to MATLAB on OS X.

It seems as though I'm simply not overriding the correct keymap. I've tried digging through the Java runtime, but I'm not familiar enough with the event dispatch model to know where to look next. Perhaps something within Java's OS-level keymap?

---

Edit: I've since done some more digging around. It appears as though the Mac version of MATLAB does not properly respect the 'consumed' property of a keyEvent. I can attach the KeyReplacementAction to either the inputMap or the keymap, and in both cases I augment the keybinding instead of replacing it. I used reflection to 'unprotect' the consume() method for AWTEvents, but the effect was the same as before.

Following the stack trace around, it appears as though the keyEvent is falling through to an instance of javax.swing.KeyboardManager. It looks like I should be able to unbind keystrokes within the KeyboardManager, but I cannot figure out how to access the instance from the MATLAB handles I have. Perhaps someone more familiar with Swing's event model and the Java debugger could get farther.

---

Edit 2: flolo's answer spurred me to look into X11's keymaps. Several notes:

• Matlab does not seem to respect ~/.Xmodmap or any currently-loaded modmaps.
• Matlab makes use of the $XKEYSYMDB environment variable if it exists at startup. Otherwise, it loads it from $MATLAB/X11/app-defaults/XKeysymDB.
• The whole $MATLAB/X11/app-defaults/ directory looks very interesting; perhaps some hackery there could make this work?
• Where are the X11 keymaps on a Mac? How does MATLAB switch to international keyboard layouts?

---

Edit 3: Hrm, I think X11 is a red herring. lsof -c MATLAB shows that it is accessing /System/Library/Keyboard Layouts/AppleKeyboardLayouts.bundle. Working on that now…

---

Edit 4: MATLAB does indeed use the system keyboard layout. I created one without any bindings as R.M. suggested. This appeared to work — MATLAB does behave properly. Unfortunately, it also breaks my custom Cocoa keybindings in all other programs. Close, but no cigar. (Close enough, in fact, that R.M. won the +500 bounty due to a brief thought that it had worked… until I attempted to compose my congratulatory comment and discovered that I couldn't navigate the text field as usual.)

Answer 1

I finally had a chance to further pursue this over the holidays. And I found a kludge of a solution. Note that this is dynamically changing the Java classes used by the Matlab GUI in runtime; it is completely unsupported and is likely to be very fragile. It works on my version of Matlab (r2011a) on OS X Lion.

Here's what I've learned about how Swing/Matlab process keypress events:

1. A keystroke is pressed.
2. The active text component's inputMap is searched to see if there's a binding for that keystroke.
   • If there is an action bound to that keystroke, dispatch that action's actionPerformed method
   • If there is a string associated with that keystroke, find the action from the text component's actionMap, and then dispatch that action's actionPerformed method
3. No matter what, as a final step, dispatch the action found within the text component's Keymap.getDefaultAction(). Here's where the problem lies.

This solution overrides the Keymap's default action with a wrapper that simply checks to see if any modifier keys were pressed. If they were, the action is silently ignored.

---

Step 1: Create a custom TextAction in Java to ignore modifier keys

import javax.swing.text.TextAction;
import javax.swing.Action;
import java.awt.event.ActionEvent;

public class IgnoreModifiedKeystrokesAction extends TextAction
{
    private static final int ignoredModifiersMask = 
        ActionEvent.CTRL_MASK | ActionEvent.ALT_MASK;
    private Action original;

    public IgnoreModifiedKeystrokesAction(Action act)
    {
        super((String)act.getValue("Name"));
        original = act;
    }

    public void actionPerformed(ActionEvent e)
    {
        if ((e.getModifiers() & ignoredModifiersMask) == 0) {
            /* Only dispatch the original action if no modifiers were used */
            original.actionPerformed(e);
        }
    }

    public Action getOriginalAction()
    {
        return original;
    }
}

Compile to a .jar:

javac IgnoreModifiedKeystrokesAction.java && jar cvf IgnoreModifiedKeystrokesAction.jar IgnoreModifiedKeystrokesAction.class

---

Step 2: Override MATLAB's default Keymap handler in both the command window and editor (from within MATLAB)

The hardest part here is getting the java handles to the command window and editor. It is dependent upon the layout and classnames of the individual editor panes. This may change between versions of Matlab.

javaaddpath('/path/to/IgnoreModifiedKeystrokesAction.jar')
cmdwin = getCommandWindow();
editor = getEditor();

for t = [cmdwin,editor]
    defaultAction = t.getKeymap().getDefaultAction();
    if ~strcmp(defaultAction.class(),'IgnoreModifiedKeystrokesAction')
        newAction = IgnoreModifiedKeystrokesAction(defaultAction);
        t.getKeymap().setDefaultAction(newAction);
    end
end

%% Subfunctions to retrieve handles to the java text pane elements
function cw = getCommandWindow()
    try
        cw = handle(com.mathworks.mde.desk.MLDesktop.getInstance.getClient('Command Window').getComponent(0).getComponent(0).getComponent(0),'CallbackProperties');
        assert(strcmp(cw.class(),'javahandle_withcallbacks.com.mathworks.mde.cmdwin.XCmdWndView'));
    catch %#ok<CTCH>
        cw_client = com.mathworks.mde.desk.MLDesktop.getInstance.getClient('Command Window');
        cw = searchChildComponentsForClass(cw_client,'com.mathworks.mde.cmdwin.XCmdWndView');
    end
    if isempty(cw)
        error('Unable to find the Command Window');
    end
end

function ed = getEditor()
    try
        ed = handle(com.mathworks.mde.desk.MLDesktop.getInstance.getGroupContainer('Editor').getComponent(1).getComponent(0).getComponent(0).getComponent(0).getComponent(1).getComponent(0).getComponent(0).getComponent(0).getComponent(0).getComponent(1).getComponent(0).getComponent(0),'CallbackProperties');
        assert(strcmp(ed.class(),'javahandle_withcallbacks.com.mathworks.mde.editor.EditorSyntaxTextPane'));
    catch %#ok<CTCH>
        ed_group = com.mathworks.mde.desk.MLDesktop.getInstance.getGroupContainer('Editor');
        ed = searchChildComponentsForClass(ed_group,'com.mathworks.mde.editor.EditorSyntaxTextPane');
        % TODO: When in split pane mode, there are two editor panes. Do I need
        % to change actionMaps/inputMaps/Keymaps on both panes?
    end
    if isempty(ed)
        error('Unable to find the Editor Window');
    end
end

function obj = searchChildComponentsForClass(parent,classname)
    % Search Java parent object for a child component with the specified classname
    obj = [];
    if ~ismethod(parent,'getComponentCount') || ~ismethod(parent,'getComponent')
        return
    end
    for i=0:parent.getComponentCount()-1
        child = parent.getComponent(i);
        if strcmp(child.class(),classname)
            obj = child;
        else
            obj = searchChildComponentsForClass(child,classname);
        end
        if ~isempty(obj)
            obj = handle(obj,'CallbackProperties');
            break
        end
    end
end

Now it's possible to define keybindings in the standard preferences window that use the option key!

---

Step 3 (optional): Remove the custom action

cmdwin = getCommandWindow();
editor = getEditor();

for t = [cmdwin,editor]
    defaultAction = t.getKeymap().getDefaultAction();
    if strcmp(defaultAction.class(),'IgnoreModifiedKeystrokesAction')
        oldAction = defaultAction.getOriginalAction();
        t.getKeymap().setDefaultAction(oldAction);
    end
end
javarmpath('/path/to/IgnoreModifiedKeystrokesAction.jar')