Extracting keyboard layouts from windows

Question

OK, this is a slightly weird question.

We have a touch-screen application (i.e., no keyboard). When users need to enter text, the application shows virtual keyboard - hand-built in WinForms.

Making these things by hand for each new language is monkey work. I figure that windows must have this keyboard layout information hiding somewhere in some dll. Would there be anyway to get this information out of windows?

Other ideas welcome (I figure at least generating the thing from a xml file has got to be better than doing it by hand in VS).

(Note: having said all which, I note that there is a Japanese keyboard, state machine and all..., so XML might not be sufficient)

UPDATE: pretty good series on this subject (I believe) here

Answer 1

Microsoft Keyboard Layout Creator can load system keyboards and export them as .klc files. Since it’s written in .NET you can use Reflector to see how it does that, and use reflection to drive it. Here's a zip file of .klc files for the 187 keyboards in Windows 8 created using the below C# code. Note that I originally wrote this for Windows XP, and now with Windows 8 and the on-screen keyboard, it is really slow and seems to crash the taskbar :/ However, it does work :)

using System;
using System.Collections;
using System.IO;
using System.Reflection;

class KeyboardExtractor {

    static Object InvokeNonPublicStaticMethod(Type t, String name,
            Object[] args)
    {
        return t.GetMethod(name, BindingFlags.Static | BindingFlags.NonPublic)
            .Invoke(null, args);
    }

    static void InvokeNonPublicInstanceMethod(Object o, String name,
            Object[] args)
    {
        o.GetType().GetMethod(name, BindingFlags.Instance |
                BindingFlags.NonPublic) .Invoke(o, args);
    }

    static Object GetNonPublicProperty(Object o, String propertyName) {
        return o.GetType().GetField(propertyName,
                BindingFlags.Instance | BindingFlags.NonPublic)
            .GetValue(o);
    }

    static void SetNonPublicField(Object o, String propertyName, Object v) {
        o.GetType().GetField(propertyName,
                BindingFlags.Instance | BindingFlags.NonPublic)
            .SetValue(o, v);
    }

    [STAThread] public static void Main() {
        System.Console.WriteLine("Keyboard Extractor...");

        KeyboardExtractor ke = new KeyboardExtractor();
        ke.extractAll();

        System.Console.WriteLine("Done.");
    }

    Assembly msklcAssembly;
    Type utilitiesType;
    Type keyboardType;
    String baseDirectory;

    public KeyboardExtractor() {
        msklcAssembly = Assembly.LoadFile("C:\\Program Files\\Microsoft Keyboard Layout Creator 1.4\\MSKLC.exe");
        utilitiesType = msklcAssembly.GetType("Microsoft.Globalization.Tools.KeyboardLayoutCreator.Utilities");
        keyboardType = msklcAssembly.GetType("Microsoft.Globalization.Tools.KeyboardLayoutCreator.Keyboard");

        baseDirectory = Directory.GetCurrentDirectory();
    }

    public void extractAll() {

        DateTime startTime = DateTime.UtcNow;

        SortedList keyboards = (SortedList)InvokeNonPublicStaticMethod(
                utilitiesType, "KeyboardsOnMachine", new Object[] {false});

        DateTime loopStartTime = DateTime.UtcNow;

        int i = 0;
        foreach (DictionaryEntry e in keyboards) {
            i += 1;
            Object k = e.Value;

            String name = (String)GetNonPublicProperty(k, "m_stLayoutName");
            String layoutHexString = ((UInt32)GetNonPublicProperty(k, "m_hkl"))
                .ToString("X");

            TimeSpan elapsed = DateTime.UtcNow - loopStartTime;
            Double ticksRemaining = ((Double)elapsed.Ticks * keyboards.Count)
                        / i - elapsed.Ticks;
            TimeSpan remaining = new TimeSpan((Int64)ticksRemaining);
            String msgTimeRemaining = "";
            if (i > 1) {
                // Trim milliseconds
                remaining = new TimeSpan(remaining.Hours, remaining.Minutes,
                        remaining.Seconds);
                msgTimeRemaining = String.Format(", about {0} remaining",
                        remaining);
            }
            System.Console.WriteLine(
                    "Saving {0} {1}, keyboard {2} of {3}{4}",
                    layoutHexString, name, i, keyboards.Count,
                    msgTimeRemaining);

            SaveKeyboard(name, layoutHexString);

        }

        System.Console.WriteLine("{0} elapsed", DateTime.UtcNow - startTime);

    }

    private void SaveKeyboard(String name, String layoutHexString) {
        Object k = keyboardType.GetConstructors(
                BindingFlags.Instance | BindingFlags.NonPublic)[0]
            .Invoke(new Object[] {
                        new String[] {"", layoutHexString},
                    false});

        SetNonPublicField(k, "m_fSeenOrHeardAboutPropertiesDialog", true);
        SetNonPublicField(k, "m_stKeyboardTextFileName",
                String.Format("{0}\\{1} {2}.klc",
                    baseDirectory, layoutHexString, name));
        InvokeNonPublicInstanceMethod(k, "mnuFileSave_Click",
                new Object[] {new Object(), new EventArgs()});

        ((IDisposable)k).Dispose();
    }

}

Basically, it gets a list of all the keyboards on the system, then for each one, loads it in MSKLC, sets the "Save As" filename, lies about whether it's already configured the custom keyboard properties, and then simulates a click on the File -> Save menu item.

Answer 2

It is a fairly well-known fact that MSKLC is unable to faithfully import & reproduce keyboard layouts for all of the .DLL files supplied by Windows–especially those in Windows 8 & above. And it doesn't do any good to know where those files are if you can't extract any meaningful or helpful information from them. This is documented by Michael Kaplan on his blog (he was a developer of MSKLC) which I see you have linked to above.

When MSKLC encounters anything it does not understand, that portion is removed. Extracting the layout using MSKLC will work for most keyboards, but there are a few–namely the Cherokee keyboard, and the Japanese & Korean keyboards (to name a few, I'm not sure how many more there are)–for which the extracted layout will NOT accurately or completely reflect the actual usage & features of the keyboard. The Cherokee keyboard has chained dead keys which MSKLC doesn't support. And the far Eastern keyboards have modifier keys which MSKLC isn't aware of–that means entire layers/shift states which are missing!

Michael Kaplan supplies some code and unlocks some of the secrets of MSLKC and the accompanying software that can be used to get around some of these limitations but it requires a fair amount of doing things by hand–exactly what you're trying to avoid! Plus, Michael's objectives are aimed at creating keyboards with features that MSKLC can not create or understand, but which DO work in Windows (which is the opposite of what the OP is trying to accomplish).

I am sure that my solution comes too late to be of use to the OP, but perhaps it will be helpful in the future to someone in a similar situation. That is my hope and reason for posting this.

So far all I've done is explain that the other answers are insufficient. Even the best one will not and can not fully & accurately reproduce all of Windows' native keyboards and render them into KLC source files. This is really unfortunate and it is certainly not the fault of its author because that is a very clever piece of code/script! Thankfully the script & the source files (whose link may or may not still work) is useful & effective for the majority of Windows' keyboards, as well as any custom keyboards created by MSKLC.

The keyboards that have the advanced features that MSKLC doesn't support were created by the Windows DDK, but those features are not officially documented. Although one can learn quite a bit about their potential by studying the source files provided with MSKLC.

Sadly the only solution I can offer is 3rd party, paid software called KbdEdit. I believe it is the only currently available solution which is actually able to faithfully decode & recreate any of the Windows supplied keyboards–although there are a few advanced features which even it can not reproduce (such as keys combinations/hotkeys which perform special native language functions; for example: Ctrl+CapsLock to activate KanaLock (a Japanese modifier layer). KbdEdit DOES faithfully reproduce that modifier layer which MSKLC with strip away, it just doesn't support this alternate method of activating that shift state if you don't have a Japanese keyboard with a Kana lock key. Although, it will allow you to convert a key on your keyboard to a Kana key (perhaps Scroll Lock?).

Fortunately, none of those unsupported features are even applicable to an on-screen keyboard.

KbdEdit is a really powerful & amazing tool, and it has been worth every penny I paid for it! (And that's NOT something I would say about virtually any other paid software…) Even though KbdEdit is 3rd party software, it is only needed to create the keyboards, not to use them. All of the keyboards it creates work natively on any Windows system without KbdEdit being installed. It supports up to 15 modifier states and three addition modifier keys, one which is togglable–like CapsLock. It also supports chained dead keys, and remapping any of the keys on most any keyboard.