.NET WebBrowser - FireBug Style Inspect HTML Element

Question

Is is possible to use .NET 3.5/4.0 to Inspect HTML Elements in a WinForm Web Browser?

Possibly using IHighlightRenderingServices Interface or Html Agility Pack?

I would like the application to function like FireBug:

Simply hover the element you want to inspect with the mouse and click on it. In versions of Firebug prior to 1.7 this automatically switches to the HTML Panel and selects the appropriate element inside the Node View.

EDIT:

Wow, I just came across http://www.selectorgadget.com/ which is exactly what I'm trying to do. It's in Javascript and after looking through the source code for the past 2 hours I still don't have a clue how to incorporate it into my program...

From what I can tell it uses tokenizing and recursive analysis of DOM elements to figure out CSS selector paths: http://www.selectorgadget.com/stable/lib/dom.js.

EDIT: Okay! I've got selectorgadget loaded into my Application. It allows you to select exactly the same HTML elements as Firebug! And even creates the Xpath query.

However, I'm using AxWebBrowser and I'm stuck on how to get it to work with HtmlAgilityPack...

    private void xpathQuery_Click(object sender, EventArgs e)
    {
        // Load Browser 
        HtmlWindow window = axWebBrowser1.Document.Window; // <---- 'object' does not contain a definition for 'Window'

        string str = window.Document.Body.OuterHtml;

        // Load HTML
        HtmlAgilityPack.HtmlDocument HtmlDoc = new HtmlAgilityPack.HtmlDocument();
        HtmlDoc.LoadHtml(str);

        //Process Xpath Query 
        HtmlAgilityPack.HtmlNodeCollection Nodes = HtmlDoc.DocumentNode.SelectNodes(xpathText.Text);

        //Print in Text box 
        foreach (HtmlAgilityPack.HtmlNode Node in Nodes)
        {
            richTextBox1.Text += Node.OuterHtml + "\r\n";
        }         
    }

Edit: I couldn't get AxWebBrowser to work with HtmlAgilityPack, so I just used the WebClient Class to load the URL and then parse it with HtmlAgilityPack.

I'm Just about finished with the Web Scraper. It functions similarly to Visual Web Ripper and all those others that cost $1,000+.

Answer 1

I have actually done this before. You have to cast the document into IExpando then you can do reflection calls against it to get members. I actually created a DynamicNode class that lets you use the dynamic keyword to interact with the document.

You might want to use the mshtml COM object instead though: Reusing MSHTML

1. Add a reference to mshtml in the COM reference list
2. Create an instance of var document = new mshtml.HTMLDocument();
3. Cast to IExpando: var window = (IExpando)document.parentWindow;
4. Create a dynamic object wrapper (see below)
5. Use dynamic keyword to interact with the document.

For example here is my dynamic node:

class DynamicNode : DynamicObject, IExpando, IEnumerable
{
    private IExpando value;

    public DynamicNode(IExpando value)
    {
        this.value = value;
    }

    public override bool TryUnaryOperation(UnaryOperationBinder binder, out object result)
    {
        switch (binder.Operation)
        {
            case System.Linq.Expressions.ExpressionType.Convert:
            case System.Linq.Expressions.ExpressionType.ConvertChecked:
                result = this.value;
                return true;
        }

        return base.TryUnaryOperation(binder, out result);
    }

    public override IEnumerable<string> GetDynamicMemberNames()
    {
        return this.value
            .GetMembers(BindingFlags.Instance | BindingFlags.Public)
            .Select(m => m.Name)
            .Distinct()
            .ToArray();
    }

    public override bool TryConvert(ConvertBinder binder, out object result)
    {
        result = this.value;
        return true;
    }

    public override bool TryGetIndex(GetIndexBinder binder, object[] indexes, out object result)
    {
        if (indexes.Length == 1)
        {
            var memberName = indexes[0].ToString();
            result = ReflectionHelpers.GetValue(this.value, memberName);
            result = DynamicNode.Wrap(result);
            return true;
        }

        return base.TryGetIndex(binder, indexes, out result);
    }

    public override bool TrySetIndex(SetIndexBinder binder, object[] indexes, object value)
    {
        if (indexes.Length == 1)
        {
            var memberName = indexes[0].ToString();
            value = DynamicNode.Unwrap(value);
            ReflectionHelpers.SetValue(this.value, memberName, value);
            return true;
        }

        return base.TrySetIndex(binder, indexes, value);
    }

    public override bool TryGetMember(GetMemberBinder binder, out object result)
    {
        if (base.TryGetMember(binder, out result))
            return true;

        result = ReflectionHelpers.GetValue(this.value, binder.Name);
        result = DynamicNode.Wrap(result);
        return true;
    }

    public override bool TrySetMember(SetMemberBinder binder, object value)
    {
        ReflectionHelpers.SetValue(this.value, binder.Name, value);
        return true;
    }

    public override bool TryInvokeMember(InvokeMemberBinder binder, object[] args, out object result)
    {
        if (binder.Name == "New")
        {
            var constructorArgs = new object[args.Length - 1];
            Array.ConstrainedCopy(args, 1, constructorArgs, 0, constructorArgs.Length);

            result = ReflectionHelpers.New(this.value, (string)args[0], constructorArgs);
        }
        else
        {
            result = ReflectionHelpers.Invoke(this.value, binder.Name, args);
        }

        result = DynamicNode.Wrap(result);
        return true;
    }

    public override bool TryInvoke(InvokeBinder binder, object[] args, out object result)
    {
        IExpando self = this.value;
        object[] constructorArgs = new object[0];

        if (args.Length > 0)
        {
            self = (IExpando)DynamicNode.Unwrap(args[0]);
            constructorArgs = new object[args.Length - 1];
            Array.ConstrainedCopy(args, 1, constructorArgs, 0, constructorArgs.Length);
        }

        result = ReflectionHelpers.Call(this.value, self, constructorArgs);
        result = DynamicNode.Wrap(result);
        return true;
    }

    private static object Wrap(object value)
    {
        if (value != null && Marshal.IsComObject(value))
            value = new DynamicNode((IExpando)value);

        return value;
    }

    public static object Unwrap(object value)
    {
        DynamicNode node = value as DynamicNode;
        if (node != null)
            return node.value;

        return value;
    }

    public IEnumerator GetEnumerator()
    {
        var members = this.value.GetProperties(BindingFlags.Public | BindingFlags.Instance);

        var indexProperties = new List<Tuple<int, PropertyInfo>>();
        var isArray = true;
        foreach (var member in members)
        {
            int value = 0;
            if (!int.TryParse(member.Name, out value))
            {
                isArray = false;
                break;
            }

            var propertyMember = member as PropertyInfo;
            if (propertyMember != null)
                indexProperties.Add(Tuple.Create(value, propertyMember));
        }

        if (isArray)
        {
            indexProperties.Sort((left, right) => left.Item1.CompareTo(right.Item1));
            foreach (var prop in indexProperties)
                yield return prop.Item2.GetValue(this.value, null);
        }
        else
        {
            foreach (var member in members)
                yield return member.Name;
        }
    }

    #region IExpando
    FieldInfo IExpando.AddField(string name)
    {
        return this.value.AddField(name);
    }

    MethodInfo IExpando.AddMethod(string name, Delegate method)
    {
        return this.value.AddMethod(name, method);
    }

    PropertyInfo IExpando.AddProperty(string name)
    {
        return this.value.AddProperty(name);
    }

    void IExpando.RemoveMember(MemberInfo m)
    {
        this.value.RemoveMember(m);
    }

    FieldInfo IReflect.GetField(string name, BindingFlags bindingAttr)
    {
        return this.value.GetField(name, bindingAttr);
    }

    FieldInfo[] IReflect.GetFields(BindingFlags bindingAttr)
    {
        return this.value.GetFields(bindingAttr);
    }

    MemberInfo[] IReflect.GetMember(string name, BindingFlags bindingAttr)
    {
        return this.value.GetMember(name, bindingAttr);
    }

    MemberInfo[] IReflect.GetMembers(BindingFlags bindingAttr)
    {
        return this.value.GetMembers(bindingAttr);
    }

    MethodInfo IReflect.GetMethod(string name, BindingFlags bindingAttr)
    {
        return this.value.GetMethod(name, bindingAttr);
    }

    MethodInfo IReflect.GetMethod(string name, BindingFlags bindingAttr, Binder binder, Type[] types, ParameterModifier[] modifiers)
    {
        return this.value.GetMethod(name, bindingAttr, binder, types, modifiers);
    }

    MethodInfo[] IReflect.GetMethods(BindingFlags bindingAttr)
    {
        return this.value.GetMethods(bindingAttr);
    }

    PropertyInfo[] IReflect.GetProperties(BindingFlags bindingAttr)
    {
        return this.value.GetProperties(bindingAttr);
    }

    PropertyInfo IReflect.GetProperty(string name, BindingFlags bindingAttr, Binder binder, Type returnType, Type[] types, ParameterModifier[] modifiers)
    {
        return this.value.GetProperty(name, bindingAttr, binder, returnType, types, modifiers);
    }

    PropertyInfo IReflect.GetProperty(string name, BindingFlags bindingAttr)
    {
        return this.value.GetProperty(name, bindingAttr);
    }

    object IReflect.InvokeMember(string name, BindingFlags invokeAttr, Binder binder, object target, object[] args, ParameterModifier[] modifiers, System.Globalization.CultureInfo culture, string[] namedParameters)
    {
        return this.value.InvokeMember(name, invokeAttr, binder, target, args, modifiers, culture, namedParameters);
    }

    Type IReflect.UnderlyingSystemType
    {
        get { return this.value.UnderlyingSystemType; }
    }
    #endregion
}

[ComVisible(true)]
public class ScriptObject : IReflect, IExpando
{
    private readonly Type type;
    private dynamic _constructor;
    private dynamic _prototype;

    public ScriptObject()
    {
        type = this.GetType();
    }

    [DispId(0)]
    protected virtual object Invoke(object[] args)
    {
        return "ClrObject";
    }

    public dynamic constructor
    {
        get { return _constructor; }
        set { this._constructor = value; }
    }

    public dynamic prototype
    {
        get { return _prototype; }
        set { this._prototype = value; }
    }

    public string toString()
    {
        return "ClrObject";
    }

    #region IReflect Members
    MethodInfo IReflect.GetMethod(string name, BindingFlags bindingAttr, Binder binder, Type[] types, ParameterModifier[] modifiers)
    {
        return type.GetMethod(name, bindingAttr, binder, types, modifiers);
    }

    MethodInfo IReflect.GetMethod(string name, BindingFlags bindingAttr)
    {
        return type.GetMethod(name, bindingAttr);
    }

    protected virtual MethodInfo[] GetMethods(BindingFlags bindingAttr)
    {
        return type.GetMethods(bindingAttr);
    }

    MethodInfo[] IReflect.GetMethods(BindingFlags bindingAttr)
    {
        return GetMethods(bindingAttr);
    }

    FieldInfo IReflect.GetField(string name, BindingFlags bindingAttr)
    {
        return type.GetField(name, bindingAttr);
    }

    FieldInfo[] IReflect.GetFields(BindingFlags bindingAttr)
    {
        return new FieldInfo[0]; // type.GetFields(bindingAttr);
    }

    PropertyInfo IReflect.GetProperty(string name, BindingFlags bindingAttr)
    {
        return type.GetProperty(name, bindingAttr);
    }

    PropertyInfo IReflect.GetProperty(string name, BindingFlags bindingAttr, Binder binder, Type returnType, Type[] types, ParameterModifier[] modifiers)
    {
        return type.GetProperty(name, bindingAttr, binder, returnType, types, modifiers);
    }

    protected virtual PropertyInfo[] GetProperties(BindingFlags bindingAttr)
    {
        return type.GetProperties(bindingAttr);
    }

    PropertyInfo[] IReflect.GetProperties(BindingFlags bindingAttr)
    {
        return GetProperties(bindingAttr);
    }

    MemberInfo[] IReflect.GetMember(string name, BindingFlags bindingAttr)
    {
        return type.GetMember(name, bindingAttr);
    }

    MemberInfo[] IReflect.GetMembers(BindingFlags bindingAttr)
    {
        return type.GetMembers(bindingAttr);
    }

    protected virtual object InvokeMember(string name, BindingFlags invokeAttr, Binder binder, object target, object[] args, ParameterModifier[] modifiers, CultureInfo culture, string[] namedParameters)
    {
        if (name == "[DISPID=0]")
        {
            return this.Invoke(args);
        }
        return type.InvokeMember(name, invokeAttr, binder, target, args, modifiers, culture, namedParameters);
    }

    object IReflect.InvokeMember(string name, BindingFlags invokeAttr, Binder binder, object target, object[] args, ParameterModifier[] modifiers, CultureInfo culture, string[] namedParameters)
    {
        return this.InvokeMember(name, invokeAttr, binder, target, args, modifiers, culture, namedParameters);
    }

    Type IReflect.UnderlyingSystemType
    {
        get { return type.UnderlyingSystemType; }
    }
    #endregion

    #region IExpando Members
    public FieldInfo AddField(string name)
    {
        throw new NotImplementedException();
    }

    public MethodInfo AddMethod(string name, Delegate method)
    {
        throw new NotImplementedException();
    }

    public PropertyInfo AddProperty(string name)
    {
        throw new NotImplementedException();
    }

    public void RemoveMember(MemberInfo m)
    {
        throw new NotImplementedException();
    }
    #endregion
}

public static class ReflectionHelpers
{
    private const BindingFlags DefaultFlags = BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static;

    public static object New(this IExpando scope, string functionName, params object[] args)
    {
        var constructor = (IExpando)scope.GetValue(functionName);
        var proto = constructor.GetValue("prototype");

        var obj = (IExpando)scope.GetValue("Object");
        var instance = (IExpando)obj.Invoke("create", new object[] { proto });
        Call(constructor, instance, args);

        return instance;
    }

    public static object Call(this IExpando function, IExpando scope, params object[] args)
    {
        object[] callArgs = new object[args.Length + 1];
        callArgs[0] = scope;
        Array.Copy(args, 0, callArgs, 1, args.Length);

        return Invoke(function, "call", callArgs);
    }

    public static void SetValue(this IExpando instance, string propertyName, object value)
    {
        if (instance == null)
            throw new ArgumentNullException("instance");

        if (string.IsNullOrEmpty(propertyName))
            throw new ArgumentException("Must specify a value.", "propertyName");

        Invoke(instance, propertyName, InvokeFlags.DISPATCH_PROPERTYPUT, new object[] { value });
    }

    public static object GetValue(this IExpando instance, string propertyName)
    {
        return Invoke(instance, propertyName, InvokeFlags.DISPATCH_PROPERTYGET, new object[0]);
    }

    public static object Invoke(this IExpando instance, string functionName, object[] args)
    {
        if (instance == null)
            throw new ArgumentNullException("instance");

        if (string.IsNullOrEmpty(functionName))
            throw new ArgumentException("Must specify a value.", "functionName");

        return Invoke(instance, functionName, InvokeFlags.DISPATCH_METHOD, args);

    }

    private static object Invoke(IExpando instance, string functionName, InvokeFlags flags, object[] args)
    {
        try
        {
            args = args.Select(arg => DynamicNode.Unwrap(arg)).ToArray();
            switch (flags)
            {
                case InvokeFlags.DISPATCH_METHOD:
                    var method = instance.GetMethod(functionName, DefaultFlags);
                    return method.Invoke(instance, args);
                case InvokeFlags.DISPATCH_PROPERTYGET:
                    var getProp = instance.GetProperty(functionName, DefaultFlags);
                    return getProp.GetValue(instance, null);
                case InvokeFlags.DISPATCH_PROPERTYPUT:
                case InvokeFlags.DISPATCH_PROPERTYPUTREF:
                    var setProp = instance.GetProperty(functionName, DefaultFlags);
                    if (setProp == null)
                        setProp = instance.AddProperty(functionName);
                    setProp.SetValue(instance, args[0], null);
                    return null;
                default:
                    throw new NotSupportedException();
            }
        }
        catch (COMException comex)
        {
            switch ((uint)comex.ErrorCode)
            {
                // Unexpected script error. This will be handled by the IProcess.UnhandledException event
                case 0x80020101:
                    return null;
                default:
                    throw;
            }
        }
    }

    private enum InvokeFlags
    {
        DISPATCH_METHOD = 1,
        DISPATCH_PROPERTYGET = 2,
        DISPATCH_PROPERTYPUT = 4,
        DISPATCH_PROPERTYPUTREF = 8,
    }
}

You can actually stick .net objects into the document this way or pull out objects and interact with them from .net. You can also eval js as strings and have it call into .net functions. Here are some code snippets of usages:

getting and setting members of a js object:

this.host.Window.eval(@" Foo = { }; ");

var foo = this.host.Window.Foo;
foo.B = 7.11;
Assert.Equal(7.11, foo.B);

calling js function from C#:

this.host.eval("function add(x, y) { return x + y; }");
var z = (int)this.host.Window.add(7, 11);
Assert.Equal(7 + 11, z);

inserting .net objects into document and calling its members from js:

this.host.Window.Custom2 = new Custom2();    
this.host.Window.eval(@"
  function test() {
    return Custom2.Test().Value;
  }");

bool success = this.host.Window.test();
Assert.True(success);

You can only stick objects that inherit from ScriptObject into the document though (defined in the codeblock above). Well I think you can put anything in there but you'll get strange behaviors if they don't implement IReflect.

Answer 2

What about firebug lite?

http://getfirebug.com/firebuglite

it's a bookmarklet that can inject most firebug functionality into anything.