Can Flutter detect a hovering stylus?

Question

Some phones, most notably the Samsung Galaxy Note line of devices, have styluses (styli?) that can be detected when they are close to the screen but not touching it. Can Flutter detect and handle that kind of event?

(what follows is my investigation on this, if you already know the answer, feel free to skip this 😄)

The Listener class can detect actions performed with a stylus when it is touching the screen and the MouseRegion class is supposed to detect actions performed with a hovering pointer. So I wrote this simple widget to test both classes:

class MyHomePage extends StatefulWidget {
  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  String  _message = "Nothing happened";
  String _location = "Nothing happened";

  void onEnter(PointerEnterEvent event) {
    setState(() {
      _message = "Pointer entered";
    });
  }

  void onExit(PointerExitEvent event) {
    setState(() {
      _message = "Pointer exited";
    });
  }

  void onHover(PointerHoverEvent event) {
    setState(() {
      _location = "Pointer at ${event.localPosition.dx} ${event.localPosition.dy} distance ${event.distance}";
    });
  }

  void onDown(PointerDownEvent event) {
    setState(() {
      _message = "Pointer down";
    });
  }

  void onUp(PointerUpEvent event) {
    setState(() {
      _message = "Pointer up";
    });
  }

  void onMove(PointerMoveEvent event) {
    setState(() {
      _location = "Pointer moving at ${event.localPosition.dx} ${event.localPosition.dy} pressure ${event.pressure}";
    });
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      body: Center(
        child: Column(
          children: [
            MouseRegion(
              onEnter: onEnter,
              onExit: onExit,
              onHover: onHover,
              child: Listener(
                  onPointerDown: onDown,
                  onPointerUp: onUp,
                  onPointerMove: onMove,
                  child: Container(
                      width: 500,
                      height: 500,
                      color: Colors.red
                  )
              )
            ),
            Text(_message),
            Text(_location)
          ]
        )
      ),
    );
  }
}

Using a bluetooth mouse, when I move the pointer over the region, the MouseRegion widget emits events, but when I do the same using the stylus, nothing happens.

However, the Listener class does emit events when I touch the region with the stylus, and the event instances even include stylus-specific information like pressure. The PointerEvent class even includes a distance field and according to its description, it is supposed to indicate the distance from the pointer to the screen, which seems to be precisely the feature I'm looking for.

This comment suggests that Flutter "is not ready" to support hoverable styluses, but he doesn't seem to be completely sure about it and it was posted a year ago, so maybe something changed.

Finally, when I hover the stylus over the screen while running the app, the following messages are shown on Android Studio's console:

D/ViewRootImpl(16531): updatePointerIcon pointerType = 20001, calling pid = 16531
D/InputManager(16531): setPointerIconType iconId = 20001, callingPid = 16531

So it does seem to detect something. It seems to me Flutter is actively discarding stylus related events and only handling mouse events since on the native side both mouse and pen actions can be handled by the MotionEvent class.

Am I missing something? Is there some other class that is able to handle that kind of event? Or some setting somewhere to enable it? Or is it really not possible at the moment?

Answer 1

I was hoping in ten minutes someone would come here and say "oh, you can just use this class, don't you know how to use google?", but apparently that's not the case. So I decided to look into Flutter's source code.

So I started from the MouseRegion class, which uses a _RawMouseRegion, which uses a RenderMouseRegion. It, then, registers some event handling callbacks using a MouseTrackerAnnotation. Instances of that class are picked up by the MouseTracker, who receives the pointer events and calls all the callbacks interested in them. That is done in the _handleEvent function, whose first two lines are:

if (event.kind != PointerDeviceKind.mouse)
      return;

So I guess I found the culprit. It's possible that this can be fixed by simply adding PointerDeviceKind.stylus to that if statement. Or maybe doing that will make the Earth start spinning backwards or something. A GitHub issue is probably the best place to get an answer for that.

But not all is lost. The MouseTracker class gets its events from a PointerRouter, whose singleton instance is available at GestureBinding.instance.pointerRouter. The PointerRouter has a addGlobalRoute method that lets you register your own callback to receive events and that includes the stylus events that MouseTracker is ignoring.

I'm sure that is not the recommended way of doing things, since it's bypassing a lot of Flutter's internal stuff and that stuff is probably there for a reason. But while there is no "official" way of doing things (and I suspect this very specific use case is nowhere near the top of their list of priorities), it is possible to work around it this way. And hey, I even found a regular widget using the PointerRouter directly as well, so it probably isn't too dangerous.

The event PointerRouter gives you is not as convenient as the one from MouseRegion, since its position is in global coordinates. But you can get the RenderBox of your widget and use globalToLocal to convert the position to local coordinates. Here is a small working example:

class MyHomePage extends StatefulWidget {
  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  //We'll use the render box of the object with this key to transform the event coordinates:
  GlobalKey _containerKey = GlobalKey();
  //Store the render box:
  //(The method to find the render box starts with "find" so I suspect it is doing
  //some non trivial amount of work, so instead of calling it on every event, I'll
  //just store the renderbox here):
  RenderBox _rb;
  String  _message = "Nothing happened";

  _MyHomePageState() {
    //Register a method to receive the pointer events:
    GestureBinding.instance.pointerRouter.addGlobalRoute(_handleEvent);
  }

  @override
  void dispose() {
    //See? We're even disposing of things properly. What an elegant solution.
    super.dispose();
    GestureBinding.instance.pointerRouter.removeGlobalRoute(_handleEvent);
  }

  //We'll receive all the pointer events here:
  void _handleEvent(PointerEvent event) {
    //Make sure it is a stylus event:
    if(event.kind == PointerDeviceKind.stylus && _rb != null) {
      //Convert to the local coordinates:
      Offset coords = _rb.globalToLocal(event.position);

      //Make sure we are inside our component:
      if(coords.dx >= 0 && coords.dx < _rb.size.width && coords.dy >= 0 && coords.dy < _rb.size.height) {
        //Stylus is inside our component and we have its local coordinates. Yay!
        setState(() {
          _message = "dist=${event.distance} x=${coords.dx.toStringAsFixed(1)} y=${coords.dy.toStringAsFixed(1)}";
        });
      }
    }
  }

  @override
  void initState() {
    //Doing it this way, as suggested by this person: https://medium.com/@diegoveloper/flutter-widget-size-and-position-b0a9ffed9407
    WidgetsBinding.instance.addPostFrameCallback(_afterLayout);
    super.initState();
  }

  _afterLayout(_) {
    _rb = _containerKey.currentContext.findRenderObject();
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      body: Center(
          child: Column(
              mainAxisAlignment: MainAxisAlignment.center,
              children: [
                Container(
                    //Event position will be converted to this container's local coordinate space:
                    key: _containerKey,
                    width: 200,
                    height: 200,
                    color: Colors.red
                ),
                Text(_message)
              ]
          )
      ),
    );
  }
}