Use Daydream Controller on HoloLens or outside Daydream?

Question

The daydream controller is awesome and we want to be able to use it in my AR app. It pairs via bluetooth to the HoloLens just fine, but not sure if I can view it in Unity.

Both HoloLens and daydream require their own Unity technical previews. The gvr Controller code is online but seems to speak directly to GVR C api.

Any thoughts on if accessing daydream controller in Unity outside the daydream tech preview is even possible?

Answer 1

It is very possible to access the daydream controller without the GVR services. I am in fact working on that myself and can share what I know.

Getting the data

Using bluetooth gatt you can view all the data available and subscribe to the ID you want. I don't know how you would do this within Hololens/Unity specifically. Basically you want to:

1. Connect to the device
2. Choose the service (0000fe55-0000-1000-8000-00805f9b34fb)
3. Choose the characteristic (00000001-1000-1000-8000-00805f9b34fb)
4. Request notifications for it (00002902-0000-1000-8000-00805f9b34fb)

Android Example:

static final UUID DAYDREAM_CUSTOM_SERVICE = UUID.fromString("0000fe55-0000-1000-8000-00805f9b34fb");
static final UUID DAYDREAM_CHARACTERISTIC = UUID.fromString("00000001-1000-1000-8000-00805f9b34fb");
static final UUID CHARACTERISTIC_UPDATE_NOTIFICATION_DESCRIPTOR_UUID = UUID.fromString("00002902-0000-1000-8000-00805f9b34fb");
...
BluetoothGattService service = gatt.getService(DAYDREAM_CUSTOM_SERVICE);
BluetoothGattCharacteristic characteristic = service.getCharacteristic(DAYDREAM_CHARACTERISTIC);
gatt.setCharacteristicNotification(characteristic, true);
BluetoothGattDescriptor descriptor = characteristic.getDescriptor(CHARACTERISTIC_UPDATE_NOTIFICATION_DESCRIPTOR_UUID);
descriptor.setValue( BluetoothGattDescriptor.ENABLE_NOTIFICATION_VALUE);
gatt.writeDescriptor(descriptor);

I suggest looking up Bluetooth Gatt to understand more about services and characteristics. I also used the BLE Scanner app on the playstore to view a lot of this information before starting with code.

Parsing the data

The device gives 20 bytes of data to work with. It is comprised of time, orientation, acceleration, raw gyro, touch position, and button flags.

Example (laying flat on a table):

5BEBFFB825FDB000041000B00000000000000000
63EFFFB825FDB000041000B00000000000000008
6C73FFB825FDB000041000B00000000000000038

Example (using touch pad):

480BFE87EB00E801841000B00000000191FBA008
4F8FFE47EB00E800441000B0000003FEB1FBA038
5893FE27EB00EFFF041000B0000003FF51FBA000

The byte definition is below:

Bytes:

  - 1: TTTT TTTT * T for time, loops
  - 2: TNNN NNKK * N is sequence number
  - 3: KKKK KKKK * IJK is orientation
  - 4: KKKI IIII
  - 5: IIII IIII
  - 6: JJJJ JJJJ
  - 7: JJJJ JOOO * MNO is acceleration
  - 8: OOOO OOOO
  - 9: OONN NNNN
  -10: NNNN NNNM
  -11: MMMM MMMM
  -12: MMMM CCCC * CDE for raw gyro
  -13: CCCC CCCC
  -14: CDDD DDDD
  -15: DDDD DDEE
  -16: EEEE EEEE
  -17: EEEX XXXX * All the X is the X touch position (8 bits)
  -18: XXXY YYYY * Y the Y touch position (8 bits)
  -19: YYYB BBBB * B the buttons (5 bits | [+][-][App][Home][Click])
  -20: Values vary

With this I have the touch pad and buttons able to work with any bluetooth device I can build apps for. In addition, you would need to add back functionality to reset the device position, control audio, etc.

Using this definition on Android:

static final int CLICK_BTN = 0x1;
static final int HOME_BTN = 0x2;
static final int APP_BTN = 0x4;
static final int VOL_DOWN_BTN = 0x8;
static final int VOL_UP_BTN = 0x10;
float xTouch=0, yTouch=0;
...
final boolean isClickDown = (data[18] & CLICK_BTN) > 0;
final boolean isHomeDown = (data[18] & HOME_BTN) > 0;
final boolean isAppDown = (data[18] & APP_BTN) > 0;
final boolean isVolMinusDown = (data[18] & VOL_DOWN_BTN) > 0;
final boolean isVolPlusDown = (data[18] & VOL_UP_BTN) > 0;

final int time = ((data[0] & 0xFF) << 1 | (data[1] & 0x80) >> 7 );

final int seq = (data[1] & 0x7C) >> 2;

int xOri = (data[1] & 0x03) << 11 | (data[2] & 0xFF) << 3 | (data[3] & 0xE0) >> 5;
xOri = (xOri << 19) >> 19;

int yOri = (data[3] & 0x1F) << 8 | (data[4] & 0xFF);
yOri = (yOri << 19) >> 19;

int zOri = (data[5] & 0xFF) << 5 | (data[6] & 0xF8) >> 3;
zOri = (zOri << 19) >> 19;

int xAcc = (data[6] & 0x07) << 10 | (data[7] & 0xFF) << 2 | (data[8] & 0xC0) >> 6;
xAcc = (xAcc << 19) >> 19;

int yAcc = (data[8] & 0x3F) << 7 | (data[9] & 0xFE) >>> 1;
yAcc = (yAcc << 19) >> 19;

int zAcc = (data[9] & 0x01) << 12 | (data[10] & 0xFF) << 4 | (data[11] & 0xF0) >> 4;
zAcc = (zAcc << 19) >> 19;

int xGyro = ((data[11] & 0x0F) << 9 | (data[12] & 0xFF) << 1 | (data[13] & 0x80) >> 7);
xGyro = (xGyro << 19) >> 19;

int yGyro = ((data[13] & 0x7F) << 6 | (data[14] & 0xFC) >> 2 );
yGyro = (yGyro << 19) >> 19;

int zGyro = ((data[14] & 0x03) << 11 | (data[15] & 0xFF) << 3 | (data[16] & 0xE0) >> 5);
zGyro = (zGyro << 19) >> 19;

xTouch = ((data[16] & 0x1F) << 3 | (data[17] & 0xE0) >> 5) / 255.0f;
yTouch = ((data[17] & 0x1F) << 3 | (data[18] & 0xE0) >> 5) / 255.0f;

This could be optimized but it assigns all the bits except for the last byte. The code value = (value << 19) >> 19 can also be value = (value >> 12) == 0 ? value : ~0x1FFF | value. It is just to extend the signed bit to a 32bit signed int.

I hope this helps and look forward to additional answers.

-- Update 2 --

After looking at the gvr code I found I had some issues with my previous assumptions. It's actually Orientation/Acceleration/Gyro. Also there was 1 more bit for sequence and 1 less for time. I've updated the byte definition and android example.

In addition the X,Y,Z values need to be scaled to floats. For Unity you could put the ints into Vector3s and then use the following. I also negated the x and y in oriVector, for Unity.

Vector3 oriVector = new Vector3 (-xOri, -yOri, zOri);
...
oriVector *= (2 * Mathf.PI / 4095.0);
accVector *= (8 * 9.8 / 4095.0);
gyroVector *= (2048 / 180 * Mathf.PI / 4095.0);

Then to get the rotation you just need the oriVector. Which is actually an axis-angle stored as: unit vector * angle.

public Quaternion orientation = Quaternion.identity;
private Quaternion controllerPoseInSensorSpace = Quaternion.identity;
private Quaternion startFromSensorTransformation = Quaternion.identity;
...
// do this bit after getting the data and scaling it
float sqrMagnitude = oriVector.sqrMagnitude;
if (sqrMagnitude > 0) {
    // extract radian angle
    float w = Mathf.Sqrt (sqrMagnitude);
    // normalize vector
    oriVector /= w;
    // set orientation space
    setOrientationInSensorSpace (w,oriVector);
}
...
// then assign to a Transform
controller.localRotation = this.orientation;
...
// sets orientation with rotation offset
void setOrientationInSensorSpace(float angle, Vector3 axis) {
    // set orientation space
    this.controllerPoseInSensorSpace = Quaternion.AngleAxis(angle*Mathf.Rad2Deg,axis);
    // rotate based on centered offset
    this.orientation = this.startFromSensorTransformation * this.controllerPoseInSensorSpace;
}
...
// after holding home for 600 milliseconds
private void setStartFromSensorTransformation() {
    Vector3 angles = this.controllerPoseInSensorSpace.eulerAngles;
    // reset rotation on Y
    this.startFromSensorTransformation.Set(0,Mathf.Sin(-angles.y * Mathf.Deg2Rad / 2f), 0, Mathf.Cos(angles.y * Mathf.Deg2Rad / 2f));
    // could also reset all, easier to work with
    //this.startFromSensorTransformation = Quaternion.Inverse (this.controllerPoseInSensorSpace);
}

That is everything related to getting daydream working with regular bluetooth devices. I also used the above C# code within Unity3D.

-- Update 1 --

Added a more complete byte definition. The values missing before were the gyro, magnetometer, and acceleration data. They each have three 13bit signed ints. There also seems to be a sequence number tucked in with the time bits.

Going forward

In order to use the device data with other platforms you would need to put the data through similar equations used for 9DoF/IMU devices. I don't have knowledge of exactly how to address this.

The last byte

This is likely reserved for flags and I'm not sure on the meaning but I have some findings to list. Version number is the firmware version of the controller.

1.0.10 (out of the box): 0xF0/0xF8
1.0.10 (previously used with gvr): 0x00/0x08/0x38/0x51
1.0.15: 0x00/0x70