bluez and service/characteristics cache issue with Android

Question

I have some trouble to communicate through BLE between an Android mobile application and a custom embedded device. The embedded device is peripheral and used a custom GATT profile define with Bluez. Everything is working fine. The problem is that sometimes, we deploy new firmware on the embedded device with new services sometimes, but also new characteristics very often. We are using the last Bluezversion (5.54)

Everything is working great with IOS. But with Android, randomly sometimes the mobile app continues to use the old services/characteristics UUID. So basically the mobile app sends a request on wrong characteristics... We are not able to reproduce the issue in a determinist way of course..

We find some stuff about cache : silabs

After looking for some answers on the internet, we find some info relating to bonding. So we tried to add this in the mobile App Android side :

Method m = device.getClass().getMethod(“removeBond”, (Class[]) null);
            m.invoke(device, (Object[]) null);

We also tried to force the refresh the cache from Android side:

final Method refresh = gatt.getClass().getMethod(“refresh”); refresh.invoke(gatt);

But it always no working correctly.. We also check the presence of client supported features and database hash in our characteristics. it seems present as expected :

./btgatt-client -i hci0 -d xxxx
[GATT client]# Service Added - UUID: 00001800-0000-1000-8000-00805f9b34fb start: 0x0001 end: 0x0005
[GATT client]# Service Added - UUID: 00001801-0000-1000-8000-00805f9b34fb start: 0x0006 end: 0x000f
[GATT client]# Service Added - UUID: 0000df00-0000-1000-8000-00805f9b34fb start: 0x0010 end: 0x0012
[GATT client]# Service Added - UUID: 0000d200-0000-1000-8000-00805f9b34fb start: 0x0013 end: 0x0023
[GATT client]# Service Added - UUID: 0000d010-0000-1000-8000-00805f9b34fb start: 0x0024 end: 0x002a
[GATT client]# Service Added - UUID: 0000d100-0000-1000-8000-00805f9b34fb start: 0x002b end: 0x0031
[GATT client]# Service Added - UUID: 0000d700-0000-1000-8000-00805f9b34fb start: 0x0032 end: 0x0045
[GATT client]# Service Added - UUID: 0000d900-0000-1000-8000-00805f9b34fb start: 0x0046 end: 0x0050
[GATT client]# Service Added - UUID: 0000d600-0000-1000-8000-00805f9b34fb start: 0x0051 end: 0x0053
[GATT client]# Service Added - UUID: 0000d500-0000-1000-8000-00805f9b34fb start: 0x0054 end: 0x0072
[GATT client]# Service Added - UUID: 0000d850-0000-1000-8000-00805f9b34fb start: 0x0073 end: 0x0075
[GATT client]# Service Added - UUID: 0000d400-0000-1000-8000-00805f9b34fb start: 0x0076 end: 0x007a
[GATT client]# Service Added - UUID: 0000da00-0000-1000-8000-00805f9b34fb start: 0x007b end: 0x0085
[GATT client]# Service Added - UUID: 0000d300-0000-1000-8000-00805f9b34fb start: 0x0086 end: 0x0099
[GATT client]# Service Added - UUID: 0000d200-0000-1000-8000-00805f9b34fb start: 0x009a end: 0x00b1
[GATT client]# Service Added - UUID: 0000d800-0000-1000-8000-00805f9b34fb start: 0x00b2 end: 0x00ba
[GATT client]# GATT discovery procedures complete

service - start: 0x0001, end: 0x0005, type: primary, uuid: 00001800-0000-1000-8000-00805f9b34fb
      charac - start: 0x0002, value: 0x0003, props: 0x02, ext_props: 0x0000, uuid: 00002a00-0000-1000-8000-00805f9b34fb
      charac - start: 0x0004, value: 0x0005, props: 0x02, ext_props: 0x0000, uuid: 00002a01-0000-1000-8000-00805f9b34fb

service - start: 0x0006, end: 0x000f, type: primary, uuid: 00001801-0000-1000-8000-00805f9b34fb
      charac - start: 0x0007, value: 0x0008, props: 0x20, ext_props: 0x0000, uuid: 00002a05-0000-1000-8000-00805f9b34fb
          descr - handle: 0x0009, uuid: 00002902-0000-1000-8000-00805f9b34fb
      charac - start: 0x000a, value: 0x000b, props: 0x0a, ext_props: 0x0000, uuid: 00002b29-0000-1000-8000-00805f9b34fb
      charac - start: 0x000c, value: 0x000d, props: 0x02, ext_props: 0x0000, uuid: 00002b2a-0000-1000-8000-00805f9b34fb
      charac - start: 0x000e, value: 0x000f, props: 0x02, ext_props: 0x0000, uuid: 00002b3a-0000-1000-8000-00805f9b34fb

service - start: 0x0010, end: 0x0012, type: primary, uuid: 0000df00-0000-1000-8000-00805f9b34fb
      charac - start: 0x0011, value: 0x0012, props: 0x08, ext_props: 0x0000, uuid: 0000df01-0000-1000-8000-00805f9b34fb

service - start: 0x0013, end: 0x0023, type: primary, uuid: 0000d200-0000-1000-8000-00805f9b34fb
      charac - start: 0x0014, value: 0x0015, props: 0x0a, ext_props: 0x0000, uuid: 0000d008-0000-1000-8000-00805f9b34fb
      charac - start: 0x0016, value: 0x0017, props: 0x0a, ext_props: 0x0000, uuid: 0000d007-0000-1000-8000-00805f9b34fb
      charac - start: 0x0018, value: 0x0019, props: 0x08, ext_props: 0x0000, uuid: 0000d006-0000-1000-8000-00805f9b34fb
      charac - start: 0x001a, value: 0x001b, props: 0x08, ext_props: 0x0000, uuid: 0000d005-0000-1000-8000-00805f9b34fb
      charac - start: 0x001c, value: 0x001d, props: 0x08, ext_props: 0x0000, uuid: 0000d004-0000-1000-8000-00805f9b34fb
      charac - start: 0x001e, value: 0x001f, props: 0x02, ext_props: 0x0000, uuid: 0000d003-0000-1000-8000-00805f9b34fb
      charac - start: 0x0020, value: 0x0021, props: 0x08, ext_props: 0x0000, uuid: 0000d002-0000-1000-8000-00805f9b34fb
      charac - start: 0x0022, value: 0x0023, props: 0x08, ext_props: 0x0000, uuid: 0000d001-0000-1000-8000-00805f9b34fb

service - start: 0x0024, end: 0x002a, type: primary, uuid: 0000d010-0000-1000-8000-00805f9b34fb
      charac - start: 0x0025, value: 0x0026, props: 0x02, ext_props: 0x0000, uuid: 0000d013-0000-1000-8000-00805f9b34fb
      charac - start: 0x0027, value: 0x0028, props: 0x08, ext_props: 0x0000, uuid: 0000d012-0000-1000-8000-00805f9b34fb
      charac - start: 0x0029, value: 0x002a, props: 0x08, ext_props: 0x0000, uuid: 0000d011-0000-1000-8000-00805f9b34fb

service - start: 0x002b, end: 0x0031, type: primary, uuid: 0000d100-0000-1000-8000-00805f9b34fb
      charac - start: 0x002c, value: 0x002d, props: 0x02, ext_props: 0x0000, uuid: 0000d103-0000-1000-8000-00805f9b34fb
      charac - start: 0x002e, value: 0x002f, props: 0x0a, ext_props: 0x0000, uuid: 0000d102-0000-1000-8000-00805f9b34fb
      charac - start: 0x0030, value: 0x0031, props: 0x02, ext_props: 0x0000, uuid: 0000d101-0000-1000-8000-00805f9b34fb

service - start: 0x0032, end: 0x0045, type: primary, uuid: 0000d700-0000-1000-8000-00805f9b34fb
      charac - start: 0x0033, value: 0x0034, props: 0x02, ext_props: 0x0000, uuid: 0000d709-0000-1000-8000-00805f9b34fb
      charac - start: 0x0035, value: 0x0036, props: 0x08, ext_props: 0x0000, uuid: 0000d706-0000-1000-8000-00805f9b34fb
      charac - start: 0x0037, value: 0x0038, props: 0x10, ext_props: 0x0000, uuid: 0000d704-0000-1000-8000-00805f9b34fb
          descr - handle: 0x0039, uuid: 00002902-0000-1000-8000-00805f9b34fb
      charac - start: 0x003a, value: 0x003b, props: 0x08, ext_props: 0x0000, uuid: 0000d707-0000-1000-8000-00805f9b34fb
      charac - start: 0x003c, value: 0x003d, props: 0x08, ext_props: 0x0000, uuid: 0000d703-0000-1000-8000-00805f9b34fb
      charac - start: 0x003e, value: 0x003f, props: 0x02, ext_props: 0x0000, uuid: 0000d708-0000-1000-8000-00805f9b34fb
      charac - start: 0x0040, value: 0x0041, props: 0x02, ext_props: 0x0000, uuid: 0000d702-0000-1000-8000-00805f9b34fb
      charac - start: 0x0042, value: 0x0043, props: 0x02, ext_props: 0x0000, uuid: 0000d705-0000-1000-8000-00805f9b34fb
      charac - start: 0x0044, value: 0x0045, props: 0x02, ext_props: 0x0000, uuid: 0000d701-0000-1000-8000-00805f9b34fb

service - start: 0x0046, end: 0x0050, type: primary, uuid: 0000d900-0000-1000-8000-00805f9b34fb
      charac - start: 0x0047, value: 0x0048, props: 0x08, ext_props: 0x0000, uuid: 0000d905-0000-1000-8000-00805f9b34fb
      charac - start: 0x0049, value: 0x004a, props: 0x02, ext_props: 0x0000, uuid: 0000d904-0000-1000-8000-00805f9b34fb
      charac - start: 0x004b, value: 0x004c, props: 0x02, ext_props: 0x0000, uuid: 0000d903-0000-1000-8000-00805f9b34fb
      charac - start: 0x004d, value: 0x004e, props: 0x08, ext_props: 0x0000, uuid: 0000d902-0000-1000-8000-00805f9b34fb
      charac - start: 0x004f, value: 0x0050, props: 0x08, ext_props: 0x0000, uuid: 0000d901-0000-1000-8000-00805f9b34fb

service - start: 0x0051, end: 0x0053, type: primary, uuid: 0000d600-0000-1000-8000-00805f9b34fb
      charac - start: 0x0052, value: 0x0053, props: 0x08, ext_props: 0x0000, uuid: 0000d601-0000-1000-8000-00805f9b34fb

service - start: 0x0054, end: 0x0072, type: primary, uuid: 0000d500-0000-1000-8000-00805f9b34fb
      charac - start: 0x0055, value: 0x0056, props: 0x0a, ext_props: 0x0000, uuid: 0000d516-0000-1000-8000-00805f9b34fb
      charac - start: 0x0057, value: 0x0058, props: 0x0a, ext_props: 0x0000, uuid: 0000d517-0000-1000-8000-00805f9b34fb
      charac - start: 0x0059, value: 0x005a, props: 0x02, ext_props: 0x0000, uuid: 0000d514-0000-1000-8000-00805f9b34fb
      charac - start: 0x005b, value: 0x005c, props: 0x02, ext_props: 0x0000, uuid: 0000d513-0000-1000-8000-00805f9b34fb
      charac - start: 0x005d, value: 0x005e, props: 0x02, ext_props: 0x0000, uuid: 0000d512-0000-1000-8000-00805f9b34fb
      charac - start: 0x005f, value: 0x0060, props: 0x08, ext_props: 0x0000, uuid: 0000d511-0000-1000-8000-00805f9b34fb
      charac - start: 0x0061, value: 0x0062, props: 0x02, ext_props: 0x0000, uuid: 0000d509-0000-1000-8000-00805f9b34fb
      charac - start: 0x0063, value: 0x0064, props: 0x08, ext_props: 0x0000, uuid: 0000d510-0000-1000-8000-00805f9b34fb
      charac - start: 0x0065, value: 0x0066, props: 0x08, ext_props: 0x0000, uuid: 0000d508-0000-1000-8000-00805f9b34fb
      charac - start: 0x0067, value: 0x0068, props: 0x08, ext_props: 0x0000, uuid: 0000d518-0000-1000-8000-00805f9b34fb
      charac - start: 0x0069, value: 0x006a, props: 0x08, ext_props: 0x0000, uuid: 0000d506-0000-1000-8000-00805f9b34fb
      charac - start: 0x006b, value: 0x006c, props: 0x08, ext_props: 0x0000, uuid: 0000d505-0000-1000-8000-00805f9b34fb
      charac - start: 0x006d, value: 0x006e, props: 0x08, ext_props: 0x0000, uuid: 0000d504-0000-1000-8000-00805f9b34fb
      charac - start: 0x006f, value: 0x0070, props: 0x08, ext_props: 0x0000, uuid: 0000d507-0000-1000-8000-00805f9b34fb
      charac - start: 0x0071, value: 0x0072, props: 0x0a, ext_props: 0x0000, uuid: 0000d501-0000-1000-8000-00805f9b34fb

service - start: 0x0073, end: 0x0075, type: primary, uuid: 0000d850-0000-1000-8000-00805f9b34fb
      charac - start: 0x0074, value: 0x0075, props: 0x08, ext_props: 0x0000, uuid: 0000d853-0000-1000-8000-00805f9b34fb

service - start: 0x0076, end: 0x007a, type: primary, uuid: 0000d400-0000-1000-8000-00805f9b34fb
      charac - start: 0x0077, value: 0x0078, props: 0x02, ext_props: 0x0000, uuid: 0000d402-0000-1000-8000-00805f9b34fb
      charac - start: 0x0079, value: 0x007a, props: 0x02, ext_props: 0x0000, uuid: 0000d401-0000-1000-8000-00805f9b34fb

service - start: 0x007b, end: 0x0085, type: primary, uuid: 0000da00-0000-1000-8000-00805f9b34fb
      charac - start: 0x007c, value: 0x007d, props: 0x02, ext_props: 0x0000, uuid: 0000da05-0000-1000-8000-00805f9b34fb
      charac - start: 0x007e, value: 0x007f, props: 0x08, ext_props: 0x0000, uuid: 0000da04-0000-1000-8000-00805f9b34fb
      charac - start: 0x0080, value: 0x0081, props: 0x02, ext_props: 0x0000, uuid: 0000da03-0000-1000-8000-00805f9b34fb
      charac - start: 0x0082, value: 0x0083, props: 0x02, ext_props: 0x0000, uuid: 0000da02-0000-1000-8000-00805f9b34fb
      charac - start: 0x0084, value: 0x0085, props: 0x08, ext_props: 0x0000, uuid: 0000da01-0000-1000-8000-00805f9b34fb

service - start: 0x0086, end: 0x0099, type: primary, uuid: 0000d300-0000-1000-8000-00805f9b34fb
      charac - start: 0x0087, value: 0x0088, props: 0x02, ext_props: 0x0000, uuid: 0000d309-0000-1000-8000-00805f9b34fb
      charac - start: 0x0089, value: 0x008a, props: 0x08, ext_props: 0x0000, uuid: 0000d30a-0000-1000-8000-00805f9b34fb
      charac - start: 0x008b, value: 0x008c, props: 0x02, ext_props: 0x0000, uuid: 0000d308-0000-1000-8000-00805f9b34fb
      charac - start: 0x008d, value: 0x008e, props: 0x0a, ext_props: 0x0000, uuid: 0000d307-0000-1000-8000-00805f9b34fb
      charac - start: 0x008f, value: 0x0090, props: 0x02, ext_props: 0x0000, uuid: 0000d306-0000-1000-8000-00805f9b34fb
      charac - start: 0x0091, value: 0x0092, props: 0x02, ext_props: 0x0000, uuid: 0000d305-0000-1000-8000-00805f9b34fb
      charac - start: 0x0093, value: 0x0094, props: 0x02, ext_props: 0x0000, uuid: 0000d304-0000-1000-8000-00805f9b34fb
      charac - start: 0x0095, value: 0x0096, props: 0x08, ext_props: 0x0000, uuid: 0000d302-0000-1000-8000-00805f9b34fb
      charac - start: 0x0097, value: 0x0098, props: 0x12, ext_props: 0x0000, uuid: 0000d301-0000-1000-8000-00805f9b34fb
          descr - handle: 0x0099, uuid: 00002902-0000-1000-8000-00805f9b34fb

service - start: 0x009a, end: 0x00b1, type: primary, uuid: 0000d200-0000-1000-8000-00805f9b34fb
      charac - start: 0x009b, value: 0x009c, props: 0x02, ext_props: 0x0000, uuid: 0000d211-0000-1000-8000-00805f9b34fb
      charac - start: 0x009d, value: 0x009e, props: 0x08, ext_props: 0x0000, uuid: 0000d210-0000-1000-8000-00805f9b34fb
      charac - start: 0x009f, value: 0x00a0, props: 0x12, ext_props: 0x0000, uuid: 0000d209-0000-1000-8000-00805f9b34fb
          descr - handle: 0x00a1, uuid: 00002902-0000-1000-8000-00805f9b34fb
      charac - start: 0x00a2, value: 0x00a3, props: 0x02, ext_props: 0x0000, uuid: 0000d208-0000-1000-8000-00805f9b34fb
      charac - start: 0x00a4, value: 0x00a5, props: 0x08, ext_props: 0x0000, uuid: 0000d206-0000-1000-8000-00805f9b34fb
      charac - start: 0x00a6, value: 0x00a7, props: 0x08, ext_props: 0x0000, uuid: 0000d207-0000-1000-8000-00805f9b34fb
      charac - start: 0x00a8, value: 0x00a9, props: 0x08, ext_props: 0x0000, uuid: 0000d205-0000-1000-8000-00805f9b34fb
      charac - start: 0x00aa, value: 0x00ab, props: 0x02, ext_props: 0x0000, uuid: 0000d204-0000-1000-8000-00805f9b34fb
      charac - start: 0x00ac, value: 0x00ad, props: 0x02, ext_props: 0x0000, uuid: 0000d203-0000-1000-8000-00805f9b34fb
      charac - start: 0x00ae, value: 0x00af, props: 0x02, ext_props: 0x0000, uuid: 0000d202-0000-1000-8000-00805f9b34fb
      charac - start: 0x00b0, value: 0x00b1, props: 0x02, ext_props: 0x0000, uuid: 0000d201-0000-1000-8000-00805f9b34fb

service - start: 0x00b2, end: 0x00ba, type: primary, uuid: 0000d800-0000-1000-8000-00805f9b34fb
      charac - start: 0x00b3, value: 0x00b4, props: 0x02, ext_props: 0x0000, uuid: 0000d805-0000-1000-8000-00805f9b34fb
      charac - start: 0x00b5, value: 0x00b6, props: 0x08, ext_props: 0x0000, uuid: 0000d804-0000-1000-8000-00805f9b34fb
      charac - start: 0x00b7, value: 0x00b8, props: 0x02, ext_props: 0x0000, uuid: 0000d803-0000-1000-8000-00805f9b34fb
      charac - start: 0x00b9, value: 0x00ba, props: 0x08, ext_props: 0x0000, uuid: 0000d801-0000-1000-8000-00805f9b34fb

uuid 2B29 and 2B2A are present in the service discovery...

it is funny also because the demon bluetoothd is crashing when I am using btgatt-client from 5.54. I have to run the test with 5.50 gatt client...

EDIT1 :

I was able to reproduce the issue on a customer device. I tried to disable the cache by putting in the bluetoothd conf under /etc/bluetooth/main.conf:

[GATT]
Cache = no

The issue is always present after this change. By the way I was trying to remove the cache by deleting the directory in /var/lib/bluetooth. But in fact I have not this directory in my case. Indeed all our main rootfs is read only on our device. So bluetoothd is not able to create the directory.. So perhaps that this is not a real cache issue ?

Answer 1

Explanation

There's an issue with some versions of Android when the bonding (with LE) operation occurs. In Nordic's source code, there's indeed a comment (source) that confirms it:

// 1. On devices running Android 4.3-5.x, 8.x and 9.0 the Service Changed
//    characteristic needs to be enabled by the app (for bonded devices).
//    The request will be ignored if there is no Service Changed characteristic.
// This "fix" broke this in Android 8:
// https://android-review.googlesource.com/c/platform/system/bt/+/239970
if (Build.VERSION.SDK_INT < Build.VERSION_CODES.M
    || Build.VERSION.SDK_INT == Build.VERSION_CODES.O
    || Build.VERSION.SDK_INT == Build.VERSION_CODES.O_MR1
    || Build.VERSION.SDK_INT == Build.VERSION_CODES.P)
        initQueue.addFirst(Request.newEnableServiceChangedIndicationsRequest());

Basically, there's a characteristic in charge of indicating whether or not the cached attributes should be updated when there's been a change in services. And it's been broken at some point.

Solution

The comment from Nordic is quite self-explanatory: you need to enable the Service Changed manually. Here's how they do it: part 1 and part 2. And here's a concise Kotlin version:

private val GENERIC_ATTRIBUTE_SERVICE =
    UUID.fromString("00001801-0000-1000-8000-00805f9b34fb")
private val SERVICE_CHANGED_CHARACTERISTIC =
    UUID.fromString("00002a05-0000-1000-8000-00805f9b34fb")
private val CLIENT_CHARACTERISTIC_CONFIG_DESCRIPTOR_UUID =
    UUID.fromString("00002902-0000-1000-8000-00805f9b34fb")

private fun enableServiceChanged(gatt: BluetoothGatt) {
    val service = gatt.getService(GENERIC_ATTRIBUTE_SERVICE) ?: return
    val characteristic = service.getCharacteristic(SERVICE_CHANGED_CHARACTERISTIC)
        ?: return
    
    val properties = characteristic.properties
    if (properties and BluetoothGattCharacteristic.PROPERTY_INDICATE === 0)
        return
    
    gatt.setCharacteristicNotification(characteristic, true)
    characteristic.getDescriptor(CLIENT_CHARACTERISTIC_CONFIG_DESCRIPTOR_UUID)?.apply {
        value = BluetoothGattDescriptor.ENABLE_INDICATION_VALUE
        internalWriteDescriptorWorkaround(gatt, this)
    }
}

private fun internalWriteDescriptorWorkaround(
        gatt: BluetoothGatt,
        descriptor: BluetoothGattDescriptor
) {
        with (descriptor.characteristic) {
            val originalWriteType = writeType
            writeType = BluetoothGattCharacteristic.WRITE_TYPE_DEFAULT
            gatt.writeDescriptor(descriptor)
            writeType = originalWriteType
        }
}