iOS Heart rate detection Algorithm

Question

I'm trying to implement heart beat recording functionality in an app i'm developing.

The preferred method of doing this is by using the iPhone's camera with the light on, having the user place their finger on the lens, and detecting fluctuations in the video feed, which correspond to the user's heart.

I found a very good starting point with the following stack overflow question here

The question provides useful code to plot a heart beat time graph.

It shows how to start an AVCaptureSession and turn the camera's light on like so:

session = [[AVCaptureSession alloc] init];  AVCaptureDevice* camera = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo]; if([camera isTorchModeSupported:AVCaptureTorchModeOn]) {     [camera lockForConfiguration:nil];     camera.torchMode=AVCaptureTorchModeOn;     //  camera.exposureMode=AVCaptureExposureModeLocked;     [camera unlockForConfiguration]; } // Create a AVCaptureInput with the camera device NSError *error=nil; AVCaptureInput* cameraInput = [[AVCaptureDeviceInput alloc] initWithDevice:camera error:&error]; if (cameraInput == nil) {     NSLog(@"Error to create camera capture:%@",error); }  // Set the output AVCaptureVideoDataOutput* videoOutput = [[AVCaptureVideoDataOutput alloc] init];  // create a queue to run the capture on dispatch_queue_t captureQueue=dispatch_queue_create("catpureQueue", NULL);  // setup our delegate [videoOutput setSampleBufferDelegate:self queue:captureQueue];  // configure the pixel format videoOutput.videoSettings = [NSDictionary dictionaryWithObjectsAndKeys:[NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA], (id)kCVPixelBufferPixelFormatTypeKey,                              nil]; videoOutput.minFrameDuration=CMTimeMake(1, 10);  // and the size of the frames we want [session setSessionPreset:AVCaptureSessionPresetLow];  // Add the input and output [session addInput:cameraInput]; [session addOutput:videoOutput];  // Start the session [session startRunning]; 

Self in this example must be an <AVCaptureVideoDataOutputSampleBufferDelegate> And will therefore have to implement the following method to obtain raw camera data:

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection { static int count=0; count++; // only run if we're not already processing an image // this is the image buffer CVImageBufferRef cvimgRef = CMSampleBufferGetImageBuffer(sampleBuffer); // Lock the image buffer CVPixelBufferLockBaseAddress(cvimgRef,0); // access the data int width=CVPixelBufferGetWidth(cvimgRef); int height=CVPixelBufferGetHeight(cvimgRef); // get the raw image bytes uint8_t *buf=(uint8_t *) CVPixelBufferGetBaseAddress(cvimgRef); size_t bprow=CVPixelBufferGetBytesPerRow(cvimgRef); float r=0,g=0,b=0; for(int y=0; y<height; y++) {     for(int x=0; x<width*4; x+=4) {         b+=buf[x];         g+=buf[x+1];         r+=buf[x+2];         //          a+=buf[x+3];     }     buf+=bprow; } r/=255*(float) (width*height); g/=255*(float) (width*height); b/=255*(float) (width*height);  float h,s,v;  RGBtoHSV(r, g, b, &h, &s, &v);  // simple highpass and lowpass filter   static float lastH=0; float highPassValue=h-lastH; lastH=h; float lastHighPassValue=0; float lowPassValue=(lastHighPassValue+highPassValue)/2;  lastHighPassValue=highPassValue;      //low pass value can now be used for basic heart beat detection   } 

RGB is converted to HSV and it is Hue that is monitored for fluctuations.

And RGB to HSV is implemented as follows

void RGBtoHSV( float r, float g, float b, float *h, float *s, float *v ) { float min, max, delta;  min = MIN( r, MIN(g, b ));  max = MAX( r, MAX(g, b ));  *v = max; delta = max - min;  if( max != 0 )     *s = delta / max; else {     // r = g = b = 0      *s = 0;      *h = -1;      return; } if( r == max )     *h = ( g - b ) / delta;  else if( g == max )     *h=2+(b-r)/delta; else      *h=4+(r-g)/delta;  *h *= 60; if( *h < 0 )      *h += 360; } 

The low pass value calculated in capureOutput: initially provides erratic data, but then stabilises to the following:

2013-11-04 16:18:13.619 SampleHeartRateApp[1743:1803] -0.071218 2013-11-04 16:18:13.719 SampleHeartRateApp[1743:1803] -0.050072 2013-11-04 16:18:13.819 SampleHeartRateApp[1743:1803] -0.011375 2013-11-04 16:18:13.918 SampleHeartRateApp[1743:1803] 0.018456 2013-11-04 16:18:14.019 SampleHeartRateApp[1743:1803] 0.059024 2013-11-04 16:18:14.118 SampleHeartRateApp[1743:1803] 0.052198 2013-11-04 16:18:14.219 SampleHeartRateApp[1743:1803] 0.078189 2013-11-04 16:18:14.318 SampleHeartRateApp[1743:1803] 0.046035 2013-11-04 16:18:14.419 SampleHeartRateApp[1743:1803] -0.113153 2013-11-04 16:18:14.519 SampleHeartRateApp[1743:1803] -0.079792 2013-11-04 16:18:14.618 SampleHeartRateApp[1743:1803] -0.027654 2013-11-04 16:18:14.719 SampleHeartRateApp[1743:1803] -0.017288 

An example of the erratic data provided initially is here:

2013-11-04 16:17:28.747 SampleHeartRateApp[1743:3707] 17.271435 2013-11-04 16:17:28.822 SampleHeartRateApp[1743:1803] -0.049067 2013-11-04 16:17:28.922 SampleHeartRateApp[1743:1803] -6.524201 2013-11-04 16:17:29.022 SampleHeartRateApp[1743:1803] -0.766260 2013-11-04 16:17:29.137 SampleHeartRateApp[1743:3707] 9.956407 2013-11-04 16:17:29.221 SampleHeartRateApp[1743:1803] 0.076244 2013-11-04 16:17:29.321 SampleHeartRateApp[1743:1803] -1.049292 2013-11-04 16:17:29.422 SampleHeartRateApp[1743:1803] 0.088634 2013-11-04 16:17:29.522 SampleHeartRateApp[1743:1803] -1.035559 2013-11-04 16:17:29.621 SampleHeartRateApp[1743:1803] 0.019196 2013-11-04 16:17:29.719 SampleHeartRateApp[1743:1803] -1.027754 2013-11-04 16:17:29.821 SampleHeartRateApp[1743:1803] 0.045803 2013-11-04 16:17:29.922 SampleHeartRateApp[1743:1803] -0.857693 2013-11-04 16:17:30.021 SampleHeartRateApp[1743:1803] 0.061945 2013-11-04 16:17:30.143 SampleHeartRateApp[1743:1803] -0.701269 

The low pass value goes positive whenever there is a heart beat. So I tried a very simple live detection algorithm which basically looks at the current value, and sees if it is positive, it also looks at the previous value, if negative it detects negative going to positive and plays a beep sound.

The problem with this is the data isn't always as perfect as the above, sometimes there's anomalous positive readings in amongst negative readings and vice versa.

A graph of the low pass value in time looks like this:

Interestingly the above anomaly is quite common, if I record a graph for a while i'll see a very similar shaped anomaly multiple times.

In my very simple beat detection algorithm, if an anomaly as shown above occurs the counted number of beats in the detection period (10 seconds) can shoot up by 4 or 5 beats. This makes the calculated BPM very inaccurate. But as simple as it is it does work around 70% of the time.

To combat this problem I tried the following.

1.Started recording last 3 low pass values in an array

2.Then looked to see whether or not the middle value had two smaller values surrounding it before and after. (Basic peak detection)

3.Counted this scenario as a beat and added it to the running total of beats in a given time.

This method is however just as vulnerable to the anomalies as any other. And actually seemed to be a worse method. (When playing live beeps after detection they seemed far more erratic than the positive to negative algorithm)

My question is can you help me come up with an algorithm that can reliably detect when a heart beat occurs with reasonable accuracy.

Another problem I realise that i'm going to have to address is detecting whether or not a user's finger is on the lens.

I thought about detecting erratic low pass values but the problem there is the low pass filter accounts for erratic values and smooths them out over time. So help there would be appreciated too.

Thanks for your time.

Answer 1

The answer to this question is a little bit involved, as you need to do several things to process the signal and there is no single "right" way to do this. However, for your filter you want to use a band-pass filter. This type of filter allows you to specify a range of frequencies that are accepted both at the high and low ends. For a human heart beat, we know what these bounds should be (no less than 40 bpm and no higher than 250 bpm) so we can create a filter that removes frequencies outside of this range. The filter also moves the data to be centered at zero, so peak detection becomes much easier. This filter will give you a much more smooth signal, even if your users increases/decreases their finger pressure (to a certain degree). After that, additional smoothing and outlier removal will also need to happen.

A specific type of band-pass filter that I have used is a butterworth filter. This is a little involved to create by hand since the filter changes based on the frequency you are collecting your data at. Fortunately, there is a website that can help with this here. If you are collecting your data at 30 fps, then the frequency will be 30 hz.

I have created a project that wraps all of this together and detects a user's heart rate well enough to include it in my app on the iOS app store. I have made the heart rate detection code available on github.

Answer 2

I presume you're using your own finger. Are you sure you don't have an irregular heartbeat? Plus, you're going to want to handle people with irregular heartbeats. In other words, you should test with a wide variety of input values. Definitely try it on your parents or other older relatives, as they might be more likely to have heart issues. Other than that, your basic problem is that your input source is going to be noisy; you're basically trying to recover signal from that noise. Sometimes that will be impossible, and you're going to have to decide if you want to bake noise into your report or just ignore the data stream when it's too noisy.

Keep trying different filter values; maybe you need an even lower pass filter. From the comments, it sounds like your low pass filter was not good; there's tons of resources on filtering out there in the web. If you've got good visualization tools that will be the best way to test your algorithm.

You can try down-sampling the data, which will smooth it out. You might also want to discard samples which lie outside a valid range, either by discarding the value altogether, replacing it with the average of the previous and next sample, and/or by clamping it to some predetermined or calculated maximum.

My biggest beef with these sorts of applications is that hiccups are treated as real, live data. One of the bikes at my gym gives useless bpm readings because, every so often, it can't find my pulse and suddenly thinks my heart's going at 300 bpm. (Which it isn't; I've asked my doctor.) For a 20-minute session the average it has is useless. I think it's more useful to see the average of the (e.g.) last ten normal beats plus the anomaly rate rather than "I tried to cram the last 20 seconds into this algorithm and here's the garbage it spat out". If you can create a separate filter that indicates anomalies, I think you'll have a much more useful application.