Breathing Detection using Kinect and OpenCV – Part 2 – Peak detection

A few days ago I published a post about how I am using a Microsoft Kinect depth camera and the OpenCV image processing library to identify a test subject from a background, and analyse the series of images from the camera to detect small movements.

The next stage is to calculate the brightness of the test subject at each frame, and turn that into a time series so we can see how it changes with time, and analyse it to detect specific events.

We can use the openCV ‘mean’ function to work out the average brightness of the test image easily, then just add it onto the end of an array, and trim the first value off the start to keep the length the same.
The resulting image and time series are shown below:

 The image here shows that we can extract the subject from the background quite accurately (this is Benjamin’s body and legs as he lies on the floor).  the shading is the movement relative to the average position.

The resulting time series is shown here – the measured data is the blue spiky line.  The red one is the smoothed version (I know I have a half second offset between the two…).

The red dots are peaks detected using a very simple peak searching algorithm.
The chart clearly shows a ‘fidget’ being detected as a large peak.  There is a breathing event at about 8 seconds that has been detected too.

So, the detection system is looking promising – I have had better breathing detection when I was testing it on myself – I think I will have to change the position of the camera a bit to improve sensitivity.

I have now set up a simple python based web server to allow other applications to connect to this one to request the data.

We are getting there.  The outstanding issues are:

  • Memory Leak – after the application has run for 30 min the computer gets very slow and eventually crashes – I suspect a memory leak somewhere – this will have to be fixed!
  • Optimum camera position – I think I can get better breathing detection sensitivity by altering the camera position – will have to experiment a bit.
  • Add some code to identify whether we are looking at Benjamin or just noise – at the moment I analyse the largest bright subject in the image, and assume that is Benjamin – I should probably have a minimum size limit so it gives up if it can not see Benjamin.
  • Summarise what we are seeing automatically – “normal breathing”, “can’t see Benjamin”, “abnormal breathing”, “fidgeting” etc.
  • Modify our monitors that we use to keep an eye on Benjamin to talk to the new web server and display the status messages and raise an alarm if necessary.

The code is available here.

Breathing Detection using Kinect and OpenCV – Part 1 – Image Processing

I have had a go at detecting breathing using an XBox Kinnect depth sensor and the OpenCV image processing library.
I have seen a research paper that did breathing detection, but it relied on fitting the output of the Kinect to a skeleton model to identify the chest area to monitor.  I would like to do it with a less calculation intensive route, so am trying to just use image processing.

To detect the small movements of the chest during breathing, I am doing the following:

Start with a background depth image of empty room.

Grab a depth image from kinect
Subtract Background so we have only the test subject.
Subtract a rolling average background image, and amplify the resulting small differences – makes image very sensitive to small movements.
Resulting video shows image brightness changing due to chest movements from breathing.
We can calculate the average brightness of the test subject image – the value clearly changes due to breathing movements – job for tomorrow night is to do some statistics to work out the breathing rate from this data.
The source code of the python script that does this is the ‘benfinder’ program in the OpenSeizureDetector archive.

    A Microsoft Kinect Based Seizure Detector?


    I have been trying to develop an epileptic seizure detector for our son on-and-off for the last year.   The difficulty is that it has to be non-contact as he is autistic and will not tolerate any contact sensors, and would not lie on a sensor mat etc.
    I had a go at a video based version previously, but struggled with a lot of noise, so put it on hold.

    At the weekend I read a book “OpenCV Computer Vision with Python” by Joseph Howse – this was a really good summary of how to combine openCV video processing into an application – dealing with separating user interface from video processing etc.   Most significantly he pointed out that it is now quite easy to use a Microsoft Kinect sensor with openCV (it looked rather complicated earier in the year when I looked), so thought I should give it a go.

    Connecting Kinect

    When I saw a Kinect sensor in a second hand gadgets shop on Sunday, I had to buy it and see what it can do.

    The first pleasant surprise that I got was that it came with a power supply and had a standard USB plug on it (I thought I would have to solder a USB plug onto it) – I plugged it into my laptop (Xubuntu 13.10), and it was immediately detected as a Video4Linux webcam – a very good start.

    System Software

    I installed the libfreenect library and its python bindings (I built it from source, but I don’t think I had to – there is an ubuntu package python-freenect which would have done it).

    I deviated from the advice in the book here, because the Author suggested using the OpenNI library, but this didn’t seem to work – looks like they no longer support Microsoft Kinect sensors (suspect it is a licensing issue…).   Also the particularly clever software to do skeleton detection (Nite) is not open source so you have to install it as a binary package, which I do not like.   It seems that the way to get OpenNI working with Kinect is to use a wrapper around libfreenect, so I decided to stick with libfreenect.

    The only odd thing is whether you need to be root to use the kinect or not – sometimes it seems I need to access it as root, then after that it works as a normal user – will think about this later – must be something to do with udev rules, so not a big deal at the moment….

    BenFinder Software

    To see whether the Kinect looks promising to use as a seizure detector, wrote a small application based on the framework in Joseph Howse’s book.   I had to modify it to work with libfreenect – basically it is a custom frame grabber.
    The code does the following:
    • Display video streams from kinect, from either the video camera or the infrared depth camera on the kinect – works!  (switch between the two with the ‘d’ key).
    • Save an image to disk (‘s’ key).
    • Subtract a background image from the current image, and display the resulting image (‘b’ key).
    • Record a video (tab key).
    The idea is that it should be able to distinguish Benjamin from the background reliably, so we can then start to analyse his image to see if his movements seem odd (those who know Benjamin will know that ‘odd’ is a bit difficult to define for him!).


    I am very pleased with the output – it looks like it could work – a few images:

    Output from Kinect Video Camera (note the clutter to make detection difficult!)
    Kinect Depth Camera Output – Note black hole created by open door.

    Depth Camera Output with background image subtracted – note that the subject stands out quite clearly.
    Example of me trying to do Benjamin-like behaviours to see if I can be detected.

    Conclusion & What Next

    Background subtraction from the depth camera makes the test subject stand out nice and clearly – should be quite easy to detect him computationally.
    Next stage is to see if the depth camera is sensitive enough to detect breathing (when lying still) – will try by subtracting an each image from the average of the last 30 or so, and amplifying the differences to see if it can be seen.
    If that fails, I will look at Skeltrack to fit a body model to the images and analyse movement of limbs (but this will be much more computationally costly).
    Then I will have to look at infrastructure to deploy this – I will either need a powerful computer in Benjamin’s room to interface with the Kinect and do the analysis, or maybe use a Raspberry Pi to interface with the kinect and serve the depth camera output as a video stream.
    Looking promising – will add another post with the breathing analysis in the new year…

    Further Development of Video Based Seizure Detector

    I have made a bit more progress with the video based epileptic seizure detector.

    Someone on the OpenCV Google Plus page suggested that I look at the Lucas-Kanade feature tracking algorithm, rather than trying to analyse all of the pixels at once like I was doing.

    This looks quite promising.  First you have to decide which features in the image to use – corners are good for tracking.  OpenCV has a neat cv.GoodFeaturesToTrack function which makes suggestions – you give it a couple of parameters, including a ‘quality’ parameter to help it choose.  This gives a list of (x,y) coordinates of the good features to track.  Note that this means ‘good’ mathematically, not necessarily the limbs of the test subject….

    Once you have some features to track, OpenCV again provides a cv.CalcOpticalFlowPyrLK, where you give it the list of features, the previous image and a new image, and it calculates the locations of the features in the new image.

    I have then gone into the fourier analysis that I have been trying for the other types of seizure detection. This time I calculate the speed of each feature over a couple of seconds, and record this as a time series, then calculate the fourier transform to give the frequency spectrum of the motion.   If there is oscillation above a threshold amplitude in a given frequency band for a specified time we raise an alarm as a possible seizure.

    The code is functioning, but is a fair way off being operational yet.  The code for this is in my OpenSeizureDetector github repository (

    The current issues are:

    • I really want to track motion of limbs, but there is no guarantee that cv.GoodFeaturesToTrack will detect these as good features – I can make this more likely by attaching reflective tape, which glows under IR illumination from the night vision camera…if I can persuade Benjamin to wear it.
    • There is something wrong with the frequency calculation still – I can understand a factor of two, but it seems a bit more than that.
    • If the motion is too quick, it looses the point, so I have to set it to re-initialise using GoodFeaturesToTrack periodically.
    • An Example of it working with my daughter doing Benjamin-like behaviour is shown below.   Red circles are drawn around points if a possible seizure is detected.
    • This does not look too good – lots of points detected, and even the reflective strips on the wrists and ankles get lost.  It seems to work better in darkness though, where I get something like the second video, where there are only a few points, and most of those are on my high-vis reflective strips.

    • It does give some nice debugging graphs of the speed measurements and the frequency spectra though.
    So, still a bit of work to do…..

    First go at a Video Based Epileptic Seizure Detector


    I have been working on a system to detect epileptic seizures (fits) to raise an alarm without requiring sensors to be attached to the subject.
    I am going down three routes to try to do this:

    • Accelerometers
    • Audio
    • Video
    This is about my first ‘proof of concept’ go at a video based system.


    I am trying to detect the shaking of a fit.  I will do this by monitoring the signal from an infrared video camera, so it will work in monochrome.  The approach is:
    1. Reduce the size of the image by averaging pixels into ‘meta pixels’ – I do this using the openCV pyrDown function that does the averaging (it is used to build image pyramids of various resolution versions of an image).  I am reducing the 640×480 video stream down to 10×7 pixels to reduce the amount of data I have to handle.
    2. Collect a series of images to produce a time series of images.  I am using 100 images at 30 fps, which is about 3 seconds of video.
    3. For each pixel in the images, calculate the fourier transform of the series of measured pixel intensities – this gives the frequency at which the pixel intensity is varying.
    4. If the amplitude of oscillation at a given frequency is above a threshold value, treat this as a motion at that particular frequency (ie, it could be a fit).
    5. The final version will check that this motion continues for several seconds before raising an alarm.  In this test version, I am just  highlighting the detected frequency of oscillation on the original video stream.


    The code uses the OpenCV library, which provides a lot of video and image handling functions – far more than I understand…
    My intention had been to write it in C, but I struggled with memory leaks (I must have been doing something wrong and not releasing storage, because it just ate all my computer’s memory until it crashed…).
    Instead I used the Python bindings for OpenCV – this ran faster and used much less memory than my C version (this is a sign that I made mistakes in the C one, rather than Python being better!).
    The code for the seizure detector is here – very rough ‘proof of concept’ one at the moment – it will have a major rewrite if it works.

    Test Set Up

    To test the system, I have created a simple ‘test card’ video, which has a number of circles oscillating at different frequencies – the test is to see if I can pick out the various frequencies of oscillation.  The code to produce the test video is here….And here is the test video (not very exciting to watch I’m afraid).
    The circles are oscillating at between 0 and 8 Hz (when played at 30 fps).


    The output of the system is shown in the video below.  The coloured circles indicate areas where motion has been detected.  The thickness of the line and the colour shows the frequency of the detected motion.
    • Blue = <3 hz="" li="">
    • Yellow = 3-6 Hz
    • Red = 6-9 Hz
    • White = >9 Hz
    The things to note are:
    • No motion detected near the stationary 0 Hz circle (good!).
    • <3hz 1="" 2="" and="" circles="" detected="" good="" hz="" li="" motion="" near="" the="">
    • 3-6 Hz motion detected near the 2,3,4 and 5 Hz circles (ok, but why is it near the 2Hz one?)
    • 6-9 Hz motion detected near the 5 and 6 Hz circles (a bit surprising)
    • >9Hz motion detected near the 4 and 7 Hz circles and sometimes the 8Hz one (?)
    So, I think it is sometimes getting the frequency too high.  This may be as simple as how I am doing the  check – it is using the highest frequency that exceeds the threshold.  I think I should update it to use the frequency with maximum amplitude (which exceeds the thershold).
    Also, I have something wrong with positioning the markers to show the motion – I am having to convert from a pixel in the low res image to the location in the high resolution one, and it does not always match up with the position of the moving circles.
    But, it is looking quite promising.  Rather computer intensive at the moment though – it is using pretty much 100% of one of the CPU cores on my Intel Core I5 laptop, so not much chance of getting this to run on a Raspberry Pi, which was my intention.

    Getting Started with OpenCV

    I am starting work on the video version of my Epileptic Seizure detector project, while I wait for a very sensitive microphone to arrive off the slow boat from China, which I will use for the Audio version.

    I am using the OpenCV computer vision library.  What I am hoping to do is to either:

    • Detect the high frequency movement associated with a seizure, or
    • Detect breathing (and raise an alarm if it stops)
    This seems quite similar to the sort of things that MIT have demonstrated some success with last year (   Their code is written in Matlib, which is a commercial package, so not much use to me, so I am looking at doing something similar in OpenCV.
    But first things first, I need to get OpenCV working.  I am going to use plain old C, because I know the syntax (no funny ‘<‘s in the code that you seem to get in C++).  I may move to Python if I start to need to plot graphs to understand what is happening, so I can use the matplotlib graphical library.
    I am using CMake to sort out the make file.  I really don’t know how this works – I must have found a tutorial somewhere that told me to create a file called CMakeLists.txt.  Mine looks like:

    cmake_minimum_required(VERSION 2.8)

    PROJECT( sd )


    ADD_EXECUTABLE( sd Seizure_Detector.c )


    Running ‘cmake’ creates a standard Makefile, and then typing ‘make’ will compile Seizure_Detector.c and link it into an executable called ‘sd’, including the OpenCV libraries.   Seems quite clever.

    The program to detect a seizure is going to have to look for changes in a series of images in a certain frequency range (a few Hz I think).   To detect this I will need to collect a series of images, process them, and do some sort of Fourier transform to detect the frequency components.

    So to get started, grab an image from the networked camera.  This seems to work:
    IplImage *origImg = 0;
    char *window1 = “Original”;
    int main() {
        camera = cvCaptureFromFile(“rtsp://”);
        if(camera!=NULL) {
        while((origImg=cvQueryFrame(camera)) != NULL) {
          procImg = cvCreateImage(cvGetSize(origImg),8,1);

    I can also smooth the image, and do some edge detection:

        while((origImg=cvQueryFrame(camera)) != NULL) {
          procImg = cvCreateImage(cvGetSize(origImg),8,1);
          //cvSmooth(procImg, procImg, CV_GAUSSIAN_5x5,9,9,0,0);
          smoothImg = cvCreateImage(cvGetSize(origImg),8,1);
          cvSmooth(procImg, smoothImg, CV_GAUSSIAN,9,9,0,0);

    Full code at

    I am about to update the code to maintain a set of the most recent 15 images (=1 second of video), so I can do some sort of time series analysis on it to get the frequencies…..