First go at a Video Based Epileptic Seizure Detector

Background

I have been working on a system to detect epileptic seizures (fits) to raise an alarm without requiring sensors to be attached to the subject.
I am going down three routes to try to do this:

  • Accelerometers
  • Audio
  • Video
This is about my first ‘proof of concept’ go at a video based system.

Approach

I am trying to detect the shaking of a fit.  I will do this by monitoring the signal from an infrared video camera, so it will work in monochrome.  The approach is:
  1. Reduce the size of the image by averaging pixels into ‘meta pixels’ – I do this using the openCV pyrDown function that does the averaging (it is used to build image pyramids of various resolution versions of an image).  I am reducing the 640×480 video stream down to 10×7 pixels to reduce the amount of data I have to handle.
  2. Collect a series of images to produce a time series of images.  I am using 100 images at 30 fps, which is about 3 seconds of video.
  3. For each pixel in the images, calculate the fourier transform of the series of measured pixel intensities – this gives the frequency at which the pixel intensity is varying.
  4. If the amplitude of oscillation at a given frequency is above a threshold value, treat this as a motion at that particular frequency (ie, it could be a fit).
  5. The final version will check that this motion continues for several seconds before raising an alarm.  In this test version, I am just  highlighting the detected frequency of oscillation on the original video stream.

Code

The code uses the OpenCV library, which provides a lot of video and image handling functions – far more than I understand…
My intention had been to write it in C, but I struggled with memory leaks (I must have been doing something wrong and not releasing storage, because it just ate all my computer’s memory until it crashed…).
Instead I used the Python bindings for OpenCV – this ran faster and used much less memory than my C version (this is a sign that I made mistakes in the C one, rather than Python being better!).
The code for the seizure detector is here – very rough ‘proof of concept’ one at the moment – it will have a major rewrite if it works.

Test Set Up

To test the system, I have created a simple ‘test card’ video, which has a number of circles oscillating at different frequencies – the test is to see if I can pick out the various frequencies of oscillation.  The code to produce the test video is here….And here is the test video (not very exciting to watch I’m afraid).
The circles are oscillating at between 0 and 8 Hz (when played at 30 fps).

Results

The output of the system is shown in the video below.  The coloured circles indicate areas where motion has been detected.  The thickness of the line and the colour shows the frequency of the detected motion.
  • Blue = <3 hz="" li="">
  • Yellow = 3-6 Hz
  • Red = 6-9 Hz
  • White = >9 Hz
The things to note are:
  • No motion detected near the stationary 0 Hz circle (good!).
  • <3hz 1="" 2="" and="" circles="" detected="" good="" hz="" li="" motion="" near="" the="">
  • 3-6 Hz motion detected near the 2,3,4 and 5 Hz circles (ok, but why is it near the 2Hz one?)
  • 6-9 Hz motion detected near the 5 and 6 Hz circles (a bit surprising)
  • >9Hz motion detected near the 4 and 7 Hz circles and sometimes the 8Hz one (?)
So, I think it is sometimes getting the frequency too high.  This may be as simple as how I am doing the  check – it is using the highest frequency that exceeds the threshold.  I think I should update it to use the frequency with maximum amplitude (which exceeds the thershold).
Also, I have something wrong with positioning the markers to show the motion – I am having to convert from a pixel in the low res image to the location in the high resolution one, and it does not always match up with the position of the moving circles.
But, it is looking quite promising.  Rather computer intensive at the moment though – it is using pretty much 100% of one of the CPU cores on my Intel Core I5 laptop, so not much chance of getting this to run on a Raspberry Pi, which was my intention.

Getting Started with OpenCV

I am starting work on the video version of my Epileptic Seizure detector project, while I wait for a very sensitive microphone to arrive off the slow boat from China, which I will use for the Audio version.

I am using the OpenCV computer vision library.  What I am hoping to do is to either:

  • Detect the high frequency movement associated with a seizure, or
  • Detect breathing (and raise an alarm if it stops)
This seems quite similar to the sort of things that MIT have demonstrated some success with last year (http://people.csail.mit.edu/mrub/vidmag/).   Their code is written in Matlib, which is a commercial package, so not much use to me, so I am looking at doing something similar in OpenCV.
But first things first, I need to get OpenCV working.  I am going to use plain old C, because I know the syntax (no funny ‘<‘s in the code that you seem to get in C++).  I may move to Python if I start to need to plot graphs to understand what is happening, so I can use the matplotlib graphical library.
I am using CMake to sort out the make file.  I really don’t know how this works – I must have found a tutorial somewhere that told me to create a file called CMakeLists.txt.  Mine looks like:

cmake_minimum_required(VERSION 2.8)

PROJECT( sd )

FIND_PACKAGE( OpenCV REQUIRED )

ADD_EXECUTABLE( sd Seizure_Detector.c )

TARGET_LINK_LIBRARIES( sd ${OpenCV_LIBS} )

Running ‘cmake’ creates a standard Makefile, and then typing ‘make’ will compile Seizure_Detector.c and link it into an executable called ‘sd’, including the OpenCV libraries.   Seems quite clever.

The program to detect a seizure is going to have to look for changes in a series of images in a certain frequency range (a few Hz I think).   To detect this I will need to collect a series of images, process them, and do some sort of Fourier transform to detect the frequency components.

So to get started, grab an image from the networked camera.  This seems to work:
IplImage *origImg = 0;
char *window1 = “Original”;
int main() {
    camera = cvCaptureFromFile(“rtsp://192.168.1.18/live_mpeg4.sdp”);
    if(camera!=NULL) {
    cvNamedWindow(window1,CV_WINDOW_AUTOSIZE);
    while((origImg=cvQueryFrame(camera)) != NULL) {
      procImg = cvCreateImage(cvGetSize(origImg),8,1);
      cvShowImage(window1,origImg);
    }
}
}

I can also smooth the image, and do some edge detection:

    while((origImg=cvQueryFrame(camera)) != NULL) {
      procImg = cvCreateImage(cvGetSize(origImg),8,1);
      cvCvtColor(origImg,procImg,CV_BGR2GRAY);
      //cvSmooth(procImg, procImg, CV_GAUSSIAN_5x5,9,9,0,0);
      smoothImg = cvCreateImage(cvGetSize(origImg),8,1);
      cvSmooth(procImg, smoothImg, CV_GAUSSIAN,9,9,0,0);
      cvCanny(smoothImg,procImg,0,20,3);
   
      cvShowImage(window1,origImg);
      cvShowImage(window2,procImg);
}

Full code at https://github.com/jones139/arduino-projects/tree/master/seizure_detector/video_version.

I am about to update the code to maintain a set of the most recent 15 images (=1 second of video), so I can do some sort of time series analysis on it to get the frequencies…..

Epileptic Seizure Detector (3)

I installed an accelerometer on the underside of the floorboard where my son sleeps to see if there is any chance of detecting him having an epileptic seizure by the vibrations induced in the floor.

I used the software for the seizure detector that I have been working with before (see earlier post).
The software logs data to an SD card in Comma-Separated-Values (CSV) format, recording the raw accelerometer reading, and the calculated spectrum once per second.  This left me with 26 MB of data to analyse after running it all night…..
I wrote a little script in Python that uses the matplotlib library to visualise it.   I create a 2 dimensional array where there is one column for each record in the file (ie once column per second).  The rows are the frequency bins from the fourier transform.  The values in the array are the amplitude of the spectral component from the fourier transform.
The idea is that I can look for periods where I have seen high levels of vibration at different frequencies to see if it could detect a seizure.  The results are shown below:
Here you can see the background noise of a few counts in the 1-7 Hz range.   The 13-15Hz signal is a mystery to me.  I wonder if it is the resonant frequency of our house?
Up to 170 sec is just me walking around the room – discouragingly little response – maybe something at about 10 Hz.  This is followed by me sitting still on the floorboard up to ~200 seconds (The 10 Hz signal disappears?)
The period at ~200 seconds is me stamping vigorously on the floorboard, to prove that the system is alive.
Unfortunately the period after 200 seconds is me lying on the floorboard shaking as vigorously as I could, and it is indistinguishable from the normal activity before 170 seconds.
So, I think attaching a simple IC accelerometer to a floorboard will not work – attaching it directly to the patient’s forearm looks very promising, but not the floorboard.
I am working on an audio breathing detector now as the next non-contact option….
The code to analyse the data and produce the above chart can be found on github.  It uses the excellent matplotlib scientific visualisation package.

Epileptic Seizure Detector (2)

Update to add another spectrum…

I have been working on setting up the Epileptic Seizure Detector.  I tried wearing it for a while, and simulating the shaking associated with a tonic-clonic seizure.   Some example spectra collected on the memory card are shown below:

This shows that the background noise level is at about 4 counts.   
Wearing the accelerometer on the biscep gives a peak up to about 8 counts at 7 Hz, but it is not well defined.  
Wearing the accelerometer on the wrist gives a much more well defined peak at 6-7 Hz. (and it raised an alarm nicely).
I have also tried an ADXL345 digital accelerometer.  The performance is similar to the analogue one, but I think it may be slightly more sensitive.  Example spectra with the accelerometer attached to the biscep are shown below.  ONe is a simulated fit.  The other is a false alarm going down the stairs.  Not that much difference!
Therefore I think there is scope for this set up to work if it is worn as a wrist watch, but just attaching it to other parts of the body may not be sensitive enough.
I wonder if I could make a wrist sensor that is watch sized, with a wireless link to a processor / alarm unit?
Not sure if I will be able to persuade Benjamin to wear a wrist sensor though….Might have to think about microphones.

Soldering onto Surface Mount ICs

I recently bought an accelerometer IC to use on my epileptic seizure detector project.  It is a tiny surface mount device as you can see below.

I gave a lot of thought to how to connect wires to it.  I did consider conductive glue, but it would be difficult to hold them all still for long enough for it to set, so I went back to solder.  This is how I did it…
1.  Mount the IC onto stripboard using apoxy adhesive:
2.  While the glue is setting, modify the soldering iron by wrapping some 1mm2 copper wire around the tip to give a very fine tip.  Use solder to increase the heat transfer between the wire and the tip:
3.  Tin the solder pads on the IC, using some very fine solder (I got some 32swg solder off ebay).
4.  Obtain some very fine copper wire (I disassembled some cheap alarm flexible cable, and used strands from that).
5.  Hold a strand of wire onto a solder pad, and touch it with the soldering iron to melt the solder and create the joint.
6.  Repeat for all connections:
7.  Route the fine wires to the copper tracks, and solder on.  I used the insulation from the original alarm  cable to prevent short circuits:
Fiddly, and not very neat, but it worked for me – it is being used in my prototype epileptic seizure detector.

Epileptic Seizure Detector (1)

Our son worried us a bit a couple of weeks ago when he had quite a nasty fit, so I have been thinking about making an alarm to warn a carer that a person in their charge is having a seizure.

There are a few different ways to do this that I have thought of:

  1. Detect Movement using an accelerometer
  2. Detect the sounds associated with the movement using a microphone
  3. Monitor the movement with a CCTV camera and use image processing to detect the abnormal movement.
I am trying option 1 (accelerometer) first, but am working on the CCTV approach in parallel by learning OpenCV.
Because our son is autistic, it will be very difficult to get him to wear a device, so I hope to detect movement through the floorboard where he sleeps, but this will be much less sensitive than detecting it directly.  Therefore, this first proof of concept version is working by attaching the accelerometer to a limb to see if I can get it working.  The issues with it are:
  1. We do not want false alarms caused by normal movement – I am addressing this by using a fourier transform to filter out only a range of frequencies of movement, in the hope that I can select the characteristic shaking of a seizure, but not detect too much normal movement.
  2. A quick shake should not raise an alarm, so to set off an alarm the acceleration in the appropriate frequency band should be more than a threshold value for a specified length of time (3 sec currently).  This will give a warning ‘pip’.   If the shaking continues for 10 sec, it raises a buzzing alarm.
  3. Sensitivity will be a problem for detecting it through the floor – will need to work on that another evening.
The system uses an Arduino microcontroller, connected to a Freescale MMA7361 three axis accelerometer.   The accelerometer is a tiny (5mm x 3mm) surface mount device, so soldering it is a challenge – you can see how I did it here.
To enable data logging so I can tune it to get the frequency response, threshold etc. the arduino is also connected to a real time clock module and a SD card module.
The completed prototype is shown below:

The code is in my Arduino Projects github repository.
And here is a simple demonstration of it working – you can hear the warning ‘pip’ and the alarm ‘buzz’ in the background when I shake my arm to simulate a seizure. 
Still quite a bit of work to do – build it on stripboard to make it more robust, then try attaching it to the floor and seeing if I can detect any signal from someone shaking.  If not, I will have to minaturise it to make it wearable, and train Benjamin to wear it….