When I first started down the road of trying to make a seizure detector (in 2013!) I looked at video processing to detect the high frequency movement (see this post). In the end I abandoned it because it was very computationally expensive, I discovered Pebble watches, and most importantly, got Benjamin to wear a watch.
Recently Benjamin has had some partial seizures that do not have limb shaking, so the watch based seizure detector does not pick them up, so I am getting back to thinking about video processing to look for him holding ‘odd’ postures for an extended period.
The first stage though is to analyse a video stream to identify ‘odd’ postures.
What do we mean by ‘Odd’ anyway?
Benjamin is autistic and has learning difficulties, so his behaviour tends to be quite unusual…..
What I mean by odd as far as this project is concerned is adopting an unusual (for Benjamin) pose, and holding it for an extended time (say 15-20 sec).
‘Odd’ poses include:
- Kneeling up in bed
- Lying in bed at an angle is not straight up-down (either way up is normal)
- Sitting on the edge of the bed.
- Lying propped up on an arm
- Arm(s) in unusual positions.
There are some photos of ‘normal’ and ‘odd’ poses here.
Options for Detecting ‘odd’ postures
The ‘Deterministic’ way that I would have thought about doing it is to take an image, look for Benjamin in it, then fit a skeleton model to calculate limb angles etc. This is possible, especially if I use a Microsoft Kinect depth camera (see here for a previous prototype that used that), but it will be very computationally expensive, and increase Benjamin’s carbon footprint even more!
Instead I am trying a machine learning approach using an Artificial Neural Network. The idea is that you train the network by showing it ‘normal’ and ‘odd’ images, then when you give it another it will estimate if it shows normal or odd behaviour.
Training Artificial Neural Network
So to do this we need sample images. I set Benjamin’s video monitor to record an image every 10 seconds, which is giving me lots of ‘normal’ images, but the problem is coming up with suitable ‘odd’ ones. To do this, I put on Benjamin’s favourite sleep suit while he was out, and wandered about his bedroom doing ‘odd’ poses and recording images.
I used a set of a few hundred of these images to train the network and ran it on real images. Unfortunately I think that rather than teaching it about odd and normal poses, I have taught it that Benjamin is normal and I am odd!!!! So I need more real Benjamin Data.
Try with More Training Images
Over the last few days I have collected one image every 10 seconds. This gives me a huge number (over 12000) images. I went through them and sorted them into ‘normal’ and ‘odd’ folders. This has got me up to nearly 800 ‘odd’ images, including the ones of me dressed as Benjamin, and a huge number of ‘normal’ ones, which include a lot where Benjamin is not in his bedroom (so do I need three categories, I wonder – ‘normal’, ‘odd’ and ‘no Benjamin’?).
So I re-trained the neural network using my ‘odd’ images, and twice the number of ‘normal’ ones sampled at random from all the normal ones. I limited the number of normal ones because I think that if you train with too many normal ones, the network gets the idea that ‘odd’ is very unlikely – when I tried it gave me 100% probability that every image was normal, which didn’t help – this is set as a parameter on my train_network script so we can fiddle with it easily.
I think produced another script that feeds all of the available categorised images through the analysis and counts how many odd images are classified as normal and vice-versa. The results are looking promising – it gave an ‘odd’ detection reliability of 96% and a false positive rate of 6%.
The odd detection failures are subtle ones – lying on front propped up on one arm etc. I think the false positives are mostly lying on back on pillows, which does not look too different to kneeling up in a similar position.
So I think continuing to collect and classify images is the way to go – I will do some more of the ‘pretend’ odd ones, because Benjamin will not do odd things on demand……
More to follow…….
The source code for this prototype is on Github: https://github.com/OpenSeizureDetector/VideoBehaviourDetector