I have been quiet for quite a while, so thought I should share an update…..
Executive Summary
The post below is quite long, but it can be summarised as follows if you do not want to read it all!:
- The data that users are contributing to the Data Sharing system is giving us the ability to assess the performance of OpenSeizureDetector and look at improving it – thank you to everyone who has contributed data so far!
- We have used the contributed data to demonstrate with confidence that OpenSeizureDetector does work and detects a large majority of seizures that involve shaking.
- Initial results from the Machine Learning (Artificial Intelligence, Neural Network) algorithm development are very promising to help improve false alarm performance – we hope to have a test version of OpenSeizureDetetctor that uses it in a few weeks.
- We always need more data to test and improve the system, so if you are not signed up to the Data Sharing system, please do, so you can help improve OpenSeizureDetector. Users who have regular seizures are particularly important to help us build a database of real seizure data.
- I am trying to decide how to publish the OpenSeizureDatabase for use by researchers – I would welcome users’ thoughts on this.
Data Sharing
Firstly, thank you so much to all the users who have been sharing their OpenSeizureDetector data using the ‘Data Sharing’ system. We are building a really useful database of data for both genuine seizures and also ‘seizure like activities’ that cause false alarms. This data is essential if we are to develop an improved seizure detection system that will both improve our seizure detection capability AND reduce false alarms.
Over the last few weeks I have been working with Jamie Pordoy (who is researching seizure detection for his PhD) to look at how best to use this data. This involves checking the data, filtering out events that appear to be tests of the system and working out how to collapse the dataset into unique events to avoid duplication.
I am pleased to say we are making good progress, and now have a system where we can test different seizure detection algorithms by running them on the data contributed to users so we can work out how the different algorithms perform compared to the original OpenSeizureDetector algorithm.
Current OpenSeizureDetector Performance (as of Version 4.0.x)
What we are finding is that the original detection algorithm which is in use by OpenSeizureDetector at the moment (Version 4.0.x of the phone app):
- has a detection reliability of about 81% for tonic-clonic seizures. (it is actually slightly higher after we were woken by an alarm at 0420 this morning, but that seizure is not included in the analysis).
- has a detection reliability of about 75% for all seizures.
- …does give quite a lot of false alarms (I still need to decide how best to calculate a false alarm rate from the available data)
I was very pleased with these results, because it is the first time we have had sufficient data that we can demonstrate with confidence that OpenSeizureDetector does work and detects a large majority of seizures that involve shaking.
We appreciate that the false alarms from the system are an issue for a lot of users, and would also like to get the detection reliability even higher, without increasing the false alarm rate.
Possible Tweaks to Improve OpenSeizureDetector
We have used the test system to try looking at different settings for the OpenSeizureDetector algorithm to try to reduce the false alarm rate, but what we found was that
- Increasing the ‘Alarm Threshold’ parameter sufficiently to reduce the number of false alarms (to ~5000) started to impact on the seizure detection reliability. We do find that setting it to about 900 reduces the number of WARNING events which makes the system quieter.
- Increasing the ‘Alarm Ratio Threshold’ parameter to 61 would halve the number of false alarms BUT would reduce the seizure detection reliability to 65% and 59% for tonic clonic and all seizure events respectively. This reduction in seizure detection reliability is significant, so the change is not recommended.
So it appears that the default settings in OpenSeizureDetector are close to optimum, so we are not likely to see significant improvements using the existing algorithm, so we will need something more complex.
Machine Learning Seizure Detection
Jamie and I are looking at using the data that users have contributed to train a Machine Learning (neural network, ‘Artificial Intelligence’) based seizure detector. There are several potential ways of designing and training this system. We have made some very good progress in the last couple of weeks, so that we have two potential models at the moment:
- One model is giving us a very good improvement in the seizure detection reliability, but is increasing the false alarm rate significantly, so will need further development.
- The other is giving modest reduction in the seizure detection reliability, but is giving a very large (factor of 3) improvement in false alarm performance.
So the two models we have at the moment are bracketing the target of having high seizure detection reliability with low false alarms.
I think there is scope to adjust the training method to sacrifice some of the false alarm performance to get the seizure detection reliability to be comparable to the existing OpenSeizureDetector reliability. If we can achieve this it will be a big improvement in the system performance.
If we can get a model that looks like it achieves this objective we will look at incorporating it into an update to OpenSeizureDetector to give users the chance to try it and see how its ‘real world’ performance compares. It is likely to be available to users who are contributing to the Data Sharing system initially, as we will need to use that infrastructure to get updates to the model to you as it is improved by more data being uploaded into the system.
Open Seizure Database Licence
The data that we have collected of measurements during seizures and seizure-like movements should be really useful for research purposes.
I am looking at making the anonymised dataset available for research purposes, so am trying to think of how to do this. The principle of OpenSeizueDetector has always been of complete openness – the source code and description of the algorithm are published for anyone to make use of, subject to an Open Source Licence.
I am looking for a suitable licence for the database but have not found one that feels quite right yet, so if anyone has suggestions, please let me know (graham@openseizuredetector.org.uk). The options I have looked at for ‘standard’ liences are:
- A very ‘open’ license that just requires users to credit OpenSeizureDetector contributors in the work that uses it. This would mean that researchers/companies could use the data without contributing back to OpenSeizureDetector.
- A ‘non commercial’ license that prevents commercial use of the data. This has the same issue as above, but would mean the data could only be used for non-commercial research purposes.
Neither of the above options quite achieves what I would like, which is that the users of OpenSeizureDetector who have contributed the data have the opportunity to benefit from it in terms of improvements to OpenSeizureDetector.
What I am proposing is a variation to the very ‘open’ licence by including a condition on the use of the data that the results from using the data must be published (as well as crediting OpenSeizureDetector for the data) – draft is here.
I would welcome users thoughts on this approach.
Conclusions
- The data that users are contributing to the Data Sharing system is giving us the ability to assess the performance of OpenSeizureDetector and look at improving it – thank you to everyone who has contributed data so far!
- We have used the contributed data to demonstrate with confidence that OpenSeizureDetector does work and detects a large majority of seizures that involve shaking.
- Initial results from the Machine Learning (Artificial Intelligence, Neural Network) algorithm development are very promising to help improve false alarm performance – we hope to have a test version of OpenSeizureDetetctor that uses it in a few weeks.
- We always need more data to test and improve the system, so if you are not signed up to the Data Sharing system, please do, so you can help improve OpenSeizureDetector. Users who have regular seizures are particularly important to help us build a database of real seizure data.
- I am trying to decide how to publish the OpenSeizureDatabase for use by researchers – I would welcome users’ thoughts on this.
