NeuroSky Mindwave & OpenViBE: A quick look at some of my EEG data

Posted by GrahamHealy on December 2, 2012 under Uncategorized | Be the First to Comment

Clustering Time-series EEG

Posted by GrahamHealy on May 19, 2012 under BCI, EEG | Be the First to Comment

In a previous post I discussed some of the difficulties involved with interpreting EEG signals and how this might effect the real-time sharing of bio-signals between people. I’ve made some progress on the EEG side of this using time-series clustering analysis to find relationships across time in the EEG data. I go into detail about it below. This approach has been useful because it has allowed me to identify similar time periods of the EEG signal. To do this I am using time-series clustering on an afternoon’s EEG data I recorded from myself as I engaged in a number of activities. Using plots/techniques like the ones below I’ve found a number of candidate patterns that might be suitable to be presented on a vibrotactile stimulator. I’ve also began to bridge how these EEG patterns are related to the activities I engage in.

Many of us are familiar with the notion that EEG (Electroencephalography) signals – and signals in general – are made up of a number of frequency components. In the case of EEG, modulations in these frequencies have been shown to be linked with cognitive processes such as attention and decision making. An example of a modulation like this is alpha wave activity that occurs when somebody closes their eyes. In the figure below (on the first signal plot) I show an example of this where I closed my eyes for a few seconds. Here we see a period of time with large oscillations that cycle roughly 10 times per second (i.e. a frequency of 10 Hz).

Closed-eye EEG alpha waves (10-20 channels Pz-Top, Fz-Bottom)

Many signals like this are detectable from the brain but it’s often the case that it’s difficult to see what frequencies a time-series signal is composed of by just looking at it. To get around this problem we can look at these signals in the frequency domain using algorithms like FFT (Fast Fourier Transform). This algorithm given a time period of signal – like that above – computes a measure of how much activity in each frequency is present for that period of time. Being able to do this means we can compare these signals by their frequency bands without being constrained by the presence of activity in other bands. This is important for EEG as activity in different frequency bands is associated with different states and processes with the brain.

In the plot below are the frequency representations of the two time-series signal plots shown above. In the first channel (red) we can see a peak of activity at around 10 Hz.

Closed-eye Alpha FFT

There are a number of other EEG phenomena such as: ERPs (Event Related Potentials) that are elicited in response to sensory events, K-complexes (in sleep), and more. Many of these phenomenon have a long history of research and can be easily elicited/detected in a healthy individual. The duration of these phenomena can have short life spans lasting only fractions of a second such as with many ERPs, while others persist in varying magnitudes throughout most of our adult lives.

Moment to moment these EEG signals change in characteristic ways that are indicative of mental state and activity. In an EEG recording we may for instance be interested in studying how a particular range of frequencies are affected by a particular task or whether there’s an observable brain response following a stimulus (such as pictures of a cat on a screen). Clinically we might be interested in observing how the patterns change in regard to events like a seizure.

In general we examine these signals for effects surrounding the time of events which we control (or at observe). For instance where we closed our eyes so as to generate alpha waves.

Since these signals are an informative measure of brain state, what can be inferred about the structure and timing of various brain states surrounding events and activities over a given period of time? Can we discover trends and patterns in these EEG signals that might allow us to better understand how we cognitively engaged with our environment in that period of time? Will this allow me to find time periods of EEG signal that contain features indicative of a particular activity I may be involved in?

I think so and I think one way of doing this is using clustering analysis on frequency domain transformations of periods of EEG data.

Clustering
Clustering involves the assignment of objects to groups (clusters) based on similarity. In our case the objects are epochs (time periods) of EEG signal.

Using clustering we can transform the time-series signal into a representation where we can look at the signal by how similar time-periods are to each other. By doing this we can expose EEG phenomenon that have that can be detected this way. Some examples are closed eye alpha or spindles (like alpha spindles seen in fatigue).

Here’s a more drawn out explanation of how clustering works in our application. Imagine I have a bag of sweets (no two the same) of a variety of prices, calories, and weights. Now imagine I’m an eccentric sweet shop owner who has just hired John to work for me. As his first job I’ve told him to group the sweets for me into similar groups on a  table as I’ll be putting them into jars later for sale in my sweet shop. He might group them into two groups by their weight seeing that there’s only two weights really (heavy and light). Alternatively, he could group them by price and weight where he might see having 3 groups as rational since there are expensive heavy ones, small heavy ones, and small light ones. Or perhaps he could group them by other criteria. If we think of an epoch of EEG signal (say 1 second) as being a sweet in this bag, we can think of the bag of sweets as  representing a stretch of time. In this particular analogy John doesn’t know that the sweets are really pieces of  EEG signal nor is he aware that each has a special number written on the inside wrapper (its time).  John just places what he sees as being similar into piles. How John knows that two sweets are similar is something we’ll address later on, but he does have a systematic way of doing this.

This analogy almost sums up what I want to express with this one addition. If I was to take a pile of these sweets (EEG epochs) that are similar and look at the time for each on the inside wrapper, would I find that they are spread out randomly in time or are they organized in such a way that is not random? And if so, what would that mean?

It would likely mean there are time periods of similar brain-states non-randomly clumped together throughout the EEG data.

Experiment
To examine some of my questions I recorded nearly 3 hours of EEG while I engaged in a variety of tasks on my computer. While recording I noted the start and end time of each activity so I could later identify which portions of the EEG corresponded with each particular activity. In the time periods between activities I was engaged in behaviors which generated a lot of movement – and hence noise on the EEG – so I did not note nor examine these later on.

I used a 2 channel EEG-SMT (based off the OpenEEG Modular design). The data stream from this device was directed to a file where it was also timestamped. Silver cup electrodes were placed at 10-20 sites Fz and Pz.Signals were referenced to an ear-clip electrode on the left earlobe. The ground electrode was placed on the chin. I used elastic bandage to keep everything in place. Reasonable efforts were made to get good signal quality by cleaning the sites on my head to minimize on the presence of noise (I’ve always found light scraping great for getting better signals when using this type of electrode). Ten20 paste was used to attach the electrodes.

Here’s a summary of what I did that afternoon:

Table 1

Analysis

Clustering analysis was used on the EEG data. To do this the EEG signal was partitioned for the 2 channels into 1 second intervals in 1 second increments. In all there were 3044 segment. Segments were only extracted for time periods where I noted I was engaged in an activity. This made it easier to select signal regions that would not be as heavily contaminated by noise introduced by movements.

The Scipy FFT algorithm was used to compute the DFT for each of the two channels. By appending the two channels FFT’s together (array of values) we can generate input data points to be used in our clustering process. Similarly, we may choose to append only a particular frequency band of both channels.

A number of agglomerative hierarchical clustering methods are available through Scipy (Single, Complete, Average, Weighted, Centroid , Median and Ward (explained here)). Each of these was tested on the FFT transformed data. Each of these methods operate on a distance matrix. This distance matrix contains a distance measurement for each pair of the 3044 time periods. A number of ways of generating distance measures are provided by Scipy library. Only a few of these seemed to give a meaningful distance measurement when comparing a pair of FFTs: cosine, correlation, canberra. Perhaps with more channels or different transforms other distance measurement methods would yield more meaningful results.

In order to interpret the outputs of this clustering process I use a cross-dendogram plot (the concept which I seen here). This is useful as it helps us to not only see what relationships exist in the data but also what relationships exist between the various clusters it generates. I developed a tool to to help do this that allows selection of a region of a cross-dendogram plot whereby it then displays FFT and distribution information about the time points represented by the selected region. It also allows some basic comparison between the selected regions.

Using the possible agglomerative hierarchical clustering methods (Single, Complete, Average, Weighted, Centroid , Median and Ward) with a number of distance measures (canberra, cosine, correlation and euclidian) and feature sets I generated a batch of dendogram plots for all permutations.

8 different feature sets were used:

Table 2

Results

Due of the large amount of data generated by the analysis process I only show a subset of it. If you’re interested in looking at the other parameter permutations you’ll find them here. I don’t show below 5 Hz because even with log scaling it makes – what I feel to be the more important – differences in higher frequencies less clear.

Below is a cross cross-dendogram plot for FFT 5 for the 8 to 12 Hz (alpha band activity). In this I show rectangular regions selected in color for which I’ve generated FFTs. What’s interesting to note here is that there’s differences on the FFT plots for the 2 channels (Pz and Fz) for these time regions.

Cross-dendogram – FFT5 ( 8-12 Hz )

FFTs for selection time regions – FFT5 ( 8-12 Hz ) – Pz on top, Fz on bottom

In the frequency plots for the regions selected in the cross-dendogram for FFT5 we can see how the clustering process made distinction between different patterns of frequency activity across the two channels.

In order to analyze how these time regions were distributed across the various activities I engaged in (show in Table 1) I used a chi-square test. Shown in the table below are the probabilities for each activity that it contained significant deviations from the expected number of time-points that should have fallen within its time range with regard to the 3 selected regions on the cross-dendogram. Although each of these probabilities are calculated independently they strongly indicate that there are patterns of 8-12Hz across the two channels that can be used to distinguish each activity.

Table 3

I followed a similar strategy of manually selecting regions of distinct clustered time epochs on cross-dendograms for FFT6 (12Hz – 16Hz) and FF7 (16Hz to 20Hz) and FFT2 (0Hz to 15Hz). These are shown here

FFT2 – Cross dendogrram plot, FFT for Pz and Oz
FFT3 – Cross dendogrram plot, FFT for Pz and Oz
FFT4 – Cross dendogrram plot, FFT for Pz and Oz
FFT5 – Cross dendogram plot, FFT for Pz and Oz
FFT6 – Cross dendogram plot, FFT for Pz and Oz
FFT7 – Cross dendogram plot, FFT for Pz and Oz

The chi-square probabilities for these can be found in this file if one is interested. Most frequency ranges don’t seem to distinguish well between activities except the 8Hz-12Hz activity (and the 4Hz to 8Hz range).

Conclusion

In this post I showed how clustering of time-series EEG data could be used to find similar portions of EEG signal spread out over time. I also demonstrated that these clusters of time periods can be indicative of particular activities undertaken in that time. I done this by clustering the FFTs in 1 second time periods from both of the EEG channels. This potentially indicates that EEG behaviors of this kind can be detected on a granularity of 1 second.

One avenue of exploration I’d like to take following these findings is to assess how I can detect features of this kind in real-time from EEG signals so as to use them as inputs to other systems whether they be human (vibrotactile EEG sharing) or artificial.

Real-time sharing of bio-signals through vibrotactile stimulation

Posted by GrahamHealy on April 29, 2012 under BCI | 4 Comments to Read

I’ve been thinking lately about how VS (Vibrotactile Stimulation) – like the buzz from your cellphone or game console controller – might work as a mechanism for real-time sharing of bio-signals. By bio-signals I mean anything recorded from the body such as EEG, ECG, PR, GSR, temperature, etc and by sharing I mean allowing somebody else in real-time to sense a representation of any of these signals though VS. In general I’m open to any other modalities through which to provide this feedback, but for practicality I think VS is a good candidate to be explored first as it works well for the applications/testing scenarios I have in mind.

What I want to do is: Use wireless bio-sensors on a person to allow their signals to be shared in real-time with other people.In particular, I’m interested in doing this with EEG as I think it could provide the most fruitful results. However, I think starting with using a bio-signal like ECG to measure heart-rate is a better starting point.

I would like to do this because I think it can facilitate a form of non-verbal communication. As humans we are good at learning patterns and interpreting their meaning. For instance I may begin to notice common patterns of behavior during when somebody tries to mislead me during a game of poker. Whether I can do this well or not is another matter. No one piece of evidence may be meaningful by itself either. For example, noticing eccentric movements of a person’s eyes in itself may not indicate anything, but in combination with other behaviors like breathing heavy or unusual periods of silence I may interpret this as indicating they’re bluffing.
I wonder how different a game of poker would be if everybody at the table was wearing a heart-rate monitor and each of the players could sense each other heart-rates in real-time?

How exactly would I present the heart-rate on the vibrotactile stimulator? Well a pretty easy scheme is to present heartbeats as pulses on the VS.

Non-verbal communication is an intricately woven part of how we go about our daily lives. We are definitely creatures who like to non-verbally communicate. A good example of this is people on the bus flirting. They may do this for a few minutes during which time no words will be exchanged and then they part ways.
I wonder if they could sense each other heart-rates how their interaction would have changed?

Conceivably it’s pretty straight forward to see how sensing somebody’s heart rate might change how you interact with that person. There’s lots of situations I could imagine where something like this could impact communication. For instance, imagine a teacher-student environment where the teacher was able to sense that a child’s heart-rate unexpectedly changed when she asked whether all students in the class did their homework last night. That kid who has been relying on his prize winning smile for so long might not be so lucky avoiding detention this afternoon. Or perhaps a military squad leader sensing in real-time that the collective heart-rate of his team has risen significantly might indicate something. I don’t know what that something is, but perhaps given the situation the squad leader might be able to make a more informed decision. It’s hard to pin down the exact value of this. I think it’s much harder though to declare it has no value though at all.

From self experience we’re familiar with how our heart-rate changes in emotional or stressful situations. This in itself makes it a good candidate as a bio-signal that might be meaningfully shared between people.

EEG signals on the other hand are something we are not use to typically sensing/interpreting unless you’re a user of neurofeedback. I don’t feel or see my or hear the EEG signals, but like my heart-rate they’re indicative of my current state.

Interpreting EEG signals is difficult even in the best of situations. It’s just a difficult process, and obtaining meaningful results takes time and care and in the end often it’s hard to shave down what you find to a succinct message that retains its original meaning.

In a lab we might for instance examine somebody’s EEG signals as they look at images on a computer screen to see if there are differences say for faces vs non-faces, or letters vs numbers. Using the time information for when we presented each picture (or letter/number) we can index the EEG signals so as to extract the relevant portions (epochs) surrounding each image presentation event to examine whether there were any differences. How we do this doesn’t matter at this point, but what’s important to note is that the EEG data without the timing information of each image is essentially useless as we no longer know where in the signal each image/character was presented. That’s what’s great about a lab, we have the instrumentation to control and capture the timing of things such as when images are presented. In the real world we are limited in what information like this we can reliably captured.

Even if you had your day’s EEG recording, without the timing information of things like sensory events you’re limited in what you can do with the data. There’s lots of logistic difficulties and drawbacks to implementing EEG outside the lab. There’s noise generated from electric equipment, movements (EMG), movements of the eyes (EOG) to name but a few. Even in the best of situations EEG signals are difficult to interpret. But they do display some immediate patterns that might make good candidates for use.

The heart-rate examples earlier were simple to understand as we have an intrinsic sort of understanding of how heart-rate works and how being aware of it might help. We don’t have this for EEG. This means we need to go about designing a system to process the raw signals captured from one individual and to transform them into a meaningful representation that can be presented to the other individual. But since we don’t know what’s exactly meaningful in the signals – or may be ultimately learned to be meaningful – there’s no clear way of doing this. This is an open ended problem; and one ripe with choice with avenues to be explored since we know of many types of patterns and phenomenon that can be detected in EEG signals.

During recording EEG signals as part of the my Ph.D. research I noticed some phenomenon were present surrounding particular events as I interacted with the subjects taking part in EEG experiments. One particular thing I noticed was at the end of experimental blocks where the subject engaged in a mentally demanding tasks often where they searched RSVP sequences for oddball images. In effect this means they silently counted the number of times they seen images that contained people from a steam of images but it’s difficult because the images go by so fast. The images were presented continuously at high speed on screen and required focus so as to not miss any of the targets (those with people). I noticed sometimes at the end of some of these blocks there was a visible periods of pronounced alpha activity. Similarly, I noticed patterns like these sometimes appearing when I rambled on for too long and they’d say “oh sorry, zoned out, what were you saying?”
I wonder what I could infer about a person’s mental state If I could sense alpha bursts like these.

Blankertz et al give a good overview of non-medical uses of BCI and describe situations and behaviors wherein alpha activity has been shown to be informative of things like performance.

I’ve another post to write that ties in with what I’ve talked about here. In that I’ll cover some of the phenomenon that can be seen in EEG and how we might leverage these to create EEG-vibrotactile communication.  Hopefully soon enough I’ll have a prototype of this built too! Other things I’d like to think and talk about are technologically enabled inter-species communication (that’s right – telepathy with my a dog).

I can think of more esoteric directions something like this could take like the technologically enabled inter-species communication with my dog (sadly, I do not have a dog). Comments? Ideas? Anyone?

BCI-Enabled Software?

Posted by GrahamHealy on April 13, 2012 under BCI, Thoughts | Be the First to Comment

With EEG sensing devices like the Emotiv EPOC becoming something that anybody can own for $299, it’s only a matter time before they’ll be supported – in some way – by everyday software applications. It’s hard to imagine how using something like  MS Word could be made easier by having a BCI, but you’ve never used a BCI have you?

There are many phenomenon present in EEG signals that indicate various subtleties of neural processes. An easy example is the P300. Many interfaces out there already use such signals to enable disabled persons to communicate with a computer by thoughts-alone method. Applications of these types harness existing neural signal phenomenon that are known to be controllable by the user. There are many ways of driving an EEG based BCI system: P300, Mu Rhythm, … And the used signals here only represent a subset of the various phenomenon and modulations known to occur in EEG.

But back to my MS Word example, it’s difficult to really justify the use of a BCI by a normal everyday user for when they use MS Word. It’s hard to imagine though in 10 years or 20 years or 100 years – if everybody for some reason wore something like an EEG everyday – that we would not use these signals to somehow enable an application like MS Word to function in a more responsive manner to their brain states. Although the investment in finding ways of utilizing such a system may seem hardly worth it, given everybody had a BCI and was using Word, this would no longer be the case.

What got me thinking about this was playing FreeCell – the card game on MS Windows – where I had hit a difficult game. What if the game knew to adjust or give hints based upon my EEG patterns? Or if MS Word could detect patterns of brain activity that might indicate I’ve difficulty doing a particular task. Or that given activity patterns of a particular type I tend to display fatigue and should be advised to take a break from writing. Could an interface adjust itself to better suite my needs by being aware of brain state? Could my Skype status someday be an icon showing my inferred brain state so that people don’t call when I’m cognitively stressed? What if someday my web browser has a plugin allowing me to share my EEG signals with say an online video site like you tube? Will I ever need to worry about privacy issues with content and social-networking websites like Facebook regarding sharing my brain data? Will my boss want to monitor my brain? Will my work be valued differently because of this? The snowballing effect seen with chains of question like this rarely converge so I’ll leave it there.

Left is now right and right is now left

Posted by GrahamHealy on October 11, 2010 under Uncategorized | Be the First to Comment

I’ve an odd thought. What if there was a machine which could rearrange the atoms in your body to move everything from the left hand side to the right, and vice versa?

I.e. you walk into this machine, and the atoms from the left hand side of your body begin to be swapped with their counterparts from the right. At the end of this process you are still facing the same direction, but where your left hand use to be there’s now your right hand, and your heart is on the wrong side!
How would you now perceive the world? Since the wiring in your brain would remain the same you would obviously see the world as having changed and not yourself. It’s a fun thought exercise trying to rearrange everything from your spatial perception of the world to imaging problems like trying to read, shake hands and not get hit by cars. Try to construct this world and walk about it. It’s difficult to try and generate this new perception of the world on the fly as you walk through it.

Pareidolia and the N170

Posted by GrahamHealy on September 18, 2010 under Uncategorized | Be the First to Comment

This post is not intended to make a scientific contribution. I am just highlighting a strange psychological phenomenon, and providing some links and thoughts.
Take all this with a pinch of salt!

Yes, Pareidolia. That thing where you see things in things when you shouldn’t, and are afraid to say it to other people around you just in case they think you are a bit cracked.
I’ve always been astonished since I was a child at how a perception of something that has remained static can change and take on an animate quality. One minute you are looking at a pile of fluff in the clouds and then suddenly you see a cat reading a book (often it’s instantaneous in that you don’t see the pile of clouds first, you just see the cat reading the book).

Our propensity towards finding faces in things makes sense though (even if they are not there). You are better off being mistaken thinking you see somebody in the bush than not spotting them at all. It’s likely a threat. Even in the brain there’s a very early response to faces and it’s one of the earliest responses we can see to a complex stimuli. This is call the N170 ERP component. I mention this not to explain what it is, but just to point out as early as 170ms after the appearance of a face on a screen your brain is already well underway in processing and integrating the perception of the face into your consciousness. Other stimuli tend to be processed in an evident fashion later on as evinced by P300 (300milliseconds onwards) and other later components. I am taking the presence of these ERP components to signify more than the lack of their presence.

Attempt 2

Posted by GrahamHealy on September 4, 2010 under Uncategorized | Be the First to Comment

I tried to keep a blog once with the intention of writing about my PhD progress. That failed for numerous reasons.
One of those is that I had an exact idea of what I was going to write and how it was going to be structured.
I think I had the wrong idea of what a blog was. So here we go again.