Convert get_level() to return floating point value.
opened this issue · comments
All headsets should return floating point sensor levels.
Doesn't regular Epoc models just return integers?
I doubt it.
oh yeah, weren't we supposed to get you an epoc csv?
Yeah we need to know what they should really be.
A dump of both running at the same time would be optimal, if possible.
with no electrodes attached or anything.
bill... in Xavier, theres an extra window for "Sequence number" and it shows this graph, that creates a line that increments up... and then drops off sharply... basically like the "counter"
i'm curious.... you said there were 2 counters... so i wonder what MiniFire's looks like, when its in Epoc+ mode.... whether theres maybe 2 or more lines?
pretty sure its a graph for the counter though. thought you might find that interesting.
i dont think i'll be able to do the CSV for you, unless i'm missing something... its asking me for a license key, or to be signed in.
i probably could sign in using my emotv id, but i'm not sure that'd be such a hot idea.
The render is not right, the data is structured differently in Epoc+.
so did we even come up with something workable with Epoc+ ?
i was kind of confused with all we had going on.
The data is decrypted but is 16 bit values instead, so the levels are not accurate at all.
I've been playing with the data to see if I can get it to match the pure eeg output.
okay i see now. can you give me a quick sample of the data you are trying to decode.
There a packet length inserted every packet and some text I added, it kind of looks like two bytes per sensor.
So I've been trying to convert the bytes to floats or doubles using struct pack and unpack.
I was getting some stuff that looked like this:
'4080.00389105'
'4080.99610901'
'4335.0000152'
'4335.003891'
'4335.99609375'
but it didn't hold up entirely.
You cannot save recording with pure.eeg without a subscription. Tomorrow I can put my headset in epoc mode and do some recordings.
MiniFire, also, take a look at the last tab, "Data Packets"
and see if under Epoc+ if it is displaying multiple lines (for the counter).... or just 1.
The packets is just 1 saw tooth line that goes up at 45 degrees and drops. If packets are lost there will be a deformation in the line.
I look at that graph alot when I am using the headset when I see a weird output in the eeg display.
Does this help with the sensor data?
Resolution: 14 bits 1 LSB = 0.51μV ( 16 bit ADC, 2 bits instrumental noise floor discarded), or 16 bits*
and also: 2 resolution: 14 bit or 16 bit per channel (0.51 µV @ 14 bit / 0.31 uV @ 16 bit)
its actually .13uV
if you check their website, on the comparison chart it shows .13uV
and on the specifications, it says .31uV
the web designer for Emotiv keeps getting it wrong, lol
and i know its .13uv because gmac clarified that once.
EPOC+ high resolution mode is the same full scale reading as 14 bit mode, so the LSB resolution is 4x smaller due to the extra 2 bits. 0.51 to about 0.13 u,V for the LSB., NOT 0.31
The 256Hz mode also applies to all channels at the same time, that’s what is commonly understood by ‘256 samples per second per channel’.
i don't quite understand what he meant by the resolution being 4x smaller... maybe he meant, that it has more floating point precision, due to the extra 2 bits.
There is a hint in there actually, reading it again.
1 LSB http://stackoverflow.com/questions/19161872/meaning-of-lsb-unit-and-unit-lsb
It makes perfect sense, actually.
so if we multiply out values for the old headset we get 2 places of precision.
4x more than that is 8 more places, 10 places total, which is the precision we see in the the pure eeg data.
Minifire, i think bill was looking for Epoc 14-bit CSV, when you get a chance.
bill, did you add the 1,2,3,4,5,6,7 into the epocplusdata.txt ?
yeah, I was counting the pairs of repeating bytes.
so the numbers that go in sequence.... those are the numbers for the sensors, found in the
sensor_bits = { } array ?
and then the two bytes of data... thats probably a big endian number, that we have to run the get_levels on?
and..... the lines that have a second group of numbers (with extra 0 padding), in the same sensor_bit subset, i'm guessing thats the 256hz part... where its updating it with additional data.
CSV coming shortly.
hey bill.... did you notice... that in the second grouping of numbers, that it goes incrementally as well..... but is half of what the first sequential number is.... kind of like how the Counters were.
so i'm taking it... that those aren't the sensor_bit numbers, but are actually the counters.
144 16 148 125 127 125 146 125 208 128 137 125 87 125 142 125 12 0 209 114 189 125 184 125 157 125 8 134 55 127 211 148 74 32 36 0 239 255 59 0 69 2 116 63 112 243 155 253 201 249 1 1 0 0 0 0 0 0 0 0 0 0 0 0
149 16 175 125 211 125 247 125 246 128 161 125 217 125 112 125 29 0 126 115 189 125 240 125 6 126 26 134 81 127 36 149
150 16 201 125 200 125 248 125 242 128 162 125 221 125 111 125 9 0 118 115 193 125 238 125 15 126 49 134 83 127 44 149 75 32 26 0 245 255 48 0 107 2 123 63 253 243 180 253 201 249 37 1 0 0 0 0 0 0 0 0 0 0 0 0
151 16 209 125 171 125 211 125 210 128 182 125 203 125 148 125 89 0 168 115 201 125 239 125 0 126 28 134 91 127 18 149
152 16 227 125 167 125 177 125 192 128 201 125 185 125 189 125 3 0 194 115 206 125 244 125 232 125 241 133 107 127 249 148 76 32 38 0 247 255 63 0 120 2 69 63 197 243 169 253 205 249 48 1 0 0 0 0 0 0 0 0 0 0 0 0
153 16 6 126 203 125 175 125 210 128 196 125 176 125 198 125 29 0 149 115 206 125 247 125 216 125 230 133 126 127 8 149
154 16 5 126 233 125 178 125 214 128 174 125 165 125 181 125 9 0 110 115 209 125 244 125 195 125 236 133 128 127 37 149 77 32 29 0 228 255 73 0 117 2 139 63 19 244 159 253 226 249 20 1 0 0 0 0 0 0 0 0 0 0 0 0
155 16 195 125 218 125 156 125 168 128 160 125 152 125 165 125 89 0 173 115 221 125 236 125 160 125 217 133 103 127 33 149
156 16 123 125 180 125 127 125 112 128 165 125 153 125 158 125 3 0 43 116 229 125 233 125 139 125 197 133 74 127 7 149 73 0 251 255 38 0 144 2 58 63 201 243 159 253 226 249 20 1 0 0 0 0 0 0 0 0 0 0 0 0
157 16 108 125 157 125 117 125 99 128 176 125 173 125 159 125 29 0 124 116 220 125 241 125 160 125 219 133 68 127 1 149
158 16 136 125 145 125 114 125 116 128 188 125 188 125 179 125 9 0 136 116 197 125 254 125 192 125 244 133 78 127 11 149 79 32 53 0 244 255 25 0 89 2 140 63 11 244 178 253 209 249 88 1 0 0 0 0 0 0 0 0 0 0 0 0
159 16 150 125 128 125 92 125 116 128 206 125 181 125 223 125 89 0 129 116 179 125 5 126 183 125 199 133 77 127 2 149
160 16 142 125 116 125 76 125 105 128 218 125 165 125 249 125 3 0 110 116 178 125 5 126 147 125 113 133 71 127 238 148 80 32 25 0 246 255 46 0 125 2 132 63 221 243 151 253 203 249 52 1 0 0 0 0 0 0 0 0 0 0 0 0
161 16 134 125 128 125 102 125 120 128 198 125 153 125 201 125 29 0 45 116 193 125 2 126 149 125 102 133 87 127 252 148
162 16 125 125 146 125 152 125 149 128 155 125 150 125 109 125 9 0 225 115 211 125 1 126 194 125 194 133 117 127 38 149 81 32 45 0 254 255 54 0 172 2 157 63 222 243 173 253 203 249 35 1 0 0 0 0 0 0 0 0 0 0 0 0
163 16 111 125 157 125 169 125 150 128 142 125 164 125 82 125 89 0 224 115 215 125 0 126 203 125 37 134 115 127 47 149
164 16 128 125 179 125 152 125 134 128 179 125 204 125 158 125 3 0 33 116 193 125 255 125 144 125 65 134 68 127 7 149 82 32 62 0 252 255 62 0 104 2 91 63 150 243 159 253 207 249 50 1 0 0 0 0 0 0 0 0 0 0 0 0
165 16 197 125 217 125 159 125 154 128 211 125 242 125 227 125 29 0 54 116 165 125 255 125 110 125 58 134 39 127 237 148
166 16 0 126 227 125 212 125 213 128 171 125 221 125 168 125 9 0 233 115 167 125 5 126 192 125 70 134 78 127 15 149 83 32 49 0 1 0 55 0 51 2 134 63 223 243 159 253 207 249 50 1 0 0 0 0 0 0 0 0 0 0 0 0
167 16 230 125 172 125 7 126 249 128 80 125 140 125 8 125 89 0 135 115 214 125 11 126 86 126 88 134 159 127 68 149
168 16 143 125 98 125 11 126 230 128 19 125 71 125 143 124 3 0 90 115 13 126 11 126 171 126 97 134 207 127 82 149 84 32 47 0 241 255 31 0 140 2 126 63 208 243 165 253 201 249 32 1 0 0 0 0 0 0 0 0 0 0 0 0
169 16 96 125 88 125 244 125 204 128 34 125 79 125 143 124 29 0 74 115 31 126 7 126 139 126 146 134 198 127 63 149
170 16 129 125 147 125 226 125 212 128 93 125 151 125 235 124 9 0 48 115 4 126 5 126 42 126 1 135 158 127 48 149 85 32 24 0 219 255 71 0 105 2 114 63 227 243 171 253 218 249 26 1 0 0 0 0 0 0 0 0 0 0 0 0
171 16 182 125 209 125 207 125 238 128 156 125 220 125 109 125 89 0 44 115 218 125 9 126 191 125 94 135 107 127 28 149
172 16 203 125 231 125 176 125 0 129 206 125 247 125 237 125 3 0 87 115 182 125 11 126 97 125 90 135 54 127 241 148 86 32 27 0 240 255 59 0 112 2 134 63 49 244 164 253 197 249 49 1 0 0 0 0 0 0 0 0 0 0 0 0
173 16 210 125 227 125 158 125 21 129 225 125 235 125 46 126 29 0 106 115 163 125 7 126 59 125 34 135 26 127 210 148
174 16 219 125 212 125 176 125 50 129 193 125 193 125 252 125 9 0 38 115 171 125 255 125 124 125 30 135 56 127 238 148 87 32 47 0 2 0 59 0 101 2 127 63 111 243 176 253 197 249 38 1 0 0 0 0 0 0 0 0 0 0 0 0
175 16 206 125 176 125 200 125 41 129 136 125 137 125 142 125 89 0 197 114 205 125 253 125 238 125 74 135 120 127 39 149
176 16 169 125 139 125 185 125 228 128 120 125 111 125 104 125 3 0 178 114 240 125 253 125 20 126 75 135 149 127 53 149 88 32 36 0 252 255 60 0 104 2 152 63 25 244 176 253 197 249 38 1 0 0 0 0 0 0 0 0 0 0 0 0
177 16 163 125 149 125 151 125 157 128 169 125 150 125 183 125 29 0 233 114 240 125 251 125 199 125 17 135 119 127 12 149
178 16 204 125 203 125 147 125 156 128 227 125 212 125 20 126 9 0 11 115 211 125 248 125 111 125 229 134 76 127 235 148
i formatted it a little for easier reading.
[counter] [sensorbits?] [precision/part] [whole] [precision/part] [whole] [precision/part] [whole] .....
maybe?
on the second half, the numbers kind of get a little funny looking...
what you probably surmised, is that the first half is probably the regular 14-bit, and
then the second half, might be the extra 2 bits,..
no... i think this is how it is.... i'll explain
89, 3, 29, 9
x, y, z, accel?
I did not reformat it:
the sensor_bits don't have anything to do with the numbers in this case. lol i think that's part of whats
involved with get_level, is that it reads the sensor_bits, and shifts appropriately, based on those numbers.
i think the first group... is a counter.
then... its like you said, 2 byte chunks, its all in big-endian format i believe.
so the first line would be.... 16,148 125,127 125,146 125,208 128, 137 125,087 125,142 125,012
then i surmise, that we see the second counter.
and then we get the same exact format as the first.... in big endian format... but you'll notice
the numbers are pretty goofy, i think its just the big endian format.
i think thats part of the floating point numbers.... that either get "attached" or added.
that's my theory, what do you think?
i think the big endian numbers for the second half, are probably so small, or so much more different than
the integers, that they look a lot different from the first set.
because consider... that big-endian format, is basically just reverse binary.
but the first block... looks very similar to the numbers i've seen in OpenVibe
big endian format.
so the first step... will be converting those 2 byte chunk into their big endian formats, and taking a look at what numbers we actually got.
yeah I've been doing just that.
I think figuring out the gyro's would be the easiest to verify.
Is there a way to put a "marker" in the decrypted output files?
http://www.scadacore.com/field-tools/programming-calculators/online-hex-converter/
perhaps this will help.
I tried converting the first number: 16,148 and i think i got 4800 in big endian.
I tried to move the headset in 1 direction at a time. Tried to atleast
256hz 16bit MEMS 128hz 16bit
[emotiv_data_2017-04-14 18-11-07.980000.txt](https://github.com/openyou/emokit/files/923058
/emotiv_data_2017-04-14.18-11-07.980000.txt)
emotiv_values_2017-04-14 18-11-07.981000.txt
hmmm, I just did that the hard way, I am going to upload a little more in a moment.
don't quote me on that, i may have done that wrong.. remembered having a heck of a time with big endian last time.
yeah, from the looks of it.... when it stores the Epoc CSV, it is storing it in the plain integer format.
but when it stores the Epoc+ data.... i'm pretty sure it stores it in the BIg-Endian format.
When I created a version of Cykit to work with OpenVibe, to stream it to its TCP writer...
i had to send it in a continuous stream in big-endian format.
i think its probably more data efficient, and thats probably their standard.
medical grade eeg probably has more sensitivity, so they probably uses floats as well.
its a pain in the butt understanding how python works with big endian, lol i think it took me like a week to grasp it, plus i was learning python 3 at the time.
in theory, my cykit should have a big endian setup.. at least for the conversion -to- big-endian.
https://github.com/CymatiCorp/CyKit/blob/master/Openvibe-Readme.md
perhaps that will help....
as you can see... openvibe lets you select settings on what type of format the raw data is in.
in my case.... i used Big-Endian 16-bit SIGNED integer.
so it was in big-endian format, but was an integer.
so i'm thinking for Epoc+ it will probably be... a Big-Endian 16-bit SIGNED float
The struct module has it builtin
Ok, I just made a file running the epoc+ along with the emokit running. I moved the headset around. Where do you want the data? The CSV is to big to post here.
pastebin?
cy = ""
values = [packet.sensors[name]['value'] for name in 'AF3 F7 F3 FC5 T7 P7 O1 O2 P8 T8 FC6 F4 F8 AF4'.split(' ')]
cy = struct.pack('>' + str(len(values)) + 'h',*values)
https://docs.python.org/2/library/struct.html
So > indicates the endianness, in this case, that would be for big.
It shows here though that... a float is 4 bytes long.... which might complicate things.
we either have to convert the two groups, together..... or separately.
Well isnt this nice, the CSV seperates the motion from the EEG.
Ok, lets think about the data you have. Given the fact that the data is in 2 separate files, you would have 2 feeds. 1 feed at 256samples, and 1 feed at 128 samples. So 2 counters.
Makes sense.
Well at least I understand this output a little better. Now I am trying to figure out what the 2 of you are talking about.
These values in the emokit decrypted CSV, what's there base?
its base 10, but i believe in its big-endian format.
you might have to do some reading up on that. its kind of a confusing format. lol
basically binary in reverse.
Bill, i tried running it through OpenVibe, as it is capable of reading several different formats,
didn't have much luck.
will have to try again later though...
<---- Started off learning Assembly. I am trying to dig into 20 year old memories of endian and conversions.
big endian can contain multiple bytes, it strings together, reversed binary data.
think thats how it works. basically its all reversed binary and concatenated.
like bill said, python contains some methods for it built into struct.
though it was confusing for me, with all the changes in unicode they did, for python 3.
got to use google on this one.
plus i think it will require a lot of python experimentation... i recommend using python from the command line on this one, and just typing in instructions... but we'll need some kind of guide... like we need to know what the values are supposed to be.
I am working on a script to take the results in the zip i posted and look for the hex equivalent of a number. Just to see if it is in there. No matter how the number is created, it will always have the same hex bytes.
The gyro is 2048 at 0 I believe.
All the available data is being collected right?
?
I believe so you can try increase the read size to see if there's more.
I think I just had a light blub moment though.
I am not looking at the data ... and i am not looking at it at this moment... but a float is 4 hex values. When you divide the column count by 4 you do not get 14 channels.
Also, there seems to be an internal clock that transmits with the data.
Maybe even 2 timers that start when the head set is powered on.
one of those characters is a byte, 8bits in a byte, two bytes per sensor=16bits, 32 bytes in the data stream.
But we may have to do the same shifting thing because of the counter.
The gyro data doesn't move very much either 2048-52 quickly glancing over the data.
The values we observe also only move one 104-110 when moving the gyro.
My point is we need to find the conversion for chr(104) -> 2048.00
and then artificially increment 104 and the conversion counts up 2049 and so on..
er 106 and 105 rather it should be noted that they are one byte position apart and the 0 value or 2048 is the same.
They are also at indexes 29(106) and 30(105).
maybe we need to use bit masks 8 * the byte index?
232,233,234,235,236,237,238,239
240,241,242,243,244,245,246,247
I was reading the half precision float standard.
The answer maybe in the wiki article.
I figured it out I'm pretty sure.
Where do I modify the python to check if there is more data
This version will attempt to read more automatically, I'm pretty sure it'll still return 32 bytes though.
Also, has all my scratch code I was using trying to find the values.
I'm pretty sure it turns out they store 4 sets of shorts, then you have to do math.
Like @warrenarea said big endian.
They share the most significant bits which subtract to 256 * 8 to shift the byte and subtract the least significant bits then add them together.
Hmm maybe not.
ok
Looking at the bits in the sensor values is interesting though:
['0', '1', '1', '1', '1', '1', '1', '0', '0', '0', '0', '1', '1', '1']
['0', '1', '1', '1', '1', '1', '0', '1', '0', '0', '0', '0', '0', '1']
['0', '1', '1', '1', '1', '1', '1', '0', '1', '1', '1', '0', '1', '0']
['0', '1', '1', '1', '1', '1', '0', '1', '0', '1', '1', '1', '0', '1']
['0', '1', '1', '1', '1', '1', '0', '1', '0', '1', '1', '1', '0', '0']
['0', '1', '1', '1', '1', '1', '0', '0', '0', '1', '1', '0', '0', '0']
['0', '1', '1', '1', '1', '1', '0', '0', '1', '0', '0', '1', '1', '1']
['0', '1', '1', '1', '1', '1', '0', '1', '0', '1', '0', '1', '1', '1']
['0', '1', '1', '1', '1', '1', '0', '1', '1', '0', '0', '0', '0', '0']
['0', '1', '1', '1', '1', '1', '1', '0', '0', '0', '0', '0', '0', '0']
['1', '0', '0', '1', '0', '0', '1', '1', '1', '0', '0', '0', '0', '1']
['0', '1', '1', '1', '1', '1', '0', '1', '0', '0', '1', '0', '1', '0']
['0', '1', '1', '1', '1', '1', '1', '1', '0', '0', '1', '1', '1', '0']
['0', '1', '1', '1', '1', '1', '0', '1', '0', '1', '0', '1', '1', '0']
['0', '0', '0', '0', '0', '0', '0', '0', '0', '1', '0', '0', '0', '0']
And here I am trying to unplug from the matrix...
8192 4096 2048 1024 512 256 128 64 32 16 8 4 2 1
'0', '1', '1', '1', '1', '1', '0', '1', '0', '1', '0', '1', '1', '0'
8022 * 0.51
4091.22
and .13 for epoc+?
yeah with two bits added, the math works better if multiplied by .13