New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
container for eye tracking related annotation information #12208
Comments
@drammock , just FYI: I'm working on this in the context of the sprint but I don't think I can label it accordingly myself. |
this has been asked in the past, basically to have arbitrary columns in
Annotations.
It's also related to the topic of metadata on epochs to "attach" to each
epoch any arbitrary
field.
from a historical perspective metadata were added before Annotations and
metadata behaves
a lot more like a dataframe.
… Message ID: ***@***.***>
|
👍 ahja nice, metadata (which I haven't worked with yet) sound like a viable round for [2] (ie., more detailled info on events). challenge for raw data remains. |
In the eyetracking analysis module, I would have functions which computes a given metric for an eyetracking event, e.g.
which takes in input a And then we can have higher level function/method/objects which will use those metric-computation function automatically and fill e.g. the epochs metadata. |
looks nice. thx @mscheltienne. However, don't we "waste energy" this way? Normal procedure is that whatever determines where a saccades is, at the same time returns the saccade parameters. How about an approach like:
which annotates |
I was applying this sentence, implementing analysis functions which compute those metrics, regardless of the eyetracking device. |
yes, but is in your scenario the |
Yes, the events should be present in the |
Let's try to solve one problem at a time.
I think it is possible to solve (1) first and independently of (2), as long as we bear in mind that the values included in As noted above, metadata dataframes are at present only supported for Epochs objects, and Annotations objects only have 4 fields (onset, duration, description, and channels). So logical options are:
|
I think As for My other thought is that annotations have a visualization component, and I don't know how we would visualize a Happy to hear others thoughts. |
@drammock , great summary/battle plan. Thx! 🙏
As @scott-huberty mentioned and as far as I have seen, at present the reading (for Regarding (b):
I agree.
I have not worked much enough with annotations (esp. their visualization) to have a strong opinion here. But I see your concerns. Also, I find
tbh, I have some concerns also here: Let me sketch the use cases which I have in mind: [2] Plot FRP as a function of the size of the incoming saccade: [3] use gaze event properties to discard independetly epoched data: Just putting this out for thought. |
update: If saccade info lives in
when epoching the data (either by using saccade events or other events) the saccade info can be passed along to the Epochs by making use of the existing functionalities for Any input/opinions from @drammock or @larsoner are welcome. I'm off for lunch now and would afterwards start working on a PR suggesting the introduction of Sidenote: @mscheltienne mentioned that we should introduce a "locking" feature/flag for annotations which makes sure that these annotations cannot be (accidentally) moved manually be the user. This also applies to saccade annotations which should never be moved manually. Definitely a separate issue though. |
I would make this attribute private, and maybe call it Lines 1088 to 1104 in 9b57c51
|
|
If someone of the core-devs could take a look at the PR, to see whether this is roughly what you had in mind, I'd be thankful (before I refactor more). |
@drammock Wide format is what we use for epochs metadata, so I'd be in favor of using it for Raw as well |
We (@eioe, @britta-wstnr and I) discussed this live today. The tentative plan is:
@eioe and @britta-wstnr please correct anything I've mis-remembered. |
Ok, but the consensus seems to be to not use DataFrames at all:
|
Yeah, I didn't understand what was meant by that 😅 Or rather, I thought the solution to that was to attach the metadata to Annotations – in whatever form we choose? Couldn't we choose a DataFrame, then? I would even go so far as to say, why not store all Annotations + metadata in a DataFrame while keeping the Annotations API the same (e.g. |
Thx @drammock for the summary of our discussion earlier today. I've just seen this now (🙈) after we talked and slightly changed plans again. Here's roughly the new idea: from mne.utils import _validate_type
from dataclasses import dataclass, fields
@dataclass(frozen=True)
class SaccadeAnnotMetadata:
sac_vmax: float = None
sac_amplitude: float = None
sac_angle: float = None
sac_startpos_x: float = None
sac_startpos_y: float = None
sac_endpos_x: float = None
sac_endpos_y: float = None
def __post_init__(self):
for field in fields(self):
_validate_type(getattr(self, field.name), float)
@dataclass(frozen=True)
class FixationAnnotMetadata:
fix_avgpos_x: float = None
fix_avgpos_y: float = None
incoming_saccade: SaccadeAnnotMetadata = None
def __post_init__(self):
_validate_type(self.fix_avgpos_x, float)
_validate_type(self.fix_avgpos_y, float)
_validate_type(self.incoming_saccade, SaccadeAnnotMetadata) Some thoughts:
|
I think, one reason that speaks against this is that different annotations may have different fields in their metadata (or no metadata at all) which would make |
ok, here's another update. Here's an examplary workflow: import numpy as np
import pandas as pd
import mne
from mne.annotations import Annotations
from mne.utils import _validate_type
class EyeEvents:
def __init__(self, annots=None, event_data=None):
self._fixations = pd.DataFrame(
columns=['annot_id', 'fix_avgpos_x', 'fix_avgpos_y', 'incoming_saccade_id']
)
self._saccades = pd.DataFrame(
columns=['annot_id', 'sac_vmax', 'sac_amplitude', 'sac_angle',
'sac_startpos_x', 'sac_startpos_y', 'sac_endpos_x',
'sac_endpos_y']
)
if (annots is not None):
_validate_type(annots, Annotations)
# make sure that we have the same number of annotations as event_data
if len(annots) != len(event_data):
raise ValueError(f'annots and event_data must have the same number ' +
f'of annotations. {len(annots)} != {len(event_data)}')
for annot, event_datum in zip(annots, event_data):
self.add_annot(annot, event_datum)
def add_annot(self, annot, event_data):
if annot['description'] == 'fixation':
# make sure that we have the correct keys in event_data
if np.all(np.isin([str(k) for k in event_data.keys()],
self._fixations.columns)):
event_data['annot_id'] = annot['_id']
tmp_df = pd.DataFrame([event_data])
self._fixations = pd.concat([self._fixations, tmp_df])
else:
raise ValueError(f'event_data has incorrect keys. Fixation events ' +
f'have columns: {self._fixations.columns} but ' +
f'event_data has keys: {event_data.keys()}.')
elif annot['description'] == 'saccade':
# make sure that we have the correct keys in event_data
if np.all(np.isin([str(k) for k in event_data.keys()],
self._saccades.columns)):
event_data['annot_id'] = annot['_id']
tmp_df = pd.DataFrame([event_data])
self._saccades = pd.concat([self._saccades, tmp_df])
else:
raise ValueError(f'event_data has incorrect keys. Saccade events ' +
f'have columns: {self._saccades.columns} but ' +
f'event_data has keys: {event_data.columns}.')
else:
raise ValueError('Invalid annotation description. EyeEvents can only ' +
'handle annotations with description "Fixation" or ' +
'"Saccade".')
if __name__ == '__main__':
# Dummy saccade data
sac_vmax = np.array([0.5, 1, 2, 3, 4]) * 123
sac_amplitude = np.array([4, 3, 2, 1, 0]) * 10
sac_angle = np.array([0, 0.2, 0.4, 0.6, 0.8]) * 360
sac_startpos_x = np.array([0, 1, 2, 3, 4])
sac_startpos_y = np.array([0, 1, 2, 3, 4])
sac_endpos_x = np.array([0, 1, 2, 3, 4]) + 10
sac_endpos_y = np.array([0, 1, 2, 3, 4]) - 10
# Dummy fixation data
fix_pos_x = sac_endpos_x
fix_pos_y = sac_endpos_y
saccade_eventdata = [{'sac_vmax': vmax, 'sac_amplitude': amp,
'sac_angle': ang, 'sac_startpos_x': sx,
'sac_startpos_y': sy, 'sac_endpos_x': ex,
'sac_endpos_y': ey} for vmax, amp, ang, sx, sy, ex, ey
in zip(sac_vmax, sac_amplitude, sac_angle,
sac_startpos_x, sac_startpos_y, sac_endpos_x,
sac_endpos_y)]
fixation_eventdata = [{'fix_avgpos_x': x, 'fix_avgpos_y': y } for x, y in
zip(fix_pos_x, fix_pos_y)]
annots_sacc = Annotations(
onset=[1, 5, 18, 33, 45],
duration=np.ones(5),
description=['saccade'] * 5
)
# We construct the fixation events so that they each occur after one of the
# saccades
annots_fix = Annotations(
onset=[sacc_on + sacc_dur for (sacc_on, sacc_dur) in zip(annots_sacc.onset,
annots_sacc.duration)],
duration=np.ones(5),
description=['fixation'] * 5
)
# We can now grab the IDs of the preceding saccad annotations and add them to the
# fixation event data
incoming_saccade_ids = annots_sacc._id
for fix_ed, sacc_id in zip(fixation_eventdata, incoming_saccade_ids):
fix_ed['incoming_saccade_id'] = sacc_id
# Now we can combine the saccade and fixation annotations
all_annot = annots_sacc + annots_fix
event_data_concat = saccade_eventdata + fixation_eventdata
# However the annotations are automatically sorted by onset time, while
# the event data are just concatenated.
# So we need to sort the event data accordingly to be able to link them.
# For this, we can use the IDs of the annotations:
ids_events_concat = all_annot._get_id(
np.hstack([annots_sacc.onset, annots_fix.onset]),
np.hstack([annots_sacc.duration, annots_fix.duration]),
np.hstack([annots_sacc.description, annots_fix.description]))
event_data_sorted = []
for id in all_annot._id:
idx = np.where(np.array(ids_events_concat) == id)[0][0]
event_data_sorted.append(event_data_concat[idx])
eye_events = EyeEvents(annots_sacc + annots_fix,
event_data_sorted)
print(eye_events._saccades)
print(eye_events._fixations) Output:
|
@drammock uf you have a sec to give this a look, that'd be great. Esp if the |
@eioe and I just had another chat about this. Here's my position after that:
Things to think about:
please feel free to raise objections or point out problems that I'm missing. |
thx @drammock
as discussed, aesthetically unpleasant, but fine for me. Will also reduce redundancy in code (which might be more important then a densly filled DF). On it.
Agree. That's why I suggested
That's also true for the
Yea, I thought about that and also @britta-wstnr had suggested that, as the numbers are smaller.
|
I did now skim read the according PR. And I do not think that is something we need to worry much in the context which we are discussing here. The kind of
Yea, this will be controversial. I think this boils down to how much the object that we are talking about is |
Hi, I'm for consistency within MNE, and here "metadata" (which is a super generic term anyway) has a different meaning than in BIDS. If we were talking about MNE-BIDS I would support your proposal for choosing a different name, but for MNE, "metadata" is the logical choice in my opinion |
+1 for |
ok, another lively discussion and pivot. 🤭 Main participants this time @britta-wstnr @drammock @larsoner .
Open questions:
Opinions on any of the points welcome. Otherwise, I'll make a decision and take opinions on the PR. |
We should probably use |
class Annotations:
"""Mock up of new `Annotations.data` attribute."""
def __init__(self):
self._data = dict()
@property
def data(self):
return self._data
foo = Annotations()
# even without a @setter I can still add new (integer!) keys:
foo.data[3] = lambda: print("value can be a callable!")
print(foo.data)
# {3: <function <lambda> at 0x7fb58260f100>} Upon init of the Annotations object we can validate whatever is passed in
I just briefly chatted with @larsoner and we're both OK with either approach. I slightly prefer approach 1. |
Thx for pointing this out. If I was to build an authoritarian regime I would not want to do it using python. Option 1. sounds good. |
Describe the new feature or enhancement
Currently, there is no dedicated way to store details about eye tracking related annotations. Several properties of saccades (e.g., peak velocity, start+end position, ...) and fixations (e.g., avg position) are useful information downstream. (This info can/will hopefully in future be generated by a [to be implemented] ET analysis module or [for now] be read from the Eyelink output file.)
Atm, annotations only allow to store
onset
,duration
,description
, (andorig_time
) as they need to be generic. So extendingAnnotations
with additional (eye tracking specific) fields is not an option.For reference: EYE-EEG stores
{'latency', 'duration', 'sac_vmax', 'sac_amplitude', 'sac_angle', 'epoch', 'sac_startpos_x', 'sac_startpos_y', 'sac_endpos_x', 'sac_endpos_y'}
for saccades and{'latency', 'duration', 'fix_avgpos_x', 'fix_avgpos_y', 'epoch', 'sac_vmax', 'sac_amplitude', 'sac_angle', 'sac_startpos_x', 'sac_startpos_y', 'sac_endpos_x', 'sac_endpos_y'}
for fixations (Code).Describe your proposed implementation
[1] With @dominikwelke, @mscheltienne , @sappelhoff we discussed having a separate structure (probably a dict) that holds the according info and can be indexed with a unique identifier linked to a given annotation object. This identifier could be a hash formed by combining onset, duration, and description.
[2] Ideally, the information can still be linked once the annotations are transferred into events. I guess, the event_dict route could be extended accordingly but have not thought this through or discussed with others.
Describe possible alternatives
As discussed above, extending the
Annotation
class does not seem a viable alternative.Alternatively, we can leave it to the user as the metrics can theoretically be calculated from the gaze data on the fly. This, however, seems to be tedious and non-trivial (esp., for parameters like peak velocity).
Additional context
@scott-huberty (and @larsoner ): do you have ideas/opinions regarding this? As you have implemented the bulk of the existing functionality regarding eye tracking data and probably already thought about similar questions. Thx!
The text was updated successfully, but these errors were encountered: