Read_raw_eyelink failure: missing headpos data in file

[scott-huberty]

Description of the problem

A researcher reported a problem with our eyelink reader and shared the problematic Eyelink file (ASCII).

There are 2 separate issues.

1. This file has an empty value for recording date/time, and our reader doesn't handle that gracefully. EDIT: The researcher actually manually removed the datetime from the ASCII file, so I think it's reasonable for us to continue assuming that EyeLink will write the datetime to the file.
2. This file says that there are head-position (x, y, distance) data, but that data isn't actually in the file, and our reader doesn't handle that either.

I'll try to open a fix this week.

Steps to reproduce

from pathlib import Path
import mne

fname = Path(".") / "to" / "problem_file.asc"
mne.io.read_raw_eyelink(fname)

Link to data

See: https://mne.discourse.group/t/problem-reading-binocular-data-with-mne-io-read-raw-eyelink/8555

Expected results

successful file read

Actual results

BUG 1:

stack trace

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[2], line 1
----> 1 raw = mne.io.read_raw_eyelink("example_data.asc")

File ~/devel/repos/mne-python/mne/io/eyelink/eyelink.py:62, in read_raw_eyelink(fname, create_annotations, apply_offsets, find_overlaps, overlap_threshold, verbose)
     32 """Reader for an Eyelink ``.asc`` file.
     33 
     34 Parameters
   (...)
     58 ``'BAD_ACQ_SKIP'``.
     59 """
     60 fname = _check_fname(fname, overwrite="read", must_exist=True, name="fname")
---> 62 raw_eyelink = RawEyelink(
     63     fname,
     64     create_annotations=create_annotations,
     65     apply_offsets=apply_offsets,
     66     find_overlaps=find_overlaps,
     67     overlap_threshold=overlap_threshold,
     68     verbose=verbose,
     69 )
     70 return raw_eyelink

File <decorator-gen-180>:12, in __init__(self, fname, create_annotations, apply_offsets, find_overlaps, overlap_threshold, verbose)

File ~/devel/repos/mne-python/mne/io/eyelink/eyelink.py:107, in RawEyelink.__init__(self, fname, create_annotations, apply_offsets, find_overlaps, overlap_threshold, verbose)
    104 fname = Path(fname)
    106 # ======================== Parse ASCII file ==========================
--> 107 eye_ch_data, info, raw_extras = _parse_eyelink_ascii(
    108     fname, find_overlaps, overlap_threshold, apply_offsets
    109 )
    110 # ======================== Create Raw Object =========================
    111 super().__init__(
    112     info,
    113     preload=eye_ch_data,
   (...)
    116     raw_extras=[raw_extras],
    117 )

File ~/devel/repos/mne-python/mne/io/eyelink/_utils.py:50, in _parse_eyelink_ascii(fname, find_overlaps, overlap_threshold, apply_offsets)
     48 raw_extras.update(_parse_recording_blocks(fname))
     49 raw_extras.update(_get_metadata(raw_extras))
---> 50 raw_extras["dt"] = _get_recording_datetime(fname)
     51 _validate_data(raw_extras)
     53 # ======================== Create DataFrames ========================

File ~/devel/repos/mne-python/mne/io/eyelink/_utils.py:190, in _get_recording_datetime(fname)
    186 fmt = "%a %b %d %H:%M:%S %Y"
    187 # Eyelink measdate timestamps are timezone naive.
    188 # Force datetime to be in UTC.
    189 # Even though dt is probably in local time zone.
--> 190 dt_naive = datetime.strptime(dt_str, fmt)
    191 return dt_naive.replace(tzinfo=tz)

File ~/miniforge3/envs/mnedev/lib/python3.10/_strptime.py:568, in _strptime_datetime(cls, data_string, format)
    565 def _strptime_datetime(cls, data_string, format="%a %b %d %H:%M:%S %Y"):
    566     """Return a class cls instance based on the input string and the
    567     format string."""
--> 568     tt, fraction, gmtoff_fraction = _strptime(data_string, format)
    569     tzname, gmtoff = tt[-2:]
    570     args = tt[:6] + (fraction,)

File ~/miniforge3/envs/mnedev/lib/python3.10/_strptime.py:349, in _strptime(data_string, format)
    347 found = format_regex.match(data_string)
    348 if not found:
--> 349     raise ValueError("time data %r does not match format %r" %
    350                      (data_string, format))
    351 if len(data_string) != found.end():
    352     raise ValueError("unconverted data remains: %s" %
    353                       data_string[found.end():])

ValueError: time data '' does not match format '%a %b %d %H:%M:%S %Y'

BUG 2:

stack trace

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[2], line 1
----> 1 raw = mne.io.read_raw_eyelink("example_data.asc")

File ~/devel/repos/mne-python/mne/io/eyelink/eyelink.py:62, in read_raw_eyelink(fname, create_annotations, apply_offsets, find_overlaps, overlap_threshold, verbose)
     32 """Reader for an Eyelink ``.asc`` file.
     33 
     34 Parameters
   (...)
     58 ``'BAD_ACQ_SKIP'``.
     59 """
     60 fname = _check_fname(fname, overwrite="read", must_exist=True, name="fname")
---> 62 raw_eyelink = RawEyelink(
     63     fname,
     64     create_annotations=create_annotations,
     65     apply_offsets=apply_offsets,
     66     find_overlaps=find_overlaps,
     67     overlap_threshold=overlap_threshold,
     68     verbose=verbose,
     69 )
     70 return raw_eyelink

File <decorator-gen-180>:12, in __init__(self, fname, create_annotations, apply_offsets, find_overlaps, overlap_threshold, verbose)

File ~/devel/repos/mne-python/mne/io/eyelink/eyelink.py:107, in RawEyelink.__init__(self, fname, create_annotations, apply_offsets, find_overlaps, overlap_threshold, verbose)
    104 fname = Path(fname)
    106 # ======================== Parse ASCII file ==========================
--> 107 eye_ch_data, info, raw_extras = _parse_eyelink_ascii(
    108     fname, find_overlaps, overlap_threshold, apply_offsets
    109 )
    110 # ======================== Create Raw Object =========================
    111 super().__init__(
    112     info,
    113     preload=eye_ch_data,
   (...)
    116     raw_extras=[raw_extras],
    117 )

File ~/devel/repos/mne-python/mne/io/eyelink/_utils.py:58, in _parse_eyelink_ascii(fname, find_overlaps, overlap_threshold, apply_offsets)
     56 # add column names to dataframes and set the dtype of each column
     57 col_names, ch_names = _infer_col_names(raw_extras)
---> 58 raw_extras["dfs"] = _assign_col_names(col_names, raw_extras["dfs"])
     59 raw_extras["dfs"] = _set_df_dtypes(raw_extras["dfs"])  # set dtypes for dataframes
     60 # if HREF data, convert to radians

File ~/devel/repos/mne-python/mne/io/eyelink/_utils.py:407, in _assign_col_names(col_names, df_dict)
    405 for key, df in df_dict.items():
    406     if key in ("samples", "blinks", "fixations", "saccades"):
--> 407         df.columns = col_names[key]
    408     elif key == "messages":
    409         cols = ["time", "offset", "event_msg"]

File ~/miniforge3/envs/mnedev/lib/python3.10/site-packages/pandas/core/generic.py:6310, in NDFrame.__setattr__(self, name, value)
   6308 try:
   6309     object.__getattribute__(self, name)
-> 6310     return object.__setattr__(self, name, value)
   6311 except AttributeError:
   6312     pass

File properties.pyx:69, in pandas._libs.properties.AxisProperty.__set__()

File ~/miniforge3/envs/mnedev/lib/python3.10/site-packages/pandas/core/generic.py:813, in NDFrame._set_axis(self, axis, labels)
    808 """
    809 This is called from the cython code when we set the `index` attribute
    810 directly, e.g. `series.index = [1, 2, 3]`.
    811 """
    812 labels = ensure_index(labels)
--> 813 self._mgr.set_axis(axis, labels)
    814 self._clear_item_cache()

File ~/miniforge3/envs/mnedev/lib/python3.10/site-packages/pandas/core/internals/managers.py:238, in BaseBlockManager.set_axis(self, axis, new_labels)
    236 def set_axis(self, axis: AxisInt, new_labels: Index) -> None:
    237     # Caller is responsible for ensuring we have an Index object.
--> 238     self._validate_set_axis(axis, new_labels)
    239     self.axes[axis] = new_labels

File ~/miniforge3/envs/mnedev/lib/python3.10/site-packages/pandas/core/internals/base.py:98, in DataManager._validate_set_axis(self, axis, new_labels)
     95     pass
     97 elif new_len != old_len:
---> 98     raise ValueError(
     99         f"Length mismatch: Expected axis has {old_len} elements, new "
    100         f"values have {new_len} elements"
    101     )

ValueError: Length mismatch: Expected axis has 8 elements, new values have 11 elements

Additional information

N/A

[larsoner]

EDIT: The researcher actually manually removed the datetime from the ASCII file, so I think it's reasonable for us to continue assuming that EyeLink will write the datetime to the file.

I think it's okay to turn this into a warning rather than an error. It's possible others have done it I suppose

[scott-huberty]

I haven't forgotten about this, just haven't figured out a fix for it yet (and I'm not sure there is a simple fix).

It's difficult to figure out the channel names from the Eyelink file, when the file provides incorrect information about this (as in this case, it tells us head position channels are there, but they aren't.). My best idea right now is to let the user pass in the channel names themselves.

I'll try to stop by the next office hours to discuss the issue.

[larsoner]

In other readers we have an exclude param

https://mne.tools/stable/generated/mne.io.read_raw_edf.html

maybe that would work?

[scott-huberty]

In other readers we have an exclude param

https://mne.tools/stable/generated/mne.io.read_raw_edf.html

maybe that would work?

Thanks @larsoner , I'll try this out!