Processes for Event-based data loading and pre-processing #514

gkarray · 2022-12-02T14:07:01Z

Issue Number:

Objective of pull request: This PR adds Processes for Event-based data loading and pre-processing.

Pull request checklist

Your PR fulfills the following requirements:

Issue created that explains the change and why it's needed
Tests are part of the PR (for bug fixes / features)
Docs reviewed and added / updated if needed (for bug fixes / features)
PR conforms to Coding Conventions
PR applys BSD 3-clause or LGPL2.1+ Licenses to all code files
Lint (flakeheaven lint src/lava tests/) and (bandit -r src/lava/.) pass locally
Build tests (pytest) passes locally

Pull request type

Please check your PR type:

What is the current behavior?

No standard Processes for handling Event-based data.

What is the new behavior?

Standard Processes for handling Event-based data (loading and pre-processing) would become available.

Currently supported Event IO Processes:
- AedatStream : Streams event data loaded from aedat4 files.
- DvStream : Streams event data from the DV software for INIvation cameras, through a TCP connection
Currently supported Event pre-processor Processes:
- BinaryToUnaryPolarity : Converts binary polarity events (0 for negative events, 1 for positive events) to unary polarity events (1 for both negative and positive events)
- EventsToFrame : Transforms sparsely represented list of Events into a densely represented frame of Events. Two modes of operation supported:
  - 2D Output (shape (W, H, 1)) : In this case, input is assumed to be unary. Output is of shape (W, H, 1) where 0 means no Event and 1 means Event (positive or negative) in the unique channel.
    Negative and positive polarities are both encoded as 1.
  - 3D Output (shape (W, H, 2)) : In this case, input is assumed to be binary. Output is of shape (W, H, 2) where 0 means no Event and 1 means negative Event in the first channel and 0 means no Event and 1 means positive Event in the second channel.
Other Process that is more general but could be used as an Event pre-processor:
- MaxPooling: Applies the max-pooling operation on incoming data with shape (W, H, C) where C is not necessarily 1 or 2.

Does this introduce a breaking change?

Yes
No

Supplemental information

mathisrichter

There is a lot here. So far, I have only had time to review the AedatDataLoader and its tests. This needs extensive revision but most of the items are small or cosmetic. Please make sure you understand the issues and fix other areas of this PR that are affected by similar problems.

I will review other parts of the PR later, ideally once you have already updated it.

pyproject.toml

src/lava/proc/event_data/event_data_loader/aedat_data_loader.py

mathisrichter · 2022-12-03T14:26:26Z

tests/lava/proc/event_data/event_data_loader/test_aedat_data_loader.py

+        # Stopping
+        data_loader.stop()
+
+        self.assertFalse(data_loader.runtime._is_running)


Can be removed. Replace with asserts on the output - see my comment above.

mathisrichter · 2022-12-03T14:27:55Z

tests/lava/proc/event_data/event_data_loader/test_aedat_data_loader.py

+        data_history = [
+            [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
+            [1, 1, 1, 1, 1, 1, 1, 1, 1],
+            [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
+            [1, 1, 1, 1, 1, 1],
+            [0],
+            [1, 1, 1],
+            [1],
+            [1],
+            [1]
+        ]
+        indices_history = [
+            [1597, 2308, 2486, 2496, 2498, 1787, 2642, 2633, 2489,
+             2488, 1596, 1729, 1727, 2500, 1780],
+            [1600, 1732, 2297, 1388, 2290, 2305, 3704, 3519, 1911],
+            [7138, 2301, 2471, 1601, 2982, 1364, 1379, 1386, 1384,
+             2983, 1390, 2289, 1401, 1362, 2293],
+            [1910, 1382, 1909, 1562, 1606, 1381],
+            [464],
+            [2323, 1908, 1393],
+            [4062],
+            [1792],
+            [3889]
+        ]


I'm assuming this comes from the DVS file we are using. Can we reduce this in size to make the test smaller and more to the point?

Now only using data from a single timestep

That may not be enough then. The test should make sure that we can read events, from multiple time steps, with different polarities, at different positions/indices. The test data should cover all of those cases (and others, if I forgot something) but does not need more. In the previous version, it seemed like it contained a bunch more cases.

Mmmh, good point. Then I guess a good middle ground could be to go up to timestep 5 (with this data). We tried to reduce the resolution of the aedat4 file, but it seems like trying to use the "crop scale" options on dv for resizing leads to weird behavior with the metadata of the file, like reducing resolution doesn't actually change the x and y indices in the expected way. Do you think it's worth trying to crop another part of the video where we may get multiple polarities, and varying event batch sizes in the first 2-3 events?

mathisrichter · 2022-12-03T14:33:22Z

tests/lava/proc/event_data/event_data_loader/test_aedat_data_loader.py

+            if expected_data.shape[0] > max_num_events:
+                data_idx_array = np.arange(0, expected_data.shape[0])
+                sampled_idx = rng.choice(data_idx_array,
+                                         max_num_events,
+                                         replace=False)


This seems to reimplement what you are testing, which is not a good way to test. Instead, make sure that the outcome is correct. Here, it should be fine to just assert that after subsampling, the number of events is the maximum. If that is not enough, you could hard-code the expected events after subsampling.

Additionally, you could write another test that runs the Process twice with different seeds and asserts that the events have been subsampled differently. This makes sure that the randomness works.

mathisrichter · 2022-12-03T14:33:59Z

tests/lava/proc/event_data/event_data_loader/test_aedat_data_loader.py

+
+        self.assertFalse(data_loader.runtime._is_running)
+
+    def test_end_of_file(self):


What does this test do?

mathisrichter · 2022-12-03T14:36:07Z

tests/lava/proc/event_data/event_data_loader/test_aedat_data_loader.py

+
+        self.assertFalse(data_loader.runtime._is_running)
+
+    def test_index_encoding(self):


What does this test do?

…/lava into dev/event_data_processes � Conflicts: � src/lava/proc/event_data/io/dv_stream.py

…/lava into dev/event_data_processes � Conflicts: � tests/lava/proc/event_data/io/test_dv_stream.py

…/lava into dev/event_data_processes

Signed-off-by: Mathis Richter <mathis.richter@intel.com>

…/lava into dev/event_data_processes

Signed-off-by: Mathis Richter <mathis.richter@intel.com>

weidel-p

After the feedback from @mathisrichter, I like most of it.
All classes and function are documented and, as far as I can see, tested.

I would like to change the folder structure, though. I dislike the "event_data" folder as the dataloader should be part of IO.
My suggestion:

lava/proc/io/
    event_data/
        aedat_stream.py
        dv_stream.py
    sink.py
    ...

lava/proc/transformations
    event_data/
        binary_to_unary.py
        event_to_frame.py 
    max_pooling.py

Signed-off-by: Mathis Richter <mathis.richter@intel.com>

…/lava into dev/event_data_processes

…_data_processes

This reverts commit 20a85bf.

gkarray and others added 4 commits December 2, 2022 09:51

Adding first round of Event Data Processes

452e68e

Adding Flattening implementation

77dc965

flattening proc, pm + all unit tests

d1ac4f5

flattening unit tests

5a89aa6

mathisrichter requested changes Dec 3, 2022

View reviewed changes

j0shcd and others added 19 commits December 5, 2022 14:05

addressed PR comments, still TODOs

9018466

addressed remaining PR comments, still TODOs

18fff7d

Applied PR comments for other processes

ddb3bb7

Applied PR comments for other tests, still TODOs

877ed67

doc strings for AedatDataLoader (+tests), events utils, PR comments

093605f

restructuring Processes

ee81533

add beginning of input process

57c8929

Merge branch 'dev/event_data_processes' of https://github.com/lava-nc…

6b15db9

…/lava into dev/event_data_processes � Conflicts: � src/lava/proc/event_data/io/dv_stream.py

refactored tests

1bd6346

WIP

be276bd

Merge branch 'dev/event_data_processes' of https://github.com/lava-nc…

4869e31

…/lava into dev/event_data_processes � Conflicts: � tests/lava/proc/event_data/io/test_dv_stream.py

refactoring event data processes (WIP)

3554e89

Merge branch 'dev/event_data_processes' of https://github.com/lava-nc…

fc79a24

…/lava into dev/event_data_processes

added tests for data loader + b-to-u

0c54139

adding complete unit test suite for MaxPooling

1bbcc66

Merge branch 'dev/event_data_processes' of https://github.com/lava-nc…

5e48595

…/lava into dev/event_data_processes

polished docstrings, split tests

34e752a

Merge branch 'main' into dev/event_data_processes

08415dc

Cleaned up DvStream and tests.

dc02db1

Signed-off-by: Mathis Richter <mathis.richter@intel.com>

gkarray requested review from weidel-p, awintel and PhilippPlank December 8, 2022 11:06

j0shcd and others added 3 commits December 8, 2022 15:17

small changes to doc strings + modifications sub_sampling

1581cac

Cleaned BinaryToUnary Process and unit tests.

5fbac8b

Signed-off-by: Mathis Richter <mathis.richter@intel.com>

Merge branch 'dev/event_data_processes' of https://github.com/lava-nc…

4ab5d1e

…/lava into dev/event_data_processes

mathisrichter and others added 5 commits December 8, 2022 15:45

Simplified the ProcModel of the BinaryToUnary Process.

082c9c9

Signed-off-by: Mathis Richter <mathis.richter@intel.com>

Fixed docstring.

7c4f1b4

Signed-off-by: Mathis Richter <mathis.richter@intel.com>

Reviewed EventsToFrame Process and unit tests.

099259f

Signed-off-by: Mathis Richter <mathis.richter@intel.com>

Reviewed EventsToFrame ProcessModel and unit tests.

3f5ee2d

Signed-off-by: Mathis Richter <mathis.richter@intel.com>

dv_stream without subsampling

283b296

weidel-p approved these changes Dec 9, 2022

View reviewed changes

j0shcd and others added 11 commits December 9, 2022 16:31

updated dv_stream + tests

6d8a4dc

moved encoding method into utils

2617193

sub-sampling and polishing other tests

a6778a4

Reviewed AedatStream unit tests.

ffac5f6

Signed-off-by: Mathis Richter <mathis.richter@intel.com>

Added missing empty line.

5a8ceb2

Signed-off-by: Mathis Richter <mathis.richter@intel.com>

Merge branch 'dev/event_data_processes' of https://github.com/lava-nc…

354340a

…/lava into dev/event_data_processes

Merge branch 'main' of https://github.com/lava-nc/lava into dev/event…

d795530

…_data_processes

^testtesttest

20a85bf

Revert "^testtesttest"

b358462

This reverts commit 20a85bf.

Merge branch 'main' into dev/event_data_processes

d1c7e89

Merge branch 'main' into dev/event_data_processes

fbe3653

mgkwill force-pushed the main branch from 527dcfa to 5ed88f7 Compare November 15, 2023 04:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Processes for Event-based data loading and pre-processing #514

Processes for Event-based data loading and pre-processing #514

gkarray commented Dec 2, 2022 •

edited

mathisrichter left a comment

mathisrichter Dec 3, 2022

mathisrichter Dec 3, 2022

j0shcd Dec 5, 2022 •

edited

mathisrichter Dec 5, 2022

j0shcd Dec 6, 2022 •

edited

mathisrichter Dec 3, 2022

mathisrichter Dec 3, 2022

mathisrichter Dec 3, 2022

weidel-p left a comment •

edited


		self.assertFalse(data_loader.runtime._is_running)

		def test_end_of_file(self):


		self.assertFalse(data_loader.runtime._is_running)

		def test_index_encoding(self):

Processes for Event-based data loading and pre-processing #514

Are you sure you want to change the base?

Processes for Event-based data loading and pre-processing #514

Conversation

gkarray commented Dec 2, 2022 • edited

Pull request checklist

Pull request type

What is the current behavior?

What is the new behavior?

Does this introduce a breaking change?

Supplemental information

mathisrichter left a comment

Choose a reason for hiding this comment

mathisrichter Dec 3, 2022

Choose a reason for hiding this comment

mathisrichter Dec 3, 2022

Choose a reason for hiding this comment

j0shcd Dec 5, 2022 • edited

Choose a reason for hiding this comment

mathisrichter Dec 5, 2022

Choose a reason for hiding this comment

j0shcd Dec 6, 2022 • edited

Choose a reason for hiding this comment

mathisrichter Dec 3, 2022

Choose a reason for hiding this comment

mathisrichter Dec 3, 2022

Choose a reason for hiding this comment

mathisrichter Dec 3, 2022

Choose a reason for hiding this comment

weidel-p left a comment • edited

Choose a reason for hiding this comment

gkarray commented Dec 2, 2022 •

edited

j0shcd Dec 5, 2022 •

edited

j0shcd Dec 6, 2022 •

edited

weidel-p left a comment •

edited