How to capture exception generated by runtime.initialize()

[waltergallegog]

Hello,
Is it possible to capture exceptions generated when calling runtime.initialize() due to issues during the building ?

In my case, I'm testing a network in Loihi2 which does not fit in the Loihi2 hardware (I'm using a number of layers higher than the limit).
The log shows that an assertion error is raised:

AssertionError: Adding <numCores>=1 exceeds maximum number of cores per chip...

However, I can not capture this exception from my code. I want to capture the exception to understand if the initialization has been successful before calling runtime.start().

I guess the issue is in how the exceptions are handled in the MultiProcessing message passage interface, but I'm not sure how it works exactly and how can I use it.

This is what I'm trying:

        compiler = Compiler()
        executable = compiler.compile(
            self.lif_interface,
            run_cfg=run_config
        )

        mp = ActorType.MultiProcessing
        runtime = Runtime(exe=executable,
                          message_infrastructure_type=mp)

        try:
            runtime.initialize()
        except AssertionError as exception:
            print(f'captured {exception}')
        else:
            # initialize() successful. call start().
            runtime.start(run_condition=RunSteps(num_steps=num_steps))

Thanks!

PS: I'm posting this here instead of sending an email as it seems more related to the lava framework (runtime, builders, MultiProcessing) than to the specifics of the loihi2 builder.

[mgkwill]

HI @waltergallegog.

In the above example you are trying to access private APIs and Lava was not designed to work in this way. Also, the error message you are trying to capture is from nxcore, the hardware interface to Loihi.

While nxcore errors are visible, they are not part of the same call stack due to the way lava-loihi and nxcore communicate and the specific error seems poorly written to pass any information back up the stack:

def addNxCores(self, numCores, allocate=True):
...
assert self.numNxCores + numCores <= self.CORES_PER_CHIP, \
            "Adding <numCores>={} exceeds maximum number of cores per " \
            "chip {}.".format(numCores, self.CORES_PER_CHIP)

I'd suggest you instantiate your runtime with higher logging level like this:

runtime = Runtime(exe=executable,
                  message_infrastructure_type=mp,
                  loglevel=logging.INFO)

You might get useful output here that you could use to decide if the test is possible.

However, I think the real problem is that the runtime does not know about the AssertionError until runtime.start(run_condition=RunSteps(num_steps=num_steps)) is called because the runtime communicates with the nxsdk_runtime_service that is the go between for lava and loihi and uses nxcore to do this communication.

You could probably setup a pretest that tries to initialize and run with the intended config and have that fail fast if you receive the AssertionError: Adding <numCores>=1 exceeds maximum number of cores per chip message and if it does not you start a real run that uses this configuration for benchmarking.