TypeError when learning full rank atoms with GreedyCDL on windows

[MerlinDumeur]

Hello,

I'm getting the following error when trying to use GreedyCDL with rank1=False and solver_z='lgcd':

.....05
joblib.externals.loky.process_executor._RemoteTraceback:
"""
Traceback (most recent call last):
  File "C:\Users\dumemer\.conda\envs\LG\lib\site-packages\joblib\externals\loky\process_executor.py", line 428, in _process_worker
    r = call_item()
  File "C:\Users\dumemer\.conda\envs\LG\lib\site-packages\joblib\externals\loky\process_executor.py", line 275, in __call__
    return self.fn(*self.args, **self.kwargs)
  File "C:\Users\dumemer\.conda\envs\LG\lib\site-packages\joblib\_parallel_backends.py", line 620, in __call__
    return self.func(*args, **kwargs)
  File "C:\Users\dumemer\.conda\envs\LG\lib\site-packages\joblib\parallel.py", line 288, in __call__
    return [func(*args, **kwargs)
  File "C:\Users\dumemer\.conda\envs\LG\lib\site-packages\joblib\parallel.py", line 288, in <listcomp>
    return [func(*args, **kwargs)
  File "~\Merlin\alphacsc\alphacsc\update_z_multi.py", line 135, in _update_z_multi_idx
    constants['DtD'] = compute_DtD(D=D, n_channels=n_channels)
  File "~\Merlin\alphacsc\alphacsc\utils\compute_constants.py", line 11, in compute_DtD
    return _compute_DtD_D(D)
  File "C:\Users\dumemer\.conda\envs\LG\lib\site-packages\numba\core\dispatcher.py", line 703, in _explain_matching_error
    raise TypeError(msg)
TypeError: No matching definition for argument type(s) readonly array(float64, 3d, C)
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "~\Merlin\alphacsc\alphacsc\convolutional_dictionary_learning.py", line 189, in fit
    res = learn_d_z_multi(
  File "~\Merlin\alphacsc\alphacsc\learn_d_z_multi.py", line 202, in learn_d_z_multi
    pobj, times = _batch_learn(greedy=True, **kwargs)
  File "~\Merlin\alphacsc\alphacsc\learn_d_z_multi.py", line 288, in _batch_learn
    z_encoder.compute_z()
  File "~\Merlin\alphacsc\alphacsc\_z_encoder.py", line 278, in compute_z
    self.z_hat, self.ztz, self.ztX = self._compute_z_aux(self.X,
  File "~\Merlin\alphacsc\alphacsc\_z_encoder.py", line 271, in _compute_z_aux
    return update_z_multi(
  File "~\Merlin\alphacsc\alphacsc\update_z_multi.py", line 78, in update_z_multi
    results = Parallel(n_jobs=n_jobs)(
  File "C:\Users\dumemer\.conda\envs\LG\lib\site-packages\joblib\parallel.py", line 1098, in __call__
    self.retrieve()
  File "C:\Users\dumemer\.conda\envs\LG\lib\site-packages\joblib\parallel.py", line 975, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "C:\Users\dumemer\.conda\envs\LG\lib\site-packages\joblib\_parallel_backends.py", line 567, in wrap_future_result
    return future.result(timeout=timeout)
  File "C:\Users\dumemer\.conda\envs\LG\lib\concurrent\futures\_base.py", line 446, in result
    return self.__get_result()
  File "C:\Users\dumemer\.conda\envs\LG\lib\concurrent\futures\_base.py", line 391, in __get_result
    raise self._exception
TypeError: No matching definition for argument type(s) readonly array(float64, 3d, C)

I'm not sure where that could be coming from. My configuration is:

• OS: Windows Server 2016/2019 both (I know)
• alphacsc v0.4.1.dev12 (installed from master)
• joblib v1.2.0

It seems to work with the l-bfgs z solver though

[MerlinDumeur]

Here are the full parameters used for instantiating GreedyCDL:

cdl = GreedyCDL(
        # Shape of the dictionary
        n_atoms=n_atoms,
        n_times_atom=n_times_atom,
        # Request a rank1 dictionary with unit norm temporal and spatial maps
        rank1=rank1,  #set to False
        uv_constraint='separate' if rank1 else 'auto',
        # apply a temporal window reparametrization
        window=True,
        # at the end, refit the activations with fixed support and no reg to unbias
        unbiased_z_hat=True,
        # Initialize the dictionary with random chunk from the data, doesn't do anything here
        D_init='chunk',
        # rescale the regularization parameter to be a percentage of lambda_max
        lmbd_max=lambda_max,  # set to 'scaled'
        reg=reg,
        # Number of iteration for the alternate minimization and cvg threshold
        n_iter=3 * int(np.ceil(100 * (n_atoms / 40))),
        eps=1e-6,
        # solver for the z-step
        # solver_z="l-bfgs",
        solver_z_kwargs={'tol': 1e-3,
                         'max_iter': 100000},
        # solver for the d-step
        solver_d='auto',
        solver_d_kwargs={'max_iter': 300},
        # sort atoms by explained variances
        sort_atoms=True,
        # Technical parameters
        verbose=1,
        random_state=0,
        n_jobs=n_jobs)

[tomMoral]

It seems that the dictionary D in your case is very large and that joblib memmaps it. This causes an issue with numba that do not seemlessly handle readonly types.

Could you try to remove (numba.float64[:, :, :],) from line 39 in alphacsc/utils/compute_constants.py and try again, to see if this solves the crash?