Adding Multi-Task ElasticNet support

[PABannier]

The goal of this PR is to add multi-task ElasticNet to the elasticnet crate.

A quick roadmap:

• Write block coordinate descent
• Write dual gap for multi-task
• Write tests for BCD
• Make CD variable names consistent with variable names in BCD
• Use a for loop when updating the residuals in CD
• Z-score + confidence level + variance
• Adapt Linfa ElasticNet API to the multi-task case
• Write tests for ElasticNet multi-task

[PABannier]

@YuhanLiin i'm implementing the Fit trait for the multi-task ENET. However, I don't know how to deal with Linfa API to handle the multi-task case. I created a MultiTaskElasticNet struct, but since both ElasticNet and MultiTaskElasticNet have the same set of parameters, I didn't create a MultiTaskElasticNetValidParams.

My question is how to restrict the trait bounds to implement the Fit trait for multi-task dataset?

[YuhanLiin]

Is it possible to make functions like coordinate_descent, duality_gap, variance_params, and compute_intercept generic across 1D and 2D arrays? That way we won't need 2 sets of similar helper functions to handle the single-task and multi-task cases.

[PABannier]

@YuhanLiin Thanks for all the remarks! I passed your comments on the latest commit.

As for making coordinate_descent generic across 1D and 2D arrays, I don't think it is a good idea. coordinate_descent and block_coordinate_descent rely on two different proximal operators and there are more for loops in block_coordinate_descent specifically designed for the multi-task case. IMHO, a wiser choice would be to let coordinate_descent and block_coordinate_descent separate and use them as backbones for different penalties in the single task or multi task case. My intuition is that in future PR we could make it such that these two functions are generic over the regularization used in a model (for now it only supports L1 + L2), but there are more complex penalties that are very useful as well (non-convex ones for instance, see SCAD or MCP).

duality_gap is also very specific in the single task or multi-task case and introducing a match syntax would make things more confusing IMHO.

For variance_params and compute_intercept I can certainly make it generic over 1D or 2D arrays.

[YuhanLiin]

Isn't this operation equivalent to r += x_j.dot(old_w_j.t())? If so you can replace these types of for loops with general_mat_mul(1, x_j, old_w_j.t(), 1, r).

[PABannier]

This operation is equivalent to np.outer in Python (see: https://numpy.org/doc/stable/reference/generated/numpy.outer.html for a more detailed explanation). I don't think there is a built-in equivalent in ndarray. It is not yet available in the ndarray crate (see: rust-ndarray/ndarray#1148) .

[YuhanLiin]

outer(x, w) is equivalent to x as a column vector multiplied with w as a row vector. The code in the ndarray PR you linked does the same thing. Using general_mat_mul allows you to add the matrix product of x and w to r in one operation. You do need to convert x and y into 2D arrays though, like in the PR.

[YuhanLiin]

Ideal way to convert 1D arrays into 2D is insert_axis. Something like x.view().insert_axis(Axis(0_or_1))

[YuhanLiin]

Making variance_params and compute_intercept generic would be great.

[YuhanLiin]

Instead of fold you can do y.iter().map(|x| x*x).sum(). Since iter is called before map it won't create a new array. For 1D arrays it's even simpler since you can just call y.dot(&y) to dot product y with itself.

[YuhanLiin]

You should also apply this change to all the other similar fold calls

[PABannier]

This AsTargets<Elem = F> is very complex to deal with. I can't do much with it, since I need to have an ArrayBase in order to call ndim(), shape() and to make the computation target - y_est. Do you know how I can circumvent this issue? I don't understand the need for an AsTarget trait in the first place. At least it should support multi-task targets.

[PABannier]

Actually it should have some way to retrieve the dimension of the targets.

[YuhanLiin]

Currently AsTargets has the method as_multi_targets, which returns a 2D array view, so you can call it to retrieve the target for both cases. For the single target case this returns an array of dimension (n, 1). This means your code needs to treat single-task and multi-task-with-only-one-task as equivalent cases. y_est will need to be a 2D array in all cases; for single-task just insert Axis(1) into y_est.

After your other PR, this should be bounded with AsMultiTargets (now that I think about it, we need AsMultiTargets as a super-trait of AsSingleTarget for this to work).

[YuhanLiin]

Use dot

[YuhanLiin]

Call iter() before map to prevent creating a new array

[PABannier]

Since #206 has been merged, ElasticNet is now easier to adapt to the multi-task case. I'm still working on it.
Roadmap before merging:

• Make variance_params generic for single task and multi-task
• Make compute_intercept generic for single task and multi-task
• Write tests for MultiTaskElasticNet (Lasso + Ridge...)

[YuhanLiin]

Work continued in #238