Solving many problems in parallel with Dask

[rileyjmurray]

CVXPY is not thread-safe. However, our users (myself included) have had success using it with Dask. Here's a helper function that folks can try if they find themselves needing to solve independent problems in parallel.

Forewarning: I haven't tested this exact function! Please post here if you encounter problems using it.

def solve_in_parallel(cvxpy_probs, num_workers: int, threads_per_worker: int, **kwargs):
    """
    * cvxpy_probs is a list of completely unrelated CVXPY Problem objects.
    * num_workers is the number of invocations of prob.solve(...) that will be active at any given time.
    * threads_per_worker is the number of threads that an individual call to prob.solve(...)
    * Any keyword arguments provided to this function that are _not_ one of the three mentioned
      above will be packed into the dict called kwargs. We'll pass the _unpacked_ version of that dict
      to the parallel calls to the CVXPY Problem objects' solve method.
    """
    import dask
    dask.config.set(scheduler='processes', num_workers=num_workers, threads_per_worker=threads_per_worker)
    def solve_passthrough(prob):
        prob.solve(**kwargs)
    lazy_results = []
    for prob in cvxpy_probs:
        lazy_data = dask.delayed(solve_passthrough)(prob)
        lazy_results.append(lazy_data)
    results = dask.compute(*lazy_results)
    return results

The product num_workers * threads_per_worker should never exceed the number of physical cores on your machine.

Ping @SteveDiamond. Someone asked about this on Discord but I can't access Discord at the moment. Can you link to this Discussion post on that thread?

Separately, we should have our web documentation point here in some kind of FAQ.