Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance issues with SCS 2.1.4 vs 2.1.2 #170

Open
b-grimaud opened this issue Sep 17, 2021 · 4 comments
Open

Performance issues with SCS 2.1.4 vs 2.1.2 #170

b-grimaud opened this issue Sep 17, 2021 · 4 comments

Comments

@b-grimaud
Copy link

b-grimaud commented Sep 17, 2021 •
edited

Specifications

  • OS: Zorin OS 16 (Ubuntu 20.04.3 LTS)
  • SCS Version: 2.1.4
  • Compiler:

Description

Using SCS 2.1.4 with cvxpy is significantly slower than with 2.1.2, and sometimes even fails.

Additional information

My particular use case requires pretty intense calculations with cvxpy. ECOS was selected by default but I had to switch to SCS because it couldn't handle some of the larger tasks.

While switching from Windows to an Ubuntu-based OS, I ran a reproducibility test to check if everything worked as expected. However, the part of my script that calls cvxpy was significantly slower, and failed at some point. I investigated further, and I noticed than when reinstalling cvxpy through conda, it automatically installed the latest version of SCS, while my previous environment used SCS 2.1.2. Switching back to that version solved the issue.

As this issue is probably specific to the problem and data I'm working with, I am just wondering if there any release notes or summary of what changed between those two versions ? I have tried looking at PRs and issues, but the only thing I could find was comparing files between the two branches, and trying to understand that kind of code is well above my abilities.

Edit : I looked a bit further into CPU usage, running a specific task with SCS 2.1.2 makes the CPU spike at about 55%, while running the same task with SCS 2.1.4 makes the CPU max out at 99 to 100%, constantly. I am using an Intel i7-10810U.
@bodono
Copy link
Member

bodono commented Sep 19, 2021

That's very strange because the main changes between those two tags are to the GPU solver (which I presume you're not using here) and adding the cmake install option (which doesn't affect python). No core part of SCS was changed. Can you run cvxpy with verbose=True in both cases and put the output here so I can take a look?

Here is the list of files changed between those two tags:

=> git diff v2.1.2 v2.1.4 --stat
 .appveyor.yml                        |  31 ----
 .bumpversion.cfg                     |   3 +-
 .github/ISSUE_TEMPLATE.md            |  28 +++
 .github/workflows/build.yml          |  21 +++
 .travis.yml                          |  36 ----
 CMakeLists.txt                       | 273 ++++++++++++++++++++++++++++
 README.md                            |  66 ++++++-
 cmake/AddInstallRPATHSupport.cmake   | 237 +++++++++++++++++++++++++
 cmake/AddUninstallTarget.cmake       |  70 ++++++++
 cmake/InstallBasicPackageFiles.cmake | 677 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 docs/scs_logo.png                    | Bin 0 -> 153979 bytes
 include/glbopts.h                    |   2 +-
 linsys/gpu/gpu.c                     |  61 +++++--
 linsys/gpu/gpu.h                     |  31 +++-
 linsys/gpu/indirect/private.c        | 140 +++++++++++++--
 linsys/gpu/indirect/private.h        |   5 +
 src/rw.c                             |   2 +-
 17 files changed, 1572 insertions(+), 111 deletions(-)

@b-grimaud
Copy link
Author

Here's the output with 2.1.2 :

    SCS v2.1.2 - Splitting Conic Solver
    (c) Brendan O'Donoghue, Stanford University, 2012

Lin-sys: sparse-direct, nnz in A = 35307
eps = 1.00e-04, alpha = 1.50, max_iters = 5000, normalize = 1, scale = 1.00
acceleration_lookback = 10, rho_x = 1.00e-03
Variables n = 7850, constraints m = 11775
Cones: linear vars: 7845
soc vars: 3930, soc blks: 2
Setup time: 6.94e-03s

Iter | pri res | dua res | rel gap | pri obj | dua obj | kap/tau | time (s)

 0| 5.82e+01  5.61e+01  1.00e+00 -5.49e+06  1.25e+04  6.44e-10  2.22e-03

100| 7.55e-05 1.00e-04 4.85e-01 -9.35e-01 7.26e-03 2.09e-13 1.31e-01
200| 3.68e-05 4.52e-05 3.31e-01 -4.90e-01 4.61e-03 8.64e-12 2.49e-01
300| 2.62e-05 3.51e-05 2.42e-01 -3.14e-01 4.68e-03 1.40e-12 3.65e-01
400| 1.91e-05 2.82e-05 1.68e-01 -1.98e-01 4.86e-03 6.15e-12 4.81e-01
500| 1.51e-05 2.08e-05 1.24e-01 -1.37e-01 4.18e-03 6.71e-12 6.04e-01
600| 1.15e-05 1.38e-05 9.17e-02 -9.64e-02 4.53e-03 5.02e-12 7.22e-01
700| 8.79e-06 1.19e-05 6.39e-02 -6.39e-02 4.35e-03 2.32e-11 8.40e-01
800| 7.83e-06 1.13e-05 4.82e-02 -4.62e-02 4.42e-03 1.04e-11 9.58e-01
900| 7.10e-06 8.95e-06 3.98e-02 -3.69e-02 4.46e-03 1.52e-11 1.07e+00
1000| 5.55e-06 7.55e-06 3.08e-02 -2.70e-02 4.78e-03 8.99e-12 1.20e+00
1100| 5.82e-06 8.30e-06 2.57e-02 -2.14e-02 4.89e-03 8.28e-12 1.32e+00
1200| 5.55e-06 7.52e-06 2.31e-02 -1.88e-02 4.87e-03 1.39e-11 1.45e+00
1300| 5.13e-06 7.20e-06 1.84e-02 -1.40e-02 4.82e-03 2.11e-11 1.59e+00
1400| 6.54e-06 8.76e-06 1.87e-02 -1.46e-02 4.48e-03 4.36e-13 1.72e+00
1500| 5.61e-06 7.04e-06 1.59e-02 -1.14e-02 4.76e-03 7.82e-13 1.85e+00
1600| 5.46e-06 7.92e-06 1.35e-02 -8.71e-03 4.95e-03 2.56e-11 1.98e+00
1700| 5.02e-06 6.54e-06 1.17e-02 -6.95e-03 4.86e-03 6.84e-12 2.11e+00
1800| 4.79e-06 7.19e-06 9.47e-03 -4.94e-03 4.63e-03 4.17e-12 2.24e+00
1900| 4.78e-06 6.11e-06 9.11e-03 -4.48e-03 4.72e-03 7.62e-13 2.37e+00
2000| 5.05e-06 7.01e-06 8.26e-03 -3.55e-03 4.77e-03 6.43e-12 2.51e+00
2100| 4.54e-06 5.38e-06 6.99e-03 -2.43e-03 4.61e-03 6.58e-12 2.63e+00
2200| 4.39e-06 6.51e-06 6.15e-03 -1.64e-03 4.54e-03 3.90e-12 2.75e+00
2300| 4.28e-06 5.66e-06 6.35e-03 -1.64e-03 4.75e-03 1.72e-11 2.88e+00
2400| 4.49e-06 6.62e-06 5.68e-03 -1.03e-03 4.68e-03 1.10e-11 3.00e+00
2500| 4.10e-06 5.51e-06 4.89e-03 -4.59e-04 4.45e-03 2.54e-12 3.12e+00
2600| 4.68e-06 6.14e-06 4.79e-03 -3.76e-04 4.44e-03 7.47e-12 3.24e+00
2700| 4.38e-06 6.21e-06 4.57e-03 -2.91e-04 4.30e-03 1.87e-12 3.36e+00
2800| 4.18e-06 5.56e-06 4.25e-03 1.47e-04 4.41e-03 4.41e-12 3.48e+00
2900| 4.50e-06 6.25e-06 3.92e-03 1.77e-04 4.12e-03 1.26e-13 3.62e+00
3000| 3.91e-06 5.73e-06 4.04e-03 2.91e-04 4.35e-03 2.53e-11 3.76e+00
3100| 4.66e-06 5.49e-06 3.59e-03 5.91e-04 4.20e-03 1.14e-11 3.88e+00
3200| 4.00e-06 5.99e-06 3.49e-03 5.81e-04 4.08e-03 9.86e-12 4.01e+00
3300| 3.84e-06 5.32e-06 3.51e-03 6.45e-04 4.17e-03 4.57e-12 4.14e+00
3400| 3.59e-06 4.74e-06 3.17e-03 9.05e-04 4.10e-03 8.48e-12 4.27e+00
3500| 4.05e-06 5.32e-06 3.04e-03 8.69e-04 3.92e-03 2.13e-12 4.39e+00
3600| 3.52e-06 5.08e-06 3.21e-03 9.10e-04 4.13e-03 5.40e-12 4.52e+00
3700| 3.69e-06 4.78e-06 2.84e-03 9.72e-04 3.82e-03 2.21e-11 4.65e+00
3800| 3.75e-06 4.52e-06 2.89e-03 8.68e-04 3.77e-03 1.32e-11 4.77e+00
3900| 3.70e-06 5.31e-06 2.93e-03 1.01e-03 3.95e-03 7.67e-12 4.90e+00
4000| 3.71e-06 5.38e-06 2.51e-03 1.08e-03 3.60e-03 7.31e-13 5.03e+00
4100| 3.94e-06 5.34e-06 2.90e-03 9.40e-04 3.85e-03 9.56e-12 5.15e+00
4200| 4.05e-06 4.75e-06 2.55e-03 1.08e-03 3.63e-03 2.51e-11 5.28e+00
4300| 3.82e-06 5.09e-06 2.42e-03 1.08e-03 3.51e-03 5.68e-12 5.40e+00
4400| 3.60e-06 5.27e-06 2.75e-03 1.01e-03 3.77e-03 8.18e-12 5.53e+00
4500| 3.29e-06 5.06e-06 2.36e-03 1.18e-03 3.55e-03 2.45e-12 5.65e+00
4600| 3.31e-06 4.49e-06 2.64e-03 9.60e-04 3.61e-03 1.18e-11 5.78e+00
4700| 3.63e-06 4.29e-06 2.46e-03 1.11e-03 3.58e-03 8.18e-12 5.90e+00
4800| 3.49e-06 4.57e-06 2.34e-03 1.08e-03 3.43e-03 1.83e-11 6.03e+00
4900| 3.40e-06 5.04e-06 2.48e-03 9.85e-04 3.47e-03 1.62e-11 6.15e+00
5000| 3.59e-06 4.71e-06 2.61e-03 8.78e-04 3.50e-03 7.99e-12 6.28e+00

Status: Solved/Inaccurate
Hit max_iters, solution may be inaccurate, returning best found solution.
Timing: Solve time: 6.28e+00s
Lin-sys: nnz in L factor: 62774, avg solve time: 2.17e-04s
Cones: avg projection time: 1.13e-05s
Acceleration: avg step time: 7.30e-04s

Error metrics:
dist(s, K) = 4.5765e-15, dist(y, K*) = 4.3368e-19, s'y/|s||y| = 2.0987e-18
primal res: |Ax + s - b|_2 / (1 + |b|_2) = 3.3716e-06
dual res: |A'y + c|_2 / (1 + |c|_2) = 5.0109e-06
rel gap: |c'x + b'y| / (1 + |c'x| + |b'y|) = 2.3090e-03

c'x = 0.0011, -b'y = 0.0034

And here it is with 2.1.4 :

SCS v2.1.4 - Splitting Conic Solver
(c) Brendan O'Donoghue, Stanford University, 2012

Lin-sys: sparse-direct, nnz in A = 35307
eps = 1.00e-04, alpha = 1.50, max_iters = 5000, normalize = 1, scale = 1.00
acceleration_lookback = 10, rho_x = 1.00e-03
Variables n = 7850, constraints m = 11775
Cones: linear vars: 7845
soc vars: 3930, soc blks: 2
Setup time: 7.55e-03s

Iter | pri res | dua res | rel gap | pri obj | dua obj | kap/tau | time (s)

 0| 5.82e+01  5.61e+01  1.00e+00 -5.49e+06  1.25e+04  6.44e-10  1.39e-03

100| 7.55e-05 1.00e-04 4.85e-01 -9.35e-01 7.26e-03 1.79e-11 2.33e-01
200| 3.68e-05 4.52e-05 3.31e-01 -4.90e-01 4.61e-03 8.67e-12 4.87e-01
300| 2.62e-05 3.51e-05 2.42e-01 -3.14e-01 4.68e-03 5.37e-14 7.78e-01
400| 1.91e-05 2.82e-05 1.68e-01 -1.98e-01 4.86e-03 1.29e-11 1.25e+00
500| 1.51e-05 2.08e-05 1.24e-01 -1.37e-01 4.18e-03 3.42e-12 1.71e+00
600| 1.15e-05 1.38e-05 9.17e-02 -9.64e-02 4.53e-03 1.69e-12 2.30e+00
700| 8.79e-06 1.19e-05 6.39e-02 -6.39e-02 4.35e-03 2.18e-11 2.70e+00
800| 7.83e-06 1.13e-05 4.82e-02 -4.62e-02 4.42e-03 1.04e-11 3.09e+00
900| 7.10e-06 8.95e-06 3.98e-02 -3.69e-02 4.46e-03 8.13e-12 3.46e+00
1000| 5.55e-06 7.55e-06 3.08e-02 -2.70e-02 4.78e-03 8.54e-12 3.80e+00
1100| 5.82e-06 8.30e-06 2.57e-02 -2.14e-02 4.89e-03 1.21e-11 4.17e+00
1200| 5.55e-06 7.52e-06 2.31e-02 -1.88e-02 4.87e-03 2.00e-11 4.52e+00
1300| 5.13e-06 7.20e-06 1.84e-02 -1.40e-02 4.82e-03 1.08e-11 4.91e+00
1400| 6.54e-06 8.76e-06 1.87e-02 -1.46e-02 4.48e-03 3.15e-12 5.35e+00
1500| 5.61e-06 7.04e-06 1.59e-02 -1.14e-02 4.76e-03 1.65e-12 5.77e+00
1600| 5.46e-06 7.92e-06 1.35e-02 -8.71e-03 4.95e-03 1.48e-11 6.23e+00
1700| 5.02e-06 6.54e-06 1.17e-02 -6.95e-03 4.86e-03 7.91e-12 6.55e+00
1800| 4.79e-06 7.19e-06 9.47e-03 -4.94e-03 4.63e-03 4.54e-12 7.00e+00
1900| 4.78e-06 6.11e-06 9.11e-03 -4.47e-03 4.72e-03 1.42e-13 7.38e+00
2000| 5.05e-06 7.01e-06 8.26e-03 -3.55e-03 4.77e-03 5.52e-13 7.71e+00
2100| 4.54e-06 5.38e-06 6.99e-03 -2.43e-03 4.61e-03 2.29e-11 8.11e+00
2200| 4.39e-06 6.51e-06 6.15e-03 -1.64e-03 4.54e-03 5.15e-12 8.48e+00
2300| 4.28e-06 5.66e-06 6.35e-03 -1.64e-03 4.75e-03 9.89e-12 8.87e+00
2400| 4.49e-06 6.62e-06 5.68e-03 -1.03e-03 4.68e-03 1.07e-11 9.23e+00
2500| 4.10e-06 5.51e-06 4.89e-03 -4.59e-04 4.45e-03 2.20e-11 9.62e+00
2600| 4.68e-06 6.14e-06 4.79e-03 -3.76e-04 4.44e-03 1.37e-11 9.97e+00
2700| 4.38e-06 6.21e-06 4.57e-03 -2.91e-04 4.30e-03 2.53e-11 1.03e+01
2800| 4.18e-06 5.56e-06 4.25e-03 1.47e-04 4.41e-03 1.33e-11 1.08e+01
2900| 4.50e-06 6.25e-06 3.92e-03 1.77e-04 4.12e-03 9.64e-12 1.13e+01
3000| 3.91e-06 5.73e-06 4.04e-03 2.91e-04 4.35e-03 1.50e-11 1.17e+01
3100| 4.66e-06 5.49e-06 3.59e-03 5.92e-04 4.20e-03 1.31e-11 1.21e+01
3200| 4.00e-06 5.99e-06 3.49e-03 5.81e-04 4.08e-03 2.64e-12 1.24e+01
3300| 3.84e-06 5.32e-06 3.51e-03 6.45e-04 4.17e-03 8.48e-12 1.28e+01
3400| 3.59e-06 4.74e-06 3.17e-03 9.05e-04 4.10e-03 1.13e-11 1.31e+01
3500| 4.05e-06 5.32e-06 3.04e-03 8.69e-04 3.92e-03 1.32e-11 1.35e+01
3600| 3.52e-06 5.08e-06 3.21e-03 9.10e-04 4.13e-03 1.89e-11 1.39e+01
3700| 3.69e-06 4.78e-06 2.84e-03 9.72e-04 3.82e-03 1.11e-11 1.42e+01
3800| 3.75e-06 4.52e-06 2.89e-03 8.68e-04 3.77e-03 1.25e-11 1.45e+01
3900| 3.70e-06 5.31e-06 2.93e-03 1.01e-03 3.95e-03 4.26e-12 1.48e+01
4000| 3.71e-06 5.38e-06 2.51e-03 1.08e-03 3.60e-03 8.84e-12 1.53e+01
4100| 3.94e-06 5.34e-06 2.90e-03 9.40e-04 3.85e-03 1.95e-11 1.56e+01
4200| 4.05e-06 4.75e-06 2.55e-03 1.08e-03 3.63e-03 2.21e-12 1.60e+01
4300| 3.82e-06 5.09e-06 2.42e-03 1.08e-03 3.51e-03 2.35e-11 1.63e+01
4400| 3.60e-06 5.27e-06 2.75e-03 1.01e-03 3.77e-03 1.30e-11 1.66e+01
4500| 3.29e-06 5.06e-06 2.36e-03 1.18e-03 3.55e-03 9.70e-12 1.70e+01
4600| 3.31e-06 4.49e-06 2.64e-03 9.59e-04 3.61e-03 4.11e-12 1.74e+01
4700| 3.63e-06 4.29e-06 2.46e-03 1.11e-03 3.58e-03 1.55e-11 1.77e+01
4800| 3.49e-06 4.57e-06 2.34e-03 1.07e-03 3.43e-03 6.16e-12 1.83e+01
4900| 3.40e-06 5.04e-06 2.48e-03 9.85e-04 3.47e-03 1.04e-11 1.88e+01
5000| 3.59e-06 4.71e-06 2.61e-03 8.78e-04 3.50e-03 4.22e-12 1.92e+01

Status: Solved/Inaccurate
Hit max_iters, solution may be inaccurate, returning best found solution.
Timing: Solve time: 1.92e+01s
Lin-sys: nnz in L factor: 62774, avg solve time: 3.91e-04s
Cones: avg projection time: 2.51e-05s
Acceleration: avg step time: 2.88e-03s

Error metrics:
dist(s, K) = 7.1054e-15, dist(y, K*) = 2.1684e-19, s'y/|s||y| = -3.4633e-17
primal res: |Ax + s - b|_2 / (1 + |b|_2) = 3.3716e-06
dual res: |A'y + c|_2 / (1 + |c|_2) = 5.0109e-06
rel gap: |c'x + b'y| / (1 + |c'x| + |b'y|) = 2.3093e-03
c'x = 0.0011, -b'y = 0.0034

@bodono
Copy link
Member

bodono commented Sep 20, 2021 •
edited

To my eye those outputs look identical (except for timing etc). Could it be something changed inside cvxpy?

@b-grimaud
Copy link
Author

b-grimaud commented Sep 20, 2021 •
edited

I investigated a bit further, and it might indeed be because of cvxpy. I thought I had 1.1.7 installed in both of my environment, but in fact what is listed in my conda environment isn't what is actually used.
It seems I cannot use scs 2.1.2 with both cvxpy 1.1.7 and 1.1.15, or at least I cannot install 2.1.4 on 1.1.7.
My 2.1.4 environment probably updated cvxpy as well without me noticing.

By the way, I noticed that installing scs through pip didn't work, as in the installation is marked as completed but running a script using SCS freezes because "SCS is not installed".
I ran into the same problem when installing cvxpy with pip, where the install fails on installing SCS.
Using conda works otherwise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bodono @b-grimaud