Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate memory usage increase with GPU CI #2803

Open
pramodk opened this issue Mar 26, 2024 · 0 comments
Open

Investigate memory usage increase with GPU CI #2803

pramodk opened this issue Mar 26, 2024 · 0 comments
Assignees
Labels
bug CI Continuous Integration

Comments

@pramodk
Copy link
Member

pramodk commented Mar 26, 2024

Overview of the CI issue

See the discussion in #2793 and my comparison mentioned in #2794 (comment):

Recently we have seen OOMs for some of the CI jobs. From the slurm reporting, it seems like some jobs of neuron are using ~30-50GB of memory.

As part of this ticket, we would like to cross-check if there has been any memory usage increase recently (~during last month).

Expected result/behavior

We are not aware of any changes that are increasing the memory of NEURON/CoreNEURON. So memory usage should be the same.

CI Logs

See links/info in #2793 and #2794 (comment).

Steps to do

  • Build today's master and some older version (e.g. from last month)
  • Build configuration is: build:neuron:nmodl:nvhpc:omp i.e. GPU build with OpenMP using NVHPC
  • And run the tests similar to test:neuron:nmodl:nvhpc:omp step
  • we need to just focus on channel-benchmarks and see memory usage (e.g. simple htop or grafana board could confirm)
@pramodk pramodk added bug CI Continuous Integration labels Mar 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug CI Continuous Integration
Projects
None yet
Development

No branches or pull requests

2 participants