Memory problem

[sujan1746]

Hi, I have used MSNoise very well.
But when I try many stations (over 40 stations), I can see the memory problems and stopped the code (crashed).
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 65.3 MiB for an array with shape (108, 79201) and data type float64
The error occurs even though there are enough memory.
It shows errors at the part of fftn, ifft, fftfreq, data.copy() in s03compute_no_rotation.py.

Even though I changed sf as np(numpy) for faster computation (almost 3times), it does not show the error but too much memories.
Usually the memory per core is counted as 0.6-0.8%, but from the middle of cc computation, it goes up until total memory of MSNoise occupy almost memory (90~98% of total memory) of the server and gets even swap memories and the memory does not reduced again, so it gets too much memories in overall. (I think fft part gets many memories)

Overall error related to fft

Process Process-19:
Traceback (most recent call last):
  File "/home/sujan/miniconda3/envs/msnoise/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/sujan/miniconda3/envs/msnoise/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/sujan/miniconda3/envs/msnoise/lib/python3.7/site-packages/msnoise/s03compute_no_rotation.py", line 551, in main
    ffts = np.fft.fftn(_data, [nfft, ], axes=[1, ])
  File "<__array_function__ internals>", line 6, in fftn
  File "/home/sujan/miniconda3/envs/msnoise/lib/python3.7/site-packages/numpy/fft/_pocketfft.py", line 815, in fftn
    return _raw_fftnd(a, s, axes, fft, norm)
  File "/home/sujan/miniconda3/envs/msnoise/lib/python3.7/site-packages/numpy/fft/_pocketfft.py", line 707, in _raw_fftnd
    a = function(a, n=s[ii], axis=axes[ii], norm=norm)
  File "<__array_function__ internals>", line 6, in fft
  File "/home/sujan/miniconda3/envs/msnoise/lib/python3.7/site-packages/numpy/fft/_pocketfft.py", line 215, in fft
    output = _raw_fft(a, n, axis, False, True, inv_norm)
  File "/home/sujan/miniconda3/envs/msnoise/lib/python3.7/site-packages/numpy/fft/_pocketfft.py", line 70, in _raw_fft
    r = pfi.execute(a, is_real, is_forward, fct)
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 168. MiB for an array with shape (139, 79233) and data type complex128

error related to data.copy()

Process Process-18:
Traceback (most recent call last):
  File "/home/sujan/miniconda3/envs/msnoise/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/sujan/miniconda3/envs/msnoise/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/sujan/miniconda3/envs/msnoise/lib/python3.7/site-packages/msnoise/s03compute_no_rotation.py", line 461, in main
    _data = data.copy()
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 81.0 MiB for an array with shape (134, 79201) and data type float64

Before the error, the memory states go up suddenly as below. (In the below memory case, it is the result of modified the code as numpy.fft.ifft. But the scipy.ifft also shows the memoryerror and gets many memories too..)

[ThomasLecocq]

first questions:

• how much RAM do you have on your machine?
• What cc_sampling_rate value did you choose?
• Are you trying to do threading (-t option) ?
• Does this problem occur after a while, or immediately?

In recent scipy versions, the numpy's FFT is called under the hood, so there should be no added memory issues. There could be a ghost from the past when scipy/numpy FFT did some caching of the memory blocks and never released them, but I don't think it's that...

[sujan1746]

• how much RAM do you have on your machine? 527671648 kB. But when I run the code with cc, it becomes 2300kB left after 2~30min and gets swap memories.
• What cc_sampling_rate value did you choose? 44Hz (Due to see high frequency 2~10Hz, preprocess_lowpass=20Hz. I thought I need to set like this for nyquist frequency)
• Are you trying to do threading (-t option) ? yes.
• Does this problem occur after a while, or immediately? After a while. About 10min after running cc.

After cc, stack shows load average problem. we have 128 cores but the average becomes over 1500 when using 30 cores. With one core, it shows 100 load average.

Actually, I remember until a few weeks ago, it does not give the problems not that much or I did not catch the problem at that time. (At that time, I used 1st - 23stations, 20Hz cc sampling rate for low frequency. 2nd - 50 stations, 44Hz cc sampling rate)

p.s. Then I will try to upgrade scipy.

[ThomasLecocq]

I don't really understand your metrics: you have 527GB RAM & 128 core machine ? What -t number are you using?

note "load average" you are talking about is CPU ? Then 100% doesn't mean anything (google will tell you why)

44Hz CC is non standard, I guess you also have to use the "lanczos resampler" to arrive there, that could lead to the preprocessing routine accumulating a bunch of temp arrays in memory. The latest development version on github solves this by executing the preprocessing in another thread which gets properly garbage-collected upon closure. If you choose to go with the dev version, WARNING, it is NOT compatible with previous databases/data structures !

[sujan1746]

In my lab, there is a server(machine) which has 527GB and 128cores. I used 30 for t number when the problem was not shown, and after the problem, I used 5~10 cores.
I thought 1 core = 1.0 load average (for 100% working), so 1500 load average for 30 cores means overload.
Could you please explain more about meaning of '44Hz CC is non standard'?
And yes, I used lanczos resampler. Then you mean, the memory usage can be due to the resampling method and the new demo version solved the problem but it is not compatible for now on?

[ThomasLecocq]

If your input data is e.g. 100 Hz, to go to 44 Hz, you are forced to use the Lanczos resampler, which is heavy computationally. You could set the CC_sampling_rate to 50 Hz, and just Decimate.

I have no idea what "load average" means in your case.

The development version (called "master" on github) is fully functional, BUT is not compatible with your existing database, because there are new elements (support for location codes, channel names) etc.

Are you using many filters? Many components (other than ZZ) ? Cross-station AND single-station ? etc... all those will add elements in memory during computation and output to disk after all slides are finished. So yes, the memory imprint can grow large for each thread. I don't know if it could grow larger than 17GB per thread, but with the preprocessing leaving some memory block behind, it could. with ZZ only you are already storing data for 780 pairs...

[sujan1746]

Ahh.. I see. I did not think about that can be a problem. I will set config using decimate.

Yeap. I see. Then I will try the demo version for other new data when using Lanczos resampler.

I just used one filter but with 9 components for cross station and single station, so I thought it would take much memories (but I could not expect over 17GB..).

I really appreciate your help and solving the problem!

[sujan1746]

And I found the reason for the load average. When I run msnoise -t 10 stack -m, it gets all cpu cores in the server. So when there are 128 cores, 10 multiprocessing gets 128 cores. I heard numpy has this problem. I will find more reason too.