aggregation operations not working with many groups

[abhi211199]

Hi, I'm new to modin.pandas, I'm trying to perform several aggregation operations on groups which were splitted from a dataframe using df.groupby(). The aggregation util works fine when I have less number of groups but fails to perform when it has more than 1000 groups.
To give an idea of the code using an example

import modin.pandas as pd
array1 = []
array2 = []
for i in range(1,2000):
     for j in range(1,10):
           array1.append(i)
           array2.append(j)

df=pd.DataFrame({'A':array1, 'B': array2})
groups = df.groupby('A')

def aggr(df):
      df['C']=df['B'].sum()
      return df

filter=[]
for A,B in groups:
     filter.append(aggr(B))

When executed, this goes on and on

but using simply pandas doesn't cause this issue.

Please let me know if I'm missing something that needs to be used while handling large number of groups. Thanks

[anmyachev]

Hi @abhi211199, thanks for the question!

What version of Modin are you using?

Spilling occurs when there is not enough memory to store all objects in distributed storage. Therefore, there may be two options: you simply do not have enough RAM to process your amount of data, or Modin in such a case uses much more memory than is required. Please provide us with this information so that we can better understand the problem.

Please note that data clearing after spilling in the temporary folder does not occur automatically. This should be done manually, so there is a chance that the temporary folder was filled with data from previous runs, which prevented your last run from running successfully.