FEAT-#4605: Implementation of Small Query Compiler to support small and empty DataFrames #5113

billiam-wang · 2022-10-10T17:16:00Z

What do these changes do?

commit message follows format outlined here
passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
signed commit with git commit -s
Resolves Handle Empty/Small Data DataFrames as a separate case #4605
tests added and passing
module layout described at docs/development/architecture.rst is up-to-date

lgtm-com · 2022-10-31T09:07:15Z

This pull request introduces 9 alerts when merging fe123de71f23dfe852bbfa6bb74ae154f3c53f94 into f492ba9 - view on LGTM.com

new alerts:

7 for First parameter of a method is not named 'self'
1 for Unused import
1 for Variable defined multiple times

lgtm-com · 2022-10-31T17:30:31Z

This pull request introduces 7 alerts when merging e0f36929ab8b4a6b5e7755714eaa72c61d891cf6 into 6f0ff79 - view on LGTM.com

new alerts:

7 for First parameter of a method is not named 'self'

lgtm-com · 2022-10-31T18:47:16Z

This pull request introduces 6 alerts when merging a3bac2c06ba1e6ce1d467f0281daf7ac870ada17 into 6f0ff79 - view on LGTM.com

new alerts:

6 for First parameter of a method is not named 'self'

devin-petersohn

Looks pretty good! Left a couple of high level comments.

modin/core/storage_formats/base/query_compiler.py

modin/core/storage_formats/pandas/query_compiler.py

Garra1980 · 2022-11-01T15:48:16Z

Do these changes need to be reflected in docs?

Signed-off-by: Bill Wang <billiam@ponder.io>

…yCompiler Signed-off-by: Bill Wang <billiam@ponder.io>

Signed-off-by: Bill Wang <billiam@ponder.io>

vnlitvinov · 2022-11-15T07:43:14Z

.github/workflows/ci.yml

+          test-small-query-compiler:
+            - 'modin/experimental/core/storage_formats/pandas/small_query_compiler.py'
+            - 'modin/core/storage_formats/pandas/query_compiler.py'
+            - 'modin/core/storage_formats/base/query_compiler.py'


this looks clever, but I think it feels temporary to me - query compilers both use stuff from lower level and are used by higher level, so this integration should eventually be tested upon change in any of the things used throughout the pipeline... this should probably stay for now, but be removed towards the shiny future where the "auto-switch" would happen

vnlitvinov · 2022-11-21T17:22:51Z

modin/experimental/core/storage_formats/pandas/small_query_compiler.py

+queries for small data and empty ``PandasDataFrame``.
+"""
+
+from modin.config.envvars import InitializeWithSmallQueryCompilers


the current design for configuration is that all config variables come from modin.config regardless of what value source they have (even though we have one value source so far)

vnlitvinov · 2022-11-21T17:24:17Z

modin/experimental/core/storage_formats/pandas/small_query_compiler.py

+from modin.utils import MODIN_UNNAMED_SERIES_LABEL
+from modin.utils import (
+    _inherit_docstrings,
+    try_cast_to_pandas,
+)


there's no need to import from single module in multiple statements

vnlitvinov · 2022-11-21T17:24:40Z

modin/experimental/core/storage_formats/pandas/small_query_compiler.py

+)
+
+
+def _get_axis(axis):


I'm pretty sure we have this code somewhere already, we should reduce the copy-pasteness of this

vnlitvinov · 2022-11-21T17:35:14Z

modin/experimental/core/storage_formats/pandas/small_query_compiler.py

+                    by_names.append(by[i].name)
+                elif isinstance(by[i], str):
+                    by_names.append(by[i])
+        if isinstance(by, pandas.DataFrame):


how can it be if by is squeezed at line 268?

vnlitvinov · 2022-11-21T18:18:20Z

modin/pandas/dataframe.py

+        if query_compiler is None and InitializeWithSmallQueryCompilers.get():
+            small_dataframe = pandas.DataFrame(
+                data=data, index=index, columns=columns, dtype=dtype, copy=copy
+            )
+            self._query_compiler = SmallQueryCompiler(small_dataframe)


this feels weird in terms of architecture

vnlitvinov · 2022-11-21T18:19:10Z

modin/pandas/series.py

        self._query_compiler = query_compiler.columnarize()
        if name is not None:
-            self._query_compiler = self._query_compiler


this removal feels completely wrong

also this is leaking architecture IMHO

vnlitvinov · 2022-11-21T18:19:25Z

modin/pandas/series.py

-    def _validate_dtypes_sum_prod_mean(self, axis, numeric_only, ignore_axis=False):
+    def _validate_dtypes_sum_prod_mean(
+        self, axis, numeric_only, ignore_axis=False
+    ):  # noqa: PR01
        """
        Validate data dtype for `sum`, `prod` and `mean` methods.

-        Parameters
+        Parameter


why change?

vnlitvinov · 2022-11-21T18:20:21Z

modin/pandas/test/dataframe/test_indexing.py

+    InitializeWithSmallQueryCompilers.get(),
+    reason="SmallQueryCompiler does not currently support IO functions.",


doesn't this "not supporting" make current PR a draft until it really does support I/O?

vnlitvinov · 2022-11-21T18:22:41Z

modin/pandas/utils.py

+    if InitializeWithSmallQueryCompilers.get():
+        return DataFrame(query_compiler=SmallQueryCompiler(df))


this feels completely off; I thought small query compiler should go the usual route for other QCs (in this case it should get picked by FactoryDispatcher according to user's settings)

YarShev · 2023-12-05T11:49:40Z

@billiam-wang, do you have a plan to continue this?

billiam-wang force-pushed the bill-sqc branch from 8c0f7f8 to fe123de Compare October 31, 2022 08:34

billiam-wang changed the title ~~Implementation of Small Query Compiler to support small and empty DataFrames~~ FEAT-#4605: Implementation of Small Query Compiler to support small and empty DataFrames Oct 31, 2022

billiam-wang marked this pull request as ready for review October 31, 2022 18:17

billiam-wang requested a review from a team as a code owner October 31, 2022 18:17

devin-petersohn reviewed Nov 1, 2022

View reviewed changes

modin/core/storage_formats/base/query_compiler.py Outdated Show resolved Hide resolved

modin/core/storage_formats/pandas/query_compiler.py Outdated Show resolved Hide resolved

billiam-wang force-pushed the bill-sqc branch 4 times, most recently from 912654e to 140d0d8 Compare November 2, 2022 22:16

billiam-wang added 17 commits November 7, 2022 11:48

Partial implementation of small query compiler

5e552ff

Working binary, datetime, and string functions for small query compiler

1c87589

Added resample and rolling implementations

05a0ee1

Added a few new functions

5722d82

Added most mapped functions

9c107aa

Passing most tests in test_general.py

a5b951c

Added changes to fix bugs, still failing tests

016de6c

Passing additional tests for series and groupby

8781a54

Signed-off-by: Bill Wang <billiam@ponder.io>

Passing most groupby tests

b6ea7ee

Signed-off-by: Bill Wang <billiam@ponder.io>

Passing almost all tests except in join sort and groupby

170f919

Signed-off-by: Bill Wang <billiam@ponder.io>

Passing all groupby tests except test_dict_agg_rename_mi_columns

3b2a02c

Signed-off-by: Bill Wang <billiam@ponder.io>

Passed all except 1 tests

d162386

Signed-off-by: Bill Wang <billiam@ponder.io>

Passing all tests and added config environment variable for SmallQuer…

ec02113

…yCompiler Signed-off-by: Bill Wang <billiam@ponder.io>

Added environment variable to control SmallQueryCompiler

85491ea

Signed-off-by: Bill Wang <billiam@ponder.io>

Removed print statements and cleaned up

075a93e

More code cleanup

f018818

Signed-off-by: Bill Wang <billiam@ponder.io>

Reset default InitializeWithSmallQueryCompilers envvar to False

2f3cf0e

Signed-off-by: Bill Wang <billiam@ponder.io>

billiam-wang added 11 commits November 7, 2022 11:49

Resolved flake and black failures

bc122f6

Signed-off-by: Bill Wang <billiam@ponder.io>

Removed print statements and fixed import statements

46e517f

Signed-off-by: Bill Wang <billiam@ponder.io>

Fixed more import statements

a080898

Signed-off-by: Bill Wang <billiam@ponder.io>

Cleaned up helper functions and fixed test_binary issues

8e5f80a

Signed-off-by: Bill Wang <billiam@ponder.io>

Fixed flake8 and black again

1be9793

Signed-off-by: Bill Wang <billiam@ponder.io>

Added docstrings for SmallQueryCompiler

a2c5a6f

Signed-off-by: Bill Wang <billiam@ponder.io>

Made flake8 and black changes

135148d

Signed-off-by: Bill Wang <billiam@ponder.io>

Passing all lint checks

c0df87a

Signed-off-by: Bill Wang <billiam@ponder.io>

Added documentation for try_cast_to_pandas

dccf80c

Signed-off-by: Bill Wang <billiam@ponder.io>

Added SmallQueryCompiler CI tests and fixed remaining bugs

40446d3

Signed-off-by: Bill Wang <billiam@ponder.io>

Changed CI test version to 3.9

43ff32b

Signed-off-by: Bill Wang <billiam@ponder.io>

billiam-wang force-pushed the bill-sqc branch from fa06e5d to 43ff32b Compare November 7, 2022 19:49

vnlitvinov reviewed Nov 21, 2022

View reviewed changes

arunjose696 mentioned this pull request Apr 29, 2024

FEAT-#4605: Handle small and empty dataframes #4606

Closed

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FEAT-#4605: Implementation of Small Query Compiler to support small and empty DataFrames #5113

FEAT-#4605: Implementation of Small Query Compiler to support small and empty DataFrames #5113

billiam-wang commented Oct 10, 2022 •

edited by dchigarev

lgtm-com bot commented Oct 31, 2022

lgtm-com bot commented Oct 31, 2022

lgtm-com bot commented Oct 31, 2022

devin-petersohn left a comment

Garra1980 commented Nov 1, 2022

vnlitvinov Nov 15, 2022

vnlitvinov Nov 21, 2022

vnlitvinov Nov 21, 2022

vnlitvinov Nov 21, 2022

vnlitvinov Nov 21, 2022

vnlitvinov Nov 21, 2022

vnlitvinov Nov 21, 2022

vnlitvinov Nov 21, 2022

vnlitvinov Nov 21, 2022

vnlitvinov Nov 21, 2022

YarShev commented Dec 5, 2023

		InitializeWithSmallQueryCompilers.get(),
		reason="SmallQueryCompiler does not currently support IO functions.",

		if InitializeWithSmallQueryCompilers.get():
		return DataFrame(query_compiler=SmallQueryCompiler(df))

FEAT-#4605: Implementation of Small Query Compiler to support small and empty DataFrames #5113

Are you sure you want to change the base?

FEAT-#4605: Implementation of Small Query Compiler to support small and empty DataFrames #5113

Conversation

billiam-wang commented Oct 10, 2022 • edited by dchigarev

What do these changes do?

lgtm-com bot commented Oct 31, 2022

lgtm-com bot commented Oct 31, 2022

lgtm-com bot commented Oct 31, 2022

devin-petersohn left a comment

Choose a reason for hiding this comment

Garra1980 commented Nov 1, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YarShev commented Dec 5, 2023

billiam-wang commented Oct 10, 2022 •

edited by dchigarev