[COMPATIBILITY] antlr4-python3-runtime 4.9.3 is no longer supported but spark is stuck with it

[vspinu]

Spark has this comment in its pom.xml:

    <!-- Please don't upgrade the version to 4.10+, it depends on JDK 11 -->
    <antlr4.version>4.9.3</antlr4.version>

Inside our company we use antlr4-python3-runtime to implement some parts of spark sql parsing. We pin the version to the version used in spark for compatibility reasons. This renders fugue and downstream packages that started including fugue as core dependency (e.g. statsforecast) incompatible with our code base.

Also #327 required that We also want to make sure 4.9.* are still supported. which didn't realize as 4.11 is now the minimal requirement.

Would it be too difficult on your side to keep antlr4 minimum version the one which is used in spark? Thanks!

[goodwanghan]

Got it, yes we can take a look. We also plan to remove antlr from our core dependency starting from Fugue 0.9.0, but that will take a while.

[goodwanghan]

Actually I don't know why it matters. Fugue has very good support of pyspark, we have never seen such a conflict.

[vspinu]

It matters only when people use antlr on python side and want to match spark's version. Spark's dependency is on java side, hence no conflicts so far :)

I understand this is rather a corner case, so it's completely understandable if you decide not to give this a priority.

[goodwanghan]

Closing because this problem should have been resolved.

[vspinu]

It's actually not. Sorry for late reply, our internal pip mirror takes at least two weeks to update the packages and I switched to a different project in meanwhile. Now I am back to updating statsforecast and it still does not work. The antlr dependency of qpd and fugue-sql-antr are still 4.11.

I might be doing something wrong though:

 $ pipdeptree -p fugue
Warning!!! Possibly conflicting dependencies found:
* <our-internal-package>==1.215.dev0
 - antlr4-python3-runtime [required: >=4.9.3,<=4.9.3, installed: 4.11.1]
------------------------------------------------------------------------
fugue==0.8.6
  - adagio [required: >=0.2.4, installed: 0.2.4]
    - triad [required: >=0.6.1, installed: 0.9.1]
      - fs [required: Any, installed: 2.4.16]
        - appdirs [required: ~=1.4.3, installed: 1.4.4]
        - setuptools [required: Any, installed: 62.1.0]
        - six [required: ~=1.10, installed: 1.16.0]
      - fsspec [required: Any, installed: 2023.9.0]
      - numpy [required: Any, installed: 1.24.3]
      - pandas [required: >=1.2.0, installed: 1.5.3]
        - numpy [required: >=1.20.3, installed: 1.24.3]
        - python-dateutil [required: >=2.8.1, installed: 2.8.2]
          - six [required: >=1.5, installed: 1.16.0]
        - pytz [required: >=2020.1, installed: 2023.3]
      - pyarrow [required: Any, installed: 11.0.0]
        - numpy [required: >=1.16.6, installed: 1.24.3]
      - six [required: Any, installed: 1.16.0]
  - fugue-sql-antlr [required: >=0.1.6, installed: 0.1.7]
    - antlr4-python3-runtime [required: >=4.11.1,<4.12, installed: 4.11.1]
    - jinja2 [required: Any, installed: 3.1.2]
      - MarkupSafe [required: >=2.0, installed: 2.1.2]
    - triad [required: >=0.6.8, installed: 0.9.1]
      - fs [required: Any, installed: 2.4.16]
        - appdirs [required: ~=1.4.3, installed: 1.4.4]
        - setuptools [required: Any, installed: 62.1.0]
        - six [required: ~=1.10, installed: 1.16.0]
      - fsspec [required: Any, installed: 2023.9.0]
      - numpy [required: Any, installed: 1.24.3]
      - pandas [required: >=1.2.0, installed: 1.5.3]
        - numpy [required: >=1.20.3, installed: 1.24.3]
        - python-dateutil [required: >=2.8.1, installed: 2.8.2]
          - six [required: >=1.5, installed: 1.16.0]
        - pytz [required: >=2020.1, installed: 2023.3]
      - pyarrow [required: Any, installed: 11.0.0]
        - numpy [required: >=1.16.6, installed: 1.24.3]
      - six [required: Any, installed: 1.16.0]
  - jinja2 [required: Any, installed: 3.1.2]
    - MarkupSafe [required: >=2.0, installed: 2.1.2]
  - pandas [required: >=1.2.0, installed: 1.5.3]
    - numpy [required: >=1.20.3, installed: 1.24.3]
    - python-dateutil [required: >=2.8.1, installed: 2.8.2]
      - six [required: >=1.5, installed: 1.16.0]
    - pytz [required: >=2020.1, installed: 2023.3]
  - pyarrow [required: >=0.15.1, installed: 11.0.0]
    - numpy [required: >=1.16.6, installed: 1.24.3]
  - qpd [required: >=0.4.4, installed: 0.4.4]
    - adagio [required: Any, installed: 0.2.4]
      - triad [required: >=0.6.1, installed: 0.9.1]
        - fs [required: Any, installed: 2.4.16]
          - appdirs [required: ~=1.4.3, installed: 1.4.4]
          - setuptools [required: Any, installed: 62.1.0]
          - six [required: ~=1.10, installed: 1.16.0]
        - fsspec [required: Any, installed: 2023.9.0]
        - numpy [required: Any, installed: 1.24.3]
        - pandas [required: >=1.2.0, installed: 1.5.3]
          - numpy [required: >=1.20.3, installed: 1.24.3]
          - python-dateutil [required: >=2.8.1, installed: 2.8.2]
            - six [required: >=1.5, installed: 1.16.0]
          - pytz [required: >=2020.1, installed: 2023.3]
        - pyarrow [required: Any, installed: 11.0.0]
          - numpy [required: >=1.16.6, installed: 1.24.3]
        - six [required: Any, installed: 1.16.0]
    - antlr4-python3-runtime [required: >=4.11.1,<4.12, installed: 4.11.1]
    - pandas [required: >=1.2.0, installed: 1.5.3]
      - numpy [required: >=1.20.3, installed: 1.24.3]
      - python-dateutil [required: >=2.8.1, installed: 2.8.2]
        - six [required: >=1.5, installed: 1.16.0]
      - pytz [required: >=2020.1, installed: 2023.3]
    - triad [required: >=0.9.0, installed: 0.9.1]
      - fs [required: Any, installed: 2.4.16]
        - appdirs [required: ~=1.4.3, installed: 1.4.4]
        - setuptools [required: Any, installed: 62.1.0]
        - six [required: ~=1.10, installed: 1.16.0]
      - fsspec [required: Any, installed: 2023.9.0]
      - numpy [required: Any, installed: 1.24.3]
      - pandas [required: >=1.2.0, installed: 1.5.3]
        - numpy [required: >=1.20.3, installed: 1.24.3]
        - python-dateutil [required: >=2.8.1, installed: 2.8.2]
          - six [required: >=1.5, installed: 1.16.0]
        - pytz [required: >=2020.1, installed: 2023.3]
      - pyarrow [required: Any, installed: 11.0.0]
        - numpy [required: >=1.16.6, installed: 1.24.3]
      - six [required: Any, installed: 1.16.0]
  - sqlglot [required: Any, installed: 18.1.0]
  - triad [required: >=0.9.1, installed: 0.9.1]
    - fs [required: Any, installed: 2.4.16]
      - appdirs [required: ~=1.4.3, installed: 1.4.4]
      - setuptools [required: Any, installed: 62.1.0]
      - six [required: ~=1.10, installed: 1.16.0]
    - fsspec [required: Any, installed: 2023.9.0]
    - numpy [required: Any, installed: 1.24.3]
    - pandas [required: >=1.2.0, installed: 1.5.3]
      - numpy [required: >=1.20.3, installed: 1.24.3]
      - python-dateutil [required: >=2.8.1, installed: 2.8.2]
        - six [required: >=1.5, installed: 1.16.0]
      - pytz [required: >=2020.1, installed: 2023.3]
    - pyarrow [required: Any, installed: 11.0.0]
      - numpy [required: >=1.16.6, installed: 1.24.3]
    - six [required: Any, installed: 1.16.0]

[vspinu]

I am really confused, while fugue-sql-antlr is no longer requiring antlr4-python3-runtime 4.11 that is not propagated to the pip package:

fugue-sql-antlr==0.1.7
  - antlr4-python3-runtime [required: >=4.11.1,<4.12, installed: 4.11.1]
  - jinja2 [required: Any, installed: 3.1.2]
    - MarkupSafe [required: >=2.0, installed: 2.1.2]
  - triad [required: >=0.6.8, installed: 0.9.1]
    - fs [required: Any, installed: 2.4.16]
      - appdirs [required: ~=1.4.3, installed: 1.4.4]
      - setuptools [required: Any, installed: 62.1.0]
      - six [required: ~=1.10, installed: 1.16.0]
    - fsspec [required: Any, installed: 2023.9.0]
    - numpy [required: Any, installed: 1.24.3]
    - pandas [required: >=1.2.0, installed: 1.5.3]
      - numpy [required: >=1.20.3, installed: 1.24.3]
      - python-dateutil [required: >=2.8.1, installed: 2.8.2]
        - six [required: >=1.5, installed: 1.16.0]
      - pytz [required: >=2020.1, installed: 2023.3]
    - pyarrow [required: Any, installed: 11.0.0]
      - numpy [required: >=1.16.6, installed: 1.24.3]
    - six [required: Any, installed: 1.16.0]

I can also confirm it directly in the installed package:

[vspinu]

Arh, ... it's still in the requirements.txt

[goodwanghan]

Ah I see, but requirements is not a part of the pypi package, the requirement should have nothing to do with it. But let me investigate. This is very weird.

[goodwanghan]

@vspinu I have found the issue and here is the PR to fix this problem. fugue-project/fugue-sql-antlr#22 Sorry it took a bit long.

0.1.8 should be ready tomorrow

[shchur]

@goodwanghan It seems that this issue is still blocked by the qpd dependency that requires antlr4-python3-runtime>=4.11.1, which should be fixed in the 0.9.0 release of fugue.

[shchur]

@goodwanghan do you have a planned date for the 0.9.0 release?

The hard dependency of fugue on antlr4-python3-runtime>=4.11.1 via qpd has been a major blocker for our project for ~1 year now (autogluon/autogluon#3458), preventing us from upgrading to statsforecast>=1.5 (released Feb 2023). The fugue-0.9.0 release where qpd becomes an optional dependency would be extremely helpful in enabling us to resolve this issue.

[goodwanghan]

@shchur I apologize for the delay. Could you try fugue 0.9.0.dev3? I think we will release 0.9.0 soon, but if you could try the latest release, it can be helpful for accelerating the release.

[shchur]

Hi @goodwanghan, thank you for the response! I just checked, pinning the dependency to fugue==0.9.0.dev3 resolves the ANTLR4 incompatibility issue on our side. I would still prefer to include the stable 0.9.0 release in our dependencies to ensure that we don't break anything for users that depend on both autogluon and fugue

[goodwanghan]

@shchur we will release Fugue 0.9.0 in two weeks. Thanks for the confirmation.