Clarify use of index-url vs. extra-index-url

[matthewfeickert]

Clarify the need(?) and use of --index-url and --extra-index-url. I might have over complicated some of the instructions given a misunderstanding of priority, but there seems to be no priority and no way to enforce priority. This I think lead to some of the confusion in #41.

References:

• index-url extra-index-url install priority order pypa/pip#5045
• index-url extra-index-url install priority order pypa/pip#8606
• pip install docs "Finding Packages" section
• Compromised PyTorch-nightly dependency chain between December 25th and December 30th, 2022.

[matthewfeickert]

Relevant thing to point to (c.f. pypa/pip#8606 (comment)) is PEP 708 – Extending the Repository API to Mitigate Dependency Confusion Attacks (currently unimplimented).

[matthewfeickert]

Note that uv differs from pip here in that it does give higher priority to --extra-index-url (which is different than I would have assumed!).

From uv pip install --help:

...
  -i, --index-url <INDEX_URL>
          The URL of the Python package index (by default: <https://pypi.org/simple>).
          
          The index given by this flag is given lower priority than all other indexes specified via the `--extra-index-url` flag.
          
          Unlike `pip`, `uv` will stop looking for versions of a package as soon as it finds it in an index. That is, it isn't possible for `uv` to consider versions of the same package across multiple indexes.
          
          [env: UV_INDEX_URL=]

      --extra-index-url <EXTRA_INDEX_URL>
          Extra URLs of package indexes to use, in addition to `--index-url`.
          
          All indexes given via this flag take priority over the index in `--index-url` (which defaults to PyPI). And when multiple `--extra-index-url` flags are given, earlier values take priority.
          
          Unlike `pip`, `uv` will stop looking for versions of a package as soon as it finds it in an index. That is, it isn't possible for `uv` to consider versions of the same package across multiple indexes.
          
          [env: UV_EXTRA_INDEX_URL=]

...

[martinfleis]

I'm curious if that's intended. I thought that their aim was to be a drop-in replacement.

[matthewfeickert]

I'm curious if that's intended.

I think it is, though we could easily check (and ask).

I thought that their aim was to be a drop-in replacement.

I believe the goal is for functionality to be achieved, not for design choices to be replicated. In the limitations section of the README they note

Limitations

While uv supports a large subset of the pip interface, it does not support the entire feature set.
In some cases, those differences are intentional; in others, they're a result of uv's early stage of development.

For details, see our pip compatibility guide.

[matthewfeickert]

Yeah, it is intentional given astral-sh/uv#2083.

[Carreau]

If there are issue with the order of indexes, one alternative would be for scientific python to have a "proxy index", that merges the json of multiple upstream index.

I'm wondering if a proxy index like that would even need a permanent server or could be hosted purely on an edge cloud as it is likely stateless.

[matthewfeickert]

@Carreau do you have examples of these proxy indexes? I haven't heard of this before, so it would be interesting to see how they work.

[Carreau]

Hum, it's theoretical, but basically you fetch multiple <repos>/simple/<package> and merge the pages. I believe I might talk about this with @ivanov last June in seattle.

Think:

import flask
import request
import bs4
app = flask.app(__file__)

repos = ['https://pypi.org/', 'https://nightly.com/']


@app.route('/simple/<package>')
def simple(package):
    pages = [requests.get(r+'/simple/'+package) for r in repos]
    bodys = [bs4.parse(p).body for p in pages]
    return HEAD +  concat(bodys) + FOOTER

That's the legacy API – I think anaconda.org only have it – but there is a new JSON API as well, so we might need some work to figure out the details.

We should only need to serve indexes, as the download come from somewhere else.

[Carreau]

Here is a poc using flask: https://github.com/Carreau/multi-index which does work locally.

It's sync, but it should not be hard to make it async with various caching.

[Carreau]

See https://github.com/Carreau/cloudflare-pypi-multi-index deployed on cloudflare workers on
https://nightly.carreau.workers.dev/nightly

$ pip install --index-url https://nightly.carreau.workers.dev/nightly --pre --upgrade ipython

Should now just work and "merge" Pypi and https://pypi.anaconda.org/scientific-python-nightly-wheels/simple/, we should be able to add any other PyPI mirror or have crazy things like /random return only a subset of the wheel, or /binary, strip all the tgz if whl are present.. for example.

If it's of interest we could have that owned by the tools team, which would simplify the above (and we could have usage metrics...)