chore: Prefer uv over pip for dependencies

[brenorb]

Description

Alter the scripts for preferring uv over pip, if it's available. Also tries pip3 if pip is not available.
uv is the modern way to manage dependencies on python, it's faster than the alternatives and written on rust.
Closes #26

Checklists

All Submissions:

• I've signed all my commits
• I followed the contribution guidelines
  [ ] I ran cargo fmt and cargo clippy before committing

[reez]

@mg-twentyone @andreasgriffin as users/contributors do you have any feedback/preferences on this?

[brenorb]

This failed test will pass once #28 is merged, it's just that problem with python version.

[mg-twentyone]

@mg-twentyone @andreasgriffin as users/contributors do you have any feedback/preferences on this?

first of all, thanks @brenorb for your PR.

uv is an interesting tool that is faster than pip for package installation and dependency resolution. It's also written in Rust, making it a good fit with UniFFI bindings from Rust lib. Additionally, it allows for more centralized dependency management using project.toml.
However, as @thunderbiscuit explains here, this is not a standard Python project. In this specific case, our bottleneck (in terms of speed) is likely the Rust compilation rather than Python package installation, so this improvement may have a non-significant impact compared to the time spent building the Rust bindings.

Anyway, imho this is not the right approach to switch from pip to uv for the following reasons:

1. using an hybrid approach increases complexity, as we would need to manage and maintain both cases.
2. IMPORTANT: uv add is not just an installer; it's a project manager. Using uv add reads requirements.txt and modifies the pyproject.toml file to include these dependencies. Furthermore, we also use generate scripts in ci/cd , which can result in a "dirty" git state (and maybe in the pipeline failure).

To summarize, imho if we want to consider this switch, we should carefully analyze all the implications and try to implement it in a more consistent way.

[brenorb]

Sorry to take a while to answer, it took a while as this is not so straightforward.

My idea was not to improve the speed, but to move from old practices to modern and standard Python practices.

TLDR

1. we can choose only one approach and stay with it.
2. re: uv
   2.1. requirements.txt is legacy and not a Python standard (per PEP 751).
   2.2. Declaring dependencies directly in pyproject.toml is the modern, standardized approach (per PEP 621).
   2.3. uv.lock is created, but it should be versioned as well. It's not standard but helps consistent and reproducible installations across machines.
   2.4. we can suppress .venv/ folder creation in uv via --active. However, a virtual environment is a must for installations which are not on isolated computers/containers.
   2.5. If we decide to stay on pure pip, then I believe requirements.txt remains necessary because pip’s dependency-group support is new and immature and still cannot install only dependencies from a pyproject.toml the without installing the project itself.

requirements.txt

Although I didn't remove the requirements.txt file in the PR, as you noticed, uv adds the dependency list to pyproject.toml as it's the modern python standard so we move away from requirements.txt. Maybe we should instead remove requirements.txt completely and simply declare dependencies in pyproject.toml.

As PEP751 states:

Unfortunately, the format [requirements.txt] is not a standard but is supported by convention. It’s also designed very much for pip’s needs, limiting its flexibility and ease of use (e.g. it’s a bespoke file format). Lastly, it is not secure by default (e.g. file hash support is entirely an opt-in feature, you have to tell pip to not look for other dependencies outside of what’s in the requirements file, etc.).

uv.lock

As you noticed, uv also creates an uv.lock file and a .venv/ (virtual environment) folder.

The uv.lock was created mirroring cargo.lock, to provide consistent and reproducible installations across machines and should be checked into version control. However, uv.lock is also not standard python. In PEP751, Python recognized "an appetite for lock files in general" and created pylock.toml standard. However, "the PEP 751-style pylock.toml files are not sufficient to replace uv.lock" as uv lead maintainer states.

.venv/

If bdk-python is created for running also in personal computers, it must be run in virtual environments because global pip install:

• Pollutes the global environment
• Can cause version conflicts with other projects
• Is not reproducible across machines

So if we are not generating the virtual environment ourselves, we should at least mention the need to run and activate a venv before running the scripts.

However, if we need to suppress the creation of the .venv/ folder for CI, we can simply add --active to the commands and uv will not create it.

pip

Nonetheless, I understand if we wish to continue with pip, since uv is not standard python. In that case, we might need to continue using requirements.txt as pip implementation is immature and can only install the full project from pyproject.toml and not just the dependencies as it does in pip install -r requirements.txt.

[andreasgriffin]

Is there any necessity to keep the requirements.txt and requirements-dev.txt? Or could they be replaced by putting the requirements in the pyproject.toml (and then in the lock file)

[brenorb]

@andreasgriffin They could be replaced by putting the requirements in the pyproject.toml if we use uv, because uv can install the requirements into a virtual environment and it can also build the whole project and install it. pip, on the other hand, can install the requirements from a *.txt file or it can install the whole library from pyproject.toml file. From my research, pip cannot install only the requirements from a pyproject.toml file. I'll run some tests this week or the next and bring updates.

[thunderbiscuit]

I want to say that I've been using uv for a few weeks now and it's... superb. Like just way better. It just... works??? Coming from pip I had just accepted my fate and the dev workflow for Python to be terrible but no more!

Thank you @brenorb for showing me this awesome tool. I definitely think we should try to support it in some capacity in bdk-python.

[brenorb]

@thunderbiscuit

YES!
I'm glad you tried it for yourself. It's hard to explain how a 3rd party tool quickly became best practice. I was thinking on how to make a case out of it, but nothing like actually experiencing it to get it.

The way I would do it would be to replace pip the instalation script (with optional fallback to pip, your call) and in the CI. Tell me how you prefer and I'll do it.