Question: What do you think about switching to uv ?

[Neko1313]

Although this is a new technology, it works much faster than poetry, and it also has all the features of working with workspaces that could be very well suited to your modular project structure

1. uv - https://docs.astral.sh/uv/
2. uv workspaces - https://docs.astral.sh/uv/concepts/projects/workspaces/

[alexanderankin]

i used to be much more averse to binary compiled software, when i worked at a company where downloading and running binary files like that went against company policy (and was strictly enforced).

now, i don't think poetry is that much of a burden to deal with, but it doesn't support workspaces - so i think it mostly depends on if we want to split up the code into multiple packages. this project used to have (actually still does have) a bunch of unclaimed packages on pypi when this was last attempted. so it is kind of a hassle to go reclaim them all or come up with a new naming scheme or something - and then make good documentation for them (there is not really much good documentation - you kind of have to read the source).

i am not convinced workspaces are out of the question - that is, publishing as different packages - but it is not a burning need at the moment either.

[Neko1313]

@alexanderankin I think that splitting into several packages within one project is not a bad practice, which will give flexibility to the project, since, let's say, milvus has several simultaneously running versions that need to be supported, which is very difficult to do with the general project approach of the general project that you are using now. If I were you, I would focus on SQLAlchemy dialects, which enable the community to expand independently of the main project.

[alexanderankin]

do you have any samples that are available to play around with? like a project with a couple of sub projects that publish separately and have separate lock files --- basically the only link they have with each other is being in the same "monorepo" and being able to be referenced as dependencies with their relative path (but also when published - still have valid resolvable dependency data so that consumers can still consume them as deps?)

[Neko1313]

@alexanderankin It may be difficult for me to find exactly such projects, but I can create an example

[alexanderankin]

needed to run poetry just now and it can neither lock from scratch or update - really moves this ticket up in terms of priority from my point of view

[CarliJoy]

I am also facing the problem that poertry seems not to be able to update the depenencies.

I think the problems in a lot of pipelines are deprecated sqlachemies that won't work with newer versions of python.

@alexanderankin could label this Issue as high priority ?

i used to be much more averse to binary compiled software, when i worked at a company where downloading and running binary files like that went against company policy (and was strictly enforced).

I not 100% follow. You can install uv with pip install uv as you can install pip install numpy, that also contains compiled code / binary blobs.
What is the difference?

[alexanderankin]

i would imagine loading a dll that is unsigned triggers the antivirus less, but I'm not sure since I don't use numpy. The company I work at now is becoming SOC 2 compliant, for example, which btw a lot of big corporate in US are, and there is a stipulation that there are restrictions on what software employees can install. I can only imagine this is a way to implement this.

I agree that this is a high priority issue, let me try to recreate the lockfile with latest versions of everything.

[Neko1313]

Guys, in general, I can try to make a full transfer to uv this New Year's weekend

[alexanderankin]

i think we are ready for that because while #946 fixes the lockfile versions, it took several minutes to lock.

it installs pretty quick though.

[alexanderankin]

i do hear pip has lockfiles now though, maybe we can also try pip in the meantime, for all i know about the python ecosystem it may have had caught back up and taken the lead

[Neko1313]

I meant not only raising dependencies, but also using a mechanism similar to the sqlalhimia dialect, as well as using a more modern dependency installer.