Skip to content

Conversation

@tangkong
Copy link
Contributor

@tangkong tangkong commented Apr 7, 2025

(Please squash this on merge)

Description

Uses cibuildwheel to compile and repair wheels for various linux and macos platforms. (not windows)

Context

pyca was missing from pypi originally because it had a naming conflict. After switching to pyca-epics, it turns out that we also just didn't understand what was necessary for packages with compiled libraries.

Long story short: I went through a lot of back and forth here, mostly revolving around trying to get the library files to be recognized properly.

Long story long:
cibuildwheel is a tool used by quite a few other open source projects, and works rather nicely. A key thing to remember is that the ci job (sometimes) builds wheels inside a docker container, so you need to make sure that any environment variables and external dependencies are installed in the right place.

Windows was skipped after I ran into an error along the lines of "pthread.h not found" Pthread is a *nix-specific construct, and building for windows seems to require using a different library entirely. I'm not in the mood to rewrite the c-extensions here so I'm going to say we don't support windows.d

The generated and repaired wheels are saved as artifacts, which I should probably hook up for automatic upload to pypi but I'll deal with that once we're more sure this all works.

Testing

I tested these wheels on my local mac and on psbuild-rocky9, followed by import pyca; print(pyca.__file__) (the ci jobs do this inside each build matrix element, but had to double check

@tangkong tangkong requested a review from ZLLentz April 7, 2025 20:40
@tangkong
Copy link
Contributor Author

tangkong commented Apr 7, 2025

I'll request a review here for now, as I think this is at a decent point and I should probably just upload to pypi and call it a day

(and before I fall down the rabbit hole of conda builds)

steps:
- uses: actions/checkout@v4

- name: Build wheels
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the PR here, this ends up building wheels marked at version 0.0.0. Is this expected for the non-tag build?

20 wheels produced in 80 minutes:
  pyca_epics-0.0.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl         476 kB
  pyca_epics-0.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl     449 kB
  pyca_epics-0.0.0-cp310-cp310-musllinux_1_2_i686.whl                           1,569 kB
  pyca_epics-0.0.0-cp310-cp310-musllinux_1_2_x86_64.whl                         1,432 kB
  pyca_epics-0.0.0-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl         476 kB
  pyca_epics-0.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl     449 kB
  pyca_epics-0.0.0-cp311-cp311-musllinux_1_2_i686.whl                           1,569 kB
  pyca_epics-0.0.0-cp311-cp311-musllinux_1_2_x86_64.whl                         1,432 kB
  pyca_epics-0.0.0-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl         477 kB
  pyca_epics-0.0.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl     450 kB
  pyca_epics-0.0.0-cp312-cp312-musllinux_1_2_i686.whl                           1,569 kB
  pyca_epics-0.0.0-cp312-cp312-musllinux_1_2_x86_64.whl                         1,433 kB
  pyca_epics-0.0.0-cp313-cp313-manylinux_2_17_i686.manylinux2014_i686.whl         477 kB
  pyca_epics-0.0.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl     450 kB
  pyca_epics-0.0.0-cp313-cp313-musllinux_1_2_i686.whl                           1,569 kB
  pyca_epics-0.0.0-cp313-cp313-musllinux_1_2_x86_64.whl                         1,433 kB
  pyca_epics-0.0.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl           470 kB
  pyca_epics-0.0.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl       450 kB
  pyca_epics-0.0.0-cp39-cp39-musllinux_1_2_i686.whl                             1,569 kB
  pyca_epics-0.0.0-cp39-cp39-musllinux_1_2_x86_64.whl                           1,432 kB

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't sorted this bit out yet. I saw this on my fork and hoped that it was just missing tags.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like all the other cibuildwheel examples end up either collecting the tag ahead of time or setting it statically. I'll deal with this in a similar way, maybe

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some more digging, setuptools_scm generates the version files on import or build. Turns out we had been using the old default wheel build method (python -m pip wheel), and switching this to python -m build generates those files properly.

@tangkong
Copy link
Contributor Author

tangkong commented Apr 8, 2025

Once we're happy with the way these builds run, I'll flip on the tag filter and only build wheels on release. The ubuntu builds take on the order of an hour and we probably don't need to be eating that many minutes

@ZLLentz
Copy link
Member

ZLLentz commented Apr 10, 2025

I think the builds look good now

Copy link
Member

@ZLLentz ZLLentz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe let's switch to tag-only, merge, tag, and give it a try

@tangkong
Copy link
Contributor Author

I still need to set up the pypi side of things, I'll run the wheels and upload to testpypi first to confirm. After that I'll get the tokens sorted

@tangkong
Copy link
Contributor Author

Merging this now, we need a clean tag to upload to pypi in any incarnation

@tangkong tangkong merged commit e61f31f into slaclab:master Apr 10, 2025
5 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants