[QDP] Initiate project QDP #786

guan404ming · 2026-01-04T17:49:45Z

Purpose of PR

Initiate QDP project

Related Issues or PRs

Changes Made

Breaking Changes

Yes
No

Checklist

Added or updated unit tests for all changes
Added or updated documentation for all changes
Successfully built and ran all unit tests or manual tests locally
PR title follows "MAHOUT-XXX: Brief Description" format (if related to an issue)
Code follows ASF guidelines

* structure * fix a lots of cuda errors * change folder name * improve

* Add Python bindings for QDP using PyO3 * Add QuantumTensor class for DLPack integration * Add TODO * Update qdp/qdp-python/src/lib.rs Co-authored-by: Ryan Huang <ryankert01@gmail.com> --------- Co-authored-by: Ryan Huang <ryankert01@gmail.com>

* add nvtx and some comments * Update qdp/docs/observability/NVTX_USAGE.md Co-authored-by: Ryan Huang <ryankert01@gmail.com> * catch cuda include path * fix prefix errors * fix precommit error and rebuild Cargo.lock * fix precommit error * fix precommit errors --------- Co-authored-by: Ryan Huang <ryankert01@gmail.com>

* add preprocessing module * Remove unused import of Cow

Signed-off-by: 400Ping <fourhundredping@gmail.com>

* async pipline * prevent error * Add TODO for dynamic chunk size tuning Added a TODO comment to tune chunk size dynamically based on GPU model or PCIe bandwidth. * update * add a test

* Integrate Apache Arrow & Parquet for data processing * Optimize Arrow Float64Array handling in io * Add chunked Arrow Float64Array support * Refactor encoding to support chunked Arrow Float64Array input * Refactor I/O and encoding documentation to remove zero-copy

* add vanilla gpu kernel Signed-off-by: ryankert01 <ryan980053@gmail.com> * Optimize amplitude encoding kernel with safety checks Refactored kernel to use multiplication for normalization and optimized memory access with double2. Added safety checks for input parameters. * Remove comments on norm validation and optimization Removed comments regarding norm checks and optimization. * add docker dev container and test Signed-off-by: ryankert01 <ryan980053@gmail.com> * add docker dev containe Signed-off-by: ryankert01 <ryan980053@gmail.com> * delete unrelated strings Signed-off-by: ryankert01 <ryan980053@gmail.com> * delete unrelated strings Signed-off-by: ryankert01 <ryan980053@gmail.com> --------- Signed-off-by: ryankert01 <ryan980053@gmail.com>

* Batch Throughput Signed-off-by: 400Ping <fourhundredping@gmail.com> * fix Signed-off-by: 400Ping <fourhundredping@gmail.com> * [chore] add comment Signed-off-by: 400Ping <fourhundredping@gmail.com> --------- Signed-off-by: 400Ping <fourhundredping@gmail.com>

* gracefully handles OOM Signed-off-by: 400Ping <fourhundredping@gmail.com> * fix Signed-off-by: 400Ping <fourhundredping@gmail.com> --------- Signed-off-by: 400Ping <fourhundredping@gmail.com>

* add a fidelity test * fix format * reformat * effectively zero

* improve amplitudeEncoders for less copy memory allocations * Fix: apply pre-commit end-of-file-fixer * use pre-processing module

Updated build instructions to include Python 3.11 environment setup.

) * [QDP] [Benchmark] Add a End-to-End benchmark to compare with speed * [QDP] improve memory management * Revert "[QDP] improve memory management" This reverts commit d909f3d.

* [QDP] Add batch encoding support * Refactor batch pre-processing

* Add Arrow IPC support * Update default frameworks to "mahout-parquet" * Add tests for Arrow IPC * Add todo to prevent oom * Add TODO for supporting multiple float types

Co-Authored-By: Ping <fourhundredping@gmail.com>

* GPU/Cuda kernel Optimizations Signed-off-by: 400Ping <fourhundredping@gmail.com> * fix Signed-off-by: 400Ping <fourhundredping@gmail.com> --------- Signed-off-by: 400Ping <fourhundredping@gmail.com>

* improve e2e benchmark * gc.coolect first

* Set float32 to default Signed-off-by: 400Ping <fourhundredping@gmail.com> * fix errors Signed-off-by: 400Ping <fourhundredping@gmail.com> * fix python bindings Signed-off-by: 400Ping <fourhundredping@gmail.com> * [chore] update log Signed-off-by: 400Ping <fourhundredping@gmail.com> --------- Signed-off-by: 400Ping <fourhundredping@gmail.com>

Signed-off-by: 400Ping <fourhundredping@gmail.com>

* DataLoader-Benchmark Signed-off-by: 400Ping <fourhundredping@gmail.com> * update output Signed-off-by: 400Ping <fourhundredping@gmail.com> * update Signed-off-by: 400Ping <fourhundredping@gmail.com> * [Fix] Remove print in Qiskit Signed-off-by: 400Ping <fourhundredping@gmail.com> * fix Signed-off-by: 400Ping <fourhundredping@gmail.com> * fix pre-commit Signed-off-by: 400Ping <fourhundredping@gmail.com> * [QDP] improve memory management * fix pre-commit Signed-off-by: 400Ping <fourhundredping@gmail.com> * update doc Signed-off-by: 400Ping <fourhundredping@gmail.com> * update doc Signed-off-by: 400Ping <fourhundredping@gmail.com> * improve memory usage * improve pool * improve for memory * fix back * merge and improve * fix error * memory release * follow suggestions * Address reviewer feedback: safety checks, error handling, and code cleanup * Revert e2e Benchmark * refactor ffi * remove redundant check * fix error --------- Signed-off-by: 400Ping <fourhundredping@gmail.com> Co-authored-by: 400Ping <fourhundredping@gmail.com>

* docs: add development guide * refactor: precommit * docs: add dev container in DEVELOPMENT * docs: add more troubleshooting

* [QDP] Fix DLPack device_id hardcoding# Please enter the commit message for your changes. Lines starting * update and move comments position * improve the test

* float32 follow up Signed-off-by: 400Ping <fourhundredping@gmail.com> * update Signed-off-by: 400Ping <fourhundredping@gmail.com> --------- Signed-off-by: 400Ping <fourhundredping@gmail.com>

* refactor: benchmark/ move into qdp-python/ * build: add benchmark group in pyproject.toml * build+docs: add make commands and update development docs * build+docs: add profiling make command and docs * fix: unit_test command to test_ * build: update e2e test and profile * docs: update development docs

* colab benchmark * add link * add link * update with colab badge * precommit * renamed folder name to notebooks * update link

* add rust linter * pre-commit Signed-off-by: Hsien-Cheng Huang <ryankert01@gmail.com> --------- Signed-off-by: Hsien-Cheng Huang <ryankert01@gmail.com>

* update link and position * update file position and cleanup script * colab link update

) * refactor: introduce 2 traits for flexible io type w/ example * linter * remove redundent test & update readme * remove numpy effort * remove numpy * remove numpy * fix test * no numpy * cleanup * fix imports * pre-commit and usize overflow check

Signed-off-by: 400Ping <fourhundredping@gmail.com>

* feat: use rust hook for precommit * Refactor clippy hook in .pre-commit-config.yaml Updated clippy hook configuration in pre-commit. * run linter

* feat: [QDP] PyTorch Tensor Detection and CPU Path Signed-off-by: Cheyu Wu <cheyu1220@gmail.com> * style: fix linter issue Signed-off-by: Cheyu Wu <cheyu1220@gmail.com> * style: linter err Signed-off-by: Cheyu Wu <cheyu1220@gmail.com> * doc: add comment for followup pr Signed-off-by: Cheyu Wu <cheyu1220@gmail.com> * revert: ipynb linter fix Signed-off-by: Cheyu Wu <cheyu1220@gmail.com> --------- Signed-off-by: Cheyu Wu <cheyu1220@gmail.com>

…751) * Double-buffered async I/O for read_parquet_batch Signed-off-by: 400Ping <fourhundredping@gmail.com> * update Signed-off-by: 400Ping <fourhundredping@gmail.com> * fix python binding error Signed-off-by: 400Ping <fourhundredping@gmail.com> * update Signed-off-by: 400Ping <fourhundredping@gmail.com> * update Signed-off-by: 400Ping <fourhundredping@gmail.com> * update Signed-off-by: 400Ping <fourhundredping@gmail.com> * update Signed-off-by: 400Ping <fourhundredping@gmail.com> * update Signed-off-by: 400Ping <fourhundredping@gmail.com> * fix build error Signed-off-by: 400Ping <fourhundredping@gmail.com> * Revert "fix build error" This reverts commit 3556b5a. * fix build errors Signed-off-by: 400Ping <fourhundredping@gmail.com> * update unit test and boundary check Signed-off-by: 400Ping <fourhundredping@gmail.com> * remove improvement 2 Signed-off-by: 400Ping <fourhundredping@gmail.com> * fix qdp-core error Signed-off-by: 400Ping <fourhundredping@gmail.com> * fix pre-commit Signed-off-by: 400Ping <fourhundredping@gmail.com> * [Fix] fix pre-commit errors & warnings Signed-off-by: 400Ping <fourhundredping@gmail.com> * fix rust linters Signed-off-by: 400Ping <fourhundredping@gmail.com> * [Fix] handle buffer pool lock poisoning Signed-off-by: 400Ping <fourhundredping@gmail.com> * [Chore] fix rust linters Signed-off-by: 400Ping <fourhundredping@gmail.com> * update Signed-off-by: 400Ping <fourhundredping@gmail.com> * Remove unused func Signed-off-by: 400Ping <fourhundredping@gmail.com> * update Signed-off-by: 400Ping <fourhundredping@gmail.com> --------- Signed-off-by: 400Ping <fourhundredping@gmail.com>

* feat: add numpy as reader * add benchmark

guan404ming · 2026-01-04T17:53:34Z

After #778 and #779, let's do the final test for qdp and get this in!

ryankert01 · 2026-01-05T01:46:55Z

#790 fixed the pre-commit issue

guan404ming · 2026-01-05T02:15:14Z

Cherry-picked #790 to preserve the sequential branch and avoid rebasing it.

* The Scaling Test (Latency vs. Qubits) Signed-off-by: 400Ping <fourhundredping@gmail.com> * fix pre-commit Signed-off-by: 400Ping <fourhundredping@gmail.com> * [Chore] make initialization clearer & clearfy doc Signed-off-by: 400Ping <fourhundredping@gmail.com> --------- Signed-off-by: 400Ping <fourhundredping@gmail.com>

Signed-off-by: Hsien-Cheng Huang <ryankert01@gmail.com>

* [Core] Update throughput benchmark to batch encoding Signed-off-by: 400Ping <fourhundredping@gmail.com> * fix conflict Signed-off-by: 400Ping <fourhundredping@gmail.com> --------- Signed-off-by: 400Ping <fourhundredping@gmail.com>

400Ping · 2026-01-06T06:40:18Z

website/community/index.md

 ## Biweekly Community Meeting
-Apache Mahout is committed to work consistently and conducts its meetings once in two weeks. To get invovled, you can access the link here. The meeting timings are communicated via official slack channel. 
+Apache Mahout is committed to work consistently and conducts its meetings once in two weeks. To get invovled, you can access the link here. The meeting timings are communicated via official slack channel.
 [Meeting Link](https://meet.google.com/hjo-njer-hzw?authuser=0&hs=122&ijlm=1767371505662&pli=1)


nit: Maybe changing this to subscribe to Mahout Google calendar.

Sure, let's do it in another pr

guan404ming · 2026-01-06T16:43:39Z

I’ve tested and verified all qdp features, and I believe this is ready to be merged into main. If any refinements are needed for QDP in the future, please open a PR targeting main.

400Ping · 2026-01-06T16:44:23Z

Nice

rich7420 and others added 30 commits November 28, 2025 13:09

[QDP] Initialize QDP, memory management, and DLPack protocol (#646)

9a96981

* structure * fix a lots of cuda errors * change folder name * improve

[QDP] Update pre-commit to add license headers for Rust (#660)

b44a4b5

[QDP] add unit test for qdp-core (#659)

c453f5f

[QDP] add integration test and introduction (#666)

7f78917

[QDP] improve memory initialization (#668)

957f569

[QDP] Add pre-processing module (#673)

038c9fd

* add preprocessing module * Remove unused import of Cow

[QDP] Add tests for pre-processing (#676)

ab2b6c9

Signed-off-by: 400Ping <fourhundredping@gmail.com>

[QDP] make dataflow async and improve amplitudeEncoder (#675)

76db5a4

* async pipline * prevent error * Add TODO for dynamic chunk size tuning Added a TODO comment to tune chunk size dynamically based on GPU model or PCIe bandwidth. * update * add a test

[QDP] Update pre-commit to add license headers for Cuda (#684)

f7ce244

[QDP] Gracefully handles OOM (#688)

5738b8d

* gracefully handles OOM Signed-off-by: 400Ping <fourhundredping@gmail.com> * fix Signed-off-by: 400Ping <fourhundredping@gmail.com> --------- Signed-off-by: 400Ping <fourhundredping@gmail.com>

[QDP] Add fidelity test (#694)

a3aeeed

* add a fidelity test * fix format * reformat * effectively zero

[QDP] improve amplitudeEncoders for less copy memory allocations (#690)

a534f4c

* improve amplitudeEncoders for less copy memory allocations * Fix: apply pre-commit end-of-file-fixer * use pre-processing module

[QDP] update uv usage in qdp-python (#698)

7da8b90

Updated build instructions to include Python 3.11 environment setup.

[QDP] [Benchmark] Add a End-to-End benchmark to compare with speed (#697

335036a

) * [QDP] [Benchmark] Add a End-to-End benchmark to compare with speed * [QDP] improve memory management * Revert "[QDP] improve memory management" This reverts commit d909f3d.

[QDP] add batch kernel support (#700)

ce4b7ca

* [QDP] Add batch encoding support * Refactor batch pre-processing

[QDP] Add Arrow IPC support (#704)

2ff6664

* Add Arrow IPC support * Update default frameworks to "mahout-parquet" * Add tests for Arrow IPC * Add todo to prevent oom * Add TODO for supporting multiple float types

[QDP] Add data loader benchmark (#701)

a13a5c8

Co-Authored-By: Ping <fourhundredping@gmail.com>

[QDP] GPU/Cuda kernel Optimizations (#706)

42397e0

* GPU/Cuda kernel Optimizations Signed-off-by: 400Ping <fourhundredping@gmail.com> * fix Signed-off-by: 400Ping <fourhundredping@gmail.com> --------- Signed-off-by: 400Ping <fourhundredping@gmail.com>

[QDP] improve e2e benchmark (#713)

c614ee1

* improve e2e benchmark * gc.coolect first

[QDP] support turing architecture GPU (#731)

328eecc

[QDP] Choose between different frameworks in benchmark (#741)

97932d0

Signed-off-by: 400Ping <fourhundredping@gmail.com>

[QDP] Enable 3060ti 3090ti cuda build sm86 (#737)

6992fc5

[docs][qdp] Add qdp development guide (#732)

f9e4c28

* docs: add development guide * refactor: precommit * docs: add dev container in DEVELOPMENT * docs: add more troubleshooting

rich7420 and others added 14 commits December 23, 2025 10:51

[QDP] Fix DLPack device_id hardcoding (#745)

aa30679

* [QDP] Fix DLPack device_id hardcoding# Please enter the commit message for your changes. Lines starting * update and move comments position * improve the test

[QDP] DLPack shape/strides: Support batch 2D tensor (#747)

03f71ec

[QDP] Add launch_amplitude_encode_f32 function (#739)

7a2b8fe

* float32 follow up Signed-off-by: 400Ping <fourhundredping@gmail.com> * update Signed-off-by: 400Ping <fourhundredping@gmail.com> --------- Signed-off-by: 400Ping <fourhundredping@gmail.com>

[QDP] Add colab benchmark example (#755)

8f20ddf

* colab benchmark * add link * add link * update with colab badge * precommit * renamed folder name to notebooks * update link

[QDP] add Rust Linter to pre-commit hook (#758)

167f743

* add rust linter * pre-commit Signed-off-by: Hsien-Cheng Huang <ryankert01@gmail.com> --------- Signed-off-by: Hsien-Cheng Huang <ryankert01@gmail.com>

[QDP] Update notebooks dir and link (#762)

ef7e10d

* update link and position * update file position and cleanup script * colab link update

[QDP] Fix Route Error in Makefile (#775)

db71abf

[QDP] Add Workflow dispatch for manual triggering (#767)

febde8c

[QDP] Rename benchmark throughput script and refresh docs (#776)

dd3856b

Signed-off-by: 400Ping <fourhundredping@gmail.com>

[QDP] Pre commit rust followup (#766)

3e781b8

* feat: use rust hook for precommit * Refactor clippy hook in .pre-commit-config.yaml Updated clippy hook configuration in pre-commit. * run linter

guan404ming changed the title ~~[QDP] Initiate QDP project~~ [QDP] Initiate project QDP Jan 4, 2026

[QDP] Numpy IO support (#777)

57f90e6

* feat: add numpy as reader * add benchmark

[QDP] fix pre-commit (#790)

c38412d

400Ping and others added 4 commits January 5, 2026 18:43

[QDP] [Numpy] add path exist check (#793)

7747bd3

Signed-off-by: Hsien-Cheng Huang <ryankert01@gmail.com>

[QDP] Fix benchmark to use batch encoding (#794)

4c99293

guan404ming marked this pull request as ready for review January 6, 2026 03:40

400Ping reviewed Jan 6, 2026

View reviewed changes

400Ping approved these changes Jan 6, 2026

View reviewed changes

guan404ming merged commit 5b6055a into main Jan 6, 2026
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[QDP] Initiate project QDP #786

[QDP] Initiate project QDP #786

Uh oh!

guan404ming commented Jan 4, 2026

Uh oh!

guan404ming commented Jan 4, 2026

Uh oh!

ryankert01 commented Jan 5, 2026

Uh oh!

guan404ming commented Jan 5, 2026

Uh oh!

400Ping Jan 6, 2026

Uh oh!

guan404ming Jan 6, 2026

Uh oh!

guan404ming commented Jan 6, 2026 •

edited

Loading

Uh oh!

400Ping commented Jan 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

[QDP] Initiate project QDP #786

[QDP] Initiate project QDP #786

Uh oh!

Conversation

guan404ming commented Jan 4, 2026

Purpose of PR

Related Issues or PRs

Changes Made

Breaking Changes

Checklist

Uh oh!

guan404ming commented Jan 4, 2026

Uh oh!

ryankert01 commented Jan 5, 2026

Uh oh!

guan404ming commented Jan 5, 2026

Uh oh!

400Ping Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

guan404ming Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

guan404ming commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

400Ping commented Jan 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

guan404ming commented Jan 6, 2026 •

edited

Loading