GitHub - NVIDIA/Model-Optimizer: A unified library of SOTA model optimization techniques like quantization, pruning, distillation, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.

NVIDIA / Model-Optimizer Public

Notifications You must be signed in to change notification settings
Fork 232
Star 1.8k

A unified library of SOTA model optimization techniques like quantization, pruning, distillation, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.

nvidia.github.io/Model-Optimizer/

Apache-2.0 license

1.8k stars 232 forks Branches Tags Activity

Notifications

About

A unified library of SOTA model optimization techniques like quantization, pruning, distillation, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.

nvidia.github.io/Model-Optimizer/

Apache-2.0 license

Code of conduct

Security policy

Custom properties

Report repository

Releases 16

ModelOpt 0.40.0 Release Latest

Used by 197

Contributors 48

+ 34 contributors

Languages

Python 98.5%
Other 1.5%