[docs] Quantization + torch.compile + offloading#11703
[docs] Quantization + torch.compile + offloading#11703stevhliu merged 9 commits intohuggingface:mainfrom
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
sayakpaul
left a comment
sayakpaul
left a comment
There was a problem hiding this comment.
Thanks for starting this. Will get you the numbers.
Code: https://gist.github.com/sayakpaul/0db9d8eeeb3d2a0e5ed7cf0d9ca19b7d Worth mentioning:
|
| ``` | ||
|
|
||
| </hfoption> | ||
| <hfoption id="group offloading"> |
There was a problem hiding this comment.
Do you think it might be better demonstrated with a more compute heavy model like Wan? This way, we can show the actual benefits of group offloading.
There was a problem hiding this comment.
Sounds good, could you get me the updated numbers for Wan with quantization/group offloading/torch.compile please?
There was a problem hiding this comment.
I think it's okay to have the Flux numbers but for the sake of code and discussions, we could have Wan.
There was a problem hiding this comment.
Ah ok, don't worry about getting the Wan numbers then!
|
|
||
| Offloading strategies move not currently active layers or models to the CPU to avoid increasing GPU memory. These strategies can be combined with quantization and torch.compile to balance inference speed and memory usage. | ||
|
|
||
| Refer to the [Compile and offloading quantized models](./speed-memory-optims) guide for more details. |
There was a problem hiding this comment.
I think #11731 can be resolved in this PR where I make a note that offloading can be combined with quantization and torch.compile
There was a problem hiding this comment.
Added your layerwise casting note in here as well :)
There was a problem hiding this comment.
Yeah feel free to close those :)
| pipeline.transformer.enable_group_offload( | ||
| onload_device=onload_device, | ||
| offload_device=offload_device, | ||
| offload_type="block_level", | ||
| num_blocks_per_group=4 | ||
| ) |
There was a problem hiding this comment.
We should use these args when a component is quantized with bitsandbytes to mitigate device mismatch issues:
But I am curious. Were you able to run the code?
sayakpaul
left a comment
sayakpaul
left a comment
There was a problem hiding this comment.
Left some more comments. LMK if they make sense.
3 participants
Follows up on #11670 and #11672 to document combinations of quantization, torch.compile, and offloading.