Hansimov's Blog

Appearance

本地运行 Qwen-VL-Chat-Int4 ​

See: Qwen-VL 量化

See: Installation of AutoGPTQ

See: Run Qwen-VL-Chat-Int4 with transformers

一键安装依赖 ​

sh
wget https://raw.staticdn.net/Hansimov/blog/main/docs/notes/scripts/qwen_vl_setup.sh -O ./qwen_vl_setup.sh && chmod +x ./qwen_vl_setup.sh && ./qwen_vl_setup.sh

脚本: https://github.com/Hansimov/blog/blob/main/docs/notes/scripts/qwen_vl_setup.sh

sh
# [~/repos/]
mkdir qwen-vl
cd qwen-vl
wget https://raw.staticdn.net/QwenLM/Qwen-VL/master/requirements.txt -O requirements.txt
pip install -r requirements.txt

pip install optimum
pip install auto-gptq --no-build-isolation

# git clone https://githubfast.com/AutoGPTQ/AutoGPTQ.git
# cd AutoGPTQ
# pip install -v .

一键运行 demo ​

sh
wget https://raw.staticdn.net/Hansimov/blog/main/docs/notes/scripts/qwen_vl_chat_int4.py -O ./qwen_vl_chat_int4.py
HF_ENDPOINT=https://hf-mirror.com python ./qwen_vl_chat_int4.py

脚本: https://github.com/Hansimov/blog/blob/main/docs/notes/scripts/qwen_vl_chat_int4.py

py
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

torch.manual_seed(1234)

model_name = "Qwen/Qwen-VL-Chat-Int4"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name, device_map="auto", trust_remote_code=True
).eval()

# 1st dialogue turn
messages = [
    {
        "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
    },
    {
        "text": "Describe in detail this image in Chinese:",
    },
]
query = tokenizer.from_list_format(messages)
response, history = model.chat(tokenizer, query=query, history=None)
print(response)


# # 2nd dialogue turn
# response, history = model.chat(tokenizer, '输出"击掌"的检测框', history=history)
# print(response)
# # <ref>击掌</ref><box>(517,508),(589,611)</box>
# image = tokenizer.draw_bbox_on_latest_picture(response, history)
# if image:
#     image.save("1.jpg")
# else:
#     print("no box")