Unsloth更快训练大模型并导出GGUF - Windows
环境搭建
系统环境
需要Nvidia显卡，至少8G显存，且专用显存与共享显存之和大于20G
建议将非安装版的环境文件都放到非系统盘，方便重装或移植
以Windows11为例，非安装环境文件都放在 E 盘下
设置自定义Path文件夹
创建 E:\mypath 文件夹，将其添加进用户环境变量Path中，之后会用
CMake
下载 CMake 的 Windows x64 ZIP 对应文件：Download CMake
解压到E:\environment\cmake
将E:\environment\cmake\bin添加到用户环境变量Path中
C++编译
下载 Community 版 Visual Studio：Visual Studio 2022 IDE
运行后选择桌面c++板块内容安装，若中途取消安装了，可以在开始菜单下栏的新增项目（或推荐项目）中找到该安装程序
这里只需要c++的编译环境，把除了C++板块以外的安装项目全部取消勾选（该板块自动勾选系统SDK等其他相关组件，不要取消勾选）
python环境
python安装
下载新版Anaconda安装程序：Download Anaconda
安装过程中将添加到环境变量等选项全部打勾
用uv管理pip包
将uv所有文件装至 E:\uv 下
符合国内环境，设置环境变量和更新源，安装到脚本常用硬盘，方便项目共用包可以硬链接节省硬盘空间
设置环境变量
PowerShell指令
[Environment]::SetEnvironmentVariable("UV_INDEX_URL","https://mirrors.huaweicloud.com/repository/pypi/simple","User")
[Environment]::SetEnvironmentVariable("UV_CACHE_DIR","E:\uv\cache","User")
[Environment]::SetEnvironmentVariable("UV_TOOL_DIR","E:\uv\tools","User")
[Environment]::SetEnvironmentVariable("UV_TOOL_BIN_DIR","E:\uv\toolsbin","User")
[Environment]::SetEnvironmentVariable("UV_PYTHON_INSTALL_DIR","E:\uv\python","User")
[Environment]::SetEnvironmentVariable("UV_PYTHON_INSTALL_MIRROR","https://ghproxy.cn/https://github.com/indygreg/python-build-standalone/releases/download","User")
[Environment]::SetEnvironmentVariable("Path","%UV_TOOL_BIN_DIR%;"+[Environment]::GetEnvironmentVariable("Path"),"User")
改变安装路径并安装
PowerShell指令
$env:UV_INSTALL_DIR = "E:\uv\bin"
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
[!tip] 如果提示在此系统上禁止运行脚本
以管理员身份打开 PowerShell，输入Set-ExecutionPolicy Remotesigned
回到前面的 PowerShell 终端继续操作
[!tip] 下载速度慢或连不上
可能需要代理连接
自主更新
uv self update
添加 python 3.11 独立文件
uv python install 3.11
uv python list # 查看是否安装成功
Git环境
安装Git：Git - 安装 Git (git-scm.com)
Nvidia CUDA 工具包
安装12.1版：CUDA Toolkit 12.1 Update 1 Downloads | NVIDIA Developer
注意运行安装程序后在安装选单界面只安装CUDA驱动和工具包，不选其他显卡驱动和Nsignt
安装完后重启电脑
输入指令查看信息
nvidia-smi # 看当前驱动最大可支持的CUDA版本
nvcc -V # 看当前安装的CUDA运行时版本
部署训练框架
创建unsloth_env文件夹，并在里面操作
uv venv -p 3.11
uv pip install .\torch-2.4.0+cu121-cp311-cp311-win_amd64.whl
uv pip install .\torchvision-0.19.0+cu121-cp311-cp311-win_amd64.whl .\torchaudio-2.4.0+cu121-cp311-cp311-win_amd64.whl
uv pip install .\flash_attn-2.6.3+cu123torch2.4.0cxx11abiFALSE-cp311-cp311-win_amd64.whl
uv pip install --no-cache-dir "unsloth[huggingface] @ git+https://github.com/unslothai/unsloth.git"
uv pip install "bitsandbytes>=0.43.3" packaging ninja
uv pip install --no-deps "xformers==0.0.27.post2"
uv pip install .\triton-3.1.0-cp311-cp311-win_amd64.whl # 官方没有win版，win版下载：https://github.com/woct0rdho/triton-windows
"triton\runtime\jit.py" 修改第717行，加入try，except self.src = None
uv pip install setuptools
uv pip uninstall transformers
uv pip install --upgrade --no-cache-dir "git+https://github.com/huggingface/transformers.git"
uv pip uninstall numpy # 不支持numpy2
uv pip install numpy==1.24.0
git clone --recursive https://github.com/ggerganov/llama.cpp
下载 llama.cpp 的可执行程序包 https://github.com/ggerganov/llama.cpp/releases 解压覆盖至 llama.cpp 目录下
unsloth更新：
uv pip install --upgrade --no-cache-dir --no-deps unsloth unsloth_zoo
".venv\Lib\site-packages\unsloth\save.py" 1028行左右修改：
from platform import system as platform_system
if platform_system() == "Windows":
quantize_location = get_executable(["llama-quantize.exe", "quantize.exe"])
else:
quantize_location = get_executable(["llama-quantize", "quantize"])
".venv\Lib\site-packages\unsloth\save.py" 1072行左右添加：
elif os.path.exists("llama.cpp/quantize.exe"):
quantize_location = "llama.cpp/quantize.exe"
elif os.path.exists("llama.cpp/llama-quantize.exe"):
quantize_location = "llama.cpp/llama-quantize.exe"
微调训练代码
from unsloth import FastLanguageModel
# from unsloth import is_bfloat16_supported
import torch
from trl import SFTTrainer
from transformers import TrainingArguments
from datasets import load_dataset
from unsloth import to_sharegpt
from unsloth import standardize_sharegpt
model_name = r"E:\AI\models\Qwen2.5-7B-Instruct"
max_seq_length = 1024 # Supports RoPE Scaling interally, so choose any!
load_in_4bit = True # 在 4 位量化中进行微调
dtype = torch.float16 # 保留此值为 None，但可以选择 torch.float16 或 torch.bfloat16，适用于较新的 GPU
r = 8 # 调整微调过程的等级。数字越大，使用的内存越多，速度越慢，但可以在更困难的任务上提高准确性。我们通常建议使用 8（适用于快速微调）到 128 之间的数字。数字过大可能导致过拟合，损害模型质量。同步 LoRA 系数。
max_steps = 60 # num_train_epochs = 1
num_train_epochs = 3 # 训练轮数
learning_rate = 5e-5 # 学习率
datasetpath = r".\alpaca_dataset.json" # Get Alpaca dataset
output_dir = "outputs"
gguf_dir = "ggufs"
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = model_name,
max_seq_length = max_seq_length,
dtype = dtype,
load_in_4bit = load_in_4bit,
)
# Load Alpaca dataset
dataset = load_dataset("json", data_files = {"train" : datasetpath}, split = "train")
dataset = to_sharegpt(
dataset,
merged_prompt = "[[{system}\n]]{instruction}[[\n{input}]]",
output_column_name = "output",
)
dataset = standardize_sharegpt(dataset)
# Do model patching and add fast LoRA weights
model = FastLanguageModel.get_peft_model(
model,
r = r,
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",],
lora_alpha = r,
lora_dropout = 0, # Supports any, but = 0 is optimized
bias = "none", # Supports any, but = "none" is optimized
# [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes!
use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context
random_state = 3407,
max_seq_length = max_seq_length,
use_rslora = False, # We support rank stabilized LoRA
loftq_config = None, # And LoftQ
)
trainer = SFTTrainer(
model = model,
train_dataset = dataset,
dataset_text_field = None,
max_seq_length = max_seq_length,
tokenizer = tokenizer,
args = TrainingArguments(
per_device_train_batch_size = 2,
gradient_accumulation_steps = 4,
warmup_steps = 10,
max_steps = max_steps,
num_train_epochs = num_train_epochs,
learning_rate = learning_rate,
fp16 = True,
bf16 = False,
logging_steps = 1,
output_dir = output_dir,
optim = "adamw_8bit",
seed = 3407,
save_strategy = "steps",
save_steps = 50,
),
)
trainer.train()
# trainer.train(resume_from_checkpoint = True) # start from the latest checkpoint and continue training.
# Save the model
model.save_pretrained_gguf(gguf_dir, tokenizer, quantization_method = "q4_k_m")
手动量化
若前步报错无法完成量化，则执行以下命令进行量化
在 E:\AI\llama.cpp 目录下
.\llama.cpp\llama-quantize.exe "ggufs\unsloth.F16.gguf" "ggufs\unsloth.Q4_K_M.gguf" q4_k_m