Profile
Back to NewsBack
GitHub Trending 19 min
Reader Mode
hiyouga/LlamaFactory: Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

hiyouga/LlamaFactory: Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

!# LlamaFactory

GitHub Repo stars</a> GitHub last commit</a> GitHub contributors</a> GitHub workflow</a> PyPI</a> Citation</a> Docker Pulls</a>

Twitter</a> Discord</a> WeChat</a> Blog</a> GitCode</a>

Open in Colab</a> Open in DSW</a> Open in Spaces</a> Open in Studios</a> Open in Novita</a>

Used by Amazon, NVIDIA, Aliyun, etc.

----

Check our new open-source project —
🐧 PenguinHarness: Local-first multi-agent app development platform - create, fine-tune, deploy AI apps automatically!

Follow us and give us a star ⭐: https://github.com/Prism-Shadow/penguin-harness

https://github.com/user-attachments/assets/9b7033e8-f08a-4c3f-bd33-547896664e6e

----

Supporters ❤️

|

Warp sponsorship
Warp, the agentic terminal for developers
Available for MacOS, Linux, & Windows | SerpAPI sponsorship | | ---- | ---- |

----

Easily fine-tune 100+ large language models with zero-code CLI and Web UI

!GitHub Trend

👋 Join our WeChat and NPU user groups.

\ English | [中文 \]

Fine-tuning a large language model can be easy as...

https://github.com/user-attachments/assets/3991a3a8-4276-4d30-9cab-4cb0c4b9b99e

Start local training:

Start cloud training:
  • Colab (free): https://colab.research.google.com/drive/1eRTPn37ltBbYsISy9Aw2NuI2Aq5CQrD9?usp=sharing
  • PAI-DSW (free trial): https://gallery.pai-ml.com/#/preview/deepLearning/nlp/llama_factory
  • AMD GPU Cloud (free credits): https://github.com/AMD-AIM/AMD_Developers_Notebooks/blob/main/en/AMD_developer_LLaMAFactory_note_en.md
Read technical notes:
  • Documentation (WIP): https://llamafactory.readthedocs.io/en/latest/
  • Documentation (AMD GPU): https://rocm.docs.amd.com/projects/ai-developer-hub/en/latest/notebooks/fine_tune/llama_factory_llama3.html
  • Documentation (ASCEND NPU): https://llamafactory.readthedocs.io/en/latest/multibackend/npu/index.html
  • Official Blog: https://blog.llamafactory.net/en/
[!NOTE]
Except for the above links, all other websites are unauthorized third-party websites. Please carefully use them.

Table of Contents

- Installation - Data Preparation - Quickstart - Fine-Tuning with LLaMA Board GUI - Build Docker - Deploy with OpenAI-style API and vLLM - Download from ModelScope Hub - Download from Modelers Hub - Use W&B Logger - Use SwanLab Logger

Features

  • Various models: LLaMA, LLaVA, Mistral, Mixtral-MoE, Qwen3, Qwen3-VL, DeepSeek, Gemma, GLM, Phi, etc.
  • Integrated methods: (Continuous) pre-training, (multimodal) supervised fine-tuning, reward modeling, PPO, DPO, KTO, ORPO, etc.
  • Scalable resources: 16-bit full-tuning, freeze-tuning, LoRA and 2/3/4/5/6/8-bit QLoRA via AQLM/AWQ/GPTQ/LLM.int8/HQQ/EETQ.
  • Advanced algorithms: GaLore, BAdam, APOLLO, Adam-mini, Muon, OFT, DoRA, LongLoRA, LLaMA Pro, Mixture-of-Depths, LoRA+, LoftQ and PiSSA.
  • Practical tricks: FlashAttention-2, Unsloth, Liger Kernel, KTransformers, RoPE scaling, NEFTune and rsLoRA.
  • Wide tasks: Multi-turn dialogue, tool using, image understanding, visual grounding, video recognition, audio understanding, etc.
  • Experiment monitors: LlamaBoard, TensorBoard, Wandb, MLflow, SwanLab, etc.
  • Faster inference: OpenAI-style API, Gradio UI and CLI with vLLM worker or SGLang worker.

Day-N Support for Fine-Tuning Cutting-Edge Models

| Support Date | Model Name | | ------------ | -------------------------------------------------------------------- | | Day 0 | Qwen3 / Qwen2.5-VL / Gemma 3 / GLM-4.1V / InternLM 3 / MiniCPM-o-2.6 | | Day 1 | Llama 3 / GLM-4 / Mistral Small / PaliGemma2 / Llama 4 |

Blogs

[!TIP]
Now we have a dedicated blog for LlamaFactory!
> Website: https://blog.llamafactory.net/en/
All Blogs

Changelog

[25/10/26] We support Megatron-core training backend with mcore_adapter. See PR #9237 to get started.

[25/08/22] We supported OFT and OFTv2. See examples for usage.

[25/08/20] We supported fine-tuning the Intern-S1-mini models. See PR #8976 to get started.

[25/08/06] We supported fine-tuning the GPT-OSS models. See PR #8826 to get started.

Full Changelog

[25/07/02] We supported fine-tuning the GLM-4.1V-9B-Thinking model.

[25/04/28] We supported fine-tuning the Qwen3 model family.

[25/04/21] We supported the Muon optimizer. See examples for usage. Thank @tianshijing's PR.

[25/04/16] We supported fine-tuning the InternVL3 model. See PR #7258 to get started.

[25/04/14] We supported fine-tuning the GLM-Z1 and Kimi-VL models.

[25/04/06] We supported fine-tuning the Llama 4 model. See PR #7611 to get started.

[25/03/31] We supported fine-tuning the Qwen2.5 Omni model. See PR #7537 to get started.

[25/03/15] We supported SGLang as inference backend. Try infer_backend: sglang to accelerate inference.

[25/03/12] We supported fine-tuning the Gemma 3 model.

[25/02/24] Announcing EasyR1, an efficient, scalable and multi-modality RL training framework for efficient GRPO training.

[25/02/11] We supported saving the Ollama modelfile when exporting the model checkpoints. See examples for usage.

[25/02/05] We supported fine-tuning the Qwen2-Audio and MiniCPM-o-2.6 on audio understanding tasks.

[25/01/31] We supported fine-tuning the DeepSeek-R1 and Qwen2.5-VL models.

[25/01/15] We supported APOLLO optimizer. See examples for usage.

[25/01/14] We supported fine-tuning the MiniCPM-o-2.6 and MiniCPM-V-2.6 models. Thank @BUAADreamer's PR.

[25/01/14] We supported fine-tuning the InternLM 3 models. Thank @hhaAndroid's PR.

[25/01/10] We supported fine-tuning the Phi-4 model.

[24/12/21] We supported using SwanLab for experiment tracking and visualization. See this section for details.

[24/11/27] We supported fine-tuning the Skywork-o1 model and the OpenO1 dataset.

[24/10/09] We supported downloading pre-trained models and datasets from the Modelers Hub. See this tutorial for usage.

[24/09/19] We supported fine-tuning the Qwen2.5 models.

[24/08/30] We supported fine-tuning the Qwen2-VL models. Thank @simonJJJ's PR.

[24/08/27] We supported Liger Kernel. Try enable_liger_kernel: true for efficient training.

[24/08/09] We supported Adam-mini optimizer. See examples for usage. Thank @relic-yuexi's PR.

[24/07/04] We supported contamination-free packed training. Use neat_packing: true to activate it. Thank @chuan298's PR.

[24/06/16] We supported PiSSA algorithm. See examples for usage.

[24/06/07] We supported fine-tuning the Qwen2 and GLM-4 models.

[24/05/26] We supported SimPO algorithm for preference learning. See examples for usage.

[24/05/20] We supported fine-tuning the PaliGemma series models. Note that the PaliGemma models are pre-trained models, you need to fine-tune them with paligemma template for chat completion.

[24/05/18] We supported KTO algorithm for preference learning. See examples for usage.

[24/05/14] We supported training and inference on the Ascend NPU devices. Check installation section for details.

[24/04/26] We supported fine-tuning the LLaVA-1.5 multimodal LLMs. See examples for usage.

[24/04/22] We provided a Colab notebook for fine-tuning the Llama-3 model on a free T4 GPU. Two Llama-3-derived models fine-tuned using LlamaFactory are available at Hugging Face, check Llama3-8B-Chinese-Chat and Llama3-Chinese for details.

[24/04/21] We supported Mixture-of-Depths according to AstraMindAI's implementation. See examples for usage.

[24/04/16] We supported BAdam optimizer. See examples for usage.

[24/04/16] We supported unsloth's long-sequence training (Llama-2-7B-56k within 24GB). It achieves 117% speed and 50% memory compared with FlashAttention-2, more benchmarks can be found in this page.

[24/03/31] We supported ORPO. See examples for usage.

[24/03/21] Our paper "LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models" is available at arXiv!

[24/03/20] We supported FSDP+QLoRA that fine-tunes a 70B model on 2x24GB GPUs. See examples for usage.

[24/03/13] We supported LoRA+. See examples for usage.

[24/03/07] We supported GaLore optimizer. See examples for usage.

[24/03/07] We integrated vLLM for faster and concurrent inference. Try infer_backend: vllm to enjoy 270% inference speed.

[24/02/28] We supported weight-decomposed LoRA (DoRA). Try use_dora: true to activate DoRA training.

[24/02/15] We supported block expansion proposed by LLaMA Pro. See examples for usage.

[24/02/05] Qwen1.5 (Qwen2 beta version) series models are supported in LlamaFactory. Check this blog post for details.

[24/01/18] We supported agent tuning for most models, equipping model with tool using abilities by fine-tuning with dataset: glaive_toolcall_en.

[23/12/23] We supported unsloth's implementation to boost LoRA tuning for the LLaMA, Mistral and Yi models. Try use_unsloth: true argument to activate unsloth patch. It achieves 170% speed in our benchmark, check this page for details.

[23/12/12] We supported fine-tuning the latest MoE model Mixtral 8x7B in our framework. See hardware requirement here.

[23/12/01] We supported downloading pre-trained models and datasets from the ModelScope Hub. See this tutorial for usage.

[23/10/21] We supported NEFTune trick for fine-tuning. Try neftune_noise_alpha: 5 argument to activate NEFTune.

[23/09/27] We supported $S^2$-Attn proposed by LongLoRA for the LLaMA models. Try shift_attn: true argument to enable shift short attention.

[23/09/23] We integrated MMLU, C-Eval and CMMLU benchmarks in this repo. See examples for usage.

[23/09/10] We supported FlashAttention-2. Try flash_attn: fa2 argument to enable FlashAttention-2 if you are using RTX4090, A100 or H100 GPUs.

[23/08/12] We supported RoPE scaling to extend the context length of the LLaMA models. Try rope_scaling: linear argument in training and rope_scaling: dynamic argument at inference to extrapolate the position embeddings.

[23/08/11] We supported DPO training for instruction-tuned models. See examples for usage.

[23/07/31] We supported dataset streaming. Try streaming: true and max_steps: 10000 arguments to load your dataset in streaming mode.

[23/07/29] We released two instruction-tuned 13B models at Hugging Face. See these Hugging Face Repos (LLaMA-2 / Baichuan) for details.

[23/07/18] We developed an all-in-one Web UI for training, evaluation and inference. Try train_web.py to fine-tune models in your Web browser. Thank @KanadeSiina and @codemayq for their efforts in the development.

[23/07/09] We released FastEdit ⚡🩹, an easy-to-use package for editing the factual knowledge of large language models efficiently. Please follow FastEdit if you are interested.

[23/06/29] We provided a reproducible example of training a chat model using instruction-following datasets, see Baichuan-7B-sft for details.

[23/06/22] We aligned the demo API with the OpenAI's format where you can insert the fine-tuned model in arbitrary ChatGPT-based applications.

[23/06/03] We supported quantized training and inference (aka QLoRA). See examples for usage.

[!TIP]
If you cannot use the latest feature, please pull the latest code and install LlamaFactory again.

Supported Models

| Model | Model size | Template | | ----------------------------------------------------------------- | -------------------------------- | -------------------- | | BLOOM/BLOOMZ | 560M/1.1B/1.7B/3B/7.1B/176B | - | | DeepSeek (LLM/Code/MoE) | 7B/16B/67B/236B | deepseek | | DeepSeek 3-3.2 | 236B/671B | deepseek3 | | DeepSeek R1 (Distill) | 1.5B/7B/8B/14B/32B/70B/671B | deepseekr1 | | ERNIE-4.5 | 0.3B/21B/300B | ernie_nothink | | Falcon/Falcon H1 | 0.5B/1.5B/3B/7B/11B/34B/40B/180B | falcon/falcon_h1 | | Gemma/Gemma 2/CodeGemma | 2B/7B/9B/27B | gemma/gemma2 | | Gemma 3/Gemma 3n | 270M/1B/4B/6B/8B/12B/27B | gemma3/gemma3n | | GLM-4/GLM-4-0414/GLM-Z1 | 9B/32B | glm4/glmz1 | | GLM-4.5/GLM-4.5(6)V | 9B/106B/355B | glm4_moe/glm4_5v | | GPT-2 | 0.1B/0.4B/0.8B/1.5B | - | | GPT-OSS | 20B/120B | gpt_oss | | Granite 3-4 | 1B/2B/3B/7B/8B | granite3/granite4 | | Hunyuan/Hunyuan1.5 (MT) | 0.5B/1.8B/4B/7B/13B | hunyuan/hunyuan_small| | InCoder-32B | 32B | Tokenizer template (v1) | | InCoder-32B-Thinking | 32B | Tokenizer template (v1) | | InternLM 2-3 | 7B/8B/20B | intern2 | | InternVL 2.5-3.5 | 1B/2B/4B/8B/14B/30B/38B/78B/241B | intern_vl | | Intern-S1-mini | 8B | intern_s1 | | IQuest-Coder (V1 Instruct/Thinking) | 7B/14B/40B | Tokenizer template (v1) | | IQuest-Q1 | 320B-A15B | Upstream template (fine-tuning unverified) | | Kimi-VL | 16B | kimi_vl | | Ling 2.0 (mini/flash) | 16B/100B | bailing_v2 | | LFM 2.5 (VL) | 1.2B/1.6B | lfm2/lfm2_vl | | Llama | 7B/13B/33B/65B | - | | Llama 2 | 7B/13B/70B | llama2 | | Llama 3-3.3 | 1B/3B/8B/70B | llama3 | | Llama 4 | 109B/402B | llama4 | | Llama 3.2 Vision | 11B/90B | mllama | | LLaVA-1.5 | 7B/13B | llava | | LLaVA-NeXT | 7B/8B/13B/34B/72B/110B | llava_next | | LLaVA-NeXT-Video | 7B/34B | llava_next_video | | MiMo | 7B/309B | mimo/mimo_v2 | | MiniCPM 4/5 | 0.5B/1B/8B | cpm4/minicpm5 | | MiniCPM-o/MiniCPM-V 4.5 | 8B/9B | minicpm_o/minicpm_v | | MiniCPM-V 4.6 | 3B/8B | minicpm_v_4_6 | | MiniMax-M1/MiniMax-M2 | 229B/456B | minimax1/minimax2 | | Ministral 3 | 3B/8B/14B | ministral3 | | Mistral/Mixtral | 7B/8x7B/8x22B | mistral | | PaliGemma/PaliGemma2 | 3B/10B/28B | paligemma | | Phi-3/Phi-3.5 | 4B/14B | phi | | Phi-3-small | 7B | phi_small | | Phi-4-mini/Phi-4 | 3.8B/14B | phi4_mini/phi4 | | Pixtral | 12B | pixtral | | Qwen2 (Code/Math/MoE/QwQ) | 0.5B/1.5B/3B/7B/14B/32B/72B/110B | qwen | | Qwen3 (MoE/Instruct/Thinking/Next) | 0.6B/1.7B/4B/8B/14B/32B/80B/235B | qwen3/qwen3_nothink | | Qwen3.5 | 0.8B/2B/4B/9B/27B/35B/122B/397B | qwen3_5/qwen3_5_nothink | | Qwen3.6 | 27B/35B | qwen3_6 | | Qwen3.8 | 27B/2.4T | qwen3_8 | | Qwen2-Audio | 7B | qwen2_audio | | Qwen2.5-Omni | 3B/7B | qwen2_omni | | Qwen3-Omni | 30B | qwen3_omni | | Qwen2-VL/Qwen2.5-VL/QVQ | 2B/3B/7B/32B/72B | qwen2_vl | | Qwen3-VL | 2B/4B/8B/30B/32B/235B | qwen3_vl | | Seed (OSS/Coder) | 8B/36B | seed_oss/seed_coder | | StarCoder 2 | 3B/7B/15B | - | | TeleChat 2-2.5 | 3B/7B/35B/115B | telechat2 | | Xing4.0-29B-A4B | 29B-A4B | xing4_0 | | Yuan 2 | 2B/51B/102B | yuan |

[!NOTE]
For the "base" models, the template argument can be chosen from default, alpaca, vicuna etc. But make sure to use the corresponding template for the "instruct/chat" models.
> If the model has both reasoning and non-reasoning versions, please use the _nothink suffix to distinguish between them. For example, qwen3 and qwen3_nothink.
> Remember to use the SAME template in training and inference.
> \*: You should install the transformers from main branch and use DISABLE_VERSION_CHECK=1 to skip version check.
> \\: You need to install a specific version of transformers to use the corresponding model.

Please refer to constants.py for a full list of models we supported.

You also can add a custom chat template to template.py.

Supported Training Approaches

| Approach | Full-tuning | Freeze-tuning | LoRA | QLoRA | OFT | QOFT | | ---------------------- | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ | | Pre-Training | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | Supervised Fine-Tuning | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | Reward Modeling | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | PPO Training | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | DPO Training | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | KTO Training | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | ORPO Training | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | SimPO Training | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |

[!TIP]
The implementation details of PPO can be found in this blog.

Provided Datasets

Pre-training datasets

Supervised fine-tuning datasets

Preference datasets

Some datasets require confirmation before using them, so we recommend logging in with your Hugging Face account using these commands.

pip install "huggingface_hub<1.0.0"
huggingface-cli login

Requirement

| Mandatory | Minimum | Recommend | | ------------ | ------- | --------- | | python | 3.11 | >=3.11 | | torch | 2.0.0 | 2.6.0 | | torchvision | 0.15.0 | 0.21.0 | | transformers | 4.49.0 | 4.50.0 | | datasets | 2.16.0 | 3.2.0 | | accelerate | 0.34.0 | 1.2.1 | | peft | 0.14.0 | 0.15.1 | | trl | 0.8.6 | 0.9.6 |

| Optional | Minimum | Recommend | | ------------ | ------- | --------- | | CUDA | 11.6 | 12.2 | | deepspeed | 0.10.0 | 0.16.4 | | bitsandbytes | 0.39.0 | 0.43.1 | | vllm | 0.4.3 | 0.8.2 | | flash-attn | 2.5.6 | 2.7.2 |

Hardware Requirement

\ estimated*

| Method | Bits | 7B | 14B | 30B | 70B | xB | | ----------------------------------- | ---- | ----- | ----- | ----- | ------ | ------- | | Full (bf16 or fp16) | 32 | 120GB | 240GB | 600GB | 1200GB | 18xGB | | Full (pure_bf16) | 16 | 60GB | 120GB | 300GB | 600GB | 8xGB | | Freeze/LoRA/GaLore/APOLLO/BAdam/OFT | 16 | 16GB | 32GB | 64GB | 160GB | 2xGB | | QLoRA / QOFT | 8 | 10GB | 20GB | 40GB | 80GB | xGB | | QLoRA / QOFT | 4 | 6GB | 12GB | 24GB | 48GB | x/2GB | | QLoRA / QOFT | 2 | 4GB | 8GB | 16GB | 24GB | x/4GB |

Getting Started

Installation

[!IMPORTANT]
Installation is mandatory.

Install from Source

git clone --depth 1 https://github.com/hiyouga/LlamaFactory.git
cd LlamaFactory
pip install -e .
pip install -r requirements/metrics.txt

Optional dependencies available: metrics, deepspeed. Install with: pip install -e . && pip install -r requirements/metrics.txt -r requirements/deepspeed.txt

Additional dependencies for specific features are available in requirements/.

Install from Docker Image

docker run -it --rm --gpus=all --ipc=host hiyouga/llamafactory:latest

This image is built on Ubuntu 22.04 (x86\_64), CUDA 12.4, Python 3.11, PyTorch 2.6.0, and Flash-attn 2.7.4.

Find the pre-built images: https://hub.docker.com/r/hiyouga/llamafactory/tags

Please refer to build docker to build the image yourself.

Setting up a virtual environment with uv

Create an isolated Python environment with uv:

uv run llamafactory-cli webui

For Windows users

Install PyTorch

You need to manually install the GPU version of PyTorch on the Windows platform. Please refer to the official website and the following command to install PyTorch with CUDA support:

pip uninstall torch torchvision torchaudio
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
python -c "import torch; print(torch.cuda.is_available())"

If you see True then you have successfully installed PyTorch with CUDA support.

Try dataloader_num_workers: 0 if you encounter Can't pickle local object error.

Install BitsAndBytes

To enable Quantized LoRA (QLoRA) on Windows, you need to install bitsandbytes.

For most users, it is recommended to install the latest official release:

pip install bitsandbytes

If you are using uv to manage your virtual environment, it is recommended to install bitsandbytes after installing the GPU-enabled version of PyTorch:

uv pip install bitsandbytes --no-deps

[!IMPORTANT] Pay attention to the CUDA Toolkit version when installing bitsandbytes. Official bitsandbytes releases are built for specific CUDA Toolkit versions. On Windows x86-64, separate builds are currently provided for CUDA 11.8–12.6 and CUDA 12.8–12.9. Support for NVIDIA RTX 50 Series GPUs (e.g., RTX 5060 Ti, sm_120) requires the CUDA 12.8–12.9 builds.

If your environment uses an older CUDA version, or you need compatibility with older Windows / PyTorch combinations, you can install the third-party precompiled Windows wheel:

pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.2.post2-py3-none-win_amd64.whl

Install Flash Attention-2

To enable FlashAttention-2 on the Windows platform, please use the script from flash-attention-windows-wheel to compile and install it by yourself.

For Ascend NPU users

To install LlamaFactory on Ascend NPU devices, please use Python 3.12 and install the extra dependencies with pip install -r requirements/npu.txt. Additionally, you need to install the Ascend CANN Toolkit and Kernels. Please follow the installation tutorial.

You can also download the pre-built Docker images:

# Docker Hub
docker pull hiyouga/llamafactory:latest-910b-ubuntu
docker pull hiyouga/llamafactory:latest-a3-ubuntu
docker pull hiyouga/llamafactory:latest-950-ubuntu
docker pull hiyouga/llamafactory:latest-910b-openeuler
docker pull hiyouga/llamafactory:latest-a3-openeuler
docker pull hiyouga/llamafactory:latest-950-openeuler

quay.io

docker pull quay.io/ascend/llamafactory:latest-910b-ubuntu docker pull quay.io/ascend/llamafactory:latest-a3-ubuntu docker pull quay.io/ascend/llamafactory:latest-950-ubuntu docker pull quay.io/ascend/llamafactory:latest-910b-openeuler docker pull quay.io/ascend/llamafactory:latest-a3-openeuler docker pull quay.io/ascend/llamafactory:latest-950-openeuler

Install BitsAndBytes

To use QLoRA based on bitsandbytes on Ascend NPU, please follow these 3 steps:

  1. Manually compile bitsandbytes: Refer to the installation documentation for the NPU version of bitsandbytes to complete the compilation and installation. The compilation requires a cmake version of at least 3.22.1 and a g++ version of at least 12.x.
# Install bitsandbytes from source

Clone bitsandbytes repo, Ascend NPU backend is currently enabled on multi-backend-refactor branch

git clone -b multi-backend-refactor https://github.com/bitsandbytes-foundation/bitsandbytes.git cd bitsandbytes/

Install dependencies

pip install -r requirements-dev.txt

Install the dependencies for the compilation tools. Note that the commands for this step may vary depending on the operating system. The following are provided for reference

apt-get install -y build-essential cmake

Compile & install

cmake -DCOMPUTE_BACKEND=npu -S . make pip install .
  1. Install transformers from the main branch.
git clone -b main https://github.com/huggingface/transformers.git
cd transformers
pip install .
  1. Set double_quantization: false in the configuration. You can refer to the example.

Data Preparation

Please refer to data/README.md for checking the details about the format of dataset files. You can use datasets on HuggingFace / ModelScope / Modelers hub, load the dataset in local disk, or specify a path to s3/gcs cloud storage.

[!NOTE]
Please update data/dataset_info.json to use your custom dataset.

You can also use Easy Dataset, DataFlow and GraphGen to create synthetic data for fine-tuning.

Quickstart

Use the following 3 commands to run LoRA fine-tuning, inference and merging of the Qwen3-4B-Instruct model, respectively.

llamafactory-cli train examples/train_lora/qwen3_lora_sft.yaml
llamafactory-cli chat examples/inference/qwen3_lora_sft.yaml
llamafactory-cli export examples/merge_lora/qwen3_lora_sft.yaml

See examples/README.md for advanced usage (including distributed training).

[!TIP]
Use llamafactory-cli help to show help information.
> Read FAQs first if you encounter any problems.

Fine-Tuning with LLaMA Board GUI (powered by Gradio)

llamafactory-cli webui

Build Docker

For CUDA users:

cd docker/docker-cuda/
docker compose up -d
docker compose exec llamafactory bash

For Ascend NPU users:

cd docker/docker-npu/

A2 with Ubuntu

docker compose --profile a2-ubuntu up -d docker compose --profile a2-ubuntu exec llamafactory-a2-ubuntu bash

A3 with Ubuntu

docker compose --profile a3-ubuntu up -d docker compose --profile a3-ubuntu exec llamafactory-a3-ubuntu bash

A2 with openEuler

docker compose --profile a2-openeuler up -d docker compose --profile a2-openeuler exec llamafactory-a2-openeuler bash

A3 with openEuler

docker compose --profile a3-openeuler up -d docker compose --profile a3-openeuler exec llamafactory-a3-openeuler bash

For AMD ROCm users:

cd docker/docker-rocm/
docker compose up -d
docker compose exec llamafactory bash

Build without Docker Compose

For CUDA users:

docker build -f ./docker/docker-cuda/Dockerfile \
    --build-arg PIP_INDEX=https://pypi.org/simple \
    -t llamafactory:latest .

docker run -dit --ipc=host --gpus=all \ -p 7860:7860 \ -p 8000:8000 \ --name llamafactory \ llamafactory:latest

docker exec -it llamafactory bash

For Ascend NPU users:

```bash docker build -f ./docker/docker-npu/Dockerfile \ --build-arg PIP_INDEX=https://pypi.org/simple \

... (README truncated for length)

Chat with me