Profile
Back to NewsBack
GitHub Trending 20 min
Reader Mode
modelscope/DiffSynth-Studio: Enjoy the magic of Diffusion models!

modelscope/DiffSynth-Studio: Enjoy the magic of Diffusion models!

8 hours ago
GitHub Trending

DiffSynth-Studio

modelscope%2FDiffSynth-Studio | Trendshift

PyPI</a> license</a> open issues</a> GitHub pull-requests</a> GitHub latest commit</a> Discord</a>

切换到中文版

Introduction

Welcome to the magical world of Diffusion models! DiffSynth-Studio is an open-source Diffusion model engine developed and maintained by the ModelScope Community team. We hope to foster technological innovation through framework construction, aggregate the power of the open-source community, and explore the rich capabilities of generative model technology!

Framework features:

  • Model Support: The framework integrates mainstream open-source Diffusion models, covering image generation, video generation, audio generation, and image quality metrics models.
  • VRAM Management: Dynamically schedules model parameters across disk, memory, and VRAM, allowing consumer-grade GPUs with low VRAM to run inference with large models.
  • Parameter Quantization: Converts model parameters to quantized precisions such as NF4 and INT8, significantly reducing the VRAM requirements of model inference and LoRA training.
  • Arbitrary Training: Almost every model that supports inference also supports training, whether base models, LoRAs, or any Adapter models with additional inputs.
  • Split Training: Uses a computational graph inference engine to track every variable in the Pipeline, splitting the training process into two stages for efficient training.
References: See also:
  • DiffSynth-ComfyUI: A ComfyUI node package built on DiffSynth-Studio, enabling you to build workflows in ComfyUI to run model inference.
  • DiffSynth-WebUI: A lightweight LoRA training tool built on DiffSynth-Studio, enabling LoRA training of models on consumer-grade GPUs.
  • ModelScope AIGC Zone (for Chinese users): Productized features powered by DiffSynth-Studio as the core inference and training engine; experience the rich potential of the open-source model ecosystem.
  • ModelScope Civision (for global users): Unlock the vast potential of the open-source model ecosystem through productized capabilities powered by DiffSynth-Studio.

Update History

DiffSynth-Studio has undergone major version updates, and some old features are no longer maintained. If you need to use old features, please switch to the last historical version before the major version update.
Currently, the development personnel of this project are limited, with most of the work handled by Artiprocher and mi804. Therefore, the progress of new feature development will be relatively slow, and the speed of responding to and resolving issues is limited. We apologize for this and ask developers to understand.
  • October 21, 2026 We now support Fibo 2.0, including unified text-to-image and image editing inference, masked and multi-reference editing, text rendering, low VRAM inference, and full/LoRA training for both text-to-image and image editing on Fibo-2.0-dev. See the documentation and example code.
  • October 16, 2026 flux-3-image-klein open-sourced, welcome a new member to the image model family! Support includes text-to-image generation, multi-reference editing, layout control, low VRAM inference, and Base training capabilities. For details, please refer to the documentation and example code. We quantized the model with EntroPack, providing pre-quantized models in 3-bit, 4-bit, and 6-bit versions at DiffSynth-Studio/flux-3-image-klein-entropack.
  • October 12, 2026 We now support distillation accelerated training based on OPD (On-Policy Distillation), which enables distillation acceleration of models at an extremely low cost. For details, please refer to Distillation Accelerated Training.
  • September 29, 2026 We are excited to open-source EntroPack, a general-purpose tensor compression library for PyTorch that provides lossless compression with exact recovery and rate-controlled lossy compression, with GPU encoding and decoding. It is integrated into DiffSynth-Studio as a quantization backend, compressing model weights at arbitrary target bitrates in both inference and LoRA training. For more information, please refer to:
* Project Page: https://github.com/modelscope/entropack * Technical Report: https://arxiv.org/abs/2609.34185 * Pre-quantized Models: https://www.modelscope.cn/collections/DiffSynth-Studio/EntroPack-Quantized-Models * Example Code: Z-Image-Turbo, Qwen-Image-2.1, MiniMax-H3, LoRA Training Example
  • September 20, 2026 Qwen-Image-2.1 was open-sourced. The model supports text-to-image generation, image editing, and RGBA output with a transparency channel. DiffSynth-Studio now provides full support for it. For details, please refer to the documentation and example code.
  • September 14, 2026 We trained and open-sourced DiffSynth-Music, a controllable music generation model based on ACE-Step, which supports five control modes: Beats, Vocals, Accompany, Prosody, and Reference. For more information, please refer to:
* Model: https://modelscope.cn/models/DiffSynth-Studio/DiffSynth-Music * Technical Report: https://arxiv.org/abs/2609.12774 * Code: ./examples/diffsynth_music/
  • September 11, 2026 We have added support for YuE2, a new music generation model. For details, please refer to the documentation and example code.
  • September 10, 2026 We have integrated LTX-2.5, the latest audio-video generation model from Lightricks. The features include text-to-audio/video with automatic duration prediction, image-to-audio/video with keyframe interpolation, audio-to-video, audio-video retake, IC-LoRA pixel spatial upscaling, text-to-audio, INT8 quantized inference, low VRAM inference, and training. For details, please refer to the documentation and code.
  • September 10, 2026 We have open-sourced DiffSynth-ComfyUI. Now you can run model inference with DiffSynth-Studio in ComfyUI.
  • September 1, 2026 We have integrated SenseNova-U1.5, SenseTime's unified multimodal model, for which we provide text-to-image generation, image editing, low VRAM inference, and training support. For details, please refer to the documentation and example code.
  • August 31, 2026 We have integrated Qwen-Video-Edit, a video editing model developed by open-source community contributor yunpeng1998 based on the image editing model Qwen-Image-Edit. This serves as an excellent example of exploring and expanding model capabilities.
  • August 25, 2026 We have open-sourced DiffSynth-WebUI, enabling one-click private deployment of LoRA training services. Combined with the model quantization feature, you can train large models even with consumer-grade GPUs.
  • August 19, 2026 We have released the model quantization feature. It provides a unified QuantizeConfig entry point supporting multiple quantization backends including bitsandbytes, torchao, and comfy-kitchen, with capabilities such as online quantization, loading pre-quantized weights, mixed quantization, saving quantized models, and quantization + LoRA training. For details, please refer to the documentation.
More
  • August 17, 2026 MiniMax-Music3 open-sourced, welcome a new member to the audio model family! Support includes text-to-music generation and low VRAM inference. For details, please refer to the documentation and example code.
  • August 7, 2026 We add support for Wan-Animate-2 in the Wan series. Given a reference image and a driving video, it makes the reference character perform the motions in the driving video, generating high-quality character animation, with both standard and distilled variants. For details, please refer to the documentation and example code.
  • August 3, 2026 MiniMax-H3 open-sourced, welcome a new member to the video model family! Support includes text-to-video-audio generation, keyframe-guided generation, reference-driven generation, low VRAM inference, and NF4-quantized inference. For details, please refer to the documentation and example code.
  • July 28, 2026 LingBot-Video open-sourced, welcome a new member to the video model family! This release includes two variants, Dense-1.3B and MoE-30B-A3B (30B total parameters, ~3B active per token), both supporting text-to-video, image-to-video and text-to-image generation, low VRAM inference, and LoRA / full training capabilities. For details, please refer to the documentation and example code. Huge thanks to NancyFyong for contributing the integration of this model!
  • July 21, 2026 We have open-sourced DiffSynth-Studio Model Integration Skills. This is a composable collection of Agent Skills that automates the entire workflow of integrating external diffusion models into DiffSynth-Studio, significantly improving the standardization and efficiency of model integration. Get started with the example!
  • June 29, 2026 Boogu-Image open-sourced. Support includes text-to-image generation, image editing, low VRAM inference, and training capabilities. For details, please refer to the documentation and example code.
  • June 24, 2026 Krea-2 is now open-source, and we have provided full support. For more details, please refer to the documentation and example code.
  • June 15, 2026 We have open-sourced Image-to-LoRA V2, compressing the hours-long training process for image style LoRAs into a single model inference step, thereby exploring a new paradigm for LoRA model training. The technical report has been released. This release includes three models:
* DiffSynth-Studio/ZImage-i2L-v2: Adapted for the Z-Image model * DiffSynth-Studio/KleinBase4B-i2L-v2: Adapted for the FLUX.2-klein-base-4B model * DiffSynth-Studio/HidreamO1-i2L-v2: Adapted for the Hidream-O1-Image model
  • June 5, 2026 Ideogram 4 open-sourced. Support includes text-to-image inference. For details, please refer to the documentation and example code.
  • May 21, 2026: Added support for image quality metrics models, including FID, CLIP, Aesthetic, PickScore, ImageReward, HPSv2, and HPSv3. For details, refer to the documentation and example code.
  • May 18, 2026 Added CPU Offload Training support. By moving model weights layer-by-layer between CPU and GPU, it significantly reduces GPU VRAM usage during training, enabling LoRA training of large models even on consumer-grade GPUs, compatible with all models. Simply add --enable_model_cpu_offload to your training command to enable (currently supports single-GPU training only). For details, see the documentation.
  • May 14, 2026 HiDream-O1-Image open-sourced, welcome a new member to the image model family! Support includes text-to-image generation, image editing, low VRAM inference, and training capabilities. For details, please refer to the documentation and example code.
  • April 28, 2026 We released Diffusion Templates, a plugin framework designed for Diffusion models that significantly lowers the barrier to training controllable generative models. Let's explore this cutting-edge technology together!
* Open-source code: DiffSynth-Studio * Technical report: arXiv * Project homepage: GitHub * Documentation: English Version | Chinese Version * Online demo: ModelScope * Model collections: ModelScope | ModelScope International | HuggingFace * Datasets: ModelScope | ModelScope International | HuggingFace
  • April 27, 2026 We support ACE-Step-1.5! Support includes text-to-music generation, low VRAM inference, and LoRA training capabilities. For details, please refer to the documentation and example code.
  • April 27, 2026: We have reinstated support for the Stable Diffusion v1.5 and SDXL models, providing academic research support exclusively for these two model types.
  • April 14, 2026 JoyAI-Image open-sourced, welcome a new member to the image editing model family! Support includes instruction-guided image editing, low VRAM inference, and training capabilities. For details, please refer to the documentation and example code.
  • March 12, 2026: We have added support for the LTX-2.3 audio-video generation model. The features includes text-to-audio/video, image-to-audio/video, IC-LoRA control, audio-to-video, and audio-video inpainting. We have supported the complete inference and training functionalities. For details, please refer to the documentation and code.
  • March 3, 2026: We released the DiffSynth-Studio/Qwen-Image-Layered-Control-V2 model, which is an updated version of Qwen-Image-Layered-Control. In addition to the originally supported text-guided functionality, it adds brush-controlled layer separation capabilities.
  • March 2, 2026 Added support for Anima. For details, please refer to the documentation. This is an interesting anime-style image generation model. We look forward to its future updates.
  • February 26, 2026 Added full and lora training support for the LTX-2 audio-video generation model. See the documentation for details.
  • February 10, 2026 Added inference support for the LTX-2 audio-video generation model. See the documentation for details. Support for model training will be implemented in the future.
  • February 2, 2026 The first document of the Research Tutorial series is now available, guiding you through training a small 0.1B text-to-image model from scratch. For details, see the documentation and model. We hope DiffSynth-Studio can evolve into a more powerful training framework for Diffusion models.
  • January 12, 2026: We trained and open-sourced a text-guided image layer separation model (Model Link). Given an input image and a textual description, the model isolates the image layer corresponding to the described content. For more details, please refer to our blog post (Chinese version, English version).
  • December 24, 2025: Based on Qwen-Image-Edit-2511, we trained an In-Context Editing LoRA model (Model Link). This model takes three images as input (Image A, Image B, and Image C), and automatically analyzes the transformation from Image A to Image B, then applies the same transformation to Image C to generate Image D. For more details, please refer to our blog post (Chinese version, English version).
  • December 9, 2025 We release a wild model based on DiffSynth-Studio 2.0: Qwen-Image-i2L (Image-to-LoRA). This model takes an image as input and outputs a LoRA. Although this version still has significant room for improvement in terms of generalization, detail preservation, and other aspects, we are open-sourcing these models to inspire more innovative research. For more details, please refer to our blog.
  • December 4, 2025 DiffSynth-Studio 2.0 released! Many new features online
- Documentation online: Our documentation is still continuously being optimized and updated - VRAM Management module upgraded, supporting layer-level disk offload, releasing both memory and VRAM simultaneously - New model support - Z-Image Turbo: Model, Documentation, Code - FLUX.2-dev: Model, Documentation, Code - Training framework upgrade - Split Training: Supports automatically splitting the training process into two stages: data processing and training (even for training ControlNet or any other model). Computations that do not require gradient backpropagation, such as text encoding and VAE encoding, are performed during the data processing stage, while other computations are handled during the training stage. Faster speed, less VRAM requirement. - Differential LoRA Training: This is a training technique we used in ArtAug, now available for LoRA training of any model. - FP8 Training: FP8 can be applied to any non-training model during training, i.e., models with gradients turned off or gradients that only affect LoRA weights.
  • November 4, 2025 Supported the ByteDance/Video-As-Prompt-Wan2.1-14B model, which is trained based on Wan 2.1 and supports generating corresponding actions based on reference videos.
  • October 30, 2025 Supported the meituan-longcat/LongCat-Video model, which supports text-to-video, image-to-video, and video continuation. This model uses the Wan framework for inference and training in this project.
  • September 23, 2025 DiffSynth-Studio/Qwen-Image-EliGen-Poster released! This model was jointly developed and open-sourced by us and Taobao Experience Design Team. Built upon Qwen-Image, the model is specifically designed for e-commerce poster scenarios, supporting precise partition layout control. Please refer to our sample code.
  • September 9, 2025 Our training framework supports various training modes. Currently adapted for Qwen-Image, in addition to the standard SFT training mode, Direct Distill is now supported. Please refer to our sample code. This feature is experimental, and we will continue to improve it to support more comprehensive model training functions.
  • August 28, 2025 We support Wan2.2-S2V, an audio-driven cinematic video generation model. See ./examples/wanvideo/.
  • August 19, 2025 Qwen-Image-Edit open-sourced, welcome a new member to the image editing model family!
  • August 15, 2025 We open-sourced the Qwen-Image-Self-Generated-Dataset dataset. This is an image dataset generated using the Qwen-Image model, containing 160,000 1024 x 1024 images. It includes general, English text rendering, and Chinese text rendering subsets. We provide annotations for image descriptions, entities, and structural control images for each image. Developers can use this dataset to train Qwen-Image models' ControlNet and EliGen models. We aim to promote technological development through open-sourcing!
  • August 4, 2025 Qwen-Image open-sourced, welcome a new member to the image generation model family!
  • August 1, 2025 FLUX.1-Krea-dev open-sourced, a text-to-image model focused on aesthetic photography. We provided comprehensive support in a timely manner, including low VRAM layer-by-layer offload, LoRA training, and full training. For more details, please refer to ./examples/flux/.
  • July 28, 2025 Wan 2.2 open-sourced. We provided comprehensive support in a timely manner, including low VRAM layer-by-layer offload, FP8 quantization, sequence parallelism, LoRA training, and full training. For more details, please refer to ./examples/wanvideo/.
  • July 11, 2025 We propose Nexus-Gen, a unified framework that combines the language reasoning capabilities of Large Language Models (LLMs) with the image generation capabilities of diffusion models. This framework supports seamless image understanding, generation, and editing tasks.
- Paper: Nexus-Gen: Unified Image Understanding, Generation, and Editing via Prefilled Autoregression in Shared Embedding Space - GitHub Repository: https://github.com/modelscope/Nexus-Gen - Model: ModelScope, HuggingFace - Training Dataset: ModelScope Dataset - Online Experience: ModelScope Nexus-Gen Studio
  • June 15, 2025 ModelScope's official evaluation framework EvalScope now supports text-to-image generation evaluation. Please refer to the best practices guide to try it out.
  • March 25, 2025 Our new open-source project DiffSynth-Engine is now open-sourced! Focused on stable model deployment, targeting industry, providing better engineering support, higher computational performance, and more stable features.
  • March 31, 2025 We support InfiniteYou, a face feature preservation method for FLUX. More details can be found in ./examples/InfiniteYou/.
  • March 13, 2025 We support HunyuanVideo-I2V, the image-to-video generation version of Tencent's open-source HunyuanVideo. More details can be found in ./examples/HunyuanVideo/.
  • February 25, 2025 We support Wan-Video, a series of state-of-the-art video synthesis models open-sourced by Alibaba. See ./examples/wanvideo/.
  • December 31, 2024 We propose EliGen, a new framework for entity-level controlled text-to-image generation, supplemented with an inpainting fusion pipeline, extending its capabilities to image inpainting tasks. EliGen can seamlessly integrate existing community models such as IP-Adapter and In-Context LoRA, enhancing their versatility. For more details, see ./examples/EntityControl.
- Paper: EliGen: Entity-Level Controlled Image Generation with Regional Attention - Model: ModelScope, HuggingFace - Online Experience: ModelScope EliGen Studio - Training Dataset: EliGen Train Set
  • December 19, 2024 We implemented advanced VRAM management for HunyuanVideo, enabling video generation with resolutions of 129x720x1280 on 24GB VRAM or 129x512x384 on just 6GB VRAM. More details can be found in ./examples/HunyuanVideo/.
  • December 18, 2024 We propose ArtAug, a method to improve text-to-image models through synthesis-understanding interaction. We trained an ArtAug enhancement module for FLUX.1-dev in LoRA format. This model incorporates the aesthetic understanding of Qwen2-VL-72B into FLUX.1-dev, thereby improving the quality of generated images.
- Paper: https://arxiv.org/abs/2412.12888 - Example: https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/ArtAug - Model: ModelScope, HuggingFace - Demo: ModelScope, HuggingFace (coming soon)
  • October 25, 2024 We provide extensive FLUX ControlNet support. This project supports many different ControlNet models and can be freely combined, even if their structures are different. Additionally, ControlNet models are compatible with high-resolution optimization and partition control technologies, enabling very powerful controllable image generation. See ./examples/ControlNet/.
  • October 8, 2024 We released extended LoRAs based on CogVideoX-5B and ExVideo. You can download this model from ModelScope or HuggingFace.
  • August 22, 2024 This project now supports CogVideoX-5B. See here. We provide several interesting features for this text-to-video model, including:
- Text-to-video - Video editing - Self super-resolution - Video interpolation
  • August 22, 2024 We implemented an interesting brush feature that supports all text-to-image models. Now you can create stunning images with the assistance of AI using the brush!
- Use it in our WebUI.
  • August 21, 2024 DiffSynth-Studio now supports FLUX.
- Enable CFG and high-resolution inpainting to improve visual quality. See here - LoRA, ControlNet, and other addon models will be released soon.
  • June 21, 2024 We propose ExVideo, a post-training fine-tuning technique aimed at enhancing the capabilities of video generation models. We extended Stable Video Diffusion to achieve long video generation of up to 128 frames.
- Project Page - Source code has been released in this repository. See examples/ExVideo. - Model has been released at HuggingFace and ModelScope. - Technical report has been released at arXiv. - You can try ExVideo in this demo!
  • June 13, 2024 DiffSynth Studio has migrated to ModelScope. The development team has also transitioned from "me" to "us". Of course, I will still participate in subsequent development and maintenance work.
  • January 29, 2024 We propose Diffutoon, an excellent cartoon coloring solution.
- Project Page - Source code has been released in this project. - Technical report (IJCAI 2024) has been released at arXiv.
  • December 8, 2023 We decided to initiate a new project aimed at unleashing the potential of diffusion models, especially in video synthesis. The development work of this project officially began.
  • November 15, 2023 We propose FastBlend, a powerful video deflickering algorithm.
- sd-webui extension has been released at GitHub. - Demonstration videos have been showcased on Bilibili, including three tasks: - Video Deflickering - Video Interpolation - Image-Driven Video Rendering - Technical report has been released at arXiv. - Unofficial ComfyUI extensions developed by other users have been released at GitHub.
  • October 1, 2023 We released an early version of the project named FastSDXL. This was an initial attempt to build a diffusion engine.
- Source code has been released at GitHub. - FastSDXL includes a trainable OLSS scheduler to improve efficiency. - The original repository of OLSS is located here. - Technical report (CIKM 2023) has been released at arXiv. - Demonstration video has been released at Bilibili. - Since OLSS requires additional training, we did not implement it in this project.
  • August 29, 2023 We propose DiffSynth, a video synthesis framework.
- Project Page. - Source code has been released at EasyNLP. - Technical report (ECML PKDD 2024) has been released at arXiv.

Installation

Install from source (recommended):

git clone https://github.com/modelscope/DiffSynth-Studio.git  
cd DiffSynth-Studio
pip install -e .

For more installation methods and instructions for non-NVIDIA GPUs, please refer to the Installation Guide.

Download Source Configuration

Before model inference and training, you can configure the model download source and other options through environment variables.
This project downloads models from ModelScope by default. For users outside China, you can download models from the ModelScope International site with the following configuration:
> export MODELSCOPE_ENDPOINT=https://modelscope.ai
>
To download models from HuggingFace, please modify the environment variables. Note that model IDs may differ across platforms:
> export DIFFSYNTH_DOWNLOAD_SOURCE="huggingface"
>

Basic Framework

DiffSynth-Studio redesigns the inference and training pipelines for mainstream Diffusion models (including FLUX, Wan, etc.), enabling efficient memory management and flexible model training.

Quick start: experience popular and the latest models:

| Architecture | Model ID | Inference | Low VRAM Inference | Full Training | Validation After Full Training | LoRA Training | Validation After LoRA Training | |-|-|-|-|-|-|-|-| | Fibo 2.0 | briaai/Fibo-2.0-dev | code | code | code | code | code | code | | Fibo 2.0 | briaai/Fibo-2.0-edit-turbo | code | code | - | - | - | - | | Qwen-Image-2.1 | Qwen/Qwen-Image-2.1 | code | code | code | code | code | code | | Qwen-Image-2.1 | Qwen/Qwen-Image-2.1-Turbo | code | code | code | code | code | code | | MiniMax-H3 | MiniMax/MiniMax-H3: FL2VA | code | code | code | code | code | code | | MiniMax-H3 | DiffSynth-Studio/MiniMax-H3-NF4: FL2VA pruned | code | code | - | - | code | code | | ACE-Step | ACE-Step/acestep-v15-xl-sft | code | code | code | code | code | code | | Z-Image | Tongyi-MAI/Z-Image-Turbo | code | code | code | code | code | code | | Krea-2 | krea/Krea-2-Raw | code | code | code | code | code | code | | Krea-2 | krea/Krea-2-Turbo | code | code | code | code | code | code |

Model overview:

  • Image generation
- Fibo 2.0: Documentation, Example code - MOSS-Image: Documentation, Example code - FLUX.3: Documentation, Example code - Qwen-Image-2.1: Documentation, Example code - SenseNova-U1: Documentation, Example code - Boogu-Image: Documentation, Example code - Krea-2: Documentation, Example code - Ideogram 4: Documentation, Example code - HiDream-O1-Image: Documentation, Example code - JoyAI-Image: Documentation

You finished the article!

Want to go deeper or explore more stories?

Full Article on GitHub Trending More Tech News
Reading Mode

Adjust typography and theme to your preference.

View Original
Related Stories
Revolut moves to self-hosted GLM
Hacker News 1 hour ago
uBlock Origin filter to block autoplay videos in TechCrunch
Hacker News 4 hours ago
Nvidia – open-source auto-ontology
Hacker News 4 hours ago
Show HN: ReBirth 338 in Browser
Hacker News 5 hours ago
Median net worth of U.S. households under 35 fell 23% from 2022 to 2025 (real)
Hacker News 8 hours ago
Related Stories

Revolut moves to self-hosted GLM

Hacker News 1 hour ago

uBlock Origin filter to block autoplay videos in TechCrunch

Hacker News 4 hours ago

Nvidia – open-source auto-ontology

Hacker News 4 hours ago

Show HN: ReBirth 338 in Browser

Hacker News 5 hours ago

Median net worth of U.S. households under 35 fell 23% from 2022 to 202...

Hacker News 8 hours ago
Chat with me