LLaMA-Factory | KVCache.AI

KT-FT v0.6.1: Closing the Loop from MoE Fine-Tuning to Local Serving

Fri, 29 May 2026 00:00:00 +0000

Previously, we published a guide on using KTransformers, LLaMA-Factory, and SGLang for low-cost local fine-tuning and inference. This post does not replace it; it focuses on what changed with KT-FT v0.6.1 and how the post-v0.6.1 serving work closes the end-to-end loop.

The goal is no longer only to run MoE SFT locally. It is to connect the whole path: fit MoE SFT on local hardware, carry the trained adapter into SGLang, and evaluate it through an OpenAI-compatible serving API. MoE training has two bottlenecks—making the job fit (GPU for attention, CPU for experts) and making the trained adapter usable in a real serving stack. v0.6.1 targets both while keeping the LLaMA-Factory training surface unchanged.

What v0.6.1 improves

The main change in v0.6.1 is the MoE SFT backend itself. Compared with earlier KT fine-tuning paths, the refactor targets three practical concerns: training speed, memory pressure, and setup friction.

In measured configurations against a ZeRO-Offload baseline, KT SFT reaches roughly 6–12x training performance. CPU memory usage drops to about half, and GPU memory is reduced further. This makes longer contexts easier to fit on the same hardware.

Results vary by model, hardware, and LoRA setup; release benchmarks will include the full configuration context.

The installation process is also cleaner. v0.6.1 collects the SFT dependency entry under:

pip install "ktransformers[sft]==0.6.1"

That entry installs the KT SFT components needed underneath LLaMA-Factory, including ktransformers, kt-kernel, transformers-kt, and accelerate-kt. For LLaMA-Factory users, the preferred path is still to use a checkout that includes the KT examples and requirements/ktransformers.txt; the integration work is tracked in LLaMA-Factory PR #10430.

We also published a separate v0.6.1 Quick Start, which readers can refer to.

Under the hood, v0.6.1 keeps the same LLaMA-Factory workflow but changes placement: attention stays on GPU with LoRA attached, routed experts can live in CPU memory, and expert computation is still exposed to PyTorch autograd so gradients can flow through the SFT job.

Beyond Quick Validation: SGLang Adapter Serving After v0.6.1

The earlier guide already covered the immediate post-training sanity check: load the adapter back into LLaMA-Factory, run a few interactive prompts, and confirm that the adapter can be used. That step is still useful. It answers a narrow but important question: did the adapter load, and does the target behavior appear?

For an end-to-end fine-tuning-to-serving loop, that is not enough. The adapter also needs to run in the same kind of environment used for benchmark traffic, automated evaluation, and application-facing APIs. In this workflow, that environment is SGLang.

For Qwen3.5 MoE, the post-v0.6.1 bridge is kt-kernel/scripts/convert_kt_to_sglang_adapter.py plus a split serving layout: expert LoRA through KT, non-expert LoRA through SGLang’s normal LoRA manager.

What KT SFT actually writes

A typical LLaMA-Factory + KT SFT output directory contains two LoRA artifacts, not one:

<KT_SFT_OUTPUT_DIR>/
  adapter_model.safetensors      # non-expert LoRA (PEFT keys, attention / linear_attn)
  fused_expert_lora.safetensors  # expert LoRA (KT fused format)
  adapter_config.json            # training metadata (rank, alpha, target_modules, ...)

Do not treat adapter_model.safetensors alone as the full fine-tuned adapter for Qwen3.5 MoE serving. Expert weights live in fused_expert_lora.safetensors until they are converted for SGLang.

Why serving is split

Expert LoRA (gate_proj, up_proj, down_proj) — --kt-expert-lora-path, KT CPU expert forward path.
Non-expert LoRA (attention / linear attention) — --lora-paths, SGLang LoRA manager.

Convert once from the raw KT-SFT directory

Run convert_kt_to_sglang_adapter.py once on the training output with --expert-output-dir and --nonexpert-output-dir. The second positional argument is a merged adapter directory (legacy/debug); for split serving, use only the two split directories with SGLang.

Output	Use in split serving
`<EXPERT_ADAPTER_DIR>`	`--kt-expert-lora-path`
`<NONEXPERT_ADAPTER_DIR>`	`--lora-paths <name>=...`
`<MERGED_ADAPTER_DIR>`	Debug only; not the split-runtime contract

python kt-kernel/scripts/convert_kt_to_sglang_adapter.py \
  saves/KT_FT_qwen35B_Moe_nekoqa_eod_240 \
  saves/KT_FT_qwen35B_Moe_nekoqa_eod_240_sglang \
  --base-model-name-or-path /path/to/Qwen3.5-35B-A3B \
  --expert-output-dir saves/KT_FT_qwen35B_Moe_nekoqa_eod_240_expert_adapter \
  --nonexpert-output-dir saves/KT_FT_qwen35B_Moe_nekoqa_eod_240_nonexpert_adapter \
  --overwrite

You should see summary lines similar to: merged ~61k tensors; expert-only ~61k tensors; non-expert-only ~380 tensors, with target_modules limited to gate_proj/up_proj/down_proj vs attention/linear-attn names respectively.

Quick pre-serve check (optional):

python - <<'PY'
import json
from pathlib import Path

expert = Path("<EXPERT_ADAPTER_DIR>")
nonexpert = Path("<NONEXPERT_ADAPTER_DIR>")

expert_cfg = json.loads((expert / "adapter_config.json").read_text())
nonexpert_cfg = json.loads((nonexpert / "adapter_config.json").read_text())

assert {"gate_proj", "up_proj", "down_proj"} <= set(expert_cfg["target_modules"])
assert not ({"gate_proj", "up_proj", "down_proj"} & set(nonexpert_cfg["target_modules"]))
assert (expert / "adapter_model.safetensors").is_file()
assert (nonexpert / "adapter_model.safetensors").is_file()
print("ok")
PY

Launch and request semantics

python -m sglang.launch_server \
  ... \
  --kt-expert-lora-path <EXPERT_ADAPTER_DIR> \
  --enable-lora \
  --lora-backend triton \
  --lora-paths <NONEXPERT_LORA_NAME>=<NONEXPERT_ADAPTER_DIR> \
  --served-model-name <SERVED_MODEL_NAME>

Request behavior:

model=<SERVED_MODEL_NAME>
=> base + KT expert LoRA (always on at startup)

model=<SERVED_MODEL_NAME>:<NONEXPERT_LORA_NAME>
=> base + KT expert LoRA + SGLang non-expert LoRA

On the expert side, serving reuses the SFT-compatible path without training state (forward_sft(save_for_backward=False)). The trained adapter can be evaluated locally without pre-merging expert weights into the base model.

Repository reference for the full Qwen3.5 loop: ktransformers/doc/en/SFT/Qwen3.5-SGLang-LoRA-Serving.md (launch flags, constraints, smoke tests). If you are comparing versions, use the split-output flags above with the current converter.

Did the adapter actually take effect?

There are two different questions here, and it is useful not to mix them.

The first is whether KT-backed SFT can train a meaningful adapter. We tested this on representative adaptation workloads including Beyond Dialogue for personalized chat and Llemr for EHR-domain adaptation. These are small-scale evaluations rather than universal claims, but they show that KT supports real adapter training rather than only loading a huge model for a demo.

The second question is whether the serving path applies the trained adapter correctly. In local NekoBench-style validation with Qwen3.5, the split LoRA serving path changed model behavior in the expected direction for a narrow personality/style adapter. The overall mean moved from 3.61 to 4.37, with stronger gains on persona and companionship-style slices. Knowledge and reasoning slices dropped, which is also expected for a narrow style-oriented adapter.

That is the intended claim: the serving path makes the adapter take effect. It does not mean every capability improves after a style fine-tune.

In practice, this is where the end-to-end loop matters: serving lets you measure adapter behavior beyond a few interactive prompts. If the adapter improves the intended behavior but hurts unrelated capability slices, you can compare dataset mix, LoRA target, rank, learning rate, or evaluation set against the same local API environment used for downstream testing.

How this post relates to the earlier guide

Topic	Earlier kvcache guide	This post (v0.6.1 + Qwen3.5 serving)
Training UI	LLaMA-Factory YAML, familiar outputs	Same surface; KT MoE SFT backend refactor underneath
Post-train check	Load adapter in LLaMA-Factory, a few prompts	Still valid as a quick sanity check
Adapter on disk	Often described as one adapter directory	KT MoE: two training artifacts (`adapter_model` + `fused_expert_lora`)
Convert for SGLang	Not the focus of the original guide	`convert_kt_to_sglang_adapter.py` with split output dirs
SGLang launch	Single adapter via `--lora-paths lora0=...`	Split: `--kt-expert-lora-path` + `--lora-paths <name>=...`
Full adapter at request time	Request `model=...:lora0`	Request `model=<base>:<nonexpert_name>`; expert LoRA is always loaded at startup

For environment setup, LLaMA-Factory + KT training, and the generic single-LoRA SGLang path, see the earlier guide. This post covers what changed afterward: the v0.6.1 MoE SFT backend and the Qwen3.5 split adapter path (converter and launch flags above).

Closing

v0.6.1 moves KTransformers fine-tuning from “possible on a workstation” toward a cleaner SFT workflow.

The backend refactor improves training performance and memory usage. The ktransformers[sft] entry lowers setup friction. The package boundary cleanup makes it clearer which components belong to SFT and which belong to inference. LLaMA-Factory remains the familiar training interface, while KTransformers provides the heterogeneous execution backend underneath.

The post-v0.6.1 serving work closes the next gap: moving the trained adapter into SGLang without pre-merging expert weights into the base model. Run convert_kt_to_sglang_adapter.py on the raw KT-SFT output with --expert-output-dir and --nonexpert-output-dir, then launch with the split flags above.

That is the KT-FT loop: train the adapter, validate it, serve it locally with SGLang, measure the behavior, revise the data or LoRA settings, and run again.

If you can run the model with KTransformers, you should be able to tune it. With the v0.6.1 SFT refactor and the post-v0.6.1 serving bridge, you can also test what you tuned in the same local environment.

KTransformers + LLaMA-Factory + SGLang: Low-Cost Local Fine-Tuning and Inference

Sat, 01 Nov 2025 00:00:00 +0000

On a local workstation, the hard part of large-model experimentation is usually the cost of bringing a large MoE model into the same loop as the user’s data and evaluation target. A researcher may want to try a domain dataset, a product prototype, or a benchmark, but the model quickly turns into a GPU-memory problem. This guide presents KTransformers, LLaMA-Factory, and SGLang as a low-cost, low-memory end-to-end path: LoRA fine-tuning stays in a familiar training recipe, KTransformers shifts the memory pressure through GPU+CPU heterogeneous execution, and the adapted model can continue into inference and benchmark testing.

Inside that workflow, LLaMA-Factory sits at the user-facing orchestration layer: it owns dataset preparation, model templates, LoRA configuration, checkpoint output, and the first chat/API validation path. KTransformers plugs in underneath as the LoRA backend engine for Attention and MoE operators, moving memory-heavy expert computation into a GPU+CPU heterogeneous path while preserving the LLaMA-Factory interface. SGLang then takes the trained adapter into the inference side of the same end-to-end flow for batch inference and benchmark traffic.

Why This Integration Matters

In the same LLaMA-Factory LoRA workflow, the KTransformers backend is the path that can handle ultra-large MoE models on commodity hardware. On DeepSeek-V2-Lite, it improves throughput and lowers GPU memory. On DeepSeek-V3 scale, the default HuggingFace path is not runnable in this 4090-class setting, while KTransformers keeps training feasible through heterogeneous placement.

LoRA BF16 with NekoQA-10K stylized dialogue	HuggingFace backend	Unsloth backend	KTransformers backend
DeepSeek-V2-Lite 14B throughput	303.58 token/s	455.37 token/s	530.38 token/s
DeepSeek-V2-Lite 14B GPU memory	32.12 GB	9.64 GB	6.08 GB
DeepSeek-V3 671B throughput	Too large to run	Not supported	40.35 token/s
DeepSeek-V3 671B GPU memory, summed across GPUs	theoretical 1400 GB	Not supported	70 GB measured peak

The 1400 GB figure is a theoretical FP16 full-parameter resident footprint. The measured KTransformers number comes from placing Attention on GPU and offloading the layered MoE workload.

Fine-Tuning Results

We validated the setup on three representative customization tasks:

Stylized dialogue, using NekoQA-10K to make a model consistently answer in a recognizable persona.
Translational-style generation, using an exaggerated Westernized translation tone.
Medical question answering, using AfriMed-QA short-answer and multiple-choice tasks.

For stylized dialogue, the fine-tuned model follows the target tone and address terms more consistently than the base model.

For the translational-style task, both DeepSeek-V2-Lite and DeepSeek-V3 improve clearly after KT-LoRA fine-tuning.

Translational-Style dataset	BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGE-1	ROUGE-2	ROUGE-L
V2-Lite, no LoRA	20.66	8.33	4.54	2.89	22.71	4.52	19.19
KT-LoRA fine-tuned V2-Lite	35.41	22.44	15.42	11.18	42.03	18.38	33.10
V3 base, no LoRA	8.49	3.34	1.62	0.96	15.91	2.55	10.07
KT-LoRA fine-tuned V3	37.02	23.70	16.21	11.49	43.43	18.96	34.54

For AfriMed-QA, KT-LoRA also improves both short-answer generation and multiple-choice accuracy.

AfriMed-QA short answer	BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGE-1	ROUGE-2	ROUGE-L
V2-Lite, no LoRA	13.58	11.12	9.10	7.23	22.48	7.81	11.73
KT-LoRA fine-tuned V2-Lite	35.90	27.63	22.99	19.15	35.25	17.50	28.44
V3 base, no LoRA	12.75	10.27	8.05	5.99	20.33	5.65	10.11
KT-LoRA fine-tuned V3	42.42	34.12	28.95	24.54	41.97	22.37	33.28

AfriMed-QA multiple choice	Accuracy
V2-Lite, no LoRA	0.0645
KT-LoRA fine-tuned V2-Lite	0.4812
V3 base, no LoRA	0.5833
KT-LoRA fine-tuned V3	0.7930

These are representative small-scale evaluations rather than a complete scaling-law study. The main takeaway is about resource cost: under the same LLaMA-Factory workflow, KTransformers makes LoRA adaptation feasible for MoE models that would otherwise exceed workstation GPU memory.

Quick Start [May be outdated, please refer to the newest blog]

Use sections 1, 2, and 5 if you only need inference. Use sections 1 through 5 if you want the full LoRA fine-tuning and inference loop.

1. Hardware Requirements

Start from the job you want to run:

For inference only, CPU requirements are lighter, but host memory still determines how large a model you can hold.
For KT LoRA fine-tuning, the CPU must support Intel AMX. Check with lscpu | grep -i amx || true.
GPU memory controls how many GPU experts you can keep resident for speed. KTransformers lets you trade GPU residency for host memory and CPU compute through placement rules.

Model	KT inference, rough starting point	KT fine-tuning reference
DeepSeek-V2-Lite-14B	3 GB GPU + 15 GB host memory	6 GB GPU + 30 GB host memory
Qwen3-30B-A3B	3 GB GPU + 30 GB host memory	5 GB GPU + 60 GB host memory
Qwen3-235B-A22B	9 GB GPU + 225 GB host memory	18 GB GPU + 450 GB host memory
DeepSeek-V3-671B	35 GB GPU + 0.65 TB host memory	70 GB GPU + 1.3 TB host memory

2. Environment and Model Preparation

Install the three layers used in this workflow: KTransformers for heterogeneous execution, SGLang for serving, and LLaMA-Factory for recipe-style fine-tuning.

# KTransformers inference kernel path.
git clone https://github.com/kvcache-ai/ktransformers.git
cd ktransformers
cd kt-kernel
./install.sh

# SGLang branch used with KTransformers.
git clone https://github.com/kvcache-ai/sglang.git
cd sglang
pip install -e "python[all]"

# LLaMA-Factory.
git clone https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e ".[torch,metrics]" --no-build-isolation

# KTransformers fine-tuning dependencies.
conda install -y -c conda-forge libstdcxx-ng gcc_impl_linux-64
conda install -y -c nvidia/label/cuda-11.8.0 cuda-runtime

# Prefer matched wheels when available to avoid local compilation.
# Match Python, PyTorch, CUDA, and ABI with your machine.
pip install ktransformers-0.4.2+cu128torch27fancy-cp311-cp311-linux_x86_64.whl
pip install flash_attn-2.8.3+cu12torch2.7cxx11abiTRUE-cp311-cp311-linux_x86_64.whl
pip install transformers==4.56.0

Use BF16 model weights for KT fine-tuning. DeepSeek-V3-671B is often distributed in FP8 form, so download a BF16 checkpoint directly or convert FP8 weights before training.

pip install -U huggingface_hub==0.34.0
huggingface-cli download --resume-download \
  Qwen/Qwen3-235B-A22B-Instruct-2507 \
  --local-dir /path/to/Qwen3-235B-A22B-Instruct-2507-BF16

3. LoRA Fine-Tuning with KTransformers

The training command stays compact. Most experiment changes should live in the LLaMA-Factory YAML.

cd LLaMA-Factory
USE_KT=1 llamafactory-cli train examples/train_lora/qwen3moe_lora_sft_kt.yaml

The important KT fields are use_kt, kt_optimize_rule, cpu_infer, and chunk_size. Choose an *-sft-* optimize rule that matches your model, CPU backend, and GPU count.

### model
model_name_or_path: /path/to/Qwen3-235B-A22B-Instruct-2507-BF16
trust_remote_code: true
template: qwen3

### method
stage: sft
do_train: true
finetuning_type: lora
lora_rank: 8
lora_alpha: 32
lora_dropout: 0.1
lora_target: all

### dataset
dataset: identity, alpaca_en_demo
cutoff_len: 2048
max_samples: 100000
overwrite_cache: true
preprocessing_num_workers: 16
dataloader_num_workers: 4

### output
output_dir: saves/qwen3moe_lora_sft_kt
logging_steps: 10
save_steps: 500
plot_loss: true
overwrite_output_dir: true
save_only_model: false
report_to: none

### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 1.0e-4
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000

### ktransformers
use_kt: true
kt_optimize_rule: examples/kt_optimize_rules/<model>-sft-amx-<gpu-count>.yaml
cpu_infer: 64
chunk_size: 2048

Training writes LoRA adapter artifacts to output_dir, usually as safetensors weights plus adapter metadata. That directory is reused by the inference steps below.

4. Quick Validation with LLaMA-Factory

Right after fine-tuning, use LLaMA-Factory for a few interactive checks. This path is meant to confirm that the adapter loads and that the target behavior appears; it is not the fastest serving path.

cd LLaMA-Factory
llamafactory-cli chat examples/inference/qwen3moe_lora_sft_kt.yaml

The inference YAML should point to the base model and adapter directory, set infer_backend: ktransformers, and keep the KT optimize rule aligned with training.

model_name_or_path: /path/to/Qwen3-235B-A22B-Instruct-2507-BF16
adapter_name_or_path: saves/qwen3moe_lora_sft_kt
template: qwen3
infer_backend: ktransformers
trust_remote_code: true
use_kt: true
kt_optimize_rule: examples/kt_optimize_rules/<model>-infer-amx-<gpu-count>.yaml
cpu_infer: 64
chunk_size: 2048

For batch evaluation through the same LLaMA-Factory stack, launch its API server with the same config:

API_PORT=8000 llamafactory-cli api examples/inference/qwen3moe_lora_sft_kt.yaml

5. Faster Serving and Benchmarking with SGLang

For benchmark runs or application-facing APIs, use SGLang with KT enabled. The serving path has three steps: convert the LoRA adapter, optionally quantize CPU-side weights, then launch the server with KT and LoRA flags.

cd sglang
python convert_lora.py <YOUR_LORA_ADAPTER_PATH>

cd ktransformers/kt-kernel
python scripts/convert_cpu_weights.py \
  --input-path <PATH_TO>/Qwen3-30B-A3B-Instruct-2507 \
  --input-type bf16 \
  --output <PATH_TO>/Qwen3-30B-A3B-Instruct-2507-INT8 \
  --quant-method int8

python -m sglang.launch_server \
  --host 0.0.0.0 \
  --port 10103 \
  --model <PATH_TO>/Qwen3-30B-A3B-Instruct-2507 \
  --mem-fraction-static 0.7 \
  --chunked-prefill-size 2048 \
  --served-model-name Qwen3-30B-A3B-Instruct-2507 \
  --tensor-parallel-size 1 \
  --kt-method AMXINT8 \
  --kt-weight-path <PATH_TO>/Qwen3-30B-A3B-Instruct-2507-INT8 \
  --kt-cpuinfer 64 \
  --kt-threadpool-count 2 \
  --kt-num-gpu-experts 1 \
  --enable-lora \
  --lora-paths lora0=<YOUR_ADAPTER_PATH> \
  --max-loras-per-batch 1 \
  --lora-backend triton

For base-model-only inference, remove the final LoRA-related flags. For Kimi K2, MiniMax M2/M2.1, and other newer model paths, use the corresponding KTransformers V0.5.0 or later instructions when FP8 or INT4 native inference is required.

Once the SGLang server is running, call it through the OpenAI-compatible API:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:10103/v1", api_key="EMPTY")

resp = client.completions.create(
    model="Qwen3-30B-A3B-Instruct-2507",
    prompt="Write quicksort in C++, Python, and Rust.",
    max_tokens=256,
)
print(resp.choices[0].text)

KT Tuning Knobs

For fine-tuning, start by changing kt_optimize_rule. Rule names usually encode the model family, whether the rule is for SFT, the CPU backend such as AMX, and the GPU count. In the LLaMA-Factory YAML, only four KT fields normally need user-side adjustment: use_kt, kt_optimize_rule, cpu_infer, and chunk_size.

For SGLang serving, reduce memory pressure in this order: lower --chunked-prefill-size for prefill OOM, lower --max-running-requests for decode OOM, reduce --kt-num-gpu-experts when GPU-resident experts are too expensive, quantize CPU weights to INT8 when host memory or bandwidth is tight, and then tune --mem-fraction-static for the target benchmark workload.

Performance and Memory

For the reported experiments, GAS=16 and qlen=512, so each optimization step processes 8192 tokens.

Model	Step time	Tokens per step	Throughput
DeepSeek-V3 671B	203 s	8192	40.35 token/s
DeepSeek-V2-Lite 14B	36 s	8192	227.6 token/s

The measured memory footprint is:

Model	GPU memory	Host memory
DeepSeek-V3 671B, 58 MoE layers out of 61	about 70 GB total GPU memory	about 1.2-1.3 TB
DeepSeek-V2-Lite 14B, 26 MoE layers out of 27	about 5.5 GB GPU memory	about 150 GB

Technical Notes

The following section condenses the original Developer Technical Notes. Blocks marked Deprecated in V2 Current describe earlier implementation details kept only as historical context.

Attention with LoRA

KTransformers provides operator injection through BaseInjectedModule, while PEFT provides LoRA layer insertion. For fine-tuning, the integration uses a KTransformersLinearLora layer that inherits from both the KT linear path and the LoRA layer path.

This keeps KT’s fast prefill_linear and generate_linear paths while adding trainable LoRA matrices. During preparation, Q/K/V/O linear layers are replaced so that the Attention block remains optimized but becomes LoRA-trainable.

MoE as a Differentiable Backend Operator

MoE parameters dominate the model size, but MoE compute is sparse. KTransformers encapsulates expert computation as a differentiable black-box operator: upstream, PyTorch sees a compact autograd node; downstream, pybind11 calls C++ kernels for forward and backward.

That backend can be selected through config. The evaluated paths include AMX BF16/INT8 and llamafile-style CPU kernels.

MoE Backward (CPU) (Deprecated in V2 Current)

Deprecated in V2 Current. In the original technical notes, MoE backward frequently needs the transposed weights $W^\top$. To avoid repeated runtime transposes, the earlier implementation precomputed and cached $W^\top$ at load time. This stored transposed-weight copy is deprecated in V2 current and should be read as historical implementation context only.

The original notes also describe caching necessary intermediate activations, such as expert projections, to reuse in backward and reduce recomputation. Treat this subsection as historical unless it is re-verified against the current V2 implementation.

Multi-GPU Loading/Training: Placement Strategy Instead of DataParallel (Deprecated in V2 Current)

Deprecated in V2 Current. The KTrainer, explicit placement, and DataParallel-avoidance details in this subsection reflect the original Developer Technical Notes and are not the current V2 behavior. They are preserved only as historical context.

In the original notes, the multi-GPU strategy was explicit placement plus model parallelism:

Deprecated in V2 Current: KTrainer prevents the entire model from being moved to one GPU.
Deprecated in V2 Current: Layers are constructed directly on target devices according to the KT config.
Deprecated in V2 Current: Automatic DataParallel wrappers are disabled when the KT path is active.
Deprecated in V2 Current: Gradients are reduced where needed, while intermediate activations stay local as much as possible.

Deprecated in V2 Current. The original notes describe this as keeping Attention and KV-related work on GPUs while MoE experts are placed on CPU and accelerated there, reducing per-GPU memory pressure without changing the user-facing LLaMA-Factory training flow. This specific placement/trainer description is deprecated in V2 current.

Limitations

The evaluation above is scoped around the low-memory training and inference path. Most measurements use single datasets and relatively small fine-tuning sets, usually no more than 20k examples. They show that LoRA adaptation can run under constrained hardware, but they are not a full study of generalization, scaling laws, multi-seed variance, or multilingual robustness.

We welcome additional community results, especially when they include the KT config, dataset samples, training/evaluation YAMLs, GPU memory, CPU memory, CPU model, and backend details. These details make performance numbers easier to compare and more useful for other developers.

Conclusion

KTransformers, LLaMA-Factory, and SGLang turn ultra-large MoE adaptation into a low-cost, low-memory workflow that runs end to end: LLaMA-Factory keeps training recipes familiar, LoRA keeps adaptation lightweight, KTransformers supplies heterogeneous placement and optimized Attention/MoE operators, and SGLang carries the inference path for benchmark or application traffic.

For smaller MoE models, the same path reduces GPU memory and improves throughput. For 671B-scale MoE models, it gives users a low-memory route where default full-GPU training is out of reach.