MakazhanAlpamys/Soup v0.5.0

v0.5.0 — Multimodal Vision Fine-tuning

on GitHub

What's New

Phase 6 — Multimodal Vision Fine-tuning

Train vision-language models (LLaMA-3.2-Vision, Qwen2-VL, Pixtral) on image+text datasets.

New Features

• modality: vision — new config option for multimodal training
• LLaVA & ShareGPT4V formats — auto-detected, with image path validation
• image_dir field in data config — resolve relative image paths
• Vision model loading — AutoModelForVision2Seq + AutoProcessor with LoRA/QLoRA
• soup init --template vision — ready-to-use config for LLaMA-3.2-Vision
• Vision stats in soup data inspect — image count, formats, missing files
• pip install 'soup-cli[vision]' — Pillow optional dependency
• soup doctor — now checks for Pillow

Example Config

base: meta-llama/Llama-3.2-11B-Vision-Instruct
task: sft
modality: vision

data:
  train: ./data/vision_train.jsonl
  format: llava
  image_dir: ./data/images

training:
  epochs: 3
  lr: 1e-5
  quantization: 4bit
  lora:
    r: 64
    alpha: 16

Stats

• 51 new tests (455 total)
• ruff lint clean
• CI green on Python 3.9/3.11/3.12

Install / Upgrade

pip install --upgrade soup-cli
pip install 'soup-cli[vision]'  # for vision support