pig-clip

pig architecture gguf clip

• natively trained multipurpose clip for gk engine
• low vram alternative/substitute of t5xxl, umt5xxl, etc.
• 80-90% smaller, more efficient and cost effective (hopefully)

rationale

since LLM in diffusion model basically serves as text encoder, which means its primary goal/function is delivering a prompt to the generator precisely, no talk, no deep reasoning, no long response, make it as simple as possible

limitation/cutoff

we stop the training when similarity reaches .95 or over 20k cycles and similarity no less than .75

how it works

simply replace --t5xxl t5xxl.gguf with --llm pig_clip-nvfp4.gguf --llm-adapter pig_t5_adapter-f16.gguf

examples

test it with pixart

ggk diffuser engine -- --diffusion-model pixart-nvfp4.gguf --vae pig_pixart_vae_fp16-f16.gguf --llm pig_clip-nvfp4.gguf --llm-adapter pig_t5_adapter-f16.gguf -p "close-up portrait of dog" --diffusion-fa -v -o out.png

test it with sd-lite

ggk diffuser engine -- --diffusion-model model.gguf --vae vae.gguf --clip_l clip_l.gguf --clip_g clip_g.gguf --llm pig_clip-nvfp4.gguf --llm-adapter pig_t5_adapter-f16.gguf -p "close-up portrait of dog" --steps 8 --cfg-scale 1 --sampling-method euler --clip-on-cpu --diffusion-fa -v -o out.png

umt5 adapter

test it with wan2.1 or wan2.2

ggk diffuser engine -- -M vid_gen --diffusion-model wan2.1_t2v_1.3b-q4_0.gguf --vae pig_wan_vae_fp32-f16.gguf --llm pig_clip-nvfp4.gguf --llm-adapter pig_umt5_adapter-f16.gguf -p "a pig moving quickly in a beautiful winter scenery nature trees sunset tracking camera" --cfg-scale 6.0 --sampling-method euler -v -n "blurry ugly bad" -W 480 -H 480 --diffusion-fa --offload-to-cpu --video-frames 14 -o out.avi

qwen3vl-layerwise adapter

test it with krea2

ggk diffuser engine -- --diffusion-model krea2_turbo-q2_k.gguf --vae pig_wan_vae_fp32-f16.gguf --llm pig_clip-nvfp4.gguf --llm-adapter pig_qwen3vl_layerwise_adapter-f16.gguf -p "a lovely pig holding a sign says Hi" --cfg-scale 1.00 --steps 8 --offload-to-cpu --diffusion-fa -v -o out.png

qwen3-4b adapter

test it with x-image

ggk diffuser engine -- --diffusion-model  x_image-nvfp4.gguf --vae pig_flux_vae_fp32-f16.gguf --llm pig_clip-nvfp4.gguf --llm-adapter pig_qwen3_4b_adapter-f16.gguf -p "cute anime style girl with pinky messy long hair blue eyes wearing a maid outfit with a long black gold leaf pattern dress and a white apron, it is a postcard held by a hand in front of a beautiful realistic city at sunset and there is cursive writing that says PIG" --cfg-scale 1.0 --steps 8 --offload-to-cpu --diffusion-fa -v -o out.png

qwen3vl-4b adapter

test it with mageflow

ggk diffuser engine -- --diffusion-model mageflow-edit-turbo-nvfp4.gguf --vae pig_mageflow_vae_fp32-f16.gguf --llm pig_clip-nvfp4.gguf --llm-adapter pig_qwen3vl_4b_adapter-f16.gguf --llm_vision mmproj-qwen3vl-4b-it-f16.gguf --ref-image sheep.png -p "a sheep in sunglasses" --cfg-scale 1.00 --steps 4 --sampling-method euler --diffusion-fa -v -o out.png

qwen3vl-8b adapter

test it with qwen-image2.1

ggk diffuser engine -- --diffusion-model qwen-image-2.1-nvfp4.gguf --vae pig_qwen_image_2.1_vae_fp32-f16.gguf --llm pig_clip-f16.gguf --llm-adapter pig_qwen3vl_8b_adapter-f16.gguf -p "an anime pig holding a sign says GGUF" -W 512 -H 512 --sampling-method euler -v -o out.png

edit image with vision clip (optional)

ggk diffuser engine -- --diffusion-model qwen-image-2.1-nvfp4.gguf --vae pig_qwen_image_2.1_vae_fp32-f16.gguf --llm pig_clip-f16.gguf --llm-adapter pig_qwen3vl_8b_adapter-f16.gguf --llm_vision mmproj-qwen3vl-8b-it-q8_0.gguf --ref-image sheep.png -p "a sheep in sunglasses" --cfg-scale 1.00 --steps 8 --sampling-method euler --diffusion-fa -v -o out.png

llada adapter

test it with llada

ggk diffuser engine -- --diffusion-model LLaDA-image-turbo-nvfp4.gguf --vae pig_flux2_vae_fp32-f16.gguf --llm pig_clip-q8_0.gguf --llm-adapter pig_llada_adapter-q8_0.gguf -p "a pig in sunglasses" --cfg-scale 1.00 --steps 4 --sampling-method euler --diffusion-fa -v -o out.png

edit image with vision clip (optional)

ggk diffuser engine -- --diffusion-model LLaDA-image-turbo-nvfp4.gguf --vae pig_flux2_vae_fp32-f16.gguf --llm pig_clip-q4_k_m.gguf --llm-adapter pig_llada_adapter-f16.gguf --llm_vision pig_llada_sigvq-q4_0.gguf --ref-image sheep.png -p "a sheep in sunglasses" --cfg-scale 1.00 --steps 4 --sampling-method euler --diffusion-fa -v -o out.png

Reference

pig engine - the new gguf compute kernels (gk)

gguf trainer - how to train your own adapter