IST Austria Distributed Algorithms and Systems Lab

Model · Image and text to text

Qwen3.8-27B-GSQ-RCO-GGUF

IST Austria Distributed Algorithms and Systems Lab

Non-uniform GGUF quantizations produced with GSQ and RCO, with a vision projector for multimodal use. This repository provides GGUF quantizations of Qwen3.8-27B at four sizes, together with the model's vision projector (mmproj) for multimodal use. In contrast to uniform quantization, which applies a single quantization type to all weight tensors, each model here assigns a separate quantization type to every tensor. The assignment is obtained by a gradient-based search that allocates precision according to per-tensor sensitivity, subject to a total size budget. The resulting files are standard GGUF and run unmodified in llama.cpp, Ollama, and LM Studio. Both methods were developed at the…

Model · Image and text to text

gemma-3-27b-it-GPTQ-4b-128g

IST Austria Distributed Algorithms and Systems Lab

This model was obtained by quantizing the weights of gemma-3-27b-it to INT4 data type. This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 75%. Only the weights of the linear operators within languagemodel transformers blocks are quantized. Vision model and multimodal projection are kept in original precision. Weights are quantized using a symmetric per-group scheme, with group size 128. The GPTQ algorithm is applied for quantization. Model checkpoint is saved in compressedtensors format. This model was evaluated on the OpenLLM v1 benchmarks. Model outputs were generated with the vLLM engine. The…