Salvatore Sanfilippo

This quants are specific for the DS4 inference engine. They may work with other inference engines or not (they should, but not the MTP model which requires a specific loader). https://github.com/antirez/ds4 Use q2 on 128 GB Mac machines, q4 on machines with ≥ 256 GB RAM, pair either with MTP for optional speculative decoding. The filename is the spec. In detail, for the q2 file: For the q4 file, only the three routed-expert classes change to Q4K. Everything else is byte-for-byte identical to the q2 recipe. The motivation behind the asymmetry: the routed experts are the majority of the parameter count but each individual expert handles only a fraction of tokens, so aggressive quantization on…