Add support for IQ quantizarions

[BruceMacD]

This change allows importing IQ type gguf quantization with ollama create.

This change carries the commit from #3657 while moving its changes around to the refactored project structure.

❯ ./ollama create nous-hermes-2-mistral:IQ_4XS -f /Users/bruce/models/nous-hermes-2-mistral/Modelfile
transferring model data 
using existing layer sha256:737258efad6ba5cf7232de66715a26cadba67b0e4bdace5cf03cf49d1e4864a0 
creating new layer sha256:d7285065edcb87b4852f1144dd090812df1b00ade49f74e234066ea9407a14bc 
creating new layer sha256:d8ba2f9a17b3bbdeb5690efaa409b3fcb0b56296a777c7a69c78aa33bbddf182 
creating new layer sha256:b2c4ee0a7317771fcbe7413c369d72ea911c63e6f52b2b0d6298a5a14c8e4983 
writing manifest 
success 

❯ ./ollama run nous-hermes-2-mistral:IQ_4XS
>>> write some python

def print_fruits(fruits):
    for fruit in fruits:
        print(fruit)

Tested with:
IQ1_S
IQ1_M
IQ2_M
IQ3_XXS
IQ3_XS
IQ3_S
IQ4_NL
IQ4_XS

resolves #3622

[sammcj]

Looks good, This will be great to have, thank you.

[sammcj]

Been running my Ollama build with this patch for the past few days without any issues, it's great to use some of the newer IQ quants!

Keen to see this merged.

[mann1x]

@BruceMacD Any idea when we can get this merged?

[oldmanjk]

Any updates?

[sammcj]

@BruceMacD would you please be able to update (rebase) your branch from main?

[BruceMacD]

ggerganov/llama.cpp@a33e6a0

[mxyng]

The test might be a flake. I suggest retrying and if it fails again, maybe ping @dhiltgen

[BruceMacD]

@sammcj sorry for the delay!

[sammcj]

Nice work, this is awesome!

[Xanton19]

Awesome, great job! Just wanted to comment that IQ3_M seems to be absent from the list (referring to "filetype.go"), which means IQ3_M would not work? Otherwise all the other quants seem to be present.

[wwjCMP]

I'm curious about which version of ollama will supported this feature.

[sammcj]

@wwjCMP there hasn't been a final release in two weeks, but v0.1.39 which is currently marked as pre-release has it - https://github.com/ollama/ollama/releases

[wwjCMP]

@wwjCMP there hasn't been a final release in two weeks, but v0.1.39 which is currently marked as pre-release has it - https://github.com/ollama/ollama/releases

thanks