LLaVA v1.6 34B can not run

[RussRobin]

Describe the issue

Issue:
Can not use LLaVA v1.6 34B

Command:

from llava.model.builder import load_pretrained_model
from llava.mm_utils import get_model_name_from_path
from llava.eval.run_llava import eval_model

model_path = .../ckpt-v1.6-34b" # I download LLaVA v1.6 34B from hugging face directly: https://huggingface.co/liuhaotian/llava-v1.6-34b

tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path=model_path,
    model_base=None,
    model_name=get_model_name_from_path(model_path)
)

Log:

Traceback (most recent call last):
  File ".../miniconda3/envs/llava/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File ".../miniconda3/envs/llava/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File ".../LLaVA/llava/serve/cli.py", line 126, in <module>
    main(args)
  File ".../LLaVA/llava/serve/cli.py", line 32, in main
    tokenizer, model, image_processor, context_len = load_pretrained_model(args.model_path, args.model_base, model_name, args.load_8bit, args.load_4bit, device=args.device)
  File ".../LLaVA/llava/model/builder.py", line 142, in load_pretrained_model
    model = AutoModelForCausalLM.from_pretrained(model_path, low_cpu_mem_usage=True, **kwargs)
  File ".../miniconda3/envs/llava/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 461, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
  File ".../miniconda3/envs/llava/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 998, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
  File ".../miniconda3/envs/llava/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 710, in __getitem__
    raise KeyError(key)
KeyError: 'llava'

Package version:
transformers==4.31.0

If i use transformers==4.41.0
The error would be:

Traceback (most recent call last):
  File ".../LLaVA/llava/eval/hf-quick-start.py", line 7, in <module>
    tokenizer, model, image_processor, context_len = load_pretrained_model(
  File ".../LLaVA/llava/model/builder.py", line 142, in load_pretrained_model
    model = AutoModelForCausalLM.from_pretrained(model_path, low_cpu_mem_usage=True, **kwargs)
  File ".../miniconda3/envs/llava/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 566, in from_pretrained
    raise ValueError(
ValueError: Unrecognized configuration class <class 'transformers.models.llava.configuration_llava.LlavaConfig'> for this kind of AutoModel: AutoModelForCausalLM.
Model type should be one of BartConfig, BertConfig, BertGenerationConfig, BigBirdConfig, BigBirdPegasusConfig, BioGptConfig, BlenderbotConfig, BlenderbotSmallConfig, BloomConfig, CamembertConfig, LlamaConfig, CodeGenConfig, CohereConfig, CpmAntConfig, CTRLConfig, Data2VecTextConfig, DbrxConfig, ElectraConfig, ErnieConfig, FalconConfig, FuyuConfig, GemmaConfig, GitConfig, GPT2Config, GPT2Config, GPTBigCodeConfig, GPTNeoConfig, GPTNeoXConfig, GPTNeoXJapaneseConfig, GPTJConfig, JambaConfig, JetMoeConfig, LlamaConfig, MambaConfig, MarianConfig, MBartConfig, MegaConfig, MegatronBertConfig, MistralConfig, MixtralConfig, MptConfig, MusicgenConfig, MusicgenMelodyConfig, MvpConfig, OlmoConfig, OpenLlamaConfig, OpenAIGPTConfig, OPTConfig, PegasusConfig, PersimmonConfig, PhiConfig, Phi3Config, PLBartConfig, ProphetNetConfig, QDQBertConfig, Qwen2Config, Qwen2MoeConfig, RecurrentGemmaConfig, ReformerConfig, RemBertConfig, RobertaConfig, RobertaPreLayerNormConfig, RoCBertConfig, RoFormerConfig, RwkvConfig, Speech2Text2Config, StableLmConfig, Starcoder2Config, TransfoXLConfig, TrOCRConfig, WhisperConfig, XGLMConfig, XLMConfig, XLMProphetNetConfig, XLMRobertaConfig, XLMRobertaXLConfig, XLNetConfig, XmodConfig, LlavaConfig, LlavaMptConfig, LlavaMistralConfig.

[itay1542]

Make sure that your model config (config.json file in huggingface) has model_type set to 'llava' and not 'llava-llama'

[chrisx599]

i meet same problem, i use transformers==4.37.2, and it return the same error like above screenshot.
and it's llava 1.6 34b, i check the config.json, model_type is "llava"

[chrisx599]

I solve the problem, it's not transformers's version problem
the model weight folder name should keep same like the original name on huggingface

because I found the function here, the author get model name use the path's last folder name
so when I change the weight folder name "llava-v1.6-34b", it worked

[RussRobin]

Great! Millions of thanks to you guys. I'll close this issue.