Using llama3 via Ollama within FunctionCallingAgentWorker

[AashiDutt]

Hello,

I want to use local llm within agent worker without using any APIs. Here is the code I'm trying to work around with

from llama_index.core.agent import FunctionCallingAgentWorker
from llama_index.core.agent import AgentRunner
from llama_index.llms.ollama import Ollama

llm = Ollama(model="llama2")

agent_worker = FunctionCallingAgentWorker.from_tools(
initial_tools,
llm = llm,
verbose = True
)

agent = AgentRunner(agent_worker)
This code returned:

Cell In[107], line 4
1 from llama_index.core.agent import FunctionCallingAgentWorker
2 from llama_index.core.agent import AgentRunner
----> 4 agent_worker = FunctionCallingAgentWorker.from_tools(
5 initial_tools,
6 llm = llm,
7 verbose = True
8 )
10 agent = AgentRunner(agent_worker)

File ~/miniconda3/envs/DL/lib/python3.10/site-packages/llama_index/core/agent/function_calling/step.py:125, in FunctionCallingAgentWorker.from_tools(cls, tools, tool_retriever, llm, verbose, max_function_calls, callback_manager, system_prompt, prefix_messages, **kwargs)
121 prefix_messages = [ChatMessage(content=system_prompt, role="system")]
123 prefix_messages = prefix_messages or []
--> 125 return cls(
126 tools=tools,
127 tool_retriever=tool_retriever,
128 llm=llm,
129 prefix_messages=prefix_messages,
130 verbose=verbose,
131 max_function_calls=max_function_calls,
132 callback_manager=callback_manager,
133 **kwargs,
134 )

...
71 )
72 self._llm = llm
73 self._verbose = verbose

ValueError: Model name llama2 does not support function calling API.

[dosubot[bot]]

The error you're encountering, "Model name llama2 does not support function calling API," indicates that the llama2 model you're attempting to use with FunctionCallingAgentWorker is not configured to support the function calling API. This is determined by a check on the model's metadata within the FunctionCallingAgentWorker's constructor, specifically looking for the is_function_calling_model property to be set to True.

To resolve this issue, you need a model that is explicitly marked to support function calling. This typically involves ensuring the model's metadata has the is_function_calling_model property set to True. Since the Ollama model or its metadata does not provide a direct way to modify this property based on the information provided, you might need to use a model that already supports function calling or find a way to adjust the metadata of the llama2 model to reflect this capability.

If you're working directly with the Ollama model or any other model and facing this issue, consider checking the documentation or source code of the model to see if there's a recommended way to enable function calling support or if a different model variant is required for your use case.

Details

Sources

• [Question]: Does OpenAIAgent support the use of customllm
• Can I use llama-index based on a locally deployed model?
• how can I use my model to predict answer?
• llama-index-core/llama_index/core/agent/function_calling/step.py
• llama-index-core/llama_index/core/llms/function_calling.py
• llama-index-legacy/llama_index/legacy/core/llms/types.py
• llama-index-core/llama_index/core/base/llms/types.py
• llama-index-legacy/llama_index/legacy/llms/ollama.py
• llama-index-integrations/llms/llama-index-llms-ollama/llama_index/llms/ollama/base.py
• llama-index-legacy/llama_index/legacy/multi_modal_llms/ollama.py
• llama-index-integrations/multi_modal_llms/llama-index-multi-modal-llms-ollama/llama_index/multi_modal_llms/ollama/base.py
• llama-index-core/llama_index/core/program/function_program.py
• llama-index-integrations/llms/llama-index-llms-llama-api/llama_index/llms/llama_api/base.py

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[AashiDutt]

@dosu how can i use tool calling without using the following code block.

`from llama_index.core.agent import FunctionCallingAgentWorker
from llama_index.core.agent import AgentRunner

agent_worker = FunctionCallingAgentWorker.from_tools(
initial_tools,
llm = llm,
verbose = True
)

agent = AgentRunner(agent_worker)`

The following remains unchanged:

`from llama_index.llms.ollama import Ollama

llm = Ollama(model="llama2")`

[dosubot]

To use tool calling with the llama2 model in Ollama without relying on the FunctionCallingAgentWorker and AgentRunner, you can create a custom subclass of LLM. This approach involves manually handling the selection and calling of tools based on your specific requirements. Here's a basic framework to get you started:

from llama_index.core.llms.llm import LLM
from llama_index.core.tools.calling import call_tool_with_selection
from llama_index.core.base.llms.types import ChatMessage, ChatResponse

class Llama2ToolCaller(LLM):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        # Additional initialization here

    def chat_with_tools(self, tools, user_msg=None, chat_history=None, verbose=False, **kwargs):
        # Implement your logic for selecting tools here
        selected_tool = self.select_tool(tools, user_msg, chat_history)
        
        # Call the selected tool
        tool_output = call_tool_with_selection(selected_tool, tools, verbose=verbose)
        
        # Format and return the response
        response = ChatResponse(content=tool_output.content, metadata=tool_output.metadata)
        return response

    def select_tool(self, tools, user_msg, chat_history):
        # Implement your tool selection logic here
        return tools[0]  # Example: simply select the first tool

This example demonstrates how to subclass LLM to manually manage tool selection and calling. You'll need to flesh out the select_tool method with your logic for choosing the right tool based on the user's message or the ongoing chat history. This custom approach provides the flexibility to handle tool calls without the need for the function calling API, accommodating the limitations of the llama2 model within the Ollama framework.

Details

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[logan-markewich]

@AashiDutt you should use ReActAgentWorker -- the function calling agent worker is only for LLMs with a specific API built-in for tool calling (this includes openai, anthropic, and mistral)