AI Agents - Reasoning, Planning, and Tool Calling
This April 2024 survey paper provides an excellent overview of the current state-of-the-art in AI agent architectures, focusing on their reasoning, planning, and tool calling capabilities.
The authors examine both single-agent and multi-agent architectures, identifying key design patterns, trade-offs, and considerations for effective agent systems.
The paper aims to communicate the current capabilities and limitations of AI agent implementations, share insights gained from observing these systems in action, and suggest important considerations for future developments in AI agent design.
The paper starts by highlighting the shift from static language models to more dynamic, autonomous agents that can engage in multi-step reasoning, planning, and interaction with external tools and data sources.
Key points from the paper
AI agents are language model-powered entities capable of planning and executing actions to achieve goals over multiple iterations.
Agent architectures can be single-agent or multi-agent, with each agent having a persona and access to various tools.
Agent personas describe the role or personality of the agent and include instructions specific to that agent, influencing the agent's behavior and performance.
Tools allow agents to interact with external data sources by pulling or pushing information.
Single-agent architectures have one language model that performs all reasoning, planning, and tool execution independently, with potential human feedback.
Multi-agent architectures involve two or more agents, which can use the same or different language models and tools, and have their own personas.
Multi-agent architectures can be categorized as vertical (with a lead agent and clear division of labour) or horizontal (all agents treated as equals, collaborating in a shared thread).
The authors identify reasoning, planning and tool use as critical capabilities for effective real-world agent performance.
Existing AI agent implementations
ReAct (Reason + Act)
Agents write a thought about the task, perform an action based on that thought, and observe the output.
This cycle repeats until the task is complete.
Improves effectiveness compared to zero-shot prompting and provides better human interoperability and trustworthiness.
Limitations: Can generate repetitive thoughts and actions, failing to create new thoughts to finish the task and exit the loop.
Incorporating human feedback during execution could increase its effectiveness and applicability.
RAISE
Builds upon the ReAct method by adding a memory mechanism that mirrors human short-term and long-term memory.
Uses a scratchpad for short-term storage and a dataset of similar previous examples for long-term storage.
Improves the agent's ability to retain context in longer conversations.
Fine-tuning the model results in the best performance, even with smaller models.
Outperforms ReAct in efficiency and output quality.
Limitations: Struggles with complex logic and agents may hallucinate with respect to their roles or knowledge.
Reflexion
A single-agent pattern that uses self-reflection through linguistic feedback.
Uses metrics such as success state, current trajectory, and persistent memory.
Uses an LLM evaluator to provide specific and relevant feedback to the agent.
Improves success rate and reduces hallucination compared to Chain-of-Thought and ReAct.
Limitations: Susceptible to "non-optimal local minima solutions" and uses a sliding window for long-term memory, limiting the volume by the token limit of the language model.
Opportunities exist to improve performance on tasks requiring diversity, exploration, and reasoning.
AutoGPT+P (Planning)
Addresses reasoning limitations for agents that command robots in natural language.
Combines object detection and Object Affordance Mapping (OAM) with an LLM-driven planning system.
Allows the agent to explore the environment for missing objects, propose alternatives, or ask the user for assistance.
Uses an image of a scene to detect objects, and an LLM selects tools (Plan, Partial Plan, Suggest Alternative, Explore) to generate goals and steps.
Works alongside a classical planner that executes the plan using Planning Domain Definition Language (PDDL).
Limitations: Accuracy of tool selection varies, and the agent may get stuck in loops or make illogical exploration decisions. Limited human interaction capabilities.
LATS (Language Agent Tree Search)
A single-agent method that synergises planning, acting, and reasoning by using trees.
Represents a state as a node and taking an action as traversing between nodes.
Uses LM-based heuristics to search for possible options and selects an action using a state evaluator.
Implements a self-reflection reasoning step that improves performance compared to other tree-based methods.
Uses environmental feedback and LM feedback to determine errors in reasoning and propose alternatives.
Performs well on various tasks but uses more computational resources and time than other single-agent methods.
Tested on relatively simple question-answering benchmarks; not yet tested on robust scenarios involving tool calling or complex reasoning.
The five main approaches to planning
task decomposition
multi-plan selection
external module-aided planning
reflection and refinement
memory-augmented planning
Not all good
The paper provides a comprehensive survey of the field of AI agent architectures.
While current approaches demonstrate impressive reasoning, planning and tool use capabilities, significant open challenges remain in developing robust, reliable, and unbiased agent systems for real-world applications.
Inconsistent and biased evaluation benchmarks
Problem: Many research teams introduce their own unique agent benchmarks, making it difficult to compare multiple agent implementations. Some benchmarks are manually scored, introducing potential bias.
Ideas to fix:
Develop standardised, widely-accepted benchmarks for evaluating AI agents across various tasks and environments.
Encourage collaboration among research teams to create diverse, unbiased, and comprehensive benchmarks.
Incorporate a mix of objective metrics (success rate, efficiency) and subjective metrics (robustness, reliability) in the benchmarks.
Data contamination and static benchmarks
Problem: Language models' training data may contain benchmark questions, leading to inflated performance scores. Static benchmarks fail to keep pace with the rapid progress of language models.
Ideas to fix:
Develop dynamic benchmarks resistant to memorisation and adaptable to the evolving capabilities of language models.
Explore the use of synthetic benchmarks tailored to specific environments or use cases while maintaining correctness and problem-solving ability.
Regularly update and expand benchmarks to match the progress of language models.
Limited scope and transferability of benchmarks
Problem: Many language model benchmarks focus on single-iteration tasks without tool calls, not accounting for agents' ability to reason over multiple steps or access external information.
Ideas to fix:
Design benchmarks that evaluate agents' reasoning abilities over multiple steps and incorporate tool usage.
Develop benchmarks that assess agents' performance in diverse environments (e.g., web browsing, command-line interfaces, video games) to test generalizability.
Include a mix of logic puzzles, real-world tasks, and domain-specific challenges in the benchmarks.
Real-world applicability and noisy data
Problem: It is unclear whether performance on current benchmarks translates to real-world performance, as real-world data can be noisy and cover a wider breadth of topics.
Ideas to fix:
Create benchmarks using real-world datasets, such as conversations with chatbots or issues raised on software development platforms.
Develop domain-specific benchmarks that closely resemble real-world tasks and challenges.
Evaluate agents' performance on a diverse range of real-world tasks to assess their adaptability and robustness.
Bias and fairness in agent systems
Problem: Language models exhibit biases in evaluation and social/fairness aspects, which can lead to faulty reasoning in agent-based implementations.
Ideas to fix:
Conduct extensive research to identify and mitigate biases within language models and agent systems.
Develop benchmarks that specifically test for bias and fairness in agent responses and decision-making.
Incorporate human evaluation alongside automated evaluation to detect and address biases in agent systems.
Train language models on diverse and inclusive datasets to reduce inherent biases.
Scalability and efficiency of agent systems
Problem: As the complexity of tasks and agent involvement increases, ensuring the scalability and efficiency of agent systems becomes challenging.
Ideas to fix:
Optimise agent architectures for scalability, enabling them to handle large-scale tasks and data.
Develop techniques for efficient resource allocation and load balancing in multi-agent systems.
Explore the use of distributed computing and parallel processing to enhance the performance of agent systems.
By addressing these issues and implementing the proposed ideas, researchers and developers can work towards building more reliable, unbiased, and robust AI agent systems that can effectively tackle real-world challenges.
Collaboration, standardisation, and continuous improvement will be key to advancing the field of AI agents.
References
AI Collaboration and Multi-Agent Systems
Weize Chen et al., "AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors" (Oct. 2023) - Discusses the dynamics of multi-agent collaboration and the emergence of behaviors in such systems.
Xudong Guo et al., "Embodied LLM Agents Learn to Cooperate in Organized Teams" (2024) - Explores how embodied LLM agents can effectively cooperate within organized team structures.
Sirui Hong et al., "MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework" (2023) - Details a framework for meta-programming that supports collaboration in multi-agent systems.
Large Language Models and Task Planning
Timo Birr et al., "AutoGPT+P: Affordance-based Task Planning with Large Language Models" (Feb. 2024) - Integrates task planning with LLMs through an affordance-based approach, enhancing automated decision-making capabilities.
Xueyang Feng et al., "Large Language Model-based Human-Agent Collaboration for Complex Task Solving" (2024) - Investigates how human-agent collaboration can be enhanced using LLMs for solving complex tasks.
Fangru Lin et al., "Graph-enhanced Large Language Models in Asynchronous Plan Reasoning" (Feb. 2024) - Presents enhancements in LLMs using graph structures to improve asynchronous planning and reasoning.
AI Ethics and Social Implications
Isabel O. Gallegos et al., "Bias and Fairness in Large Language Models: A Survey" (Mar. 2024) - Provides a comprehensive review of the biases present in LLMs and discusses fairness considerations.
Peter S. Park et al., "AI Deception: A Survey of Examples, Risks, and Potential Solutions" (Aug. 2023) - Surveys the risks associated with AI deception and proposes potential solutions to mitigate these issues.
Enhancing LLM Capabilities and Understanding
Karl Cobbe et al., "Training Verifiers to Solve Math Word Problems" (Nov. 2021) - Focuses on developing verifiers within LLMs to enhance their ability to solve math word problems accurately.
Na Liu et al., "From LLM to Conversational Agent: A Memory Enhanced Architecture with Fine-Tuning of Large Language Models" (Jan. 2024) - Explores the transformation of LLMs into conversational agents with enhanced memory capabilities.
Zijun Liu et al., "Dynamic LLM-Agent Network: An LLM-agent Collaboration Framework with Agent Team Optimization" (2023) - Discusses a framework for optimizing collaboration between LLMs and other agents in team settings.
Last updated
Was this helpful?