Maciej Besta, Florim Memedi, Zhenyu Zhang, Robert Gerstenberger, Guangyuan Piao, Nils Blach, Piotr Nyczyk, Marcin Copik, Grzegorz Kwaśniewski, Jürgen Müller, Lukas Gianinazzi, Ales Kubicek, Hubert Niewiadomski, Aidan O'Mahony, Onur Mutlu, Torsten Hoefler:
Demystifying Chains, Trees, and Graphs of Thoughts
(arXiv:2401.14295. Jan. 2024)
Abstract
The field of natural language processing (NLP) has witnessed significant progress in recent years, with a notable focus on improving large language models' (LLM) performance through innovative prompting techniques. Among these, prompt engineering coupled with structures has emerged as a promising paradigm, with designs such as Chain-of-Thought, Tree of Thoughts, or Graph of Thoughts, in which the overall LLM reasoning is guided by a structure such as a graph. As illustrated with numerous examples, this paradigm significantly enhances the LLM's capability to solve numerous tasks, ranging from logical or mathematical reasoning to planning or creative writing. To facilitate the understanding of this growing field and pave the way for future developments, we devise a general blueprint for effective and efficient LLM reasoning schemes. For this, we conduct an in-depth analysis of the prompt execution pipeline, clarifying and clearly defining different concepts. We then build the first taxonomy of structure-enhanced LLM reasoning schemes. We focus on identifying fundamental classes of harnessed structures, and we analyze the representations of these structures, algorithms executed with these structures, and many others. We refer to these structures as reasoning topologies, because their representation becomes to a degree spatial, as they are contained within the LLM context. Our study compares existing prompting schemes using the proposed taxonomy, discussing how certain design choices lead to different patterns in performance and cost. We also outline theoretical underpinnings, relationships between prompting and others parts of the LLM ecosystem such as knowledge bases, and the associated research challenges. Our work will help to advance future prompt engineering techniques.
Documents
download article:
Recorded talk (best effort)
BibTeX
@article{besta2024topologies, author={Maciej Besta and Florim Memedi and Zhenyu Zhang and Robert Gerstenberger and Guangyuan Piao and Nils Blach and Piotr Nyczyk and Marcin Copik and Grzegorz Kwaśniewski and Jürgen Müller and Lukas Gianinazzi and Ales Kubicek and Hubert Niewiadomski and Aidan O'Mahony and Onur Mutlu and Torsten Hoefler}, title={{Demystifying Chains, Trees, and Graphs of Thoughts}}, journal={arXiv:2401.14295}, year={2024}, month={Jan.}, source={http://www.unixer.de/~htor/publications/}, }
