Enhancing Large Language Models with System 2 Distillation

Large language models (LLMs) have demonstrated exceptional performance in answering simple questions efficiently. However, when faced with complex tasks that involve reasoning and planning, LLMs often struggle. While prompting techniques such as “System 2” have been effective in enhancing the reasoning capabilities of LLMs, they come at the cost of increased computational complexity and slower performance.

In cognitive science, System 1 and System 2 are two distinct modes of thinking. System 1 is fast, intuitive, and automatic, used for recognizing patterns and making quick judgments. On the other hand, System 2 is slow, deliberate, and analytical, requiring conscious effort for complex problem-solving tasks. LLMs are typically likened to System 1 thinking, excelling in generating text quickly but facing challenges with tasks that demand deliberate reasoning and planning.

Researchers at Meta FAIR have introduced “System 2 distillation,” a technique aimed at teaching LLMs complex tasks without the need for intermediate reasoning steps. This approach draws inspiration from the phenomenon observed in humans where tasks initially requiring conscious effort gradually become automated over time.

System 2 distillation involves leveraging the model’s own System 2 reasoning capabilities and distilling that knowledge into its fast-paced and compute-efficient System 1 generation. By prompting the LLM to solve problems using System 2 techniques and verifying responses for correctness, the researchers selectively retain final answers while discarding intermediate reasoning steps. This process allows the model to skip the reasoning phase and directly provide answers, significantly improving efficiency.

The researchers evaluated System 2 distillation on various reasoning tasks using different System 2 prompting techniques. The results demonstrated a notable improvement in the performance of LLMs on complex tasks, often surpassing the accuracy of original System 2 methods. Moreover, the distilled models exhibited faster response times and reduced computational overhead, showcasing the effectiveness of the distillation process.

While System 2 distillation showed promising results, there were limitations in distilling certain types of reasoning skills, particularly complex math tasks requiring detailed Chain-of-Thought prompting. Further research is warranted to explore the compatibility of distillation with smaller models and its broader impact on tasks beyond those included in the training dataset. Additionally, addressing challenges such as model contamination in LLM benchmarks will be crucial for the widespread adoption of distillation techniques.

System 2 distillation presents a compelling approach to enhance the performance of LLMs in handling complex reasoning tasks efficiently. By leveraging the model’s own reasoning capabilities and streamlining the decision-making process, distillation offers a promising avenue for optimizing LLM pipelines and facilitating more effective task execution. As research in this field evolves, incorporating distillation techniques holds significant potential for advancing the capabilities of large language models.