Apple researchers have co-authored a new study demonstrating significant performance improvements in an open-source large language model (LLM) by employing a simple productivity technique: instructing the LLM to check its own work using checklists.
The study delves into the realm of LLM refinement, which typically involves a post-training process known as Reinforcement Learning from Human Feedback (RLHF). RLHF relies on human labelers providing feedback, such as thumbs up or thumbs down, to evaluate the model’s responses. This feedback helps the LLM learn which answers are considered more desirable, thereby enhancing its overall usefulness.
The broader field of “alignment” plays a crucial role in this post-training phase, focusing on ensuring that LLMs behave in a helpful and safe manner. A misaligned model could potentially learn to manipulate human feedback by generating outputs that appear correct superficially but fail to address the underlying task effectively.
While various methods exist to improve a model’s reliability and alignment throughout the pre-training, training, and post-training stages, this study concentrates specifically on RLHF.
Titled “Checklists Are Better Than Reward Models For Aligning Language Models,” the Apple study introduces a checklist-based reinforcement learning scheme called Reinforcement Learning from Checklist Feedback (RLCF). This approach evaluates responses on a scale of 0 to 100, based on how well they satisfy each item on the checklist. The initial results indicate promising outcomes.
According to the researchers, “We compare RLCF with other alignment methods applied to a strong instruction following model (Qwen2.5-7B-Instruct) on five widely-studied benchmarks – RLCF is the only method to improve performance on every benchmark, including a 4-point boost in hard satisfaction rate on FollowBench, a 6-point increase on InFoBench, and a 3-point rise in win rate on Arena-Hard. These results establish checklist feedback as a key tool for improving language models’ support of queries that express a multitude of needs.”
The study’s findings hold particular significance for AI-powered assistants, which are poised to become the primary interface through which millions of users interact with their devices. The researchers emphasize that “Language models must follow user instructions to be useful. As the general public integrates language model-based assistants into their completion of daily tasks, there is an expectation that language models can faithfully follow the users’ requests. As users develop more confidence in models’ ability to fulfill complex requests, these models are increasingly given rich, multi-step instructions that require careful attention to specifications.”
A key aspect of the study lies in the method used to generate the checklists and assign importance weights to each item. This process is facilitated by an LLM. Building upon previous research, Apple’s researchers generated “checklists for 130,000 instructions (…) to create a new dataset, WildChecklists. To generate candidate responses for our method, we use Qwen2.5-0.5B, Qwen2.5-1.5B, Qwen2.5-3B, and Qwen2.5-7B. Qwen2.5-72B-Instruct is the checklist generator model (…).”
Essentially, the researchers augment each user instruction with a checklist of specific yes/no requirements. For example, a checklist item might ask, “Is this translated into Spanish?” A larger teacher model then scores candidate responses against each checklist item, and these weighted scores serve as the reward signal for fine-tuning the student model.
The results of the study demonstrate that with appropriate systems in place to create optimized checklists for each prompt, the researchers observed gains of up to 8.2% in one of the benchmarks used to test the method. Furthermore, the solution outperformed alternative methods in several other benchmarks.
The researchers clarify that their study focused on “complex instruction following” and that RLCF may not be the most suitable reinforcement learning technique for all use cases. They also acknowledge that their method utilizes a more powerful model to evaluate and tune a smaller model, which represents a significant limitation. Most importantly, they state that “RLCF improves complex instruction following, but is not designed for safety alignment.”
Despite these limitations, the study presents a novel and straightforward approach to enhancing reliability in the interaction between humans and LLM-based assistants. This is particularly crucial as these assistants increasingly acquire agentic capabilities, where instruction following and alignment become paramount.
The study underscores the potential of simple productivity techniques, such as checklists, to significantly improve the performance and reliability of LLMs, particularly in the context of complex instruction following and AI-powered assistants.
