Scoring, Reasoning, and Selecting the Best!
Ensembling Large Language Models via a Peer-Review Process

Zhijun Chen†, Zeyu Ji†‡, Qianren Mao, Hao Wu, Jinhuan Song‡, Junhang Cheng, Bangjie Qin, Zhuoran Li, Jingzheng Li, Kai Sun, Zizhe Wang, Yikun Ban, Zhu Sun, Xiangyang Ji, Hailong Sun*, Xiao Huang
Beihang University, Beijing, China
The Hong Kong Polytechnic University, Hong Kong, China
† Equal contribution. ‡ Core experimental contributors. * Corresponding author.
Paper Code arXiv HF Paper 📑 Blog

Abstract

We propose LLM-PeerReview, an unsupervised LLM Ensemble method that selects the most ideal response from multiple LLM-generated candidates for each query, harnessing the collective wisdom of multiple models with diverse strengths. LLM-PeerReview is built on a novel, peer-review-inspired framework that offers a transparent and interpretable mechanism, while remaining fully unsupervised for flexible adaptability and generalization. Specifically, it operates in three stages: For scoring, we use the emerging LLM-as-a-Judge technique to evaluate each response by reusing multiple LLMs at hand; For reasoning, we can apply a straightforward averaging strategy or a principled graphical model-based truth inference algorithm to aggregate multiple scores to produce a final score for each response; Finally, the highest-scoring response is selected as the best ensemble output. LLM-PeerReview is conceptually simple and empirically powerful. Our results across four datasets show that the two variants of the proposed approach outperform the advanced model Smoothie-Global by 6.9% and 7.3% points, across diverse task types including factual recall QA, math reasoning, and instruction following.

Notably, we also establish a carefully curated benchmark suite for LLM Ensemble, integrating 12 methods across four classic datasets and three task families, all evaluated under a rigorous and consistent protocol. We hope this repository will help researchers reproduce the LLM Ensemble baselines.

Overview of LLM-PeerReview

Overview of the LLM-PeerReview framework

The proposed LLM-PeerReview contains three steps:
(1) Scoring: For a given query, after each LLM independently generates a response (analogous to a submitted academic paper), LLM-PeerReview applies the LLM-as-a-Judge technique (and the proposed flipped-triple scoring trick), treating each model as a reviewer to assign scores to all candidate responses;
(2) Reasoning: LLM-PeerReview then uses a truth inference algorithm—analogous to a senior reviewer—to estimate a final score for each response;
(3) Selecting the best: Finally, for each query, LLM-PeerReview selects the response with the highest final score as the ensemble output—analogous to how a senior reviewer chooses the most ideal paper from a specific submission pool.

Motivation / Introduction

The artificial intelligence domain has undergone a massive transformation recently, driven by the emergence of Large Language Models (LLMs) such as Gemini, GPT-4, Llama, and DeepSeek. The success of these models has triggered a surge in research activity, with over 182,000 models now available on Hugging Face.

Behind this research enthusiasm, we can observe two main points: 1) Persistent performance concerns: Although large language models can be easily deployed for zero-shot or in-context few-shot inference, they still face common performance issues, such as limited accuracy, hallucinations, and misalignment with human goals; 2) The varying strengths and weaknesses of LLMs: These models display significant behavioral differences, primarily driven by variations in their architecture, scale, training data, dictionary, tokenization and methodology. Consequently, their responses to the same prompt often diverge. With the above two points in mind and inspired by the spirit of Ensemble Learning, it is reasonable to suggest that relying on a single LLM—even one with a high public ranking or other criteria—may not be the optimal strategy for every user query. Instead, it might be more advantageous to simultaneously consider multiple LLM candidates (which are usable out-of-the-box) and leverage their distinct strengths. This concept is the core focus of the burgeoning field of LLM Ensemble.

As LLM Ensemble gains increasing attention, one well-established class of solutions—ensemble-after-inference (also known as post-hoc ensemble) methods—has emerged. These methods include the following two representative approaches:

  • Selection-then-regeneration approaches rely heavily on curated task-specific data and require fine-tuning additional models, which severely restricts their generalization and adaptability across different domains.
  • Similarity-based selection approaches suffer from coarse-grained designs and naive strategies (e.g., using shallow metrics like BLEU), failing to fully utilize the deep semantic information required for optimal selection.

When we revisit this research problem, we ask the most fundamental question: In the real world, how would humans select the most ideal text from a set of candidate texts? Perhaps the most immediate and relatable real-world example is: the academic peer-review process. Motivated by this, we propose a new, fully unsupervised LLM Ensemble method called LLM-PeerReview.

Experiment Setup

Datasets and evaluation. We evaluate four widely used datasets spanning three task categories: (1) Factual Recall: TriviaQA evaluates the accuracy of model responses to factual questions across various domains, including history, science, and geography. (2) Arithmetic Reasoning: GSM8k and MATH assess basic arithmetic and more advanced mathematical reasoning, respectively, with accuracy as the evaluation metric, focusing on correct numerical answers. (3) Instruction Following: AlpacaEval tests models' ability to follow various instructions. We use GPT-4o-mini to evaluate the accuracy of model responses, assessing whether the model’s response exceeds the reference answer in the dataset. We use the same dataset versions as prior studies.

Seed LLMs and baselines. We evaluate our method using four widely adopted open-source LLMs: Llama-3.1-8B-Instruct, Mistral-7B-Instruct, Qwen2-7B-Instruct, and Qwen2.5-7B-Instruct. We compare LLM-PeerReview with two categories of baselines. (1) Single LLMs: The four LLMs above. (2) LLM Ensemble baselines: (i) Random is a random-selection baseline that simply returns the response from a randomly chosen LLM. As one of the simplest ensemble strategies, it has previously been applied to dialogue tasks; (ii) Smoothie-Global, Smoothie-Local, and Agent-Forest are recently proposed, strong similarity-based ensemble methods; (iii) GaC is a representative token-level ensemble-during-inference approach. It constructs a unified vocabulary that merges the individual dictionaries of LLMs and samples from their distributions over that vocabulary; (iv) SAFE and CoRE are two recent ensemble-during-inference methods. SAFE adopts speculative decoding and can be coupled with different base ensemble methods, yielding SAFE-GaC and SAFE-UniTE. CoRE uses token- and model-level consistency to assign weights during aggregation, yielding CoRE-GaC and CoRE-UniTE; (v) MAD is a classic multi-model collaboration method that can be applied to LLM Ensemble, where models refine their responses through debate and select the final answer by majority voting.

Results

Through comprehensive experiments, we reached the following conclusions:

The main results are shown in the table. First, by examining the results in the “Single LLM” and “LLM Ensemble” sections of the table, one key finding is that both LLM-PeerReview and LLM-PeerReview-W consistently outperform any single LLM and all LLM Ensemble baselines across all datasets. In the last column, which presents the average performance, our two variant methods (with results of 67.4% and 67.8%) surpass the strongest single model, Qwen2.5, by 4.7% and 5.1%, respectively, and outperform the advanced ensemble method, Smoothie-Global, by 6.9% and 7.3%. These results directly demonstrate the effectiveness of our method, as it achieves superior performance by integrating the collective knowledge of multiple models across factual-recall QA tasks, math reasoning tasks, and instruction-following tasks. Also, the ensemble task across these four datasets is challenging, as the performance of the four LLMs varies significantly for each dataset.
In the figure, the upper subplot presents a radar chart of individual LLM performance, while the lower subplot displays the win-tie-loss chart for models on the challenging instruction-following dataset, AlpacaEval. This figure highlights that models with the best overall performance may underperform on specific tasks compared to those with weaker overall results. In summary, the results in the table and figure demonstrate that a strong LLM does not excel across all datasets. Each model has its strengths and weaknesses, highlighting the substantial practical significance of LLM Ensemble.
In the “Our variants” of the table, we present the performance of using a single LLM as a judge to select the optimal response. From the average performances in the last column of the table, we observe that these variants perform quite well (surpassing the overall best model, Qwen2.5, in 3/4 cases). On the other hand, when comparing the performance of these variants with that of our prototype LLM-PeerReview, it becomes clear that aggregating and averaging the scores from multiple judges is beneficial.
By observing the average results in the table, we find that LLM-PeerReview-W leads to further performance gains compared to LLM-PeerReview. In the left subplot of the figure, we observe subtle variations in the transition matrices learned for each model. On the other hand, the right subplot in the figure, displaying positive correlation coefficients, demonstrates that our method can effectively identify stronger and weaker judges.
(1) It is intuitive that, in addition to our recommended flipped-triple scoring method, several variant scoring methods could be employed. (Their definitions are provided in the caption of the table.) Overall, the performance of these four variants follows the order: quadruple-half > flipped-triple > double > single. Variants quadruple-half, flipped-triple, and double all offer noticeable de-biasing performance advantages over the single-scoring strategy. On the other hand, in terms of theoretical computational complexity, the complexities of single/double/flipped-triple/quadruple-half are O(J)/O(J2)/O(J)/O(J!), respectively, with flipped-triple having the lowest computational complexity among the de-biased strategies. Further, in the table, we present the scoring efficiency of these four strategies. Compared to strategies double and quadruple-half, strategy flipped-triple is the most time-efficient. (2) Furthermore, the remarkable performance gain—from 59.7 with the original single to 66.8 with the proposed flipped triple—validates the profound debiasing impact of our scoring technique. This technique serves as a crucial cornerstone of LLM-PeerReview.
The table compares their post-generation inference costs, excluding the shared response-generation stage. On GSM8K and MATH, LLM-PeerReview requires substantially less GPU time. This advantage mainly comes from its simpler interaction pattern: our scoring stage requires only one round, in which each judge outputs a few scalar scores, whereas MAD performs two debate rounds and requires every model to generate much longer revised full responses in each round. Also, as shown in the table, our final selecting step incurs virtually no additional computational cost.
As discussed in Section 2.4, our framework allows for flexibility in reducing the judge count during deployment to further enhance efficiency, building upon the efficient flipped-triple scoring strategy. The Appendix also shows that LLM-PeerReview remains effective in a lightweight setting: using fewer judges during scoring still outperforms the baselines.
In the figure, we conduct a further analysis of how different scoring levels influence the performance of flipped-triple judges. For each scoring level, we have carefully crafted meaningful descriptions and corresponding prompts. Under these conditions, these judges exhibit varying performance, showing no consistent tendencies across the levels of 3, 5, 7, and 10. In addition, as indicated by the prompts in the Appendix, for the main experiment in the table, the scoring levels for the four datasets were 5, 3, 3, and 10, respectively.
We provide more comprehensive experimental analyses in the Appendix, including performance results of LLM-PeerReview variants using the “Single” scoring strategy, win-tie-loss charts across all models on four datasets, full analysis for the transition matrix, additional results using varying numbers of judges, the performance of LLM-PeerReview with 13B-scale LLMs, more efficiency/cost analysis for LLM-PeerReview and four case studies across the four datasets respectively.

BibTeX


@misc{chen2026scoringreasoningselectingbest,
      title={Scoring, Reasoning, and Selecting the Best! Ensembling Large Language Models via a Peer-Review Process}, 
      author={Zhijun Chen and Zeyu Ji and Qianren Mao and Hao Wu and Jinhuan Song and Junhang Cheng and Bangjie Qin and Zhuoran Li and Jingzheng Li and Kai Sun and Zizhe Wang and Yikun Ban and Zhu Sun and Xiangyang Ji and Hailong Sun and Xiao Huang},
      year={2026},
      eprint={2512.23213},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2512.23213}, 
}