# Same paper. Different places.

Peers put ten papers in order, twice. An order is easier to compare than a score, and it shows where a paper sits among its neighbors.

- [Rank ten papers](/ballot/)

## A rank needs no shared scoring scale.

One researcher is sparing with an 8 out of 10. Another gives almost everything an 8. To combine their scores, you first have to guess how each one uses the scale.

Asking each peer to put ten papers in order removes that step. Every paper gets a place from first to tenth, and a place means the same thing to everyone: this one ahead of that one.

A rank still depends on which nine papers sit beside yours. That is why the sets overlap and why each paper needs several readers, not one. Several comparisons from different peers give the cohort more than a single list to work with.

**Two orders, kept apart.** Each peer orders the same ten papers twice. First by expected research impact over two years: which papers will other researchers build on and cite? Then by interest: which talks would you go to in person? The first order feeds the shortlist for full review. The second goes to people planning events and never to a publication decision. A paper can lead one list and not the other.



## Ten distinct peers per paper, on each question.

These figures describe the design capacity of a cohort of 1,000 submissions. No cohort has run yet.

| Quantity | At 1,000 submissions |
| --- | --- |
| Completed double ballots | 1,000 |
| Placements per question | 10,000 (1,000 ballots × 10 papers) |
| Average sets that contain a given paper | 10 |
| Distinct peers required per paper, per question | At least 10 |
| Finalists (about the top 30% by expected impact) | About 300 |
| Full reviews (3–5 per finalist team) | 900–1,500 |

Two orders from one peer count as one peer. A team that submits several papers owes one double ballot for each.

**A worked example with missing ballots.** If every ballot comes back, each paper appears in ten sets on average, which just meets the minimum. Suppose only 90% come back. That is 900 ballots and 9,000 placements, so the average paper has 9 peers, and some have fewer. Those papers are not counted as ranked. The cohort would replace missing ballots with extra assignments until every paper has ten distinct peers.

**Why overlap matters.** Sets are balanced and connected, so a paper that drew easy company in one set can be compared through another. A raw average of within-set positions can favor papers that happened to sit among weaker neighbors, so the cohort aggregates comparisons across sets instead.

## What remains uncertain.

This is a proposed method. Nothing here shows yet that peer orders pick better papers than a committee does.

- **Prediction.** Whether a frozen impact order tracks later citations is a question for a pilot. The two-year horizon is a wording choice that gives "impact" a time frame an author can understand and an analyst can test.
- **Familiarity.** Peers rank summaries, not full manuscripts. A summary can flatter a paper or miss its flaw, which is why finalists get full reviews.
- **Two measures.** An impact order is not an Impact Factor. A journal's [Impact Factor](https://clarivate.libguides.com/jcr/) counts citations in a reporting year to items from the two preceding years, divided by eligible items. A ballot forecast and a paper's lifetime citations are different measures.
- **Talk interest.** Whether local interest becomes attendance can only be checked once meetings happen.

## How a pilot would be tested.

Would a fresh group of peers choose the same papers? That is what a pilot should answer.

1. **Freeze the usual shortlist.** A journal, society or track picks one decision it already makes and records its current result before any ballot opens.
2. **Collect overlapping orders.** Submitting teams order overlapping sets of ten. No result is shown until orders are in.
3. **Freeze the predictions.** Impact orders are fixed before anyone looks up later citations.
4. **Compare at the same size.** Put the two shortlists side by side, read the papers that appear on only one, and measure how often the shortlist changes near the cutoff.
5. **Report both outcomes, with uncertainty.** Compare local talk interest with actual attendance once meetings take place. Report that result separately from impact.

## Try it on ten papers.

A practice set of ten historical papers. Put them in order twice and see where your two lists differ.

- [Rank ten papers](/ballot/)

---
Source: https://tenpeers.com/method/
