The TenPeers Society

Who chooses the papers?

The people who submit them. Bring a manuscript and you get a ballot. If your paper advances, you read other finalists in full.

Two researchers made from printed ivory paper lean in to point at different parts of the same attention map.
Two researchers, one disputed figure

Illustrative

Inside a ballot

Ten people can read the same paper differently.

Each letter stands for one peer, in both rows. Each number shows where that peer put one paper in their set of ten. First place is 1. Peer D put this paper fifth for impact and first for a talk.

Expected impact: each peer's rank for the paper

  • 1A
  • 2B
  • 2C
  • 5D
  • 1E
  • 3F
  • 4G
  • 2H
  • 6I
  • 1J

Talk interest: each peer's rank for the paper

  • 7A
  • 4B
  • 5C
  • 1D
  • 2E
  • 5F
  • 8G
  • 3H
  • 6I
  • 7J

Illustrative ballots for one paper. Both rows show the same ten peers. Peers do not see other peers' ranks until both of their own orders are committed. The published rule combines the impact placements with full reviews; the talk placements go to workshop programming.

The author

Bring a paper.

Submit the manuscript you already have. Approve the summary that other researchers will rank.

Author guide ↗

The peer

Return the read.

Each submitting team ranks ten papers. One peer from each finalist team (the top 30% of papers by expected impact) writes three to five full reviews of other finalists.

Try a peer ballot ↗

The cohort

Decide together.

The published rule combines the impact ballot with the full reviews. Talk interest guides the local workshop program.

Read the method ↗

After the decision

What belongs on a researcher's record?

Suppose your paper missed publication, but several peers put it first on their list of talks. An acceptance line on a CV would lose that information. A TenPeers record is designed to keep the two placements with the topic and the number of papers in that month's cohort.

You choose whether to share an unpublished result. Someone opening your profile would see what fellow researchers saw in that particular paper. When your next paper enters a ballot, its readers make a fresh judgment.

Researcher records are planned for a later release; they are not part of the first cohort.

The picture on this page

Is attention explanation?

In a 2019 movie-review experiment, the strongest highlight moves from "waste" to "was." The prediction stays at 0.01. Sarthak Jain and Byron C. Wallace used examples like this to question what an attention map can tell us about a prediction. Sarah Wiegreffe and Yuval Pinter raised a different test: could a model learn a different way of assigning attention that also works across reviews?

  • Figure. Sarthak Jain and Byron C. Wallace, "Attention is not Explanation," NAACL 2019, Figure 1. © 2019 Association for Computational Linguistics. Two details cropped and enlarged, with the authors' reported prediction scores reset as type. Paper and figure · CC BY 4.0
  • Response. Sarah Wiegreffe and Yuval Pinter, "Attention is not not Explanation," EMNLP-IJCNLP 2019. Paper
  • Illustration. The two paper researchers are an AI-generated TenPeers editorial illustration of a published exchange between two author teams.

Take the other side of a submission.

Try the two ballots on ten historical papers. Rank them twice, then compare your orders.