Illustrative

Example issue, The inference issue

When should a small model ask for help?

Mira Vale (example author). Efficient inference.

Abstract

Can a routing threshold keep a large model off routine cases while sending it the decisions that need another read?

An encoder makes the first prediction. Moving a confidence threshold changes the fraction of cases a second model sees. In 100 constructed example cases, a strict threshold sends 12 to the second model, a middle one sends 29, and a loose one sends 57.

What the authors tested

A small encoder makes the first prediction on every case. A confidence threshold decides which cases also go to a larger second model. The authors moved the threshold through three settings and counted how many of 100 example cases reached the second model at each.

Figure 1. Cases sent for a second read

Figure 1: Cases sent for a second read Horizontal bar chart. Cases sent to the second model out of 100: strict threshold 12, middle threshold 29, loose threshold 57. 0255075100Strict 12 Middle 29 Loose 57 Cases sent to the second model, out of 100 Threshold
Text alternative: a strict threshold sends 12 of 100 cases to the second model, a middle threshold sends 29, and a loose threshold sends 57.Looser thresholds send more cases to the second model. Counts are out of 100 example cases.Source: constructed values for this illustrative page. Not measured data.

Limitations

The trade-off depends on how well confidence holds up on new data. The illustrated routing counts do not establish accuracy on future traffic.

Treat the saving as a measurement on these example cases, and test it on your own traffic before using it in a deployment plan.

Cite this example

Illustrative record. There is no DOI; the identifier is assigned on publication.

Vale, M. "When should a small model ask for help?" TenPeers, The inference issue (example issue, illustrative; not a published paper).
Identifier: assigned on publication.
Cohort: 100 eligible papers (example). Expected impact 8th of 100; talk interest 2nd of 100.

Back to the sample issue