News Classification: Test Confidence Before Automating Alerts

A classifier can confidently return a label and still generate the wrong alert. Check what the classifier’s probability means for the articles your application actually receives.

On November 11, 2026, Google launched the Gemini Decisions API for bounded decisions such as classification with probabilities, including multiple choices. Google explains calibrated confidence as a means to route cases for which the classifier is uncertain to a fallback or human reviewer. This design structure is useful for news classifiers, but a statement regarding the calibration of the classifier does not ensure the accuracy of your news labels.

The release question is more narrow: at what confidence threshold will the classifier be useful and what will be missed?

Define the alert before testing confidence

A topic label and an alert decision answer different questions. An article can belong to a product-safety category because it discusses recall policy without reporting a recall that requires action.

For a recall-monitoring application, define the positive class as an article reporting a newly announced recall affecting a product on the monitored list. A retrospective about an old recall is negative under that rule. If the product or announcement date cannot be resolved, label the case for review rather than quietly changing the definition.

Webz.io’s News API provides article text and metadata in structured JSON, along with topic and entity enrichments. Topical classification can help refine the candidate stream, and a custom decision layer can evaluate that stream with respect to the rule. The application owns the rule, its evaluation labels, and the threshold that determines delivery.

Write the rule where reviewers can apply it consistently. Ask two reviewers to label a small shared sample and resolve disagreements. A model can’t be calibrated against a definition that changes reviewer to reviewer.

Check calibration against labeled articles

Calibration concerns groups of predictions. According to Scikit-learn’s calibration documentation, if a classifier is well-calibrated, among the cases assigned a probability of 0.8, around 80% would belong to the positive class. This doesn’t mean that cases assigned a probability of around 0.8 are definitely classified correctly or that the event reported by an article actually occurred.

Use a representative labeled sample from your feed, including articles that should not generate alerts. Don’t limit yourself to obvious cases. Ensure near-identical reports of the same story do not appear on both sides of the development and test split.

For each probability band, find the average predicted probability and compare it to the positive fraction. If the average is high and positive fraction is low, calibration is poor for that band.

You will use only one development set to analyze score bands and to decide on thresholds. You will set aside a test set to check the policy you’ve decided on. Once you see the results of the test set, you will validate the policy again on a new set of data if you change the policy based on those results.

An example using recall-alerts

In this constructed example, suppose there are 400 news articles in the development set. Two hundred and forty of those articles are identified by human reviewers as meeting the criteria to be a recall-alert. The classification team has created a rule to automatically provide those alerts using a classifier, and has defined two possible policies.

At a threshold of 0.80, 200 articles are automatically provided. One hundred and eighty of those articles are classified correctly. Therefore, alert precision is 90% (180/200). The automated route captures 75% (180/240) of the relevant articles.

At a threshold of 0.95, the policy automatically provides 80 articles, and 78 of those articles are classified correctly. Therefore, in this case, precision is 97.5% (78/80). The automated route captures 32.5% (78/240) of the relevant articles.

If every article outside automatic delivery goes to review, the queue contains 200 articles at the lower threshold and 320 at the higher threshold.

If it is operationally required to provide alerts automatically with at least 97% precision, and the team has the capacity to review 320 articles per batch of this size, then the higher threshold policy meets the operational requirement. In the sample, the lower threshold policy does not meet the operational requirement. In all cases, the results of a policy need to be validated on new data to give a high degree of confidence in the results. Eighty automated decisions in a sample do not allow one to make firm statements about the future with that degree of certainty.

Look at the misses, not just the alerts

The remaining 162 relevant articles under the higher threshold are not delivered automatically. Just because an article is not delivered automatically does not mean the article is rejected; if there are workers available to review articles in the queue, the article can be reviewed and either delivered or rejected. Measure how long articles remain in the review queue and the size of the review queue, in addition to automating the workflow, to determine that the workflow is functioning adequately.

Where sample sizes permit, evaluate where the errors occur by source type and language. If local reports are often missed, raising confidence requirements may remove the early coverage the monitoring product was designed to capture. Thresholds cannot repair missing context.

Reviews of low scoring articles should also be done. If only the uncertain middle is reviewed, the negatives, which may be incorrect, remain invisible. Use a random audit sample from below the review cutoff if your production policy excludes a part of the feed.

Recheck after adjustments to the feed or the classifier

Evaluation records should connect each article reference to its reviewer label and model score, as well as the classifier version and delivery threshold. Upon a change to the model or label rule, compare the new policy on the same reference set, and then on new articles.

Changes in source mix can also affect the result. Should the feed add a language or a new publisher type, test this content before applying the old threshold without qualification. During rollout, compare delivered alerts to completed reviews, and ensure these are in accordance with the agreed rule. Should a segment of the feed produce an excessive number of errors, deliver alerts from that segment manually, and do not mask the issue by stating that alerts in general are being delivered with very high accuracy.