News Search API Benchmarks: Count Every Query in Your Trial

A news search API can return excellent results when a request succeeds and still leave an AI agent waiting without usable evidence. Before comparing relevance scores, check which searches entered the average, which were excluded, and whether the responses arrived within your application’s deadline.

NEEDLE, Keenable’s live search benchmark, published a fresh run on November 2, 2026, at 07:10 UTC. Its current methodology assigns zero to service failures but excludes account and plan errors from the mean. That distinction gives developers a practical starting point: measure retrieval quality and the ability to complete your workload separately.

Read the benchmark’s exclusions first

NEEDLE evaluates returned rankings, titles and snippets without fetching pages or reranking results. Its protocol separates service errors from account errors, and its latency figures cover successful search calls. Keenable operates the benchmark and discloses that its judge has not undergone a systematic human-agreement audit.

Those rules describe the measurement. They do not establish how your own account, client, traffic or deadlines will behave. A quota exclusion may be reasonable when comparing engines, yet an exhausted account still prevents your application from retrieving evidence. Keep that operational outcome visible without mislabeling it as poor relevance.

Define a useful response before running queries

Start with questions your application actually receives. Save their exact wording, required languages, publisher restrictions, publication window and maximum results. Replace relative dates such as “yesterday” with absolute timestamps and a time zone. Record when each test ran, because a live news index can change between requests.

Define completion as receiving a valid, parseable response before a fixed deadline. Define usefulness separately: for example, at least one of the first five results must support the requested event, entity and publication period. Write down that rule before inspecting providers’ responses. Otherwise, favorable examples can quietly change the acceptance standard.

Use an example question such as “Which European automakers announced production cuts this week?” Record the exact week and what counts as an announcement. An article about a possible future reduction should not pass a rubric requiring a confirmed announcement. Review the supporting passage rather than accepting a matching company name.

Include a few questions for which your reference review finds no qualifying announcement. Label these as expected-empty cases and score correct abstention separately from evidence retrieval. Do not place them in a table whose success rule requires an article. This prevents a system from earning credit simply by returning something for every question.

Report completion and usefulness together

The following figures are hypothetical. They illustrate a trial design, not measurements of Webz.io, NEEDLE or another provider. Each setup receives the same 20 logical queries, and a useful query is counted only when its response meets the deadline and relevance rule.

Setup B looks stronger when usefulness is calculated only among completed queries: 10 divided by 12. Setup A delivers useful evidence for more of the original workload: 12 divided by 20. Neither percentage replaces the other. Report all three rates so the reader can distinguish answer quality from missing responses.

Also report the sample size alongside each rate. Twenty queries can expose a logging or configuration problem, but a small trial does not establish a universal reliability ranking. Repeat the workload across relevant times and query categories before making a procurement decision.

Separate failures from empty and irrelevant responses

Assign each logical query an outcome and preserve the underlying request details. Useful categories include service timeout, server error, account restriction, rate limit, invalid response, valid empty response and completed response without useful evidence. Keep the original status and error message so reviewers can revisit your classification.

A valid empty result is a completed search, not automatically an outage. It may be correct when no matching news exists. An irrelevant result is also different from a transport failure. Conversely, a valid response arriving after the application’s deadline should remain a deadline miss in the operational view.

A sample record could read: “Q07; completed; 4 results; useful: no; reason: all reports predate the requested window.” Another could read: “Q08; deadline missed; one retry; account restriction reported.” These labels describe observations without claiming that the provider’s search engine failed whenever the client received no evidence.

Count retries against the same deadline

Give each original query one identifier and each retry its own attempt number. Start the timer before the first request and stop when the final response reaches the application or the deadline expires. Include backoff and parsing time. A request that eventually succeeds after several retries may still be unsuitable for an interactive agent.

Choose the same overall deadline and retry budget for comparable workflows. Record client configuration and account limits, and diagnose local failures separately from provider failures. Preserve every original outcome if you rerun a corrected configuration; publish the corrected trial as a separate run rather than silently replacing unfavorable rows.

For latency, show the distribution of completed queries together with the number that missed the deadline. A fast median among completed calls cannot describe requests that never completed. Keep the deadline visible beside the chart or table, and state whether timings measure individual attempts or the entire logical query.

Test the content your agent will actually consume

Webz.io’s News Search API accepts natural-language queries and supports filters including dates, language, country and source. Its product page describes matching article information, relevant content chunks, relevance information and credit usage. Test those returned passages against your rubric instead of assuming a feature description guarantees useful evidence for every question.

A contextual news search comparison can help shortlist capabilities. Your trial should then hold the evaluated content budget constant and inspect whether passages support the requested claim. If you separately test full articles or downstream answer generation, label that stage and its inputs; do not merge it into a snippet-only retrieval score.

Export the query list, timestamps, account restrictions, outcome labels, retry history, relevance decisions and reviewer notes with the summary table. Record observed credit usage where available, and mark missing cost data as unknown. Before sharing the results, reconcile attempted queries with completed responses and every remaining outcome so no request disappears from the report.