Abstract
We want to test whether a system helps someone complete a particular task, including the checking and correction it requires. A draft can contain the right facts and still miss an exception, omit a source or propose an action that has not been approved.
The initial speech programme examined language switches, terminology, numbers and accents, and investigated changes in general-language recognition. Speaker-assignment and transcription errors were evaluated separately. This paper extends that evaluation approach with a proposed record for wider model and workflow experiments. It separates output errors, source support, tool actions, review effort and recovery after a failure.
The speech programme conducted adapted-checkpoint comparisons with unchanged baselines and assessed recognition and speaker-assignment errors separately. Comparative findings are under internal review. Broader workflow and video comparisons in this paper are further test plans.
1. Make the task the unit of comparison
Define a task with an observable end state: a specification ready for a named reviewer, a source-grounded answer with unresolved evidence visible, or a document-derived draft that retains the information required for a decision. The acceptance conditions should say what makes the result useful and what must still be checked.
The task record should identify permitted inputs, the output form, relevant constraints and which actions are excluded. It should also include the current way of completing the work. Without a baseline, it is difficult to know whether an AI workflow saves effort or merely moves that effort to a different person.
The same task can contain several measures. A transcript can be graded for recognition accuracy, a document extraction for field correctness and a final requirement for whether its supporting source is present. Those component measures should feed a task-level decision rather than substitute for it.
2. Include cases that should cause hesitation
A useful set includes normal cases and cases where proceeding would be wrong. Missing source material, conflicting dates, uncertain speakers, an ambiguous document image or an unconfirmed tool result can all change the correct next step.
We would organize cases by the difficulty they introduce and preserve their original source grouping. Evaluation should not reward a model for a confident completion when the right behavior is to name the missing fact. It should separately record whether a person can see why the result remains incomplete.
Multimodal cases should vary which source supplies the important information. A statement found only in the recording, a clause found only in the document and a location found only in a visual annotation test different parts of the system. Controlled conflicts between them reveal whether the workflow has retained their relationship.
3. Use a record that makes failures explainable
A comparison needs enough information for someone to reconstruct its conditions. The proposed record captures:
- Task and sources: objective, source identifiers, permitted processing and expected output.
- Configuration: model revision, adaptation method, context preparation, tool set and decoding or execution settings.
- Output: deliverable revision, extracted observations, source references and open questions.
- Action state: proposed, attempted, confirmed or unresolved operations, with the current approval boundary.
- Review: errors found, corrections made, information the reviewer had to seek and the final acceptance decision.
The record should avoid adding private source material to a public benchmark by default. Reproducibility can require controlled access to permitted data or a fictional equivalent, with the limits of that substitute stated clearly.
4. Separate quality, authority and effort
Quality includes recognition, extraction, interpretation, source support and the usefulness of the final deliverable. Each error should be classified by where it entered the workflow. A misheard term should not be confused with a reasoning error introduced after a correct transcript.
Authority concerns whether the system used permitted information and respected the action boundaries defined for the task. Completion achieved by crossing those boundaries is not a successful task result. An evaluation should record the attempted boundary crossing even if a downstream service rejected it.
Effort includes the time and attention needed to inspect, correct and accept the work. A fast draft can create an expensive review if the person has to reopen every source. We would measure correction effort alongside generation and tool costs, under declared workload assumptions.
These dimensions should remain visible separately. A single headline score can hide a tradeoff between a polished output, a difficult review and a failure to respect a permission boundary.
5. Compare interventions under the same conditions
For model adaptation, hold source access and task conditions steady while comparing the untuned baseline with context improvements and a fine-tuned candidate. For an agentic change, hold the model and task set steady where possible while changing the orchestration or review structure.
Where a comparison changes several components together, report it as a system comparison. It may be valuable, but it cannot isolate the contribution of a specific fine-tuning intervention. Keep preprocessing, language coverage, runtime settings and hardware context in the experiment record.
A person should inspect consequential claims against the source. Automated checks are useful for structural constraints, known expected fields and repeatable tasks; model-based grading can help organize comparisons. Disagreement between those measures is itself worth investigating.
6. Examine interruptions and incomplete outcomes
A workflow test should include interrupted reads, unknown write outcomes, repeated requests and a change in access between steps. The result should show whether work was preserved, whether uncertain actions were reconciled and whether the workflow resumed under current authority.
Some cases should end without completing the original request. A blocked source or an unconfirmed approval can make completion inappropriate. The useful outcome is then a preserved deliverable, a clear account of the missing condition and a bounded next step.
That behavior should be judged against the task specification, not marked as failure simply because the agent stopped. Conversely, continuing through an unresolved condition should not receive credit solely for reaching an output.
7. Publish the conditions with the result
A public result should identify the task set, number and nature of cases, model and system configuration, grading method and limitations. It should report meaningful failure cases and the variation across tasks, not only an average or a best example.
Use the findings to decide whether to extend an experiment, change the workflow or stop pursuing an intervention. Report the conditions and uncertainty behind that decision.
Comparing NVIDIA, Google and Apple models
The ongoing research set includes Nemotron 3 Nano Omni, NVIDIA specialist speech models, Google Gemma 4 and Gemma 3n, and Apple AFM 3. Select exact variants before comparison: Gemma 4 audio input is available in E2B, E4B and 12B, while Omni’s model card specifies English. Evaluate only the languages and input types that the chosen model supports.
Use separate task sets for image or document understanding, audio transcription, audiovisual reasoning and spoken responses. For a model that returns text, measure any speech synthesis stage separately and include it in end-to-end latency. Test speaker attribution independently from recognition.
For every cloud Mac run, record the chip, memory, operating system, model revision, framework or runtime, quantization, input duration, frame sampling, peak memory and latency. Record cloud calls and data movement explicitly. A cloud-hosted Mac is remote infrastructure; inference on it should not be described as processing on the customer’s own device.
Compare unchanged open-weight checkpoints before attributing improvements to adaptation. For Apple-managed models, record the model variant, framework configuration and APIs exercised.
References and source notes
References checked on 14 October 2026.