CUSTOM MODELS · SPEECH, VISION AND TEXT
Speech and vision.
Models for both.
We research, prototype and build with speech and vision models. Our work covers multilingual transcription, speaker attribution and spoken dialogue, alongside image, document and video understanding.
Our initial six-month programme included a Hindi–English adaptation study. It produced prototypes, data-processing workflows, baseline configurations, evaluation procedures and an internal technical report. Comparative findings are under review; selected prototypes are being hardened for pilots.
SPEECH RESEARCH
Languages, speakers
and spoken interaction.
Our speech capabilities span Indic languages, Arabic and European languages including German, French and Spanish. We investigate recognition, speaker-attributed transcripts and interactive dialogue, with language coverage and evaluation defined for each system.
The initial programme includes a Hindi–English adaptation study, a pyannote-based speaker-attribution prototype and a Moshi feasibility study. These are specific investigations within the broader speech capability.
Read the initial programme tracks ↓VISION RESEARCH
Images, documents
and moving scenes.
Our vision research examines how models interpret images, document layouts and video frames, and how those observations can be checked against the source. Ongoing model investigations include NVIDIA Nemotron, Google Gemma and Apple AFM 3.
Controllable generation is a related programme: our “Making AI Work” video prototypes investigate character and object continuity, action timing and selective repair. Visual understanding and generation have different tasks and evaluation criteria.
ONGOING RESEARCH · VISION AND SPEECH
Models we can host.
Models we access on Apple platforms.
Our ongoing work includes NVIDIA and Google open-weight models and Apple’s AFM 3 family, with experiments on a cloud-hosted Mac alongside the wider AWS research environment.
| Model family | Vision and speech capabilities | Access and research role |
|---|---|---|
| NVIDIA Nemotron 3 Nano Omni | Images, video, audio and text as inputs; text as output. The model card specifies English support. | Open weights. A candidate for joint visual and spoken-source understanding, document analysis and transcription. |
| Google Gemma 4 | Images and video frames across the family; E2B, E4B and 12B also accept audio. Outputs are text. | Open weights under Apache 2.0. Candidates for compact multimodal experiments; audio support depends on the variant. |
| Google Gemma 3n | Text, image, video and audio inputs; text outputs. | Open weights under Gemma terms. An additional candidate for constrained-device research. |
| NVIDIA speech models | Parakeet for recognition; Canary for recognition and translation; Canary-Qwen for speech-language tasks; Magpie for speech synthesis. | Specialist candidates with checkpoint-specific licenses and language coverage. These are distinct from a combined vision-and-speech model. |
| Apple AFM 3 Core and Core Advanced | Core supports image understanding. Apple describes Core Advanced as natively multimodal, including dictation and expressive voices. | Apple-managed models accessed through supported frameworks and operating systems. |
| Apple AFM 3 Cloud, Cloud Pro and ADM 3 Cloud | Cloud models address multimodal reasoning and complex tasks; ADM 3 Cloud specializes in image generation and editing. | Apple’s Private Cloud Compute path. Cloud execution is recorded separately from inference on the Mac. |
Understanding speech and generating a spoken reply are different capabilities. Likewise, reading video frames does not establish video generation or lip-sync support. Check the selected model’s inputs, outputs and runtime requirements before choosing it for a task.
Cloud Mac experiments examine runtime compatibility, quality, latency and memory use. A rented Mac remains remotely hosted infrastructure, even when a model executes on that machine. We record any additional cloud-model calls separately.
Primary references: NVIDIA Omni model card; Gemma 4 model card; Gemma 3n model card; NVIDIA speech model guide; Apple AFM 3 research overview; Apple Foundation Models framework.
Read the comparison method ↗STUDIES IN THE INITIAL PROGRAMME
Three problems
in conversational speech.
| Track | Work completed | Current stage |
|---|---|---|
| Hindi–English adaptation study ↗ | Qwen3-ASR adaptation workflow: dataset preparation, transcript normalization, supervised experiments and adapted-checkpoint comparisons against unchanged baselines. | Prototype and comparative findings under internal review before pilot release. |
| Speaker-attributed conversation analysis ↗ | pyannote-based pipeline: speaker changes, overlap, configuration experiments and targeted speech-segmentation adaptation. Transcription and speaker-assignment errors evaluated separately. | Prototype; consistency across unfamiliar speakers and recording conditions, plus operational hardening, remain ongoing. |
| Interactive spoken dialogue ↗ | Moshi-based feasibility study: local inference, controlled interaction experiments and assessment of further adaptation requirements. | Exploratory research. Broader validation and pilot readiness are later milestones. |
TRANSCRIPTION RESEARCH
Language changes.
The terms matter too.
We examined Hindi–English conversations containing language switches, specialist terminology, numbers and varied accents. Adapted checkpoints were compared with their unchanged foundation-model baselines. The programme also investigated recognition quality on general-language speech after adaptation.
The prototype supports repeatable experiments and structured error analysis. We are reviewing checkpoint selection, comparative findings and local deployment characteristics before pilot release.
Read the adaptation note ↗RESEARCH AUTOMATION
Agentic engineering.
Human research decisions.
AI-assisted workflows supported dataset inspection, preprocessing, experiment configuration, evaluation scripts, error categorization and documentation. Humans reviewed data acceptance, evaluation criteria and the interpretation of results.
Deployment preparation investigated AWS GPU infrastructure for training and controlled evaluation, and Apple Silicon for local inference experiments. Checkpoint conversion, runtime compatibility, memory profiling and accuracy after compression are part of that preparation.
Explore research automation and agentic systems ↗FROM RESEARCH TO PILOTS
What comes
next.
Pilot preparation focuses on reproducibility, installation, monitoring and performance under representative conditions. Customer-specific adaptation and broader validation continue alongside internal review.
Ongoing NVIDIA, Google and Apple model investigations cover vision and speech, including document layouts, images and video frames.