Best speech recognition API should probably be judged by error type, not leaderboard rank

I don’t think “best speech recognition API” can be answered by leaderboard rank alone.

WER is useful, but it flattens errors that do not have the same cost.

Missing “uh” is not the same as missing “don’t.”

A benchmark should probably separate error types:

entity errors
negation errors
domain-term errors
speaker attribution errors
timestamp errors
redaction errors
diarization drift
codec/noise sensitivity
language switching problems
partial/final instability
latency under realtime use

For a realtime ASR system like Smallest AI Pulse, I’d evaluate the stream itself, not just the final transcript. The test should include partial events, final corrections, timestamps, diarization/redaction behavior and whether the transcript is usable while speech is still happening.

That is a different evaluation method from offline transcription.

A system can have a decent broad WER and still fail badly if:

the wrong speaker gets tagged
a phone number is wrong
redaction misses sensitive info
timestamp points to the wrong moment
partial text triggers the wrong app action
final text arrives too late for realtime use

Raw error count is not enough.

Error severity matters.

What would you include in a proper speech recognition API evaluation?

Hi :)

TDR: looking for tips finding institutions that hire REs without a PhD / getting RE jobs.

I am European, graduated from my Master's in NLP last year and soon after I started working as a MLE for a tech company where most of my work revolves around pretraining and post-training LLMs. I really liked the job at first because it felt very research-y at the time and I loved my time writing my thesis.

However, it seems like I am going to be doing agentic systems for the foreseeable future. It hasn't been that much yet but I can see it coming. Moreover, it seems like AutoML is going to be pushed internally.

This is not the turn I want my career to take. Looking at my options in industry outside of my current company, the situation does not seem better at all. Hence I have been trying to find RE type jobs but I honestly don't know where they are hiding these positions lmao. It feels like either you know of institutions that hire these kind of positions already or it is impossible to come across these jobs so of course I come to reddit for your wisdom: which EU institutions regularly hire REs? How would you go around finding and getting these kind of jobs?

Ideally I'd find an institution that doesn't pay like crap (BSC I am looking at you) and isn't weird about language (ejem Inria ejem) but oh welp.

Obviously, I know my career wants are harder without a PhD. I am actively looking for one that fits me but I feel like my PhD chances would be better if I had more research experience (right now just 6 months but at a prestigious-ish institution) so my plan is to PhD out of my company or look for RE positions.

I’m classifying scientific abstracts written in Portuguese into two temporal categories: "Old" vs "Recent". I tested several models, but two of them are giving me massive red flags:

• DeBERTa (base): Getting exactly 100% accuracy on the test set.
• Albertina (a Portuguese DeBERTa-based model): Getting exactly 0% accuracy on "Old" and 100% on "Recent".
• Note: Other models like mBERT, XLM, and local Portuguese BERTs (Bertimbau) are getting realistic results around 75-85%.

What I've already tried (Data Cleaning): Knowing that 100% accuracy screams data leakage, I went aggressive on the preprocessing:

1. Used regex to replace all dates, citation years (1900-2026), and any 4 consecutive digits with a [HIDDEN_DATE] tag.
2. Removed all DOIs, URLs, emails, and modern copyright strings (e.g., "Open Access", "Creative Commons").
3. Removed all <tags> in case the modern abstracts were scraped differently from the old PDFs.

And Albertina is completely flipped.

So i have questions

1. What other structural artifacts in scientific abstracts could DeBERTa be exploiting to perfectly separate decades-old texts from modern ones? Length bias? OCR noise?

2. Why would Albertina (and only Albertina) completely invert the predictions? Is there a known issue with id2label mapping inheritance when loading specific pre-trained models from the HF Hub? How do I force the correct mapping?

3. Would running SHAP on the DeBERTa model be the best next step to highlight the exact tokens causing the 100%? Have you successfully used SHAP to debug leakage in text classification?

Hi, everyone! I am doing my PhD in Language Studies. I do not have a full tech background since I am more of a linguist that dabbled a little bit on NLP, so would like your help, suggestions or any recommendation. I am currently working on creating Universal Dependencies (UD) treebanks for minority languages, then using these UD treebanks to model parsers. I am planning to use rule-based parsing because I wouldn't have enough resources for data to run other models. One of the professors told me I could also use statistical methods, but if I am more comfortable with rule-based parsing, then I should just focus on it.

For the rule-based parsing, I would be trying to set rules similar to GiellALT. And for the evaluation of the scores, I am planning to use MaChAmp and the Stanza tagger.

1. Do you have any suggestions for me what books or publications I could read that would be relevant for my topic? I've been going through some published materials in ACL, but I would need help to narrow my focus.

2. Do you have suggestions about the modeling or evaluation using rule-based parsers? I haven't done much parsing before but I have done dependeny treebanking. Would just like to ask if there is something like a rundown or any materials that could help me fully understand parsing techniques.

Thank you for your patience! I am still learning about parsing and NLP in general.

I've been testing different approaches for long recordings lately, and this is the part I keep struggling with

The transcription itself usually isn't the issue anymore. Even with my old Whisper/Pyannote setup, I could get pretty decent results.

It's everything around it.

Multiple speakers mean checking diarization. Long recordings mean chasing timestamps. If the audio contains more than one language, there are usually extra things to verify. And once you have several hours of material, manually checking all of that gets painful.

I recently decided to stop optimizing the individual pieces and instead look for something that could handle more of them in one workflow.

I ended up testing Speechmatics and liked that transcription, speaker separation and word-level timestamps were handled together. It was particularly convenient for the multilingual recordings I was working with.

The biggest difference wasn't that the transcript suddenly became perfect. It was that I had fewer things to fix before I could actually use it.

So what are people using right now for this? I'm mainly interested in longer recordings with several speakers. Bonus points if it handles multiple languages without needing a bunch of manual intervention.

For english all works out basically fine either be local or cloud but disaster arrives when its non english, most often needs a VLM then while if youre on OCR, it fails depending on the script

this is where it actually torn apart- 

Most of it because the ocr models are built around latin, left to right and once character at a time assumptions. whereas Arabic breaks that immediately like the letters change the shape by position and reading direction is right to left so without contextual letterform handling you get garbled output and mixing rtl and ltr scrambles the line order

Indic scripts miss segmentation and whole syllables just drop. Cjk throws thousands of characters and sometimes vertical text at models which are tuned for alphabet. Once a page is mixed-language a single global language guess falls apart and you need detection down at token level. Additionally accuracy quietly craters on low resource languages as the training data has taught it . So rather leaning on to the LLM only there are some tooling good or aimed at some specific language criterias. like paddleocr is the open go to for CJK, Surya covers wide range fr and tesseract really works well on clean print. the general llms read a surprising amount but theyre inconsistent, pricey at volume and hallucinate on mixed pages. although the mixed script or vlm based parsing either Surya or paddleocr locally or for cloud llamaparse or any other which routes per element and pull ahead of the plain ocr engine

Do you guys use tooling for the multilingual processing or have you worked ona  project that needs this or have you made anything of your own for multilingual

Hi! Need some crowd wisdom here. I'm completely stuck on my own.

The thing is, I don't come from a tech or ML background, but I have 20+ years of experience in cross-cultural content adaptation and mono/multilingual workflows (RU-RU, RU-EN, EN-RU, FR-RU).

A couple of years ago I started translating Korean into Russian, without knowing Korean (still don't). And no, I'm not lazily editing an LLM-generated zero-shot — with KO-RU that trick doesn't work, the languages are too distant.

My prehistoric… sorry, my pre-LLM translation approaches turned out to work surprisingly well with AI. According to models, the methods I’d been using all along now have fancy industry-ish names: hypothesis space management, adversarial/contrastive probing, multi-model differential analysis, tracking shifts in model behavior, and so on.

And here’s a list of some recurring failure patterns I’ve observed: premature semantic concretization, compounded coherence degradation, analysis–synthesis gap, asymmetric handling of low-salience pragmatic markers, etc. The distance between the languages made a lot of otherwise hidden failure patterns quite visible.

None of this happened inside any industry pipeline, though. So I found the closest publicly available thing to an established evaluation setup for translation quality, the WMT25 General MT human-evaluation data (EN-RU), and I ran a small self-diagnostic to see how my judgment held up there. The result was quite unexpected. On one hand, my assessment was mostly in line with the official scores. On the other hand, I noticed some errors that weren't reflected in the annotations, I saw considerable variation in human scores, and what really surprised me were the human references themselves: some looked like they needed another editing pass.

Honestly, I've never quite understood that thing with “single gold standard” reference for literary or conversational language. This practice doesn't seem like an effective approach in the long run. Well, I guess there's nothing that the industry doesn't already know, but my take is: teach a model to fish rather than just keep feeding it answers.

By “teaching models to fish,” I don't mean simply giving them multiple equally valid reference translations (that's the obvious idea). What I mean is teaching a model to recognize the techniques behind good translations and building up a “toolbox” of those techniques — things it could then reuse instead of having to solve the same kind of problem from scratch every time.

Models tell me that moving away from reliance on a single fixed reference is already a recognized tension in MT evaluation. Reassuring, in a way: I'd apparently arrived at a similar place independently, from the translator's side rather than the ML side.

Yay, I found my new calling, I thought. Let’s turn it into an actual job! But for two months now I've been trying to figure out how to position my experience. Models suggest starting with roles where there's no immediate need for technical skills, like Red Teaming or Eval Strategy. I'm personally more drawn to research, observation, and diagnostics, though I realize that's probably too bold even to dream about at this point. So, I genuinely can't tell which door I'm supposed to knock on.

My questions are:

• How would you position someone with my experience and way of thinking? I'm mostly looking for honest calibration.
• No industry experience, remote-only work — what are realistic entry points for someone with these constraints?

Thanks for any thoughts!

P.S. The KO-RU translation turned out great — readers call it “divine” and never suspected a human-AI collaboration behind it :)

Bodhan AI, an IIT Madras-incubated initiative working with AI4Bharat, has released models for speech, OCR and translation across Indian languages. Consider a student asking:
“Photosynthesis ka process Telugu lo explain cheyyandi.”
A real AI tutor may need to handle:
Speech → transcription → translation → retrieval → answer
Each stage can fail.
Good Hindi ASR doesn’t guarantee good performance for a child’s regional accent. Good printed OCR doesn’t guarantee handwritten Telugu. And the most worrying case is when the pipeline works, but the final answer is confidently wrong.
So instead of one “multilingual accuracy” score, shouldn’t we evaluate language, accents, code-mixing, noise, handwriting and factual correctness separately?
Which failure worries you most?

Hi all, looking for some kind of tool or library that I can use to figure out the countability of a given noun, preferably multilingually. Something that would give results like "countable:true", "countable:false", or "both" if applicable. NLTK doesn't seem to have anything like that. Trying to avoid LLMs, but it's looking like that might be the best option...

I happen to be in a position where I have to read a lot of reviews, and increasingly everything seems AI-generated. I am honestly starting to question my sanity. Is everything truly written by AI, or have we always written this way? Anyone else asks this question themselves?

We just released the Last Translation Benchmark paper. In a massive crowdsourcing effort we collected 3456 unique hard-to-translate examples that break state-of-the-art translation models, and which can be used for more reliable evaluation.

I think this is a nice intro project to NLP. I can go in cookbook style. Suits me cause my knowledge is spotty with holes all over. If I think of it as a generic starter cookbook, I can document my work into something worth sharing. I'm actually thinking of a better ways to organize but without running a fat LLM, but that's a longer discussion.

Starting with recorded speech, I want to transcribe each file and split the text into tagged paragraphs. In my mind's eye they're post-it notes, and they come in infinite shapes and color schemes.

I've tried many STT solutions, there's plenty for Linux. But I haven't found anything to organize messy text. Not summarize or query, split into their small meaningful pieces of a puzzle. All the STT solutions I tried either produce one big block of text or have a lot of one-liner words and phrases. It takes too much time to cleanup into manageable paragraphs.

I don't want to edit any of the content, that's why I keep thinking it should be a cheaper job, hardware wise. I have a measly 2GB GPU, MX330 to run small STT models.

You know of anything that already does this? Self-hosted of course. If an LLM is needed, it should let me connect to a local one. If there's nothing close to such a workflow and i'd have to build it, I found txtai but all the examples seem to start from sqlite databases. Dunno how to get there from text.

Any suggestions appreciated, thanks.

Should I argue against the rejection? I believe I should.

It came as a shock though.

context: AAAI usually instructs authors to attach the reproducibility checklist after the references and just before the document ends. These pages don't count against the main paper's page limit.

This year there has been a change in the rule, but the authors kit remained the same. The spill to the 10th page was due to that reproducibility checklist.

Third-year PhD student, NLP, mostly LLM-based reasoning. Given a collection of a private organization's HR policy documents (100-500 PDFs), find all pairs of clauses that contradict each other. There's a mountain of work on NLI-style contradiction classification, but that assumes someone gives you the sentence pair. Here, the pair is the problem. With about 1-2k clauses, you're looking at millions of candidate pairs. So brute-force pairwise LLM calls are out, and whole-document prompting fails for the usual lost-in-the-middle reasons. The closest work I found generates synthetic contradictions in synthetic corpora to test detectors. I borrowed the evaluation idea by injecting contradictions into corpora. I also used a university HR handbook and one dataset with existing external annotations, contractNLI (made for the NLI task by Stanford). I used this one as well because it has real contradictions. But this one is quite different. In this dataset, the task formulation is like hypothesis versus clause, whereas in the first two datasets, I do clause-to-clause comparison. So I built a two-stage pipeline. First, retrieval with a HyDE-style approach where the query is a hypothetical, *contradicting* version of each clause. Then, recall-based candidate retrieval (LLM), followed by precision-based verification with an LLM, where each candidate pair is re-read within its source documents. The contributions: I used contextual sentences guided by Anthropic, which helped retrieval, and showed that a document’s surrounding context helped precision. Agentic verification (tools, multi-step) actually underperformed a single prompt. As a case study, I ran the pipeline on a public government policy corpus. It found a few genuine contradictions. I have a few questions. Am I missing a community? I can't believe nobody works on this. I've looked at legal NLP (ContractNLI, etc.), requirements engineering conflict detection, and RAG-conflict work. They're all adjacent, but none does discovery over a real multi-document policy corpus. Is there a literature I don't know the name of? My PI is leaning toward a lower-tier conference or journal. Is this the kind of paper that has a chance at a first-tier NLP venue, or is my PI just being realistic? If you were strengthening this in one month, what would you add? I already have NLI, direct-prompting, and agentic baselines. Happy to share more details in comments. Mostly, I want to know whether this problem is as understudied as it looks from where I'm sitting, or whether I formulated the task the wrong way.

Hi everyone,

I’m wrapping up my PhD in NLP (3-year system, graduating next January) after starting in early 2024 right as the LLM trend began exploding.

With the sheer volume of papers being published lately (especially over the past year), I feel that the standalone value and rarity of individual publications have decreased. Given this paper inflation, it's hard to gauge where my publication record objectively stands compared to other fresh PhD graduates.

The three-year timeframe felt too short and rushed; it is disappointing that just a few rejections meant the graduation was already looming. My research focus and first-author record (excluding co-authored papers) are as follows:

• Research Focus: Trustworthy LLMs, Legal Tech
• First-Author Publications:
  • EMNLP (Main) x 2
  • ACL (Findings) x 1
  • IP&M (Information Processing & Management, Journal) x 1
  • Many(4papers) under review papers...

Questions:

1. Objective Standing: Considering the current paper volume inflation in the NLP/LLM field, where does my first-author publication record place me compared to other fresh PhD graduates?
2. Career Path (Academia vs Postdoc vs Industry): Is it realistic to apply directly for going to indusrty, or would it be wiser to do a 1-2 year postdoc to build a stronger CV and try to acedemia? (Academia is my top choice, but I'm open to industry positions if needed.)
3. Field Outlook: How is the current sentiment in academia and the job market regarding research in Trustworthy LLMs and Legal Tech?

I’d appreciate any honest feedback or insights from current professors, postdocs, or industry researchers. Thanks!

With ASR and speech models improving rapidly, I'm trying to understand where trained phoneticians still provide value in commercial Speech AI projects.

Which tasks still genuinely require human phonetic expertise rather than general language annotators or automated tools?

For example, I'm thinking about pronunciation evaluation, phonetic transcription, speech-data QA, accent analysis, error analysis, segmentation and evaluation of ASR/TTS outputs.

I'd be especially interested in examples from people who have worked on commercial speech datasets or speech AI projects.

​

A thing I got wrong for a long time: I evaluated speed by how long a response took to finish. That's the intuitive metric and it's close to irrelevant for interactive work.

What you actually perceive is time to first token. The gap between hitting enter and seeing something happen. Once tokens are flowing, your brain treats the thing as working and you start reading. A response that begins in 300ms and finishes in 20 seconds feels fast. A response that begins in 3 seconds and finishes in 12 feels slow, despite being objectively quicker. The total is what you'd measure and the opening is what you'd feel.

This has a few consequences that took me a while to connect.

Buffering anywhere in the path is much more expensive than it looks on a graph. Any layer that collects a full response before passing it on converts a fast-feeling interaction into a slow-feeling one while barely moving the total. If you're evaluating anything that sits between you and a provider, this is the property to test, and it won't show up in a benchmark that measures completion time.

Provider degradation shows up here first. Before anything goes red on a status page, TTFT drifts. Elevated queueing, longer scheduling delays, the request eventually completes fine so nothing registers as an incident. If you're only watching totals and error rates you'll miss a bad hour entirely while your experience quietly gets worse.

Reasoning models scramble the whole framing. If a model thinks before it emits, TTFT as classically defined can be many seconds even when nothing is wrong. Whether that reads as broken or as working depends almost entirely on whether the interface shows you something during the wait. Same latency, different experience, decided by UI.

And averages hide the thing you care about. TTFT distributions are skewed — a good median with an ugly tail feels unreliable in a way that a slightly worse median with a tight tail doesn't. People remember the bad turns. p95 is the honest number and mean is the flattering one, which is presumably why mean is the one that gets published.

I watch this closely because I run a layer that adds a hop, and a hop is exactly the kind of thing that can quietly ruin TTFT if you build it lazily (routera . one, my project, stating it rather than being coy). Anything in the path has to earn its place on this metric specifically.

Caveat: for batch or background work none of this applies, and optimizing TTFT there is wasted effort. Throughput and cost are the right metrics when nobody's watching the screen. It's specifically interactive coding where the opening moment dominates.

What I haven't figured out: how to communicate provider slowness to a user without training them to ignore you. Show a warning whenever p95 ticks up and it becomes wallpaper within a week. Show nothing and people blame your product for someone else's bad afternoon. I don't think there's a clean answer but I'd like to hear from anyone who thinks they've found one.

We're producing lcalized video content across different markets and the workflow is a nightmare right now. Basically we need something that can take a finished video, swap the voiceover into 4 or 5 languages and ideally keep the background audio intact underneath. The bigger issue is scale, we can't do this one by one manually for every single asset. What I'm really trying to find is an API that handles video dubbing at scale, something we can pipe into our existing pipeline without rebuilding everything from scratch. We already use a tool for AI-generated soundtracks that syncs music to the video's emotional pacing automatically, which honestly saves us maybe 40+ hours a month on audio production alone, so I know the API ecosystem for video audio tools is maturing. Anyone know if a dedicated dubbing API exists that handles lip sync, voice cloning or at least clean audio replacement across languages? Even a rough cost ballpark would help us figure out if its viable.

Heyy Everyone!

I hope everyone is doing great. Currently, I am applying for a programme related to data science of Multi modal human communication. My background is in English studies; however, I need some quantitative proof as evidence related to this programme. Which I don't have, can a postgraduate diploma in computational linguistics or any research. I have explored but I couldn't be able to find any source to be done.

I can jump in python and NLP training. But is there any source from where I can get computational linguistics Training, if it's for 6 months I um up for it.

So, is there anyone who can help me regarding this. I would really appreciate your guidance.

Hi,

I'm relatively new to the world of transcription. As a ESL person i'm looking for state of the art model for bi-lingual transcription. Currently focusing on russian/english pair but also interested in spanish/english and chinese/english in the futre.

On top of that I'm interested in these models to be steerable with a prompt (i.e. asking to format the content in a certain way or remove filler words)

Third most important parameter is latency (ideally looking at least 500ms response time).

So far I've found most reliable

OpenAI gpt-transcribe and gpt-live-transcribe, also older gpt-4o-transcribe are most reliable for russian/english pair and transcription speed.

I've looked at gemini 3.5 flash transcribe - but the latency isn't that great.

Anything else i should be aware of?

Hello everyone. I am a linguistics researcher and have a complex issue where manual annotation of speech would take an extensive amount of time and labor.

I have hours of conversational data where each speaker (A & B) is recorded separately. I also have transcripts of each speaker's utterances. Although MFA has been mostly great at aligning, there are still some misalignment that can wreck the entire text grid (therefore the time-speech alignment).

Some issues I think might contribute to the misalignment:

• Faint audio picked up from speaker B on speaker A microphone
• Relatively long silences (when speaker B is talking)

"Uhms" and "ahs", other non-lexical expressions, false-starts, are all in the transcriptions.

My goal is to extract word-/phrase-level onsets and duration (phonemes are not necessary). The extraction of these times is not a problem I have.

I have tried adjusting --beam, and --silence_weight, --cleanup_textgrids (with /some/ success). I have also tried VAD segmentation with mfa segment, but that ended up being worse than regular alignment without segmentation. I understand there is no chance of a 100% accurate alignment as these are long audios, but even 80% I would be happy with that I can then manually edit and look over. I am seeking advice on how to improve alignment on long these audios.

My current ideas are:

1. Segment audio manually and stitch together (which comes with it's own technical issues)
2. Filter audio such that noise is reduced, and speech (of speaker A) is amplified
3. Play around more with mfa parameters such as beam and silence weights

Any advice would be appreciated. Thank you.

I'm a linguist with a strong interest in phonetics and I'm currently preparing to work freelance on Speech AI projects, particularly speech data annotation, phonetic transcription, pronunciation evaluation and QA.

I'm comfortable with IPA and I'm currently developing my practical skills in Praat, Audacity and speech-data workflows. I'm also starting to explore ASR evaluation and tools such as Whisper.

For those working in Speech AI or language technology: which skills do you actually use in your day-to-day work?

In particular, I'd be interested in knowing:

• How much acoustic analysis with Praat is actually required?
• How important are Python and R?
• Which annotation tools are commonly used?
• What kinds of tasks specifically benefit from phonetic expertise?
• Which skills would you recommend learning before taking on paid Speech AI projects?

I'm particularly interested in industry rather than academic careers.