Aditya Yadavalli

selected publications

1. EMNLP

   PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data

   Watts, Ishaan, Gumma, Varun,  Yadavalli, Aditya, Seshadri, Vivek, Swaminathan, Manohar, and Sitaram, Sunayana

   In Proc. EMNLP 2024

   Abs Bib PDF

   Evaluation of multilingual Large Language Models (LLMs) is challenging due to a variety of factors – the lack of benchmarks with sufficient linguistic diversity, contamination of popular benchmarks into LLM pre-training data and the lack of local, cultural nuances in translated benchmarks. In this work, we study human and LLM-based evaluation in a multilingual, multi-cultural setting. We evaluate 30 models across 10 Indic languages by conducting 90K human evaluations and 30K LLM-based evaluations and find that models such as GPT-4o and Llama-3 70B consistently perform best for most Indic languages. We build leaderboards for two evaluation settings - pairwise comparison and direct assessment and analyse the agreement between humans and LLMs. We find that humans and LLMs agree fairly well in the pairwise setting but the agreement drops for direct assessment evaluation especially for languages such as Bengali and Odia. We also check for various biases in human and LLM-based evaluation and find evidence of self-bias in the GPT-based evaluator. Our work presents a significant step towards scaling up multilingual evaluation of LLMs.

   @article{Watts2024PARIKSHAAL,
     title = {PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data},
     author = {Watts, Ishaan and Gumma, Varun and Yadavalli, Aditya and Seshadri, Vivek and Swaminathan, Manohar and Sitaram, Sunayana},
     journal = {In Proc. EMNLP},
     year = {2024}
   }

2. EACL Findings

   AccentFold: A Journey through African Accents for Zero-Shot ASR Adaptation to Target Accents

   Owodunni, Abraham*,  Yadavalli, Aditya*, Emezue, Chris*, Olatunji, Tobi*, and Mbataku, Clinton

   In Proc. EACL Findings 2024

   Abs Bib PDF

   Despite advancements in speech recognition, accented speech remains challenging. While previous approaches have focused on modeling techniques or creating accented speech datasets, gathering sufficient data for the multitude of accents, particularly in the African context, remains impractical due to their sheer diversity and associated budget constraints. To address these challenges, we propose AccentFold, a method that exploits spatial relationships between learned accent embeddings to improve downstream Automatic Speech Recognition (ASR). Our exploratory analysis of speech embeddings representing 100+ African accents reveals interesting spatial accent relationships highlighting geographic and genealogical similarities, capturing consistent phonological, and morphological regularities, all learned empirically from speech. Furthermore, we discover accent relationships previously uncharacterized by the Ethnologue. Through empirical evaluation, we demonstrate the effectiveness of AccentFold by showing that, for out-of-distribution (OOD) accents, sampling accent subsets for training based on AccentFold information outperforms strong baselines a relative WER improvement of 4.6%. AccentFold presents a promising approach for improving ASR performance on accented speech, particularly in the context of African accents, where data scarcity and budget constraints pose significant challenges. Our findings emphasize the potential of leveraging linguistic relationships to improve zero-shot ASR adaptation to target accents

   @article{Owodunni2024AccentFoldAJ,
     title = {AccentFold: A Journey through African Accents for Zero-Shot ASR Adaptation to Target Accents},
     author = {Owodunni, Abraham* and Yadavalli, Aditya* and Emezue, Chris* and Olatunji, Tobi* and Mbataku, Clinton},
     journal = {In Proc. EACL Findings},
     year = {2024}
   }

3. ACL

   SLABERT Talk Pretty One Day: Modeling Second Language Acquisition with BERT

   Yadavalli, Aditya*, Yadavalli, Alekhya*, and Tobin, Vera

   In Proc. ACL 2023

   Abs Bib PDF

   Second language acquisition (SLA) research has extensively studied cross-linguistic transfer, the influence of linguistic structure of a speaker’s native language [L1] on the successful acquisition of a foreign language [L2]. Effects of such transfer can be positive (facilitating acquisition) or negative (impeding acquisition). We find that NLP literature has not given enough attention to the phenomenon of negative transfer. To understand patterns of both positive and negative transfer between L1 and L2, we model sequential second language acquisition in LMs. Further, we build a Mutlilingual Age Ordered CHILDES (MAO-CHILDES) – a dataset consisting of 5 typologically diverse languages, i.e., German, French, Polish, Indonesian, and Japanese – to understand the degree to which native Child-Directed Speech (CDS) [L1] can help or conflict with English language acquisition [L2]. To examine the impact of native CDS, we use the TILT-based cross lingual transfer learning approach established by Papadimitriou and Jurafsky (2020) and find that, as in human SLA, language family distance predicts more negative transfer. Additionally, we find that conversational speech data shows greater facilitation for language acquisition than scripted speech data. Our findings call for further research using our novel Transformer-based SLA models and we would like to encourage it by releasing our code, data, and models

   @article{Yadavalli2023SLABERTTP,
     title = {SLABERT Talk Pretty One Day: Modeling Second Language Acquisition with BERT},
     author = {Yadavalli, Aditya* and Yadavalli, Alekhya* and Tobin, Vera},
     journal = {In Proc. ACL},
     year = {2023},
     volume = {abs/2305.19589}
   }

4. TACL

   AfriSpeech-200: Pan-African Accented Speech Dataset for Clinical and General Domain ASR

   Olatunji, Tobi, Afonja, Tejumade,  Yadavalli, Aditya, Emezue, Chris Chinenye, Singh, Sahib, Dossou, Bonaventure F. P., Osuchukwu, Joanne, Osei, Salomey, Tonja, Atnafu Lambebo, Etori, Naome, and Mbataku, Clinton

   Transactions of the Association for Computational Linguistics 2023

   Abs Bib PDF

   Africa has a very poor doctor-to-patient ratio. At very busy clinics, doctors could see 30+ patients per day—a heavy patient burden compared with developed countries—but productivity tools such as clinical automatic speech recognition (ASR) are lacking for these overworked clinicians. However, clinical ASR is mature, even ubiquitous, in developed nations, and clinician-reported performance of commercial clinical ASR systems is generally satisfactory. Furthermore, the recent performance of general domain ASR is approaching human accuracy. However, several gaps exist. Several publications have highlighted racial bias with speech-to-text algorithms and performance on minority accents lags significantly. To our knowledge, there is no publicly available research or benchmark on accented African clinical ASR, and speech data is non-existent for the majority of African accents. We release AfriSpeech, 200hrs of Pan-African English speech, 67,577 clips from 2,463 unique speakers across 120 indigenous accents from 13 countries for clinical and general domain ASR, a benchmark test set, with publicly available pre-trained models with SOTA performance on the AfriSpeech benchmark.

   @article{Olatunji2023,
     title = {AfriSpeech-200: Pan-African Accented Speech Dataset for Clinical and General Domain ASR},
     author = {Olatunji, Tobi and Afonja, Tejumade and Yadavalli, Aditya and Emezue, Chris Chinenye and Singh, Sahib and Dossou, Bonaventure F. P. and Osuchukwu, Joanne and Osei, Salomey and Tonja, Atnafu Lambebo and Etori, Naome and Mbataku, Clinton},
     journal = {Transactions of the Association for Computational Linguistics},
     year = {2023},
     volume = {11},
     pages = {1669-1685}
   }

5. Interspeech

   Multi-Task End-to-End Model for Telugu Dialect and Speech Recognition

   Yadavalli, Aditya, Mirishkar, Ganesh, and Vuppala, Anil Kumar

   In Proc. Interspeech 2022

   Abs Bib PDF

   Conventional Automatic Speech Recognition (ASR) systems are susceptible to dialect variations within a language, thereby adversely affecting the ASR. Therefore, the current practice is to use dialect-specific ASRs. However, dialect-specific information or data is hard to obtain making it difficult to build dialect-specific ASRs. Furthermore, it is cumbersome to maintain multiple dialect-specific ASR systems for each language. We build a unified multi-dialect End-to-End ASR that removes the need for a dialect recognition block and the need to maintain multiple dialect-specific ASRs for three Telugu regional dialects: Telangana, Coastal Andhra, and Rayalaseema. We find that pooling the data and training a multi-dialect ASR benefits the low-resource dialect the most – an improvement of over 9.71% in relative Word Error Rate (WER). Subsequently, we experiment with multi-task ASRs where the primary task is to transcribe the audio and the secondary task is to predict the dialect. We do this by adding a Dialect ID to the output targets. Such a model outperforms naive multi-dialect ASRs by up to 8.24% in relative WER. Additionally, we test this model on a dialect recognition task and find that it outperforms strong baselines by 6.14% in accuracy.

   @inproceedings{Aditya2022Interspeech,
     author = {Yadavalli, Aditya and Mirishkar, Ganesh and Vuppala, Anil Kumar},
     booktitle = {Proc. Interspeech},
     title = {Multi-Task End-to-End Model for Telugu Dialect and Speech Recognition},
     pages = {1387--1391},
     doi = {10.21437/Interspeech.2022-10739},
     year = {2022}
   }

6. NAACL-SRW

   Exploring the Effect of Dialect Mismatched Language Models in Telugu Automatic Speech Recognition

   Yadavalli, Aditya, Mirishkar, Ganesh, and Vuppala, Anil Kumar

   In North American Chapter of the Association of Computational Linguistics Student Research Workshop 2022

   Abs Bib PDF

   Previous research has found that Acoustic Models (AM) of an Automatic Speech Recognition (ASR) system are susceptible to dialect variations within a language, thereby adversely affecting the ASR. To counter this, researchers have proposed to build a dialect-specific AM while keeping the Language Model (LM) constant for all the dialects. This study explores the effect of dialect mismatched LM by considering three different Telugu regional dialects: Telangana, Coastal Andhra, and Rayalaseema. We show that dialect variations that surface in the form of a different lexicon, grammar, and occasionally semantics can significantly degrade the performance of the LM under mismatched conditions. Therefore, this degradation has an adverse effect on the ASR even when dialect-specific AM is used. We show a degradation of up to 13.13 perplexity points when LM is used under mismatched conditions. Furthermore, we show a degradation of over 9% and over 15% in Character Error Rate (CER) and Word Error Rate (WER), respectively, in the ASR systems when using mismatched LMs over matched LMs.

   @inproceedings{Aditya2022NAACLSRW,
     author = {Yadavalli, Aditya and Mirishkar, Ganesh and Vuppala, Anil Kumar},
     booktitle = {North American Chapter of the Association of Computational Linguistics Student Research Workshop},
     title = {Exploring the Effect of Dialect Mismatched Language Models in Telugu
     Automatic Speech Recognition},
     publisher = {Association for Computational Linguistics (ACL)},
     year = {2022}
   }