Aditya Yadavalli

selected publications

1. EMNLP

   PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data

   Watts, Ishaan, Gumma, Varun,  Yadavalli, Aditya, Seshadri, Vivek, Swaminathan, Manohar, and Sitaram, Sunayana

   In Proc. EMNLP 2024

   Abs Bib PDF

   Evaluation of multilingual Large Language Models (LLMs) is challenging due to a variety of factors – the lack of benchmarks with sufficient linguistic diversity, contamination of popular benchmarks into LLM pre-training data and the lack of local, cultural nuances in translated benchmarks. In this work, we study human and LLM-based evaluation in a multilingual, multi-cultural setting. We evaluate 30 models across 10 Indic languages by conducting 90K human evaluations and 30K LLM-based evaluations and find that models such as GPT-4o and Llama-3 70B consistently perform best for most Indic languages. We build leaderboards for two evaluation settings - pairwise comparison and direct assessment and analyse the agreement between humans and LLMs. We find that humans and LLMs agree fairly well in the pairwise setting but the agreement drops for direct assessment evaluation especially for languages such as Bengali and Odia. We also check for various biases in human and LLM-based evaluation and find evidence of self-bias in the GPT-based evaluator. Our work presents a significant step towards scaling up multilingual evaluation of LLMs.

   @article{Watts2024PARIKSHAAL,
     title = {PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data},
     author = {Watts, Ishaan and Gumma, Varun and Yadavalli, Aditya and Seshadri, Vivek and Swaminathan, Manohar and Sitaram, Sunayana},
     journal = {In Proc. EMNLP},
     year = {2024}
   }

2. EACL Findings

   AccentFold: A Journey through African Accents for Zero-Shot ASR Adaptation to Target Accents

   Owodunni, Abraham*,  Yadavalli, Aditya*, Emezue, Chris*, Olatunji, Tobi*, and Mbataku, Clinton

   In Proc. EACL Findings 2024

   Abs Bib PDF

   Despite advancements in speech recognition, accented speech remains challenging. While previous approaches have focused on modeling techniques or creating accented speech datasets, gathering sufficient data for the multitude of accents, particularly in the African context, remains impractical due to their sheer diversity and associated budget constraints. To address these challenges, we propose AccentFold, a method that exploits spatial relationships between learned accent embeddings to improve downstream Automatic Speech Recognition (ASR). Our exploratory analysis of speech embeddings representing 100+ African accents reveals interesting spatial accent relationships highlighting geographic and genealogical similarities, capturing consistent phonological, and morphological regularities, all learned empirically from speech. Furthermore, we discover accent relationships previously uncharacterized by the Ethnologue. Through empirical evaluation, we demonstrate the effectiveness of AccentFold by showing that, for out-of-distribution (OOD) accents, sampling accent subsets for training based on AccentFold information outperforms strong baselines a relative WER improvement of 4.6%. AccentFold presents a promising approach for improving ASR performance on accented speech, particularly in the context of African accents, where data scarcity and budget constraints pose significant challenges. Our findings emphasize the potential of leveraging linguistic relationships to improve zero-shot ASR adaptation to target accents

   @article{Owodunni2024AccentFoldAJ,
     title = {AccentFold: A Journey through African Accents for Zero-Shot ASR Adaptation to Target Accents},
     author = {Owodunni, Abraham* and Yadavalli, Aditya* and Emezue, Chris* and Olatunji, Tobi* and Mbataku, Clinton},
     journal = {In Proc. EACL Findings},
     year = {2024}
   }

3. ACL

   SLABERT Talk Pretty One Day: Modeling Second Language Acquisition with BERT

   Yadavalli, Aditya*, Yadavalli, Alekhya*, and Tobin, Vera

   In Proc. ACL 2023

   Abs Bib PDF

   Second language acquisition (SLA) research has extensively studied cross-linguistic transfer, the influence of linguistic structure of a speaker’s native language [L1] on the successful acquisition of a foreign language [L2]. Effects of such transfer can be positive (facilitating acquisition) or negative (impeding acquisition). We find that NLP literature has not given enough attention to the phenomenon of negative transfer. To understand patterns of both positive and negative transfer between L1 and L2, we model sequential second language acquisition in LMs. Further, we build a Mutlilingual Age Ordered CHILDES (MAO-CHILDES) – a dataset consisting of 5 typologically diverse languages, i.e., German, French, Polish, Indonesian, and Japanese – to understand the degree to which native Child-Directed Speech (CDS) [L1] can help or conflict with English language acquisition [L2]. To examine the impact of native CDS, we use the TILT-based cross lingual transfer learning approach established by Papadimitriou and Jurafsky (2020) and find that, as in human SLA, language family distance predicts more negative transfer. Additionally, we find that conversational speech data shows greater facilitation for language acquisition than scripted speech data. Our findings call for further research using our novel Transformer-based SLA models and we would like to encourage it by releasing our code, data, and models

   @article{Yadavalli2023SLABERTTP,
     title = {SLABERT Talk Pretty One Day: Modeling Second Language Acquisition with BERT},
     author = {Yadavalli, Aditya* and Yadavalli, Alekhya* and Tobin, Vera},
     journal = {In Proc. ACL},
     year = {2023},
     volume = {abs/2305.19589}
   }

4. Interspeech

   Multi-Task End-to-End Model for Telugu Dialect and Speech Recognition

   Yadavalli, Aditya, Mirishkar, Ganesh, and Vuppala, Anil Kumar

   In Proc. Interspeech 2022

   Abs Bib PDF

   Conventional Automatic Speech Recognition (ASR) systems are susceptible to dialect variations within a language, thereby adversely affecting the ASR. Therefore, the current practice is to use dialect-specific ASRs. However, dialect-specific information or data is hard to obtain making it difficult to build dialect-specific ASRs. Furthermore, it is cumbersome to maintain multiple dialect-specific ASR systems for each language. We build a unified multi-dialect End-to-End ASR that removes the need for a dialect recognition block and the need to maintain multiple dialect-specific ASRs for three Telugu regional dialects: Telangana, Coastal Andhra, and Rayalaseema. We find that pooling the data and training a multi-dialect ASR benefits the low-resource dialect the most – an improvement of over 9.71% in relative Word Error Rate (WER). Subsequently, we experiment with multi-task ASRs where the primary task is to transcribe the audio and the secondary task is to predict the dialect. We do this by adding a Dialect ID to the output targets. Such a model outperforms naive multi-dialect ASRs by up to 8.24% in relative WER. Additionally, we test this model on a dialect recognition task and find that it outperforms strong baselines by 6.14% in accuracy.

   @inproceedings{Aditya2022Interspeech,
     author = {Yadavalli, Aditya and Mirishkar, Ganesh and Vuppala, Anil Kumar},
     booktitle = {Proc. Interspeech},
     title = {Multi-Task End-to-End Model for Telugu Dialect and Speech Recognition},
     pages = {1387--1391},
     doi = {10.21437/Interspeech.2022-10739},
     year = {2022}
   }