LLM-Based Automatic Sentence Generation for AAC Devices

Date & Time

Wednesday, March 20, 2024 - 2:20 PM PDT  

Location

Elite 1-3  

Description

This work presents a system for the automatic generation of complete sentences using large language models (LLMs) based on the selected combinations of symbols for AAC devices. It first uses speech-to-text (STT) to transcribe spoken utterances to texts for extracting the context of the given situation, then it takes input in terms of selection of symbols from the AAC user. The contextual information as well as the selected symbols are then used as inputs to an LLM, GPT-4 in our case, to generate a complete sentence suitable for AAC-based communications. Then a text-to-speech (TTS) plays the speech using the generated sentence. For enabling natural communication, the system needs to guarantee real-time operation by using LLM, STT, and TTS application programming interfaces (APIs) in a sequential manner, and careful prompting techniques to generate proper sentences suitable for the context of AAC device users. In addition to these, this work presents methods to dynamically change the layout of the symbols based on the context learned from the STT results as well as previous conversations, for efficiently utilizing the limited screen real estate. It also presents the results of user studies for people with communication disorder, autism spectrum disorder (ASD), and brain lesions.

This Presentation Link is provided by the Presenter(s) and not hosted by the Center on Disabilities at CSUN. The Center on Disabilities has confirmed, as of March 27, 2024, content linked is relevant to the presentation, but has not been reviewed for accessibility nor will the Center on Disabilities attempt to remediate any accessibility issues in the linked content. Please contact the Presenter(s) with any accessibility concerns.

 

Audience

• Higher Education
• Information & Communications Technology
• Disability Specific
• K-12 Education
• Research & Development

Audience Level

Intermediate  

Session Summary (Abstract)

The presentation will describe the system integration of LLM, STT, and TTS for enabling natural communication for AAC devices by exploiting the capabilities of LLM for generating complete sentences given proper prompting.  

Primary Topic

Augmentative and Alternative Communications (AAC)  

Secondary Topics

• Artificial Intelligence (AI) & Machine Learning (ML)
• Emerging Technologies
• Engineering

 

Session Type

General Track  

Presenters

• Bowon Lee
  Inha University/SpeechWise Inc.
• Jong Hwan Na
  Inha University/SpeechWise Inc.

Add to Outlook/iCal (How it works)

Back to Session List