Audio search engine based on joint embedding

Abstract

Audio retrieval based on language allows users to search for audio content using natural language queries. This technology, which has gained popularity in recent years, has numerous applications in fields such as entertainment, education, and healthcare. To achieve our goal, we conducted several tests and validated our results using a phonetic subtitle dataset, converting the sentences into vectors using sBert. We extracted log mel spectrograms from the corresponding audio files. Our analysis was further deepened by applying a convolutional neural network (CNN) architecture to extract features from the log mel spectrograms. We then calculated the similarity with subtitles using the cosine metric. This research underscores the potential for enhanced audio retrieval systems, paving the way for more intuitive and effective methods for accessing audio information. Keywords: Language-based audio retrieval, natural language queries, log mel spectrogram, sBert