LLM-Modulo-Rec: Leveraging Approximate World Knowledge of LLMs to Improve eCommerce Search Ranking Under Data Paucity

Effective ranking of products relevant to a user's query and interest is the main goal of e-commerce product ranking. In this context, ranking irrelevant products or those mismatched with the intent of the user query results in sub-optimal user experience. Providing high-quality, relevant search rankings requires large labeled datasets for training powerful deep learning (DL) based ranking pipelines. However, such large datasets are costly and time-consuming to obtain. Another important facet that influences search ranking quality is the intent and ambiguity in the user's search query. Hence, data paucity and query ambiguity are two ever-present challenges impeding the success of modern deep learning (DL) based e-commerce ranking models. In this work, we present the first ever investigation of employing large-language models (LLMs) as approximate knowledge sources to counter these challenges and improve the performance of off-the-shelf ranking models, under data paucity and query ambiguity. Specifically, we undertake the first ever investigation of developing an LLM-Modulo method to improve the search ranking performance of off-the-shelf ranking models. Our framework utilizes LLMs not only to characterize intent ambiguity and augment training data, but also to generate challenging hard-negative examples to enable ranking models to better distinguish fine-grained relevance levels. Our experiments demonstrate notable performance improvements in ranking quality of these off-the-shelf models, when employed in an LLM-Modulo manner.