Confidence-Aware Ranking with Embedding-based Models in Recommendation Systems

Joel Machado Pires

Confidence-Aware Ranking with Embedding-based Models in Recommendation Systems

Recommender systems (RecSys) enhance information retrieval efficiency across domains such as e-commerce, streaming platforms, and smart cities by delivering personalized content that aligns with user preferences. These systems address user-item relationships through matrix factorization and graph neural networks (GNN). Despite advancements in accuracy, existing approaches often narrowly focus on predictive performance while neglecting the broader utility of confidence estimation. This estimation is important for quantifying the certainty behind recommendations, especially fostering trust when there is a need for a balance between risk and reward. By utilizing confidence, recSys can mitigate uncertainties stemming from data noise and model limitations. Integrating confidence metrics enhances collaborative filtering through refined user-group weighting, improves model robustness via co-training and data augmentation, and optimizes ranking strategies by prioritizing reliable recommendations. Conversely, neglecting confidence risks user disengagement, suboptimal decisions, and revenue loss due to unchecked inaccuracies. By embedding confidence-aware mechanisms, next-generation RecSys can transcend traditional accuracy-centric paradigms, fostering systems that balance performance, adaptability, and user-centric interpretability. Existing approaches to confidence estimation face critical limitations. Parametric methods rely on rigid distributional assumptions, risk convergence issues, and inflexible uncertainty modeling. Non-parametric techniques, such as neural network-based probabilistic calibration, remain confined to classification tasks, failing to address regression scenarios like rating prediction. Many confidence models operate independently of core recommendation processes, limiting their adaptability and calibration impact. Notably, the literature neglects the integration of confidence estimation into graph neural network (GNN)-based RecSys and lacks strategies to leverage confidence for enhancing ranking quality. This study aims to enhance ranking performance in collaborative filtering-based recommender systems by integrating confidence estimation into matrix factorization, GNN, and deep GNN architectures. Using the MovieLens, Amazon Beauty, and Jester Joke datasets, the work evaluates models through RMSE, MAE, and NDCG metrics, focusing on improving NDCG@3 and NDCG@10 via confidence-aware mechanisms. Key objectives include assessing the efficacy of cosine similarity between user-item embeddings as a confidence measure, identifying optimal confidence thresholds to balance recommendation reliability and coverage, and analyzing how confidence-error relationships influence ranking outcomes. We observed that confidence estimation improved model performance, achieving NDCG gains of up to 15%, enhancing top-ranking accuracy, and revealing a triangular-shaped relationship between modeled confidence and absolute error.

Frederico Araújo Durão

Leandro Balby Marinho (UFCG)

http://lattes.cnpq.br/3728312501032061

Yuri de Almeida Malheiros Barbosa

http://lattes.cnpq.br/6396235096236217

Ricardo Araújo Rios (UFBA)

http://lattes.cnpq.br/0427387583450747

Tiago de Oliveira Januario - (Boston University)

http://lattes.cnpq.br/7975984954243833

18 Mar, 2025

8:30 AM