Separating resonant signals from background is a major challenge in studies of heavy baryon decays, especially when several states overlap. In this work, the decay \(\Lambda_{b}^{0} \rightarrow \Lambda_{c}^{+} \bar{D}^{*0} K^{-}\) is studied using simulated LHCb detector data. This decay channel is relevant for understanding the formation of exotic states such as pentaquarks, first observed by the LHCb Collaboration in related \(\Lambda_{b}^{0}\) decays [1], [2]. Monte Carlo simulations are used to model both resonant and non-resonant contributions to the decay.
The decay phase space is explored using Dalitz plots, which clearly show resonant structures and the kinematic boundary of the decay (Fig. 1). Based on these simulated data, several machine learning classifiers, including Random Forest, Gradient Boosting, and neural networks, are trained using invariant mass and angular variables. Their performance is evaluated using ROC curves and AUC values, showing good separation between signal and background. To better understand how the models make decisions, SHAP feature importance analysis is applied [3], indicating that invariant mass variables play the dominant role in the classification. This study shows that combining Dalitz plot analysis with interpretable machine learning methods is an effective approach for studying complex \(\Lambda_{b}^{0}\) decay dynamics.

Keywords: pentaquarks, Dalitz plot, Monte Carlo, machine learning, SHAP, RapidSim