JOURNAL OF HIGH ENERGY PHYSICS, vol.2020, no.11, 2020 (SCI-Expanded)
Decays of the 125 GeV Higgs boson into a Z boson and a rho (0)(770) or (1020) meson are searched for using proton-proton collision data collected by the CMS experiment at the LHC at root s = 13 TeV. The analysed data set corresponds to an integrated luminosity of 137 fb(-1). Events are selected in which the Z boson decays into a pair of electrons or a pair of muons, and the rho and phi mesons decay into pairs of pions and kaons, respectively. No significant excess above the background model is observed. As different polarization states are possible for the decay products of the Z boson and rho or phi mesons, affecting the signal acceptance, scenarios in which the decays are longitudinally or transversely polarized are considered. Upper limits at the 95% confidence level on the Higgs boson branching fractions into Z rho and Z phi are determined to be 1.04-1.31% and 0.31-0.40%, respectively, where the ranges reflect the considered polarization scenarios; these values are 740-940 and 730-950 times larger than the respective standard model expectations. These results constitute the first experimental limits on the two decay channels.