The pressure dependence of a large number of phonon modes in CaFe₂As₂ with energies covering the full range of the phonon spectrum has been studied using inelastic x-ray and neutron scatterings. The pressure range was large enough to cover the first-order phase transition into the so-called collapsed phase where the c-axis contracts by about 6% whereas a and b axes expand by about 1.5%. Our main result is that pressure-induced phonon frequency shifts are well explained by the changes in relevant bond lengths throughout the pressure range, including those associated with the first-order phase transition. Specifically, the frequencies of phonons polarized in the ab plane as well as the Fe-As bond lengths change little across the phase transition. On the other hand, the transverse-acoustic phonons propagating along the c direction stiffen very significantly in response to the large contraction of the bonds along the c axis. Nonmagnetic density-functional calculations describe the frequencies in both the zero pressure and in the collapsed phase in a satisfactory way if based on the respective experimental crystal structures. This suggests that there is no need to invoke changes in magnetic moments on Fe atoms to explain the pressure-induced frequency shifts.