Structural transformation effect on a photoinduced (PI) charge-transfer process leading to a phase transition in RbMn[Fe(CN)6], which exhibits a magnetic phase transition with a hysteresis loop between 230 and 300 K, has been investigated by Raman spectroscopy. The valence states of the transition metal ions and the lattice symmetry are reflected in the vibrational frequency of the adjacent CN ions. The charge-transfer phase transition from the high-temperature (HT) phase to low-temperature (LT) phase can be induced by the photo-excitation with 395 nm light in resonance with the ligand-to-metal charge transfer (LMCT) band. The conversion efficiency is lowered with decreasing temperature within the hysteresis loop. However, upon further lowering the temperature, the photo-conversion restarts in the residual HT configuration units and domains around 50 K. The behavior is ascribed to the change of the potential curves in the excited electronic state due to the lattice deformation caused by the cubic-to-tetragonal phase transition in the surrounding lattice.