Management-by-walking-around (MBWA) is a widely adopted technique in hospitals that involves senior managers directly observing frontline work. However, few studies have rigorously examined its impact on organizational outcomes. This study examines an improvement program based on MBWA in which senior managers observe frontline employees, solicit ideas about improvement opportunities, and work with staff to resolve the issues. We randomly selected hospitals to implement the 18-month-long, MBWA-based improvement program; 56 work areas participated. We find that the program, on average, had a negative impact on performance. To explain this surprising finding, we use mixed methods to examine the impact of the work area's problem-solving approach. Results suggest that prioritizing easy-to-solve problems was associated with improved performance. We believe this was because it resulted in greater action-taking. A different approach was characterized by prioritizing high-value problems, which was not successful in our study. We also find that assigning to senior managers responsibility for ensuring that identified problems get resolved resulted in better performance. Overall, our study suggests that senior managers' physical presence in their organizations' front lines was not helpful unless it enabled active problem solving.