A study of refactorings during software change tasks

Developers frequently undertake software change tasks that could be partially or fully automated by refactoring tools. As has been reported by others, all too often, these refactoring steps are instead performed manually by developers. These missed opportunities are referred to as occasions of disuse of refactoring tools. We perform an observational study in which 17 developers with professional experience attempt to solve three change tasks with steps amenable to the use of refactoring tools. We found that the strategies developers use to approach these tasks shape their workflow, which, in turn, shape the opportunities for refactoring tool use. We report on a number of findings about developer strategies, demonstrating the difficulty of aligning the kind of refactoring steps that emerge during a change task based on the strategy with the tools available. We also report on findings about refactoring tools, such as the difficulties developers face in controlling the scope of application of the tools. Our findings can help inform the designers of refactoring tools.


| INTRODUCTION
Few software changes are alike: they vary in size, in complexity, and in their motivating reasons (e.g., to add functionality and to fix a bug), among other differences. 1,2Despite this substantial variability, one similarity is that software change tasks almost always involve modification of code by a developer.Several studies [3][4][5][6] have shown that some portion of the modifications developers undertake as part of change tasks are refactorings, namely, modifications that alter the structure of a system while preserving its behavior. 7As a simple example, a developer may change the name of a method to better reflect its purpose.If the developer recognizes that this renaming operation is a refactoring automated by their development environment, they can invoke the automated support to update the method name and all references (e.g., call sites) to the new name.Common refactorings like rename, extract, and move 8 occur both during software changes that are purely quality-improving and-more commonlyduring changes to software functionality. 5,9,10 1 Rfactorings: For descriptions of refactorings referred to in this paper, consult the Appendix 3.
The automation of refactoring operations is intended to enable developers to modify code faster and with fewer errors.This is a seductive proposition and has led to many research and development efforts aimed at enabling and supporting a broad set of refactorings in commonly used integrated development environments (IDEs).For example, the standard installations of mainstream IDEs for the popular object-oriented language Java (e.g., Eclipse 11 ) and C# (e.g., Visual Studio 12 ) each offer their users automated tools for over 40 refactoring operations.
Despite a long-standing prevalence of refactoring tools in development environments, it is widely recognized that this refactoring support is not used as often as would be expected, and in fact, these tools are disused. 13To address this disuse, researchers have studied a variety of different approaches, including improving developers' awareness of the tools, [14][15][16] improving the usability 17 and correctness 18,19 of the tools, and investigating factors affecting developers' acceptance of automation. 20These approaches generally assume that increased use of automated refactorings is largely about the tools: if developers are made aware of tools and the tools are made usable, then the tools will be used.
We take a different perspective.We believe that to be both useful and used, tools must not only automate code transformations but must fit into a developer's overall workflow.In other words, we pursue an approach that is about the humans first and the tools second.
We thus choose to investigate the broader context of refactoring during software change tasks by considering how developers approach the tasks, how their approach enables or blocks opportunities to use a refactoring, and their use and success with refactorings, whether manual or automated.We perform an observational study to investigate, in a controlled environment, how 17 experienced developers approach three software change tasks that are amenable to the use of automated refactorings.With this study, we explore three research questions: RQ1: What strategies do developers use to approach software change tasks which include refactorings?RQ2: How often do developers use automated support for refactoring versus proceeding manually?RQ3: How do developers experience refactoring tools?
We found that the participants made progress on the tasks utilizing one of three strategies: (1) a Local strategy in which the participant's workflow largely relies on frequent code changes guided by local compiler errors, (2) a Structure strategy in which the participant builds up an understanding of the code structure and uses that structure to guide their changes, and (3) an Execute strategy in which the participant guides their changes by managing and executing tests.
While refactoring tools were used by several participants from each strategy group, the tools were most commonly used by participants who employed the Structure strategy.This increased usage seems to be due to the tools being more aligned with this strategy.Participants using other strategies struggled to use the tools as the tools did not align with their workflows.For instance, participants using a Local strategy desired to follow a change step-by-step: the extensive changes supported by a refactoring tool do not align with the need for these developers to see each change as it occurs.Refactoring tools would need to operate differently for participants using the Local and Execute strategies to make use of the tools.
We also report novel findings of how developers use refactoring tools, such as gathering information from a tool's "preview view" to guide manual code changes or relying on diff-tools like git to learn about the impact that the refactoring tool had on their code.This work also provides detailed examples of previously suggested barriers to the use of refactoring tools, such as lack of trust or a gap between the operation needed in the eyes of the developer and the operations supported by tools.On the basis of our findings, we propose several changes to refactoring tool workflows.
This paper makes four contributions: • It reports on an observational study that enables investigation of how software developers approach software change tasks amenable to refactoring.We provide a replication package to allow others to undertake a similar study or use the experimental materials for other refactoring studies (Section 8).
• It introduces a categorization of workflows developers use to approach change tasks in terms of Local, Structure, and Execute strategies.This categorization may be useful for empirical studies in software evolution.
• It presents a series of findings of how developers approach refactoring operations that can inform practitioners and toolmakers, such as the need to control the scope of code transformations provided by the tools.
• It demonstrates the need to study refactoring operations through the broader context of how software developers work.
We begin by placing our study in the context of earlier studies of automated refactoring (Section 2).We then describe the study (Section 3), present the results (Section 5), discuss threats (Section 6), and consider implications (Section 7).We summarize our findings in the final section of the paper (Section 8).
The potential benefits of refactoring tools have been recognized for over three decades. 21Much of the research that has been conducted about refactorings has focused on making refactoring tools correct and usable.With respect to these tools, researchers have looked at improving correctness 22 by means of different implementation techniques, 18,[23][24][25] checking a minimal set of preconditions, 19 and removing bugs. 26[28][29][30][31] The refactoring tools that exist in commonly used development environments reflect some of these research findings.
In this paper, we accept these tools as they are, focusing instead on how developers make use of them.We flip the focus by presenting developers with tasks that are amenable to the use of refactorings and observing how they proceed.With this focus, our investigation is more similar to research conducted about how developers use refactoring tools what experiences motivate use or disuse.
An early example of such a study was undertaken by Murphy-Hill et al. 6 In this study, data were gathered about developer interactions with environments that included refactoring tools, as well as from version control repositories into which developers stored code.
Consideration of both tool use and version histories is common in refactoring research for two reasons: (1) it enables identification of when refactoring tools were invoked and whether that use resulted in a lasting code change, and (2) it enables mining when refactorings might have occurred based on how the code evolved as seen through analysis of the version history.Murphy-Hill et al. used this information to characterize how often different refactoring operations were invoked, completed, and integrated into the code repository.They found that 90% of refactorings were performed manually and conducted followup interviews with five participants to understand why.These participants self-reported not using tools due to lack of awareness of tools, lack of trust in tools, no opportunity to use them, and tools disrupting their workflow.
In another study, Murphy-Hill et al. investigated specific usability problems with the extract-method refactoring in an observational study. 17They instructed participants to apply the extract method refactoring to source code while the experimenters recorded any usability problems they encountered.They observed several usability problems that could lead to tool disuse, such as the inability to select code that formed statements and incomprehensible error messages.
Vakilian et al. also investigated refactoring tool use by analyzing data from IDEs that captured developer interactions.They investigated how developers invoked, configured, and applied refactoring tools and found further evidence for disuse. 13They conducted interviews with a subset of the participating developers (11 participants) to understand broad issues affecting the use of refactorings.On the basis of their interview data, the authors suggest factors that might impact developers' choice to use or disuse refactoring tools, such as lack of awareness of tools, lack of trust, and unpredictable tools.
Silva et al. used an interesting methodological approach to investigate why developers use refactorings. 5They detected refactoring commits to a number of GitHub open-source projects and immediately query contributors for the motivations behind their refactoring operations.They found that refactoring activity is mainly driven by changes in requirements or functional changes.They also ask contributors if the refactoring was performed manually or using a tool.Over half of the respondents report doing it manually and mention several factors contributing to this choice, including lack of awareness, lack of trust, the complexity being too high or too low, and lack of IDE support.Kim et al. were also interested in how software developers experience refactorings. 32They conducted a survey of Microsoft employees and more in-depth interviews with a specific refactoring team and an analysis of that team's version history.Their participants self-report doing 85% of refactorings manually.In their study, the meaning of the term "refactoring" was broader, and refactoring operations like remove parameter were used interchangeably with large quality-improving program changes that occurred over several years.The latter category of refactorings is less amenable to direct tool support and varies substantially from the smaller operations for which automated support appears in development environments.This definition of refactoring is nonetheless common in industry and can lead to ambiguity in study results when the type of refactoring is not defined.
Liu W. and Liu H. were interested in what motivates developers to apply refactorings and focused their study on the extract method.
Through an analysis of version histories of open-source projects and interviews of developers, they found reuse to be the main motivation for performing this specific refactoring. 33Paixão et al. also investigated why developers apply refactorings and mined code changes from open-source communities, finding that refactorings are mainly driven by the intent to introduce or enhance features. 9o other studies that summarize perspectives on refactoring from the industry are conducted by Sharma et al 14 and Leppänen et al. 34 These studies investigated both kinds of refactoring and suggest factors that deter from tool use, such as lack of exposure to tool and differential code bases 14 and lack of trust in tools. 34Despite trust being a recurring theme in these studies, it is not well understood what developers refer to when they complain of lack of trust, nor how to address it in tools.Whereas some researchers argue that trust requires correct tools, others speculate that trust is tied to the tools' transparency and predictability. 13,35veral studies refer to the usability of refactoring tools without defining what usability is.We previously argued for the utility of lab studies with practitioners to investigate this facet of refactoring tools. 36We later introduced a theory of refactoring tool usability. 37We developed this theory by investigating the transcripts from the study reported in this paper from the perspective of usability as defined by the ISO 9241-11. 38The ISO definition focuses on qualities of the developer's experience of the human-computer interaction that occurs when they use a refactoring tool, such as satisfaction.In contrast, this paper focuses on understanding how the act of refactoring-with or without the use of an automated refactoring tool-fits into developer workflows.The analysis of the transcripts used to develop the theory of usability is distinct from the analysis of study data reported in this paper.The details of the study method are also unique to this paper.
The theory we presented says: Software developers employ refactoring tools to help prepare or complete functional changes to a software system.Software developers seek these tools to be reasonable to locate, and for the tools to help them assess the efficiency of the tool, in terms of the costs and benefits of the tool, before its use.To enable effective use in multiple situations, software developers seek to guide how a tool changes the source code for a system; this ability to tailor how a tool works can improve the efficiency of the tool for the developer.Software developers also seek refactoring tools to explain their impact to source code so that the software developer can understand the effectiveness of the tool.Software developers also expect tools to communicate clearly and directly in terms that match how software developers perceive refactoring operations.These characteristics in a refactoring tool increase the satisfaction of the software developer using a refactoring tool.
The existing literature shows that developers do perform refactorings during software change tasks and often do so manually: Kim et al. found that developers at Microsoft self-report doing 86% manually, 32 Murphy-Hill et al. found 90% to be manual, 6 and both Negara et al. and Silva et al. reported more than half of the refactorings they studied to be performed manually.The existing literature then hones in on why existing tool support is disused, tacitly assuming that the automated refactoring tools are appropriate for how developers work.In this paper, we flip the focus, presenting developers with tasks that are amenable to the use of refactorings and observing how they proceed.Unlike previous observational studies, 16,17 we do not ask developers to perform refactorings.Instead, we ask them to perform realistic software change tasks that include code changes amenable to automated refactoring support.We then observe what they do and what they say regardless of whether they use or disuse the tools and conduct immediate follow-up interviews to discuss their experiences.With this approach, we question the tacit assumption that refactoring tools as defined are an adequate substitution for developer's manual processes and open up opportunities to consider how to help developers complete change tasks that include refactorings.

| STUDY
To support the investigation of how software developers use refactoring tools in software change tasks, we performed an observational study.
We recruited software developers with professional experience and asked them to solve a set of software change tasks while following a think-aloud protocol.After their work on the tasks, we performed a semi-structured interview.This format enabled us to compare how multiple developers approached the same three software change tasks and compare their interview transcripts, as well as to ground the interviews in the activities observed during their work on the tasks.
The tasks were performed on a scaled-down version that we created of the Apache Commons-Lang project 39 and were modeled after commits to popular open source systems, verified to contain refactorings 3 that previous work has identified to be performed manually more often than with automated tools: change-signature, inline-method, and move-method.This makes the tasks realistic and relevant to understand refactoring tool use and disuse.
A full experimental package is available to support replication (Section 8).

| Participants
We recruited 19 individuals with professional development experience in a North American city using snowball recruiting seeded by company contacts known to the authors.Participants filled out an online presurvey to determine eligibility and give consent.To be a participant, an individual needed at least 1 year of professional (employed and paid) experience with Java (or a similar) programming language, be able to provide a definition of refactoring, and be able to describe a few refactoring operations.
Two of the 19 admitted individuals experienced challenges during the experiment that led us to exclude them from the participant pool and analysis: one encountered technical problems with the IDE that required stopping the experiment, and one was unable to make progress on simple tasks.The results from these two participants are not included in the analysis and results presented in this paper: in the remainder of this paper, we use "participants" to mean the 17 remaining participants and use the notation P x to reference the experiences of a particular numbered participant.To maintain consistency with the data package, we keep the original indexing of the remaining participants.This explains the existence of P 18 and P 19 despite the results pertaining to only 17 participants.Table 1 presents an overview of the participants.
For the 17 participants, the mean (and average) number of years of work experience was 10, with the range of experience being from one to more than 20.Two participants did not have previous experience in Java but were accepted due to extensive experience (≥10 years) with other object-oriented languages.Only two of the participants presented as female.Four participants were enrolled as students.Each participant was allowed to choose between study locations hosted by the experimenter (the first author of this paper) on the university campus or to host the experimenter at their workplace.Seven participants chose the campus location, and 10 chose to provide a meeting room at their workplace.All participants received a $20 gift card as a token of appreciation.

| Experimental session
The experimental session was conducted in person with one participant at a time.We provided each participant with the same laptop, system, and tasks.The first author of this paper acted as an experimenter and was present in the room during the experiment.The experimenter administered the consent form, the tasks, answered questions, gave prompts, and took notes during the tasks.The experimenter was positioned such that she could see the screen and prompt the participant if he stopped describing what he was doing.The experimenter also made notes of events that occurred during the tasks that could be investigated further in the interview segments.
After obtaining consent, an experimental session began with an explanation of the experiment setup with an introduction to the scope of the tasks, the participant's role, and the experimenter's role.The participant was told that they would be asked to undertake three provided change tasks in succession on a codebase loaded into a development environment.The participant was told that there was no set time per task but that the experimenter would indicate if it was time to move to the next task to keep the overall session within 2 h and make time for an interview segment at the end.All experimental sessions were completed within 2 h.
The tasks were given in order, with the task description being revealed to the participant when that task began.For each task, the participant was given a task description on paper and asked to describe their plan for completing the task.Knowing the upfront plan enabled the experimenter to detect deviations from the plan and to prompt for events or experiences that triggered changes to the plan.The participant was told that they would not be bound to their plan.
The participant was asked to think aloud as they worked and were prompted if they lapsed in voicing their thoughts and actions.The experimenter answered questions asked by the participant, such as helping to orient them in the development environment.Answers related to questions about the tasks themselves were limited to contextual information from the original commits (e.g., motivation for the task) and the scope of the tasks (e.g., questions related to client code).
For each session, the participant's screen and audio were recorded, and their code changes saved.At the end of a task, the experimenter asked: 1. Which source code changes did you make in order to solve this task?2. Do you know of any tools that could have automated any of the changes you made? 3. Are there any changes you are unsure if you got right?
The experimenter also asked for clarifications on the participant's actions and strategies.Participants might also be asked to explain specific things they had done or why they changed their plans while the task was fresh in their minds.If they expressed different strategies throughout the task, they may be asked how they would have solved it if they were to do it again.To avoid bias, the experimenter took care to not use refactoring names until after a participant did and subsequently used the name that the participant chose in conversation.
After all three tasks were over, the experimenter immediately conducted a longer interview with that participant.The interview included questions about their past experience with refactoring tools and what impacted their choices to use or not to use these tools during the tasks.
T A B L E 1 Overview of all the admitted participants Note: Rows denote number of years of professional experience, gender, and the editors that participants reported familiarity with.Thirteen participants reported that they were familiar with the editor that was used in the study, IntelliJ. a

| Experimental system and tasks
The change tasks used in the study are based on actual changes to open-source systems.We sourced these change tasks from commits to open-source systems.We looked for commits that contained refactorings that have been identified by earlier research as more often manually performed and that were understandable and replicable by experienced developers in a reasonable time.We investigated the dataset from Silva et al 5 and used a state-of-the-art tool, RefactoringMiner, 3 to mine several additional repositories, including Apache Commons-Lang.
The three change tasks are based on existing commits to open-source systems: the first two from Apache Commons-Lang and one from Quasar. 40We ensured that the descriptions of tasks did not instruct developers to perform specific refactorings, nor did the instructions include the word refactoring.Indeed, out of the three tasks, only the first task is a pure refactoring, that is, a code change that only alters structure and not functionality.This task involves moving methods around, whereas the other two tasks alter APIs and thus require corresponding changes to tests.Previous studies have identified such API-level changes as containing refactorings. 4e initial (6) pilot sessions were performed on the systems in which the commits were detected.The pilot participants solved the first two tasks in Apache Commons-Lang and the third task in Quasar.The pilot sessions revealed that it was challenging for participants to switch context between multiple software projects and to solve tasks in source code about an unfamiliar domain and with frameworks or coding style that were unfamiliar to them.We acted on this feedback by recreating a part of Apache Commons-Lang as a stand-alone system and mapping the three commits onto this system.The two commits that came Apache Commons-Lang were easy to map to our system; we adjusted the third to fit the new code context verified that the task contained the same refactorings as in the original commit by using the RefactoringMiner tool.
The target system was a scaled-down version we created of the Apache Commons-Lang project, which provides common helper methods such as string manipulation and basic numerical methods.This system is self-contained yet large enough to simulate a realistic project, comprising 78 K lines of Java, version 1.8, and 1335 JUnit tests.The system is built using Maven 41 and version-controlled through git.Participants worked on a provided Macbook Air, 13", 2017, and were asked to use IntelliJ CE IDEA 2017.3.
This system was chosen because of its relative simplicity: the target domain of Apache Commons-Lang should be familiar to most Java programmers; it is relatively simple to scale it down in size and complexity (e.g., removing external dependencies) so that an experienced developer can become familiar with the code in a short time; it is large enough that a developer will benefit from using tools to change the code; and finally, the system follows common programming patterns such as JUnit tests and Maven standard layout for files and consists mainly of utility classes with static methods.In order to solve all three tasks, participants need to change (e.g., move, alter, or delete) 45 methods distributed across 5 files.The pilot participants were not counted among the study participants.

| Task 1: Organize test methods
In this task, a participant is asked to reorganize 12 listed test methods from two large existing test classes into a new class, such that similar testing functionality is gathered together.This task replicates part of an Apache Commons-Lang commit 42 that reorganized their test methods.
To complete this task, a participant needed to create a new class into which the 12 listed methods were to be placed.For the code to compile, there was also a need to include the required imports and resolve a reference in the moved methods to a local constant defined in the original test class.The constant could be resolved by duplicating it in the new test class, by increasing visibility of the originally defined constant, or replacing references to it with its value.
Work on this task could benefit from the help of such automated refactorings as move method and extract-class.The mining tool shows this task as move-method and extract-class.

| Task 2: Removing redundant methods
In this task, a participant is asked to remove two methods, which are negated versions of two other methods in the API.The two other methods depend on the negated versions.Listing 1 shows one of the original method pairs, and Listing 2 shows part of the solution for that pair.
Each of the four methods has a single JUnit test and Javadocs.This task replicates an Apache Commons-Lang commit, 43 which is discussed in this pull request. 44e minimal changes required to solve this task are to remove the two designated methods and their test methods and then to restore behavior in the project.Restoring behavior requires either inlining the methods before removal or reimplementing the callers.
Work on this task could benefit from the inline-method refactoring.RefactoringMiner shows this task as inline-method.

| Task 3: Overloaded method reduction
In this task, a participant is asked to remove particular functionality from a designated class.The class consists of eight method pairs that offer functionality related to reading from and writing to objects of some data type.In each method pair, one method implements some standard behavior, and the other method takes an extra flag argument that lets callers toggle some special behavior.The first method is a wrapper to the second method, calling the second method with the extra flag argument set to a particular value.Listing 3 provides one example of such a method pair.All methods in this class have multiple tests each.Both the standard behavior and the special behavior are tested.
A participant is guided to remove the parameter such that clients can no longer detect the functionality (e.g., changing the visibility of methods is not sufficient).Each method with the flag has an overloaded version without the flag in the original code.This task replicates part of a commit to Quasar that was collected and analyzed by Silva et al. 5 We mapped the kind of code change from this commit onto a class in our experimental system and verified that it was an equivalent refactoring both manually and using RefactoringMiner. 3o complete this task, all eight method declarations should have the boolean parameter removed, and all calls should be updated.The method pairs should be reduced into one method, such that only the wrapper method signature exists and contains all of the logic.All logic that relies on the parameter should perform as if the parameter was false.For example, an if-branch that executes if and only if the parameter is true can be removed altogether.All tests associated with the boolean parameter are duplicated and can be removed.The many changes needed to be performed are intended to make automated support more attractive to a developer.
Work on this task can benefit from the safe-delete, inline-method, or the change-signature refactoring.RefactoringMiner shows this task as inline-method.

| DATA AND ANALYSIS
The study data comprised approximately 32 h of study sessions.We transcribed the audio portion of the video captured from the 17 experimental sessions and created one transcript for each session from the recorded screen capture and audio capture information.In these transcripts, we marked the sections related to interview questions and answers.In total, the 17 transcripts comprise 134,947 words and can be accessed in the replication package along with the screen recordings, participants' solution source code, and a report from the mining tool on refactorings that can be detected in each of their solutions.This data are analyzed in several steps.
The experimenter (the first author of this paper) performed the initial labeling of the transcripts for easier retrieval of points of interest by marking sections that pertain to events in the videos or comments made by participants.The labels are shown in Table 2 and identify actions or events that were significant to the participants' individual workflows, such as their stated plans for how to solve the tasks, which refactoring tools they named or invoked, other tools they invoked, and actions they took to validate that their changes were correct before continuing their work.
While labels pertaining to plans or sentiments could be identified solely by the transcript and labels pertaining to invocations solely by the video, the remaining labels needed to be determined by reviewing both audio and video.The labeling was discussed with, and reviewed by, the second author of this paper.
We then created summaries of the plans and actions that each participant used to approach each task.In order to develop a schema that captured a participant's approach to each task, we included in the summary schema the following information: quotes that indicate their plans or intent (if any such were made); a natural-language summary of their approach, written by the experimenter; which refactoring tools they mentioned, tried, and used with and without prompts; other tools that they used to solve the task; how they guided their changes (e.g., emphasis on navigating to and fixing compiler errors, avoiding errors, and top-down in the file); and how they verified their changes.The set of possible refactoring tools is taken from the IDE that was used. 45If a participant mentions or invokes a refactoring tool, then we consider them aware of the tool; if they invoke the tool, then we consider that they tried it; and in order to use a refactoring tool, the participant needs to both apply it and continue working with the resulting code.
T A B L E 2 Labels used for initial labeling of transcripts

Plan (keywords)
The participant expresses actions they intend to take to solve the task.Keywords include short, descriptive statements such as break-fix or systematic.

Change Validation (Method)
The participant takes an action to validate their change.Method is a short, descriptive statement of the method they employed such as static (e.g., compiling, syntax errors, and warnings), dynamic (e.g., running tests), and compare (e.g., using git diff of comparing to commented out code).

Named (Refactoring)
The participant names a refactoring tool.Refactoring tools include the common refactoring tools: Rename, Move, Inline, et.c.

Invoked (Tool)
The participant invokes a refactoring tool.Tools include the common refactoring tools: Rename, Move, Inline, etc. as well as Find Usages, etc.

Restart
The participant restarts the task.

Mistake
The participant makes a coding mistake such as changing the wrong code location.
Error (Not) Understand (Refactoring/Other) The participant encounters an error that they do (not) understand, either arising from a refactoring too, the compiler, or tests.Mark errors that are significant to their work, not all encountered errors.Mark source of error (e.g., Move and compiler) where nonobvious from transcript.

Trust
The participant expresses a sentiment related to tool use or disuse and trust or lack of trust.

Predictable
The participant expresses a sentiment related to tool use or disuse and predictability or lack of predictability.

Refactoring Insight
The participant makes an insightful statement regarding tool use or disuse that may be useful to retrieve later.
The experimenter (the first author of this paper) created the summaries by applying the schema to all participants' workflows on each task.
The schema was developed by the two authors of this paper by reviewing several video recordings and discussing participant workflows.The experimenter reviewed the video and audio recording of each participant to create a summary for that participant.Figure 1 shows the schema completed for a participant working on the second task.In this summary, the participant's intent is represented by a quote that summarizes their intended plan and then the experimenter filled out the Approach Summary describing what actual actions the participants took, problems they encountered (tool error), and how they organized their changes ("Isolate changes and keeps tests running by introducing a shared dependency as an intermediate step.").Then records about refactoring tools are listed, such as tools the participant mentions, tries to use, or uses, as well as other tools they relied on, how they guided their changes, and how they verified their changes.To verify that the summaries appropriately capture a participant's workflow, the second author of this paper reviewed several summaries, comparing them with the video recordings.The two authors also discussed any difficult to interpret actions by developers.This process results in 51 summaries.
Once the set of 51 summaries was created, we compare them and look for patterns in the following information: the intent that participants express and the actions they used to change the code, guide their changes, verify their changes, and recover from unwanted errors.On the basis of the patterns between summaries, we identify different strategies that participants employed in each task.

| RESULTS
Overall, participants were able to make good progress on the tasks: all participants made some progress on all three tasks.Furthermore, all participants completed Task 1, 16 (94%) completed Task 2, and 12 (70%) completed Task 3. Twelve (70%) of participants completed all tasks.The progress made by participants on the tasks provided substantial opportunities to observe the workflow and tool use during each task.We describe our results in terms of the research questions posed in Section 1 and use the notation P x to reference to the experiences of a particular numbered participant.
We consider the three research questions outlined in Section 1 in turn.

| RQ1:
What strategies do developers use to approach software change tasks which include refactorings?
The participants took a variety of approaches to the three software change tasks.Applying the analysis described in Section 4, we identified three distinctly different strategies for interacting with the code: 1. Local, in which participants guided their changes by local and immediate information such as feedback from the compiler.

2.
Structure, in which participants guided their changes by code structure such as callers, conditional branches, and references.
F I G U R E 1 Task analysis schema for P 9 on Task 2 3. Execute, in which participants guided their changes by executing the test suite.
Figure 2 shows how the strategies used distribute across each participant's approach to a task.Each of the strategies was successfully employed by one or more participants to solve each task, indicating that a task did not dictate which strategy to use; instead, the choice of strategy was dependent on the combination of the individual and the task, with some participants changing strategies between tasks.Figure 3 further shows that there was no one dominant strategy used for a task.
Despite the variety of strategies, participants using different strategies still often produced the same source code solution in the end.For example, an Execute participant may first isolate the functional changes that needs to be performed in order to achieve the task goal and then update the code to match this new behavior.Meanwhile, a participant with a Local strategy may first alter the code that seems to implement the exact behavior that is required to change, then-almost accidentally-propagate the change to tests by iteratively fixing compiler errors.A partici- We group the workflows first by task, then by strategy, so that the 17 workflows from Task 1 is represented by the three leftmost bars, the 17 workflows from Task 2 by the three middle bars, and the 17 from Task 3 in the three rightmost bars.Within each group, the three different-colored bars indicate how the strategies were distributed across participant's workflows for that task.For example, in Task 1, there were seven participants who employed a workflow that matched the Local strategy, six participants with a workflow matching the Structure strategy, and four matching the Execute strategy The strategies were distributed both across participants (x-axis) and tasks (y-axis).In total, 22 workflows were identified as Local, 13 were identified as Structure, and 16 were identified as Execute.As can be seen in P 4 , P 16 , and P 18 , individual participants sometimes changed strategies between tasks pant using a Structure approach will likely make the same code changes but may make them by studying program structure rather than isolating either local changes or changes oriented to the program execution.

| Local
Participants who used a Local strategy had workflows that were characterized by interacting predominately with information that was immediately available to them (i.e., local information) and changing elements that were present in their editor.Their workflows included actions that intentionally provoke feedback from the IDE such as deleting or commenting out code elements that have references elsewhere in the code, thereby introducing compiler errors or "red lines."When stating their intent, participants made comments such as P 4 on Task 1: "copy-paste them all from one place to another and see what breaks" and P 2 on Task 3: "So what I'm going to do now is rely on my friendly compiler to do this.So now things are going to break."Their workflows were oriented around introducing, navigating between, and fixing this feedback.As a result, the source code frequently underwent long periods in noncompiling states, and the impact that source code changes had on their tests in Tasks 2 and 3 was often not known until at the very end.
For example, in Task 2, participants who used a Local strategy in this task (P 2 , P 5 , P 8 , P 12 , P 15 , P 18 , P 19 ) began by deleting the method that contained the implementation, which leads to compiler errors in the caller.In order to "fix" the compiler error, one needs to either obtain the previously deleted code or reimplement the method from scratch.Only three of the seven participants obtained the previous implementation (through git, undo, or comments), whereas the remaining four reimplemented the method from scratch.All of these four spent significant time on resolving bugs or errors that appeared in their reimplementation.In comparison, only one other participant with a non-Local strategy (P 16 ) had a similar problem.
Participants who relied on the Local strategy were able to guide their navigation and changes by utilizing compiler errors and warnings.Yet, when they encountered errors that they did not understand, they had little choice but to "backtrack" and redo the task, hoping that their guidance would work better the next time.For instance, P 2 had to restart Task 3 after encountering a compiler error that they did not understand.This error originated from a binding change of an overloaded method that they had not noticed.They said "Just confused by this error.Wrong first argument type.That's weird.(…) So I don't understand the error."Since the participant did not understand the error or how to solve it, their solution was to restart and redo the previous steps in another order.They did not run into the same error again.On the same task, P 6 decided where to work based on how many compiler errors arose and described that they would monitor the number of red lines that a change induced and if above a certain threshold, they would undo that change and look for any other place to work.

| Structure
Participants who used a Structure strategy were orienting their workflows around source code structure, such as considering references or callers to an element.When starting a task, they typically explored code from one or more starting points using structural queries before altering the program.The understanding a participant gained about the code structure through their upfront explorations was used to formulate plans for their changes according to the impact it would have on other places in the code.
In Task 2, four participants (P 4 , P 6 , P 7 , and P 14 ) used this strategy.All of them formed plans to inline the method that was to be removed to its caller and executed this plan either manually or using tools.For example, P 7 stated in his upfront plan that "Any invocations would need to be dealt with.So what I'd like to do, is look in StringUtils, look where these are used (..) and then depending on the implementation of the anynot I may just be able to inline them."When they changed code, they intentionally propagate changes across code structures by means of navigational tools or refactoring tools.Where other participants might organize their changes so that they can act on local information (Local) or around the execution of tests (Execute), these participants made efforts at organizing changes along structural edges even when it required more navigation or "leaving behind" problems in the code.
Both users of the Local and Structure strategy delayed test changes to the end of Tasks 2 and 3.This may not be intentional-participants did not express such an intent-but may be simply due to the compiler errors they needed to navigate during the tasks.This is in stark contrast to participants who used the Execute strategy who relied heavily on the tests' behavior throughout the entire task.

| Execute
Participants using an Execute strategy used a workflow that was oriented towards executing the tests and, thus, keeping the code compiling so that the tests could be run throughout the task.At the beginning of tasks, they explored the codebase using structural navigational tools, similarly to someone using Structure strategy, but for the purpose of locating the relevant tests and exploring the call graph.
These participants searched for a start location that allowed them to isolate functional changes from syntactic changes.For example, for Task 3, many of these participants began by locating the program element responsible for the behavior of interest and, rather than removing it, made a controlled change to the functionality while preserving program structure.The controlled change allowed them to run the test suite and locate the tests impacted by the purely functional change.In this way, the participants could safely redefine the test suite to the target state of the software change before making the structural changes to the code.For example, P 11 described a strategy for Task 3 as follows: One strategy would be to change the internal definition of the methods first so that it always executes as if it was false.That will reveal the test cases that I need to look at more closely.
In contrast, participations using Local or Structure strategies typically relegated interacting with the test code until the very end.In order to keep the tests running, these participants traversed the call graph to locate "leaf nodes," nodes that do not call other methods.This helps them limit the impact of their code changes and test each change before moving on the next.For example, P 18 used this strategy on Task 3 and noted, If I change it in a method where it is being passed down, it is going to lead to a lot of cascading effects, that are going to have to change other methods […].Whereas if I do it just here, I am going to only focus on this method and worry about everything else, like up top, later.

| Summary and relationship to earlier work
We described three strategies that participants used to approach tasks: Local, Structure, and Execute.In considering how developers approach a change task, these findings are similar to those who have studied general, non-refactoring specific change tasks.Several of these earlier works have reported the use of structural investigations as a strategy for pursuing a task (e.g., Ko et al. 46 ).Fewer of these studies have described a focus on the execution of the system as a strategy; the closest we are aware of is Maleej et al.'s report on the use of a debugging strategy by industrial developers. 47We are not aware of descriptions of the use of a strategy similar to Local for solving tasks; however, developers have reported avoiding refactoring tools in favor of compiler errors. 6,13,16This study is the first to our knowledge that observes these different strategies on the same tasks.

| RQ2: How often do developers use automated support for refactoring versus proceeding manually?
We investigated how often participants used refactoring tools to solve tasks.We analyzed 51 task workflows, one for each participant working on each task.Recall that we consider a participant to use a refactoring tool if they apply it and continue working with its output.
Of the 51 workflows, 17 workflows (33%) contained one or more recorded uses of refactoring tools.In the remaining 34 workflows (67%), participants solved the tasks without using refactoring tools.As seen in Table 3, this was true both for more and less experienced participants.
Even in an environment where experienced developers are primed to use refactoring tools, and the tasks are amenable to use, participants did not use refactoring tools extensively.
Each task in our study included steps amenable to consideration as refactorings (Section 3.3), which means that when a participant does not use a refactoring tool on a task, it is an occurrence of tool disuse.Table 3 shows how the refactoring tool use and disuse are distributed across Overview of all the refactoring tools that participants used to solve tasks and the experience of each participant  participants and tasks.We can see that all tasks had three or more participants that used refactoring tools, showing that each task was indeed amenable to the use of refactoring tools.Furthermore, we observe that different participants used refactoring tools on different tasks, indicating that participants were not just invoking tools at every opportunity.
We also investigated whether tool use occurred more often in workflows that followed certain strategies.Because participants sometimes change strategies, we consider, for each participant's workflow on each task, which strategy they employed and whether they used refactoring tools or not. Figure 4 shows how tool use and disuse occurred in workflows from each strategy.We see that the workflows in which a participant employed a Structure strategy more often include refactoring tool use.
We also considered whether the participants who did not use tools had created solutions that were not amenable to automated refactoring tools, which would mean that these participants did not encounter opportunities to use the tools.We ran the RefactoringMiner 3 tool on all participant's code solutions and identified 385 refactorings in the code they produced.These were distributed across all participants' code solutions for all tasks.This means that every participant solved every task by changing the code in ways that were amenable to a refactoring tool being used.
By comparing these results to actual use, we observe that even some participants who did use tools missed several opportunities for use by solving parts of the task manually.We describe this further when answering RQ3.
We also collected all of the refactoring invocations that participants performed throughout all three tasks and found that in total, they invoked refactoring tools 100 times.Figure 5 shows how the invocations are distributed across participants.We see that even participants who did not use refactoring tools tried to invoke them, which invites the question of why these participants did not end up using tools.

| Summary and relationship to earlier work
We observed whether participants used automated support for refactoring or proceeded manually.In the 51 different cases that the three tasks posed across all 17 participants, we found that in 17 (33%) cases, the participants used a refactoring tool, whereas in the remaining 34 cases (67%), participants solved the task by changing code manually.These numbers are aligned with previous findings in which developers self-report to use tools less than half of the time. 5,13,32,48These studies investigate various granularity of changes: our cases are modeled off of commits and are therefore most comparable with studies like Silva et al. 5 However, by analyzing the refactoring opportunities that were identified in participants' source code, as was done by Murphy-Hill et al, 6 we find that even participants who use tools on a task missed a number of opportunities for tool use by solving some of it manually.
The number of times a participant employed each strategy in their workflow and used or did not use a tool in the same workflow.
Each participant solved three task, for a total of 51 workflows

| RQ3: How do developers experience refactoring tools?
We investigated how participants experienced the refactoring tools that they interacted with during the study.Previously, we reported cases in which a participant used tools; here, we also consider cases in which the tool was not used, in particular where participants had experiences that led them to avoid the tool or not retain its effect on the code.
The participants who used a refactoring tool had to go through the following process: 1 they must be be aware of the tool (e.g., by recalling a tool that can help or recognizing that a tool might be available even if they do not know the name of the tool), 2. they must try to use the tool (e.g., by locating and invoking it), 3. they must successfully apply the tool to the code (e.g., avoiding errors and not clicking "cancel"), and 4. they must continue working with the tool's output (i.e., not reverting the application).
Many of the participants who did not use tools also went through some of this process but had experiences in various stages of this process that ultimately led them to not use tools.
For each task, we consider a participant aware of a refactoring tool if they mention refactoring tools that can help them during the task or subsequent interview segment or if they try to invoke one.This means that a participant may be aware of a tool that may help them in one task while not in another task.We consider that a participant tried a tool if they invoke it, regardless of whether the tool produces errors, they cancel the invocation, revert what the tool did, or continue working with the resulting code.Only in the latter case do we consider the tool used.
Figure 6 shows, for each participant and each task, whether the participant indicated awareness of refactoring tools, whether they tried invoking a refactoring tool, and whether they used a refactoring tool.The figure shows that most participants showed awareness of refactoring tools during the study, but far fewer ended up using one.In fact, in all tasks, there occurred a drop-off from awareness of refactoring tools to the actual use of the tools.For Task 1, 12 (71%) of participants showed awareness of a refactoring tool, but only six (35%) of participants used such a tool.
For Task 2, six (35%) of participants showed awareness, and three (17%) exhibited use, and for Task 3, 14 (82%) showed awareness, and eight (47%) made use of a refactoring tool.Only three participants (17%) made use of refactoring tools in two tasks (P 7 , P 14 , and P 19 ) despite all but one participant indicating awareness of refactoring tools.
In total, out of the 51 opportunities for tool use that the 17 participants had across all three tasks, participants expressed awareness of tools in 32 (63%) cases, and out of these 32 cases, participants tried a refactoring tool in 23 cases (72%).Out of the 23 cases in which participants tried The number of refactoring tool invocations that each participants performed across all tasks a refactoring tool, they ended up using a tool in 17 (74%) cases.We see that for every step, there is a drop-off of around 30 % resulting in tools being used in only 33% of cases.To understand the drop-off from awareness to attempted use and from attempted use to use, we inspect the transcripts, including the answers to interview questions, to learn why participants whose to not continue the process of tool use.Similarly, we investigate, for participants who used tools, how they experienced them.In each of these steps, we observed a number of factors that impacted how participants experienced the tools.

| Awareness
Participants were mostly aware of refactoring tools.In Task 1, 12 participants (71%) showed awareness of tools, all of whom indicated that they could use move or extract.In Task 2, six participants (35%) showed awareness of tools, five of whom indicated that they could use inlinemethod, and one participant (P 6 ) mentioned safe delete.In Task 3, 14 participants (82%) indicated awareness and referred to change-signature, inline, or safe-delete.All three tasks presented the 17 developers with 51 cases in which they had an opportunity to use a refactoring tool, and participants were aware of tools in 32 (63%) cases.From these 32 cases, participants moved on to try a tool in 23 cases (72%), while in the remaining nine cases, they did not.We observed multiple reasons why the remaining participants decided not to even try the tool to help with a task.The main reason given by participants was that they had proceeded too far in the change before realizing a tool could help them.Another recurring reason was that the tool might negatively impact later steps in their workflow.
In Task 1, P 13 expressed that they realized after the fact that they could have used a tool, whereas P 16 did look for a way to invoke move from the "structure" view, which shows a truncated view of the class by listing class member declarations such as methods and fields.They failed to do so and said, "I spent a couple of minutes and didn't get it working.I decided it is faster to do manually." In Task 2, P 16 made the manual change before realizing that they could use inline, and P 6 did not invoke safe-delete because they did not "trust" the tool.When asked to explain, this participant expressed that they did not trust that the tool would present them with information about the code that would teach them as much as they would learn by manually changing it: "I prefer to see these things happening step by step because I know what's there, I can double-check any of the files that are to be changed and see if there are any other changes that need to be done in them." In Task 3, five participants (P 3 , P 6 , P 13 , P 15 , P 18 ) did not try refactoring tools despite being aware of them.P 3 mentioned inline but said they would look for an automated approach only if they wasted too much time doing it by hand and if they trusted their tests.P 13 , P 15 , and P 18 F I G U R E 6 Each task presented each participant with the opportunity to use refactoring tools, for a total of 51 opportunities.In each of these 51 cases, we investigate whether the participant was aware of refactoring tools, tried to use tools, and used a refactoring tool once or more named change-signature during the interview but said that they were not sure how much it would have helped them.P 13 and P 18 also did not realize upfront that they could use this tool, whereas P 15 feared that it either would not help them or impact code that they did not yet want to change: Change signature might have helped you but not really.It would tell me that the method with the signature which is without the boolean already exist.So not really.I also don't want to remove all the callers of the method because I want to look at them and see whether or not it makes sense for them to be removed or whether they should stay (P 15 ).
A similar sentiment was expressed by P 6 , who mentioned safe-delete but said they did not trust the tool and explained: "I would have the problem of doing a refactoring and not knowing if the behavior of the test should be the same or the behavior of the code."They added, "I don't want to rely on a tool that will not help me become familiar with the source code," explaining that they gained an understanding of the code by making changes manually.If they were to make changes in the future, they would be faster due to this experience, while using a tool would deprive them of gathering similar knowledge.
In summary, we found that participants who were aware of tools without trying them experienced the tool as unable to present them with satisfactory information about the code and unable to distinguish test code from functional code.Both of these problems were noted by participants as making later steps in their workflow more difficult, both during the same task and if they were to work with the same codebase again.
These experiences have not, to our knowledge, been previously reported.We also observed experiences that support previously reported factors that motivate disuse, like realizing too late that a refactoring tool can be used 6 and lack of trust. 5,6,13,34

| Tries
In the 23 cases where participants tried refactoring tools, they made one or more invocations of the tools that are listed in Table 4.In 17 of the cases (74%), this resulted in use, whereas in the remaining six, participants did not end up using any refactoring tool on that task.We had expected that participants who experienced so-called barriers to use 17 -warnings, problems, or errors-might avoid using tools; however, although participants did encounter such barriers in these six cases, so did the participants who went on to use tools.In fact, in every task, most of the sparticipants who tried a refactoring tool encountered some variation of tool warnings, problems, errors, or the introduction compiler errors, and while these experiences led participants to abandon the tool in Tasks 1 and 2, participants repeatedly relied on tools in Task 3, despite encountering barriers to use.In some cases, these were even the same participants.
In Task 1, P 2 and P 18 had experiences when invoking the tools that led them to avoid using them.Both participants tried the move refactoring but had problems invoking the right version of move due to incorrectly assuming that the tool would understand their code selection as arguments to the refactoring.Unlike previously reported problems with selecting statements correctly, 17 these participants did make a correct selection of the methods they wanted to move in the editor.Then they right-clicked somewhere on the selected text and invoked move in the refactoring menu, thereby indicating that they wanted to move all of the selected methods.However, the tool failed to recognize the selection altogether and instead interpreted the invoking to apply to the surrounding class, thereby launching move-class instead of move-method.Both participants repeated this interaction several times before giving up and indicating that they might as well move the methods manually, referring to the simplicity of the task.
In Task 2, all participants who tried a refactoring tool (P 7 , P 9 , P 10 , P 14 ) initially invoked inline-method.These participants experienced a problem that has previously been reported for other tools: misinterpreting error messages. 17When invoking inline on the method to be inlined, the tool produced an error due to this method's code structure.All four participants misinterpreted the error as relating to the structure of the caller.As a result, all four participants performed extract-local-variable in the caller in an attempt to bypass the error, with no success.By Note: The refactoring, on the left, was invoked on the code elements seen in the middle.For example, both inline-constant and inline-method were invoked.The problem with test code highlighted the participants' dependence on diff tools to understand the tool's impact: out of the nine participants who tried a refactoring tool in Task 3, seven participants (P 4 , P 5 , P 7 , P 9 , P 10 , P 16 , P 19 ) resorted to git to disambiguate callers both in the source code file and the test file after applying the refactorings.These participants inspected review views and problems views before applying the tool, yet found that afterwards, they needed to know what the code used to do or how it had changed.Several participants expressed dislike over this workflow.P 9 and P 7 made the following comments: Some of these tests should be failing because they should be testing behavior that is no longer supported.So I should go through and look at the tests.Maybe I should have done this before deleting the parameter because now it's going to be hard to look at the tests and know what they were doing before.Oops (P 9 ).
By invoking the refactoring tool it changed code I wasn't looking at.So a good way to see those changes is through the git integration.If this wasn't in a git repository, I'd be more screwed (P 7 ).
In order to limit problems related to code impact, some participants carefully orchestrated the order in which they applied refactoring tools to the call graph, starting on what they referred to as "leaf nodes" or nodes with few callers.In this way, they controlled the impact of the tool because they made sure that there were only ever a few test callers to propagate changes to.P 16 employed this workflow throughout the entire class, whereas others like P 4 , P 7 , P 9 , and P 10 realized only partway through.In this case, the order in which the tools were applied impacted the cost associated with using them.
Participants experienced code layout as relevant in Tasks 1 and 3.In the first task, P 14 and P 19 used the moverefactoring to move one method and indicated that using the tool was not worth it.They continued the task by manually moving code.P 14 commented that the layout of the code (i.e., the methods being declared after one another) made the manual approach easier: "It's nice that these are all together.Then I can just wholesale them over.If they were scattered throughout then maybe it would be worth the extra steps."Similarly, in Task 3, many of the participants (e.g., P 8 , P 11 , P 12 , P 14 ), who manually inlined methods to their wrapper method, observed the convenience of the wrapper being located just above each method declaration, allowing them to make only a single text selection.Even P 9 and P 14 , who initially used the inline-method on this task, eventually reverted to this workflow.
Among the participants who successfully used refactoring tools throughout the entirety of Task 3, we observed an interesting workflow.Two participants used "problem" views or "preview" views to learn about the code before changing it manually by acting on the information these views presented.In Task 3, P 5 used safe-delete and invoked it twice for each of the eight methods that had to be changed: the first time, they used the preview view to navigate to code that would be impacted by the change and updating some of these elements manually before invoking it again and applying the refactoring.P 8 used a similar workflow, where they invoked change-signature and inspected the "problem" view before canceling the refactoring and locating the problematic elements in the code, changing them, and reapplying the refactoring.This also provides empirical evidence for the claim posed by Fleming et al. that a developer who is refactoring needs to forage for cues for how the refactoring should be performed and that refactoring tools help present such cues. 49 summary, we observed that participants' success with tools was impacted by experiences with the tool's impact on test code, difficulties understanding the tool's impact on code, and code layout.The participants who successfully used tools were careful with the order in which they applied refactorings or used the information presented by the tools to organize their manual changes.To our knowledge, there is no earlier work that brings up this distinction of test code from functional code in refactoring tools, nor reports of developer reliance on diff tools like git to learn about a tool's impact after it is applied.Some earlier works have indicated that developers avoid using refactoring tools due to fear of merge conflicts or code ownership. 32The following observations are, to our knowledge, novel: that participants organize refactoring applications along the call graph, use information from the tool to organize manual changes, and that code layout impacts the choice to use or not use tools are.

| Experiences and strategies
We looked into whether participant strategies impacted their experience of tools.Figure 7 shows the drop-off trend relative to their strategies.
We see that a drop-off happens more often when the participant uses a Local strategy and less often when they use a Structure strategy.This shows that participants who used a Structure approach are not the only ones who are aware of tools, but awareness led to use more often in this group.We speculate that participants who employed the other strategies might more often encounter experiences with the tools that lead them to avoid using them.
the code change that was supported by the tool.The gulf between tools and workflows may be larger than has been previously considered for refactoring tools.
Future studies should consider developer approaches to tasks in more detail.Perhaps it is possible to guide a developer to a strategy and workflow that enables the use of refactoring tools through suggestions.In other cases, such as where code layout enables easier manual changes, developers may benefit from help with a cost-benefit analysis to evaluate whether using a refactoring tool is worthwhile.Alternatively, there may be other tools more useful to developers, such as ones that help assess the impact, risk, or consequence of changes to the code.Such a tool may support manual changes a developer is performing that may or may not be similar to refactoring operations.
participant used a tool if they continued working with the output of the tool.ExCon = Extract Constant; InCon = Inline Constant; RC = Rename Class; MM = Move Method; ESc = Extract Superclass; ELVar = Extract Local Variable; IM = Inline Method; ChS = Change Signature.
Not completely.I think if I'd played a little more I could have excluded part of the refactoring.I'd like to say I want to go ahead with the refactoring in this area and not in this area, because if I could exclude the tests I could better see the impact of what tests are actually broken.But you want to have the proper changes done inside the code (P 19 ).