Following up internet-delivered cognitive behaviour therapy (CBT): A longitudinal qualitative investigation of clients' usage of CBT skills

Background: While the acquisition and application of Cognitive Behaviour Therapy (CBT) skills is a core component and likely mechanism of effect maintenance in all CBT-based treatments, the extent of post-therapeutic CBT skills usage among internet-delivered CBT (iCBT) clients remains under-researched. Method: Nested within a pragmatic randomized controlled trial, 241 participants received an 8-week supported iCBT intervention for anxiety and/or depression and answered open-ended questions about their use and experience of CBT skills at 3-, 6-, 9-, and 12-month follow-up. Recurrent, cross-sectional qualitative analysis following the descriptive and interpretive approach was used to create a taxonomy, through which all qualitative data was coded. Results: In total, 479 qualitative responses across 181 participants were analysed. Participants reported using a wide range of CBT skills and associated helpful and hindering experiences and impacts. The reasons for discontinued CBT skills usage were diverse, ranging from rare adverse effects to healthy adaptation. Conclusion: The study shows how clients receiving iCBT in routine care learn CBT skills during treatment and utilize them in productive ways post-treatment. Findings coincide with similar research in face-to-face CBT and may inform future research to drive innovation and iCBT intervention development.


| INTRODUCTION
Cognitive Behaviour Therapy (CBT) today may be considered an umbrella term for a large variety of evidence-based psychological interventions, principally uniting behavioural and cognitive approaches, and in recent years also acceptance and mindfulnessbased approaches under its wings. The common goal across CBTs is a behavioural adaptation, which is achieved through the implementation of specific evidence-based techniques and strategies (sometimes referred to as CBT skills) designed to drive universal change principles like context engagement and attention and cognitive changes (Hayes & Hofmann, 2018;Mennin et al., 2013). CBTs of differing complexity and length have been developed, and research confirms their effectiveness for various psychological issues across a broad range of symptomology (e.g., Forman et al., 2007;D. A. Richards et al., 2016;Wiles et al., 2013).
While sharing a common goal and hypothesizing similar mechanisms of change (Mennin et al., 2013), the particular evidence-based techniques and strategies implemented in CBT now stretch from behavioural techniques like activity scheduling and exposure (Foa & Kozak, 1986;Lewinsohn & Graf, 1973) and cognitive techniques like cognitive restructuring and distancing (Beck & Clark, 1997;Zettle & Hayes, 1988) to also include problem solving, distress tolerance and other practical coping skills (D'Zurilla & Nezu, 2007;Linehan, 2013) as well as relaxation, mindfulness and acceptance-focused techniques (Manzoni et al., 2008;Masedo & Rosa Esteve, 2007). The degree to which these and other CBT techniques or skills are evident in a given treatment may differ depending on its focus, mode of delivery and length; however, irrespective of the specific techniques introduced by the treatment, the independent application of these techniques by clients during treatment as well as after treatment has ended represents a core component of all CBT-based interventions (Hayes & Hofmann, 2018;Hundt et al., 2013).
While much remains unknown about the mechanisms of change that account for the positive effects of CBT, it has been argued that in fact the acquisition, use and practice of evidence-based CBT techniques and strategies (referred to as CBT skills from here on) plays a mediating role in how initial change occurs as well as in how this change is maintained into follow-up (Hollon et al., 2006;Strunk et al., 2013). Available research has predominately investigated the relationship between CBT skills usage and symptom alleviation during treatment, finding various forms of CBT skills usage to be related to depression and anxiety symptom outcomes of varying levels of severity, in adults and adolescents, receiving individual CBT, group-based CBT or internet-delivered CBT (Hawley et al., 2017;Terides et al., 2018;Webb et al., 2016Webb et al., , 2019. Little is known about how CBT effects are maintained after active treatment ends. Here, some have suggested that the change that occurs during treatment (e.g., reductions in cognitive reactivity and internalizations of therapists; Knox, 2003;Segal et al., 1999) permanently affects clinical outcomes, while others highlight the role of compensatory skills (e.g., CBT skills) to address fluctuations in clinical outcomes (Hollon et al., 2006). CBT skill acquisition and performance during active treatment, as well as ongoing CBT skills practice after it has ended, have been shown to relate to lower relapse rates and better depression outcomes during follow-up (Michalak et al., 2008;Morgan et al., 2014;Powers et al., 2008;Strunk et al., 2013). Similarly, there is some preliminary research suggesting the same might be true for anxiety outcomes (Kim et al., 2008). Furthermore, a number of qualitative studies complement these findings, showing that many CBT completers perceive CBT skills as helpful and continue to apply them after therapy has ended. More importantly though, these studies also highlight perceived barriers to CBT skills usage as well as the fact that CBT skills usage may occur in idiosyncratic and implicit ways as individuals adapt, change and integrate skills into their daily lives (e.g., French et al., 2017;Glasman et al., 2004).
Taken together, the available research is only beginning to paint a full picture of the curative and preventative, clinical and clientdefined effects CBT skills may have. Especially, little is known about the degree to which the ever-growing number of clients receiving internet-delivered CBT (iCBT) around the world acquire CBT skills during treatment and practice them after their mostly short-term, minimally supported active treatments have ended (Folker et al., 2018;Titov et al., 2018). While to our knowledge two studies have shown CBT skills usage during iCBT to be related to post-treatment clinical outcomes to date (Forand et al., 2018;Terides et al., 2018), only a limited number of qualitative studies have touched on posttherapeutic CBT skills usage after iCBT (Berg et al., 2020;Halmetoja et al., 2014). Halmetoja et al. (2014) qualitatively followed-up a cohort of participants who had received iCBT for Social Anxiety Disorder 4 years after treatment had ended and discovered that at least some had continued working with the material covered during their iCBT programme. Some participants had, for example, set themselves exercises based on their treatment or searched for new related content-in other words had used CBT skills. Moreover, the authors found that participants who adopted this active approach in caring for their own mental health also reported better anxiety outcomes throughout follow-up. Similarly, Berg et al. (2020) described distinct differences between those who actively applied therapeutic learning posttreatment and those who were more passive in their approach based on their qualitative interviews with adolescent iCBT completers

Key Practitioner Message
• Routine care clients learn CBT skills during iCBT and utilize them afterwards.
• A wide range of cognitive and behavioural skills were used after treatment had ended.
• Skills usage resulted in helpful impacts such as reduced symptoms and coping.
• Reasons for ceased skills usage spanned from adverse effects to healthy adaptation. 6-month post-treatment. Overall, the authors concluded that their participants had gained important knowledge during iCBT but that the degree to which this knowledge was concrete, explicit and actively implemented varied-in a way positioning therapeutic leaning along a continuum that ranges from general knowledge gain to productive use of CBT skills.
Previous quantitative research has highlighted iCBT's potential for long-term clinical effects (Buntrock et al., 2016;Fogliati et al., 2016;Ruwaard et al., 2010); however, regarding posttherapeutic CBT skills usage as a possible mechanism of effect maintenance, both of the aforementioned studies present with a number of limitations. Both utilized rather small and selective samples, in that only 10 to 12 participants were recruited from studies, which themselves had recruited participants via social media and other advertisements rather than from routine care services. In addition, while Berg et al. (2020) followed-up participants soon after iCBT had ended, in Halmetoja et al.'s (2014) case, there were many years between treatment and follow-up, with neither study capturing how CBT skills usage evolves over time. Therefore, exploring CBT skills usage across larger and routinely treated samples will shed further light into the processes and experiences associated with the longitudinal use of CBT skills. This is important so as to establish iCBT's ability to reliably affect mechanisms underpinning psychological symptomatology and thereby produce long-term change (Mogoaşe et al., 2017).

| Aims and research questions
The aim of this study was to explore and map out the use of CBT skills following completion of iCBT treatment for anxiety and depression, delivered as part of routine mental health care. In this vein, the research questions of interest in this study were: Do clients who received iCBT report using CBT skills at follow-up? Which CBT skills do they use and which do they stop using? What are the experiences associated with CBT skills usage after iCBT and what are the reasons why some may stop using CBT skills? 2 | METHOD 2.1 | Design A recurrent, cross-sectional qualitative study was conducted, which was nested within the follow-up period of a pragmatic randomizedcontrolled trial evaluating the effectiveness and cost-effectiveness of iCBT for anxiety and depression (D. Richards et al., 2020). Results from the parent trial showed that those in the iCBT group experienced significantly larger reductions in depression and anxiety symptoms from baseline to 8 weeks than those in the waitlist control group (d = 0.55-0.63), with significant further improvements in symptoms observed in the iCBT group from 8 weeks to 12-month follow-up (for details, see D. Richards et al., 2020).
For the current study, a longitudinal qualitative design was chosen to allow for the exploration of the idiosyncratic use and experience of using CBT skills across four follow-up time points, ranging from 3-month follow-up (shortly after the end of treatment) to 12-month follow-up. Post-therapeutic skills usage represents an under-researched and dynamic concept, likely occurring along a continuum between formal and informal, explicit and implicit (French et al., 2017;Glasman et al., 2004), and as such would have only been partially addressed by quantitative measures. A recurrent, crosssectional frame was selected to suit the study's primary interest of exploring CBT skills usage at follow-up on the level of the entire sample rather than focusing in on intra-individual changes in CBT skills usage only (Grossoehme & Lipstein, 2016).

| Setting
The current study was conducted within routine care delivered through the Improving Access to Psychological Therapies (IAPT) service in the United Kingdom. This service aims to increase access to evidence-based treatments within a stepped care model of mental healthcare by tailoring the intensity of the intervention offered to a client's symptomology based on an assessment of their needs. Clients presenting with mild to moderate symptoms of depression and anxiety are offered Step 2 low-intensity interventions such as guided selfhelp, internet-delivered CBT or group well-being psychoeducation.
These interventions are usually delivered by Psychological Wellbeing Practitioners, paraprofessionals with graduate degrees in Psychology specially trained in low-intensity CBT interventions, who provide support and monitor progress throughout treatment. Referral occurs through general practitioners, allied health services or self-referral.
Ethical approval for the larger trial including this study was obtained from the National Health Service Health Research Authority (REC reference: 17/NW/0311).

| Participants
Clients were eligible for participation if they were aged over 18 and a Step 2 service user, so had been deemed suitable for low-intensity interventions in terms of their clinical presentation, determined by a score of ≥9 on the PHQ-9 and/or ≥8 on the GAD-7 (see parent trial protocol for details, D. Richards, Duffy, et al., 2018). They were further telephone-screened by clinicians for suitability for iCBT, defined as willingness to engage and access to the internet. They then completed the Mini International Neuropsychiatric Interview 7.0.2 (M.I.N. I.7.0.2) diagnostic interview to establish a primary diagnosis of depression or anxiety. Exclusion criteria were suicidal or self-harm risk (score >2 on the PHQ-9 Item 9 and as expressed during clinical interview), substance misuse, a psychotic illness diagnosis and currently receiving another form of psychological treatment. Clients invited to the trial provided informed consent to participate by way of an electronic signature. See Table A1 for sample characteristics.

| Treatment
A total of 361 participants were allocated to the depression or anxiety arm and then individually randomized, stratified by primary diagnosis, to iCBT intervention (n = 241) or waiting-list control group (n = 120) in a 2:1 ratio. The iCBT programmes used were SilverCloud Health's 'Space from Depression', 'Space from Anxiety' and 'Space from Depression and Anxiety', principally composed of CBT content customized for depression, anxiety, and comorbid presentations. The seven-core modules are delivered in an interactive manner (including journal entries and use of tools) and non-linear pattern (i.e., the user can decide the order of modules they want to complete) on a Web 2.0 platform. Within the modules, participants receive CBT-focused psychoeducational content to develop relevant skills such as understanding emotions (self-monitoring and cognitive distancing), flexible thinking (cognitive reframing/restructuring), problem solving, facing fears (graded exposure), mindfulness and identification and scheduling of enjoyable and motivational activities (behavioural activation).
Depending on the needs of the user, there are additional modules the supporter can unlock which feature content for specific difficulties (e.g., worry management, sleep hygiene and behavioural experiments).
For outlines of the programmes, as well as the unlockable modules, see Table A2. Participants are encouraged to practice these skills through the use of interactive tools such as mood monitoring, worksheets and audio meditation exercises. Supporters monitored participants' progress through the intervention and were advised to provide six online reviews to each participant over the 8-week intervention, spending approximately 15 min per review per user. In practice, this varied somewhat with some participants receiving more than six reviews or being contacted via telephone. Reviews aimed at promoting engagement with the intervention by providing encouragement and feedback on work completed week by week. The effectiveness of these interventions has been demonstrated in the treatment of depression and anxiety (D. Richards et al., 2020), and they adhere to the National Institute for Health and Care Excellence guidelines (National Institute for Health and Care Excellence, 2009, 2016).

| Open-ended questions
In order to explore participants' idiosyncratic use of CBT skills, including their experiences surrounding these skills and reasons why they may have stopped using certain skills, participants were asked to answer two open-ended questions in writing and online at 3-, 6-, 9and 12-month follow-ups.

| Clinical measures
Patient Health Questionnaire-9 Item (PHQ-9) This is a well-validated, self-report instrument used widely for the screening and diagnosis of depressive symptoms experienced over the past 2 weeks (Kroenke et al., 2001). The patient answers nine questions concerning symptom presence based on criteria for a major depressive disorder diagnosis across a 4-point Likert scale (0 = 'not at all' and 3 = 'nearly every day'). Scores range from 0 to 27, with larger scores indicating greater severity and frequency of symptoms.
Generalized Anxiety Disorder-7 Item (GAD-7) The GAD-7 is a self-administered questionnaire assessing the frequency of anxiety symptoms over the past 2 weeks (Spitzer et al., 2006). Patients rate the frequency of their anxiety symptoms by answering seven items using a scale of 0 = 'not at all' to 3 = 'nearly every day'. Greater severity or frequency of symptoms is reflected in higher scores. For further details on measures, materials and procedures, please refer to the parent study (D. Richards, Duffy, et al., 2018;Richards et al., 2020).

| Data analysis
Qualitative analysis resulting in the development of a CBT skills usage taxonomy, as well as the subsequent coding of all available qualitative data in line with that taxonomy, was informed by Elliott and Timulak's (2005) descriptive-interpretative approach, which has previously successfully been used with qualitative data of a similar nature (Jardine et al., 2020;D. Richards, Dowling, et al., 2018). Qualitative responses were analysed across the three open-ended questions. An initial taxonomy was developed by the first author of this study on the basis of qualitative data collected from 35 participants at 3-month follow-up through the following steps. Following data screening and cleaning, all data were broken down into individual meaning units.
Next, an organizing structure for the data, made up of the domains of investigation covered by the open-ended questions, was developed.
These domains were (1) CBT skills used across follow-up, (2) CBT skills whose use was discontinued during follow-up, (3)  Following this, approximately 60 randomly selected responses across all time points were analysed, and the taxonomy was refined until no new meanings emerged and saturation was deemed likely.
At this point, the analysis and taxonomy were audited a second time by the second author and a detailed guide on its use was compiled.
Two independent coders (the sixth and seventh author) were then trained to code all qualitative data across all time points in line with the taxonomy. During coder training, the taxonomy guide was reviewed, 21 responses were co-rated and discrepancies and subtle differences in the understanding of different codes were discussed and resolved.
Interrater-reliability assessments between the first author and the two independent coders were carried out before data coding commenced and repeated at the half-way point to assess for potential coder drift. Percentage agreement, Fleiss' kappa and its bootstrapped confidence interval were calculated in line with and utilizing R code provided by Zapf et al. (2016). The following cut offs for Fleiss' kappa and its confidence interval were implemented: 0.0-0.4 poor to fair agreement, 0.41-0.60 moderate agreement, 0.61-0.8 substantial agreement and 0.81-1 almost perfect agreement (Landis & Koch, 1977). Once all data were coded, specific coding issues coders flagged were solved through consensus between the first author and the two independent coders. Coders were encouraged to report specific responses, whose meaning may not have been sufficiently addressed in the taxonomy yet to the first author. The first author then carried out validity checks across all categories, addressed inconsistencies in coding and further refined the taxonomy in consultation with the second author. At this point, one further sub-category was added and one sub-category was broken further down. Given that all taxonomy refinements occurred at the sub-category level, the likelihood that all main categories were saturated was high.
Data analysis was led by the first author, an early career researcher and counselling psychologist trained in and practicing a number of psychotherapeutic approaches including CBT. The two independent coders were in the process of completing a master's degree in applied psychology and had no allegiance to any psychological approach. Auditing of the analysis was completed by the second author, a senior researcher with extensive experience in qualitative research, who has training in CBT but whose psychotherapeutic orientation lies outside of CBT. Reflexivity was an ongoing process facilitated by frequent discussions between research team members, bearing in mind the limitations of how data was collected (e.g., being collected in writing meant participants could not be asked to clarify their answers) and remaining sensitive to the various possible interpretations of participants' responses. Any differences along demographic, outcome and engagement variables between participants who responded and those who did not respond to open-ended questions were explored using Chi-squared, Mann-Whitney and t tests in SPSS version 26.

| RESULTS
Of the 241 participants randomized to the iCBT treatment group,  Table A1  ). Text answer length ranged from 2 to 1243 characters, with an average length of 104 characters. Interrater-reliability assessments suggested substantial agreement before data coding commenced (71.2% among three raters; Fleiss' K 0.78, 95% CI 0.73, 0.83) and at the half-way point of coding (72.5% among three raters; Fleiss' K 0.78, 95% CI 0.72, 0.84). Among behavioural skills, mindfulness and meditation (e.g., 'I also like using the meditation techniques to bring things back to the breath'), relaxation and breathing exercises (e.g., 'I listened to the relaxation videos') and activation and engaging in activities to improve one's mental health (e.g., 'I try to plan things into my schedule I know will help also, such as seeing people, making phone calls/facetimes, and exercising') were cited most often, with 50 to 57 out of 181 (28%-31%) reporting having used at least one of these skills throughout follow-up (see Figure 1 for details). The number of people reporting they had not used any CBT skills recently increased from 5% at 3 months (6/120) to just under 14% at 12-month follow-up (17/125).

| CBT skills used across follow-up
For full descriptions and quotes associated with each reported CBT skill see Table A3.
3.2 | CBT skills whose use was discontinued during follow-up  Table A3 for a full description of categories and associated quotes.

| Helpful and hindering experiences/impacts related to post-therapeutic CBT skills usage
Participants associated helpful as well as hindering experiences with using CBT skills after their active treatment had ended. Overall, categories of helpful experiences/impacts were more common than hindering experiences/impacts, with 73% (133/181) of participants who answered at least one open-ended question at one follow-up time point reporting a helpful experience/impact whereas only 28% (50/181) of those reporting a hindering experience (see Table 1).

| Helpful experiences/impacts
Qualitative analyses suggested eight different categories of helpful experiences or impacts in relation post-therapeutic CBT skill usage.
Thirty-six participants (20%) out of 181 reported experiencing fostered insight and flexibility (e.g., 'as it helps me realize that my thoughts are not set in stone and they can be changed.'). Ten participants (5%) reported increased self-compassion and self-kindness (e.g., 'Recognizing this has been useful in enabling me to recognize that I am doing everything that I can') and self-efficacy and self-confidence (e.g., 'It is helping to boost my […] confidence as I feel I am achieving something.'), respectively. Furthermore, feeling calm and relaxed (e.g., 'a great way to relax and de stress'), reduced symptoms and letting go of worries (e.g., 'has helped reduce my anxiety') and coping and problemsolving (e.g., 'This meant I got stuff done and actually didn't worry!') as a result of using CBT skills were reported by 23 (13%), 55 (30%) and 49 (27%) of the 181 participants, respectively. In addition, 45 out of 181 participants made nondescript positive statements about their CBT skills usage across the time points, which were recorded as nonspecific helpful experiences (e.g., 'I find it helps lots').
F I G U R E 1 Spontaneously named CBT skills used across follow-up. Note. This figure demonstrates the frequency of participants' reports of CBT skills used at each of the follow-up time points. As at 3-month follow-up 47 of the 120 participants who responded at this time point were still receiving support, figures at this time point may also reflect some in-treatment skills usage. (a) Techniques and strategies involving thought processes or relying predominantly on 'thinking about' one's self, experience and behaviour. (b) the 'worry tree' exercise, a worksheet involving yes/no questions aimed at distinguishing between worries that can be acted on and those that cannot. (c) the 'worry time' strategy involves participants noting down worries they recognize during the day but delaying engaging in this worry until a designated time each evening. (d) Behavioural techniques and strategies, techniques and strategies that require participants 'doing' something or aim at changing behaviour [Colour figure can be viewed at wileyonlinelibrary.com] Besides these helpful impacts of skills usage, 84 of 181 (46%) participants also described various ways in which they were proactively engaging in the continued use of CBT skills, relating to ongoing acquisition of skills, selection and adaptation of skills and their integration with daily living. Especially earlier on into follow-up a limited number of participants (13/181) reported their skills usage as 'work in progress' (see Table 1). These participants appeared to be still in the process of acquiring CBT skills (e.g., '… am getting to under-  Finally, a number of participants (30/181) described continuous use of skills, often suggesting that skills had been integrated into daily life. Continuous skills practice reaching from frequent use (on a concrete level) to increasingly automatic or implicit skill application (on an abstract level) was reported (e.g., '… after several months of trying this it has become more natural. My thoughts still start of negative but almost immediately and without thinking I try to put a more positive spin on it.').

| Hindering experiences/impacts
Participants reported four categories of experiences or impacts that hindered them in the application or execution of CBT skills. At times participants reported skills as insufficiently effective or not doing what they are supposed to do (13/181; e.g., 'I understand the premise and process behind everything on Silvercloud but it hasn't changed my thought process') or even that they experienced them problematic or upsetting in some way (4/181; e.g., 'I find it relatively discouraging and damaging to categorize these thoughts as unhelpful in whichever way as to me, the thoughts are my own and thus I feel that my own thoughts are the issue and that I myself am therefore being unhelpful.'). In addition, difficulties in the execution of skills, (25/181; e.g., 'It is hard using [challenging/changing thoughts] as I am often anxious or worrying and I am not even aware of it -worrying has become F I G U R E 2 CBT skills whose use was discontinued during follow-up. Note. This figure summarizes the frequency of participants' reports of CBT skill use they had ceased at each of the follow-up time points. As at 3-month follow-up 47 of the 120 participants who responded at this time point were still receiving support, figures at this time point may also reflect some in-treatment processes [Colour figure can be viewed at wileyonlinelibrary.com] my default. I find it a struggle to catch my thoughts …') and daily and situational barriers (24/181; e.g., 'with frequent 300-mile round trips to try to help mum and the on-going chaos of building works at home I have let this lapse (perhaps when I need it most).') were reported as hindering skills usage.

| Reasons for/experiences associated with discontinued CBT skills usage at follow-up
Participants reported six categories of reasons for why they had stopped using CBT skills during follow-up (see Table 2). Thirty-five of T A B L E 1 Helpful and hindering experiences/impacts related to post-therapeutic CBT skills usage Furthermore, 24 of 181 participants cited insufficient effectiveness of CBT skills in the context of ceasing their use, describing them as either not helping or not getting to the core of their problems (e.g., 'I'm less inclined to use as they don't really work for me'). There was also a small proportion of participants (14/181) who reported CBT skills to have worsened their symptoms, leading them to discontinue them eventually. These upsetting effects of CBT skills usage often related to increases in such symptoms as anxiety, worry or panic but increases in negative thoughts and self-criticism were also described (e.g., 'I stopped using breathing techniques as they seemed to induce panic').
At the same time, there was also a proportion of participants for who the discontinuation of skills usage appeared to be linked to positive mental health or adaptation. Thirty-two of 181 participants reported that in fact they were not using skills anymore because their symptoms had improved to the point that skills were not needed anymore (e.g., 'Have felt much better-and haven't needed to use [mindfulness] so much.'). Similarly, 12 participants stated they had discontinued a particular skill to use another skill they perceived superior or more effective (e.g., 'The relaxation and breathing exercises didn't do much for me-I prefer something more proactive in helping my thought process.'). Finally, 13 participants also described intentions or plans for resumed CBT skills practice (e.g., '… so I've decided to implement them again in order to lift my mood and wellbeing').

| DISCUSSION
The current study set out to explore and map out CBT skills usage following the completion of routinely delivered iCBT for anxiety and depression. Qualitative data on client experiences were collected longitudinally across four follow-up time points from a large sample, representing a major strength of this study. Analyses showed how a wide range of CBT skills are used by individuals after they have finished iCBT treatment. Moreover, they shed light into the experiences and impacts associated with this skills usage, which are often positive and adaptive but at times can be negative and counterproductive too.
Furthermore, while participants also reported having stopped using one or multiple CBT skills during follow-up, the reasons for this were diverse, ranging from adverse effects of skills usage to healthy adaptation.
These findings add to the iCBT literature by showing clientdefined outcomes beyond the clinical effectiveness reported on by the parent trial (D. Richards et al., 2020) and are testimony to the continued benefits CBT-based intervention can exert even after treatment has ended. As such, the findings extend upon previous research on clients' continuous use of CBT skills after face-to-face CBT (French et al., 2017;Glasman et al., 2004) to include minimally supported iCBT-further establishing iCBT's ability to produce change at a meaningful and sustainable level (Mogoaşe et al., 2017;Strunk et al., 2007). Interestingly, proactive and ongoing engagement in the use, selection and practice of CBT skills emerged as a helpful experience from the analysis. While this category may be understood as 'client involvement' in the context of previous research on helpful impacts in psychotherapy (Quick et al., 2018;Timulak, 2007), it could also be construed as client agency, which in the psychotherapy context is defined as 'expectations related to the active, purposeful use of psychotherapy to meet needs, solve problems, and make life changes' (Coleman & Neimeyer, 2015, p. 3). Extending this idea to after treatment has ended, it can be argued that by actively practicing and purposefully selecting CBT skills to meet mental health needs and solve problems (e.g., findings described under symptom-driven selection and combination of skills, adaptation and idiosyncratic use of skills and continued skills practice & integration with daily living), participants were in fact demonstrating agency.
Beyond being an outcome of CBT skills usage, acting with agency may have also allowed some clients to overcome hindering, counterproductive experiences such as difficulties in skill application or daily and situational barriers to enable them to select the right skill for their Of significance was also the findings that while many participants had stopped using CBT skills at follow-up for some, this was actually due to having improved to the point of not needing skills anymore.
Different theories exist that link various skill deficits (e.g., problem solving, social and emotion regulation skills) to the occurrence and maintenance of specific disorders as well as psychopathology generally (Hopko et al., 2001;Lukas et al., 2018;Nezu, 1987). In this vein, participants' reports of not needing skills anymore might in fact represent a bridging of these skills deficits, highlighting how the perceived usefulness of skills usage may actually be dependent on the presence and specific type of skill deficit in each individual in the first place.

| Limitations
The study has several limitations. First, generalizability of the results from this study is limited by missing data and the nature of the qualitative methodology employed. The 25% of participants who did In principle, all CBT skills reported by participants in this study aligned with the iCBT intervention they received. Given some differences in the symptom-specific CBT skills covered within the three programmes and the option of unlockable content (see Table A2), which specific CBT skills participants encountered during their treatment varied somewhat between participants. Also, there were a handful of participants (<5%), who mentioned techniques or strategies that were not covered by the iCBT intervention (e.g., one participant cited not using the rubber-band strategy for self-harm urges anymore; another reported using/ceasing the use of a gratitude jar).
Specific techniques like these may have been introduced by supporters as per routine treatment protocols, which encourages them to supplement iCBT content where deemed clinically necessary.
Also, we did not assess participants use of CBT skills prior to commencing treatment, meaning that we cannot say for certain that participants did not use the skills they reported already ahead of iCBT treatment. Nevertheless, we would expect that participants with significant knowledge of CBT were the exception in the current study, given that the service hosting the study used iCBT as a step two intervention (i.e., a first-line treatment) for mild to moderate symptoms only.
Finally, due to the nature of the collected data (i.e., utilization of open-ended questions rather than pre-prepared checklists) and the cross-sectional qualitative analysis applied, we were unable to further analyse changes over time in CBT skills usage and associated experiences for the purpose of this report. Further research to explore patterns of intra-individual changes in CBT skills usage and associated experiences over time as well as the relationship between CBT skill usage and clinical outcomes across follow-up will be essential in furthering our understanding in this respect.

| CONCLUSION
In conclusion, the current study provides evidence that those who receive iCBT in routine care not only learn CBT skills during treatment but utilize them after active treatment had ended. This CBT skill usage appeared linked to helpful impacts, linking in with hypotheses around post-therapeutic skills usage as a mediator of effect maintenance after iCBT. However, given the limitations of the current study, quantitative research, employing longitudinal mediation analysis will be crucial to more fully explore these hypotheses. Here, of particular interest will be distinctions between formal, so concrete CBT skill usage via the iCBT platform, for example, and informal, implicit or adapted CBT skills usage described by some participants in this study. Understanding these intricates of skills usage better in addition to the rich accounts of CBT skills usage provided by participants in this study will provide the best possible steppingstone to drive innovation and development of even better iCBT interventions of the future.

ACKNOWLEDGEMENTS
We wish to thank the R&D and clinical team members at Berkshire NHS Foundation Trust service and employees at SilverCloud Health for assisting trial execution and data curation. We also thank the many patients who volunteered their time and efforts to participate in our trial.  Abbreviations: GAD-7, generalized anxiety disorder-7 item; IQR, interquartile range; MINI, mini international neuropsychiatric interview 7.0.2; PHQ-9, patient health questionnaire-9 item; SD, standard deviation.

CONFLICT OF INTEREST
T A B L E A 2 Outline of modules in space from depression and space from anxiety programs Getting started An outline of CBT basics and thoughtsfeelings-behaviours cycles relating to depression. Users become more aware of mood and their situation.
Getting started An outline of CBT basics and thoughtsfeelings-behaviours cycles. Users begin noting their difficulties with anxiety.
Understanding feelings Encouraging a greater understanding of mood and emotions, how thoughts, physical reactions, and behaviours are connected in influencing how we feel.
Understanding feelings Encouraging a greater understanding of the connection between thoughts, feelings and behaviours. Exploring aspects of the impact of lifestyle choice and purposeful relaxation on these.
Boosting behaviour Encouraging the user to plan enjoyable activities that promote a sense of achievement. Identifying tasks to target the physical reactions to distress and noting behaviour traps.
Facing your fears Defining the users' fears, exploring graded exposure and defining a fear hierarchy. Experimenting with facing fears to reduce anxiety. Encouraging the user to attempt to accept fear and embrace uncertainty.
Spotting thoughts Further awareness of unhelpful thinking patterns, recognizing distorted thoughts and addressing the outcomes of such negative thought cycles.

Spotting thoughts
Flagging anxious thoughts and noticing their automatic acceptance. Weakening the power of thoughts by practicing balanced acceptance.
Challenging thoughts Identification of hot thoughts and thinking errors to tackle and dispute negative patterns.
Challenging thoughts Identification of hot thoughts and thinking errors to tackle and dispute negative patterns.
Bringing it all together Considering perspective changes and skills acquired. Outlining steps for future progress including normalizing set-backs, maintaining well-being and social support.
Bringing it all together Considering perspective changes and skills acquired. Outlining steps for future progress including normalizing set-backs and maintaining well-being and social support.

Core beliefs (unlockable content)
Particular targeting of deeply-held core beliefs underlying unhelpful thoughts. Users identify healthy and unhealthy beliefs and learn strategies to generate more balanced core beliefs.
Managing worry (unlockable content) Improving knowledge of worry and recognizing practical and hypothetical worries. Implementing strategies to manage worries, for example, using the worry tree.

Mini modules (unlockable across programs by supporters)
Sleep Education about sleep disturbance and sleep hygiene. Applying techniques of sleep monitoring.

Relaxation
Psychoeducation regarding the benefits of relaxation as a skill. Relaxation techniques are covered.

Self-esteem
Psychoeducation around self-esteem, self-talk and self-compassion. Users reflect on how they view and talk to themselves. Activities promote learning to counter negative self-talk and cultivate self-compassion.
Employment support Encouraging users experiencing employment related stress to identify the impact on their mental health. Introducing techniques to reduce this impact and the use of tools to cope and proactively manage the situation.

Behavioural experiments
Guiding users in this CBT technique, which includes devising and conducting tests regarding the automatic negative thinking that maintains their low mood/anxiety. Users reflect and learn from the outcomes.

Anger
For users experiencing excessive/inappropriate anger. Introducing the thoughts-feelings-behaviours cycle to assist in monitoring their anger and its function and identifying when it is a problem. Highlighting unhelpful thinking styles.
Grief and loss Psychoeducation regarding common emotional reactions to loss and the dual-process model of coping with bereavement. Reflecting on the user's unique grief response, encouraging healthy expressions of feelings and restoration of their lives. Planning for triggers or anniversaries, reflecting on life values and developing ways to remember the person they lost.

Communication and relationships
What healthy communication in relationships looks like. Identifying different communication styles, reflecting on the user's own style, guidance towards improving communication skills, and consideration of other factors that are crucial to effective communication.
Note. The comorbid Space from Depression and Anxiety program is composed of modules from the depression program with an integrated Managing Worry module. Core Beliefs and Facing Your Fears are unlockable modules. Breathing and relaxation exercises that aim and reducing tension and relaxing the body, but do not specifically address or 'target' thought processes like mindfulness exercises and mediation.
'I have definitely started finding time on an evening to practice relaxation' 'I use relaxation techniques frequently'

Mindfulness and meditation
Beyond mere relaxation, mindfulness and mediation encompasses elements of becoming aware of one's thought processes and 'clearing' one's mind.
'Focusing on breath and staying focused in the present rather than on spiralling thoughts' 'Meditation is the main one I use'

Time-out and taking 'me time'
Taking time-out and/or 'me time' within this category related to either selfregulation in distressing circumstances or self-care practices and involved participants dedicating physical time to either cause.
'Also taking time out for myself, reading for an hour, or just walking away from a situation and taking 10 mins to relax' 'Reserving some "me time"'

Social support
Social support in this instance encompasses participants talking to or spending time with others in order to help them cope better with their mental health problem.
'I have tried talking to my boyfriend more about the issues' 'Taking through my problems with a family member'

Activation and activities
This category refers to participants engaging in activities in order to improve their mental health and mood-Encompassing both activity scheduling and activities that could be classed as exposure.
'I also followed the exercise advice and exercise daily, along with making time every day to engage in an activity that I enjoy' 'Jumping into action before talking myself out of it. I was able to attend social events where I would have cancelled' 1.2.6. Lifestyle changes (diet, sleep etc.) This category refers to participants making changes in their daily lives with the aim of improving their wellbeing. These changes may relate to either sleep, diet or other daily activities.
'Reading before bed rather than watching tv or being on my phone' 'Have given up alcohol, changed jobs' 1.2.7. Goal-oriented and solutionbased strategies The defining feature of goal-orientated and solution-based strategies within this category is that they serve the achievement of a specific goal, targeting such concepts as motivation, problemsolving and stress.
'Breaking tasks down-Doing them for 5 minutes and then if I do not want to carry on come back to it later.' 'The goal setting tool has, and I think will, continue to help me a lot.' This category refers to discontinued cognitive techniques and strategies related to worry and how to cope with it (including worry time and worry tree as described above).
'I stopped using the worry tree' 'I was using the decision tree before which was useful.' 'Most strategies did not work very effectively for me.' 'Some of the time this does not work'

Difficulties in skill application
Other than the previous category, this category refers to participants' perceptions of executing and applying the skills as difficult, hard or challenging. This difficulty may occur independent of the perceived effectiveness of skills and relates to the process of using skills rather than the outcome of skills usage.
'However, I find it hard to control the way I think.' 'This can sometimes be quite difficult.'

Situational and daily barriers to skills usage
Participants citing events in their lives (daily or extraordinary) as influencing them in their experience of skills usage were recorded within this category.
'I've changed jobs (after 7 years) recently which was a major life event for me and this was a hard situation to navigate. I've been worrying and ruminating a lot.' 'I am finding it difficult to set aside that "me time"' 3.2.4.CBT skills as upsetting or problematic in some way This category refers to participants not just experiencing skills usage as lacking in efficacy but actually describing skills usage as having negative effects in some way.
'I find it relatively discouraging and damaging to categorise these thoughts as unhelpful in whichever way as to me, the thoughts are my own and thus I feel that my own thoughts are the issue and that I myself am therefore being unhelpful.' 'Also introducing worry time became a negative part of my day' This category represented responses which suggest that participants stopped using skills because they felt their symptoms had improved to the point that they did not need to use skills anymore to manage them.
'I have not considered using it recently as i am more low than anxious I think' 'Do not need to do that much now as feeling better.'

Experiences of skills worsening symptoms
Participants who indicated that using skills had made them feel worse in some way are recorded in this category.
'[…] opens up old wounds and makes me very sad and panicked instead of helping me to get through things' 'For me [breathing techniques] induce panic' 4.7. Intentions for future/resumed skill usage When participants indicated that they are intending to use skills in the future, learn new skills or return to the iCBT platform in the near future, then their responses were counted here.
'I see a counsellor once a week to focus on the PTSD and anxiety. I only have one session left which is a little stressful-I may use silver cloud more because of this.' 'I would like to log back into SilverCloud and run through all the techniques again'