SEARCH

SEARCH BY CITATION

Abstract

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Mechanism of action
  6. Sacral nerve stimulation and constipation
  7. Sacral nerve stimulation in children with constipation
  8. Adverse events with sacral nerve stimulation and constipation
  9. Discussion
  10. Disclosure
  11. References

Background:

For over 10 years sacral nerve stimulation (SNS) has been used for patients with constipation resistant to conservative treatment. A review of the literature is presented.

Methods:

PubMed, MEDLINE and Embase databases were searched for studies demonstrating the use of SNS for the treatment of constipation.

Results:

Thirteen studies have been published describing the results of SNS for chronic constipation. Of these, three were in children and ten in adults. Test stimulation was successful in 42–100 per cent of patients. In those who proceeded to permanent SNS, up to 87 per cent showed an improvement in symptoms at a median follow-up of 28 months. The success of stimulation varied depending on the outcome measure being used. Symptom improvement correlated with improvement in quality of life and patient satisfaction scores.

Conclusion:

SNS appears to be an effective treatment for constipation, but this needs to be confirmed in larger prospective studies with longer follow-up. Improved outcome measures need to be adopted given the multiple symptoms that constipation may be associated with. Comparison with other established surgical therapies also needs consideration. Copyright © 2012 British Journal of Surgery Society Ltd. Published by John Wiley & Sons, Ltd.


Introduction

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Mechanism of action
  6. Sacral nerve stimulation and constipation
  7. Sacral nerve stimulation in children with constipation
  8. Adverse events with sacral nerve stimulation and constipation
  9. Discussion
  10. Disclosure
  11. References

It has been estimated that, worldwide, around 14 per cent of adults suffer from constipation. The incidence is higher in older patients, in females and in those of lower socioeconomic status1. Constipation has been defined by various consensus groups, such as the Rome Foundation, which developed the Rome III criteria. According to these, two of six symptoms must be present for 6 months or more. These include straining at stool, passing hard stools, sensation of incomplete emptying, sensation of anorectal obstruction, self-digitation and a defaecation frequency of less than three times per week2. These diagnostic criteria reflect the variety of symptoms that can be present with constipation, along with abdominal pain, bloating, and frequent laxative and enema use.

The pathogenesis of constipation can be broadly subdivided into disorders of colonic motility and disorders of evacuatory dysfunction; there is a degree of overlap in many patients3. Current medical treatment includes dietary advice, drug treatment (laxatives, enemas, suppositories) and biofeedback4.

Surgery has a role in carefully selected patients if these options fail. In those with slow-transit constipation, an antegrade continence enema procedure or colectomy with or without stoma formation has been advocated3. Although often successful in childhood, the antegrade continence enema in adults is associated with a high complication rate, with a reoperation rate of up to 88 per cent and a failure rate of 48–59 per cent in the medium term. Colectomy often improves defaecation frequency but symptoms of abdominal pain persist in 41 per cent of patients, reflecting the panenteric motility disorder that exists in many subjects. Following colectomy, 14 per cent of patients develop diarrhoea and/or faecal incontinence and 18 per cent adhesional small bowel obstruction5.

Evacuatory dysfunction may be related to rectal hyposensitivity, functional pelvic floor dyssynergia or mechanical outlet obstruction. For those with hyposensitivity or dyssynergia, abdominal or perineal surgery is not an option. In those with mechanical dysfunction, operations can be divided into ‘hitching’ procedures (such as rectopexy) or ‘excisional’ procedures (for example stapled transanal rectal resection). The literature in this area is complicated owing to heterogeneity in the groups of patients studied. Rectal intussusception and rectocele are often observed in patients with normal evacuation, and the relevance of such abnormal findings on proctography remains unclear. Reports of worsened constipation with posterior rectopexy have swung current treatment algorithms towards ventral mesh in rectopexy, which is often coupled with colpopexy in women with inadequate support of the vaginal vault. Stapled transanal rectal resection remains an alternative treatment option, although in the authors' opinion the reported complication rate is unacceptably high and they do not advocate its use.

Sacral nerve stimulation (SNS) was first described by Tanagho6 in 1988 for the treatment of urinary voiding dysfunction. It was during initial studies for this indication that the effect on bowel function was first appreciated. During initial testing, often referred to as peripheral nerve evaluation (PNE), external electrical stimulation is delivered via a temporary lead inserted percutaneously through the third sacral foramen to the S3 sacral nerve7. After a screening period of 2–3 weeks, the patient's response to treatment is assessed objectively by reference to bowel or bladder diaries and/or a questionnaire. If sufficient clinical improvement is demonstrated, the patient is offered implantation of a permanent system. A similar protocol has been used for the assessment of the response of patients with constipation.

Methods

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Mechanism of action
  6. Sacral nerve stimulation and constipation
  7. Sacral nerve stimulation in children with constipation
  8. Adverse events with sacral nerve stimulation and constipation
  9. Discussion
  10. Disclosure
  11. References

An extensive review was carried out of all the evidence published on the use of SNS for chronic constipation. An electronic search was performed of MEDLINE, PubMed and Embase databases, with the keywords and/or medical subject headings (MeSH) ‘constipation’, ‘sacral nerve stimulation’ and ‘sacral neuromodulation’. Further studies were identified by cross-referencing from relevant articles and abstracts. Only studies that reported the clinical outcome of SNS for constipation in the English language were included in this review. The aim of the review was to assess the clinical efficacy of SNS for constipation based on the current published literature.

Mechanism of action

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Mechanism of action
  6. Sacral nerve stimulation and constipation
  7. Sacral nerve stimulation in children with constipation
  8. Adverse events with sacral nerve stimulation and constipation
  9. Discussion
  10. Disclosure
  11. References

The mechanism of action of SNS is unclear. It is likely to involve multiple physiological effects, in particular modulation of afferent and local reflex pathways8. The S2 root has the greatest afferent outflow from the sacral plexus9; however, stimulation of this root often leads to activation of sciatic motor fibres that can lead to buttock and leg pain. Therefore, stimulation of the S3 root is often the preferable option for therapy, providing maximal sacral afferent stimulation in the absence of unwanted side-effects.

Many studies have described changes in anorectal sensory and motor function as a result of SNS, although there does not appear to be a consistent pattern to these changes10. A double-blind randomized study assessing the effect of SNS on rectal sensation in patients with rectal hyposensitivity suggested that modulation of afferent pathways may be important as improvement in threshold and maximum tolerated volumes to balloon distension occurs with therapy11.

Giani and colleagues12 demonstrated an increase in the latency of cerebral somatosensory evoked potentials in patients with faecal incontinence and constipation undergoing SNS. Griffin and co-workers13 examined the effects of posterior tibial nerve stimulation and SNS on the somatosensory cortex in an animal model. An increase in amplitude of primary cortical evoked potentials was seen during stimulation. These findings suggest that SNS alters the representation of the lower gastrointestinal tract in the sensory cortex.

The effects of SNS on colonic motility were investigated by Dinning and colleagues14. They showed that SNS at suprasensory threshold significantly increased the antegrade pancolonic wave sequence in patients with slow-transit constipation. This corresponded to an increase in frequency of defaecation and a reduction in laxative use. However, stimulation at subsensory threshold, as used routinely in clinical practice, may not have an effect on motility15. Nevertheless, Kamm and co-workers16 showed normalization of whole-gut transit time at 6 months' follow-up in half of patients undergoing SNS for slow-transit constipation using stimulation at subsensory threshold. This corresponded to an increase in frequency of defaecation. In contrast, SNS in patients with faecal incontinence may reduce anterograde colonic transit17. The difference in the effect of stimulation under different conditions remains unclear, but SNS may act to normalize transit by modulation of the enteric nervous system or reflex pathways at the spinal cord level.

Sacral nerve stimulation and constipation

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Mechanism of action
  6. Sacral nerve stimulation and constipation
  7. Sacral nerve stimulation in children with constipation
  8. Adverse events with sacral nerve stimulation and constipation
  9. Discussion
  10. Disclosure
  11. References

Thirteen published studies were found reporting the use of SNS for constipation (Table 1). Ten involved adult subjects, including two double-blind crossover studies and three retrospective reviews. Three studies reported the use of SNS in children and adolescents.

Table 1. Results of published studies of sacral nerve stimulation for constipation
ReferenceYearStudy typeLevel of evidence*Patient groupType of constipationNo. who had PNENo. with permanent SNSPNE success rate (%)Follow-up
  1. *According to the Centre for Evidence Based Medicine, University of Oxford29. PNE, peripheral nerve evaluation; SNS, sacral nerve stimulation; NA, clear data not available.

Ganio et al.182001Prospective uncontrolledIVAdultsIdiopathic10NANA10 days
Malouf et al.192002Prospective uncontrolledIVAdultsSlow colonic transit8NANA3 weeks
Kenefick et al.202002Prospective uncontrolledIVAdultsSlow colonic transit and evacuatory dysfunction441006 months (mean)
Kenefick et al.21*2002Double-blind crossoverIIbAdultsSlow colonic transit and evacuatory dysfunctionNA2NA4 weeks
Humphreys et al.22*2006Prospective uncontrolledIVChildrenIdiopathicNA15NA13 months (mean)
Holzer et al.232008Prospective uncontrolledIVAdultsSlow colonic transit and evacuatory dysfunction1984211 months (median)
Roth et al.24*2008Prospective uncontrolledIVChildrenIdiopathicNA17NA27 months (median)
Naldini et al.252010RetrospectiveIVAdultsSlow colonic transit1596042 months (mean)
Kamm et al.162010Prospective uncontrolledIIbAdultsSlow colonic transit and evacuatory dysfunction62457328 months (median)
Sharma et al.262011RetrospectiveIVAdultsSlow colonic transit and idiopathic21115234 months (median)
Govaert et al.272012RetrospectiveIIbAdultsSlow colonic transit and evacuatory dysfunction117685837 months (median)
van Wunnik et al.282012RetrospectiveIVChildrenSlow colonic transit and evacuatory dysfunction1312926–12 months
Knowles et al.112012Double-blind crossoverIIbAdultsEvacuatory dysfunction13118519 months (mean)

In 2001 Ganio and colleagues18 described 40 patients suffering from a variety of functional anorectal and urinary disturbances, of whom 12 had constipation. These patients were subjected to 10 days of PNE, and ten completed a week's course of stimulation. This group showed no increase in the weekly number of bowel movements, but there was improvement in initiating evacuation and a reduction in the number of unsuccessful visits to the toilet and time required to evacuate.

In 2002 Malouf and co-workers19 exposed eight patients to 3 weeks of PNE, all of whom had slow-transit constipation. Two patients demonstrated an increase in frequency of spontaneous defaecation over the test period and an improvement in bowel function as assessed by visual analogue scale (VAS); the remaining six showed no improvement.

More convincing results were obtained by Kenefick et al. in 200220. Four women with constipation were exposed to PNE and then permanent SNS. Two patients had slow-transit constipation and two had dyssynergia on defaecation proctography. An increase in the number of spontaneous defaecations, and an improvement in abdominal pain and bloating were demonstrated at 6-month follow-up. Improvements in the Cleveland Clinic constipation score30 and in a VAS score were seen. In a subsequent study, Kenefick and co-workers21 described a double-blind placebo-controlled crossover study of two of these patients. Improvement in defaecatory frequency, abdominal pain, bloating and VAS score was demonstrated when the stimulator was switched on, compared with when it was off.

In 2008, Holzer and colleagues23 described 19 patients who underwent temporary SNS. Eight of these had slow-transit constipation and nine had evacuatory dysfunction. Only eight patients were deemed eligible for permanent stimulation, on the basis of improvement in evacuation frequency to achieve two or more defaecations per week or defaecation without digitation. Clinical benefit was maintained with chronic stimulation at a median follow-up of 11 months, with an associated reduction in Cleveland Clinic constipation scores (from median (range) 23 (18–27) at baseline to 8 (4–13) at 12 months) and Short Form 36 (SF-36®; QualityMetric, Lincoln, Rhode Island, USA) quality-of-life (QoL) scores31.

A retrospective study by Naldini and co-workers25 in 2010 described the outcome of 15 patients with slow-transit constipation who had undergone PNE. Of these, nine met the study criteria for placement of a permanent implant. The criteria for permanent SNS were disappearance of the need for laxatives or enema use, occurrence of spontaneous defaecation or an increase in SF-36® QoL score. At 6 months' follow-up there was a significant increase in the number of bowel movements per week (from mean 1·8 at baseline to 3·3 at 6 months), with a concurrent reduction in the number of enemas required. The Cleveland Clinic constipation score improved from 24 at baseline to 12 at 6 months, with improvement in all domains of the SF-36®.

The largest prospective study to date was carried out in 2010 by Kamm et al.16. In this multicentre international study 62 patients underwent test stimulation, 81 per cent of whom had slow-transit constipation. Forty-five (73 per cent) progressed to permanent stimulation. The criteria for progression to permanent SNS were a frequency of evacuation of three or more per week, a reduction of 50 per cent or more in the number of evacuations associated with straining or a 50 per cent decrease in the sensation of incomplete evacuation. The patients were followed up for a median of 28 months. Thirty-nine (87 per cent) of the 45 who had undergone permanent stimulation showed an improvement in symptoms. Significant improvements were seen in frequency of defaecation, straining, sensation of incomplete evacuation, abdominal pain and bloating. The Cleveland Clinic constipation score improved from 18 at baseline to 10 at latest follow-up, with a VAS score that rated overall bowel function increasing from 8 to 66 (where 0 represented worst function and 100 best function). Significant improvements were seen in four of the eight domains of the SF-36® health questionnaire. Rectal sensation was altered by stimulation, with a reduction in the sensory, urge and maximal thresholds to rectal balloon distension. The colonic transit time in half of the patients with slow gut transit at baseline had normalized by 6 months' follow-up and this corresponded to an increase in defaecatory frequency.

Sharma and colleagues26 published a retrospective study in 2011. Twenty-one patients underwent temporary stimulation, 18 of whom had slow-transit constipation. Of these, 11 went on to have a permanent implant. The frequency of defaecation increased from a median of one per week at baseline to four per week at 34 months' follow-up. Eight of these 11 patients completely stopped using laxatives.

A recent retrospective review by Govaert and co-workers27 described 117 patients who had undergone PNE. Of these 68 (58·1 per cent) proceeded to permanent SNS. Some of the patients in this report were part of the multicentre study published by Kamm et al.16. There was an initial significant improvement in the Cleveland Clinic constipation score from mean(s.d.) 17(3·8) to 10·2(8·3) at first follow-up. This improvement was maintained at a median follow-up of 37 months. Data regarding pathogenesis of constipation were incomplete, but both patients with slow-transit constipation and those with evacuatory dysfunction appeared to benefit from treatment.

One study reported on possible predictors of successful SNS for constipation. Carriero and colleagues32 suggested that psychological assessment may aid in the selection of patients. In this study, 68 patients with slow-transit constipation were offered psychological assessment using the Minnesota Multiphasic Personality Inventory (MMPI) 2 questionnaire33, of whom 45 completed the questionnaire. Thirteen had a normal score and underwent PNE. Of these, 11 proceeded to permanent stimulation. Nine of the patients who had refused to complete the MMPI-2 questionnaire, or had an abnormal score, also underwent PNE. Of these, only three progressed to permanent stimulation.

Sacral nerve stimulation in children with constipation

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Mechanism of action
  6. Sacral nerve stimulation and constipation
  7. Sacral nerve stimulation in children with constipation
  8. Adverse events with sacral nerve stimulation and constipation
  9. Discussion
  10. Disclosure
  11. References

Two papers from the USA have reported the effects of SNS in children with severe dysfunctional elimination syndrome. This syndrome is described as a broad spectrum of functional disturbances involving the urinary and lower gastrointestinal systems in the absence of any obvious anatomical or neurological deficiency. One of the problems experienced by these children was constipation. The first study, reported by Humphreys and colleagues in 200622, described 23 children, of whom 15 had constipation-type symptoms. The age of the patients ranged from 6 to 15 years. After a mean follow-up of 13 months, constipation symptoms improved in 12 of the 15 patients treated with permanent SNS. The second study, published by Roth and co-workers24 in 2008, described 17 children with constipation-type symptoms; in 12 children these symptoms either resolved or improved following SNS. The age range in this study was 8–17 years. In both studies it is unclear how these symptoms improved, as no objective measurements were described.

Van Wunnik et al.28 described a retrospective study, in which 13 adolescent girls, aged 10–18 years, underwent PNE for constipation; all had failed conservative therapy. Twelve had placement of a permanent implant. At follow-up of between 6 and 12 months, there was an improvement in the number of defaecations per week, abdominal pain, straining and in Cleveland Clinic constipation score. Absenteeism from school was noted in ten of these patients at baseline, but at 6 months' follow-up all were attending school regularly.

Adverse events with sacral nerve stimulation and constipation

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Mechanism of action
  6. Sacral nerve stimulation and constipation
  7. Sacral nerve stimulation in children with constipation
  8. Adverse events with sacral nerve stimulation and constipation
  9. Discussion
  10. Disclosure
  11. References

A retrospective review by Maeda and colleagues34 described the suboptimal outcome and complications of SNS for constipation observed in an individual unit. Of 38 patients who had undergone permanent stimulation, 22 had an adverse event specifically related to the treatment. Nearly half of these events resolved with reprogramming, but a third needed a further surgical procedure. Kamm and co-workers16 reported 61 adverse events specifically related to the treatment in 45 patients with a permanent implant. Another 40 adverse events were attributed to the underlying constipation or to a new unrelated diagnosis. Two-thirds of the SNS-related adverse events were self-limiting, and most required reprogramming of the device. Although the frequency of these adverse events appears high, the proportion of subjects affected is unclear. The figures appear to be comparable with those documented for use of SNS for faecal incontinence. Maeda et al.35 reported that 85 per cent of those with a permanent implant for faecal incontinence reported an adverse event, with the majority related to loss of efficacy.

Discussion

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Mechanism of action
  6. Sacral nerve stimulation and constipation
  7. Sacral nerve stimulation in children with constipation
  8. Adverse events with sacral nerve stimulation and constipation
  9. Discussion
  10. Disclosure
  11. References

Although chronic constipation is common, surgical treatment remains complex and requires a sound understanding of the underlying aetiological factors contributing to the patient's symptoms. Constipation should be managed initially using a combination of pharmacological agents, rectal irrigation and/or behavioural treatments. There is a close association between severe constipation and psychological disorders, including a past history of physical or sexual abuse. Addressing these problems is mandatory before considering any form of surgical intervention.

SNS appears to be an effective treatment for constipation; however, research to date has been predominantly confined to small, low-level evidence studies with most lacking a coherent definition of constipation. Although SNS appears to have great potential in treating patients with slow-transit constipation, its use in routine clinical practice remains uncommon. This is likely to be related to the cost of this therapy, which results in reluctance by healthcare providers to fund the treatment in the absence of more robust data. Although patients with evacuatory dysfunction have also benefited from treatment, it remains unclear which patients are most likely to benefit from the therapy. Given the size of the studies and the inconsistency of the outcomes reported to date, it would be difficult to perform any meaningful summative analysis of the data.

Constipation is associated with a wide range of clinical features. This is important to remember when evaluating studies assessing treatment as multiple outcome measures are often recorded. The most common outcome measured in these studies was a change in the frequency of defaecation. Few studies described other clinical features, such as abdominal pain, straining and the sensation of incomplete emptying. Only Kamm and colleagues16 sought to address how the majority of these symptoms were affected. It is worth noting that, although many of these symptoms improved significantly, none completely resolved. Because of the complexity of the symptoms, it may be that complete resolution is impossible to achieve by a single intervention. With this in mind, the aim of any therapeutic intervention for constipation should be for the patient to achieve acceptable bowel function and QoL. To this end, constipation-related symptom and QoL scores, and patient-based VAS scores (assessing overall satisfaction with bowel function) may be better assessments of efficacy. As with all studies of functional bowel disorders, the risk of a placebo effect in those undergoing an operative intervention is high. The two double-blind crossover studies11, 21 demonstrated a true clinical effect from therapy, supported by outcome data showing ongoing benefit from SNS in the long term.

The success rate of PNE for constipation in adults ranged from 42 per cent23 to 100 per cent20 in Kenefick's series of four patients. However, the largest prospective series achieved a success rate of only 73 per cent16. The reported success rates of PNE when used for faecal incontinence are between 74 per cent36 and 90 per cent37. This suggests that the proportion of patients with constipation progressing from PNE to permanent stimulation is lower than that seen in the faecal incontinence group. However, the evidence base of SNS for constipation is considerably smaller than that for faecal incontinence. It may be that the 2–3-week period of temporary stimulation, although seemingly suitable for patients with faecal incontinence, may not be sufficient for those with constipation. Govaert and colleagues27 suggested that age is a significant predictive factor for successful PNE, in favour of younger patients, whereas Carriero and co-workers32 suggested that psychological assessment may aid the selection of patients.

So far, follow-up of these patients has been relatively short. The longest duration of follow-up has been a median of 37 months. It is unclear whether any early improvement in constipation is maintained in the longer term. A gradual loss of efficacy over the long term in some patients undergoing SNS for faecal incontinence has been reported38 and this might also be expected to occur in the constipation group. In some patients reprogramming of the stimulation parameters may allow restoration of clinical effect.

With the exception of a reduction in rectal sensation urge and maximum threshold to balloon rectal distension, there do not appear to be any other consistent anorectal physiological changes. This is analogous to findings for SNS for faecal incontinence, where there are no consistent changes in physiological parameters10, 39. There is limited evidence that other forms of neurostimulation, including percutaneous and transcutaneous posterior tibial nerve stimulation and acupuncture, are beneficial in treating slow-transit constipation. This suggests that the mechanism of action, as with faecal incontinence and urological disorders, is mediated predominantly by the stimulation of afferent pathways. An increase in colonic motility has been described in patients with slow-transit constipation14, 16. As noted previously, it is not clear from the literature whether slow colonic transit- or evacuatory dysfunction-type constipation is likely to respond better to SNS.

All the reported studies were uncontrolled, with no direct comparison with any other treatment. Most studies were retrospective case series, providing level IV scientific evidence29. The exceptions to this are the double-blind crossover studies11, 21 and the larger cohort studies of Kamm et al.16 and Govaert and co-workers27, which provide level IIb evidence.

All patients had failed conservative measures, such as biofeedback and drug treatment. Although not mentioned explicitly, it can be assumed that mechanical causes of constipation were ruled out by the authors. Constipation often has a mixed aetiology, and slow-transit and evacuatory disorders can be present in the same patient to varying degrees. Surgery should be considered only in patients who have undergone full medical and psychosocial evaluation, and in whom conservative measures, including psychiatric treatment where necessary, have been exhausted. SNS allows a minimally invasive trial of therapy with low morbidity and in the authors' opinion should be the first-line surgical option for those with slow-transit constipation and/or rectal hyposensitivity. Further evidence is required to ascertain whether SNS is truly beneficial in those with pelvic floor dyssynergia. An algorithm for the management of chronic idiopathic constipation is shown in Fig. 1.

thumbnail image

Figure 1. Proposed treatment algorithm for the treatment of constipation. This algorithm applies to patients in whom Hirschsprung's disease and megarectum have been excluded. Although colectomy, antegrade continence enema (ACE) and stapled transanal rectal resection (STARR) are accepted treatments at other institutions, they are not part of the authors' practice. *More than 50 per cent of patients with pelvic floor dyssynergia have rectal hyposensitivity and 60 per cent have coexisting slow-transit constipation; a trial of sacral nerve stimulation (SNS) can be considered in those with coexisting aetiology. VMR, ventral mesh rectopexy

Download figure to PowerPoint

Based on the evidence from this limited number of studies, SNS appears to be an effective treatment for chronic constipation. Larger clinical and cost-effectiveness studies comparing SNS with alternative treatments are required. Better defined measures of constipation using improved symptom and constipation-specific QoL scores should be used. Patients often complain of multiple symptoms, and frequency of defaecation, although sometimes important, is not the only endpoint.

References

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Mechanism of action
  6. Sacral nerve stimulation and constipation
  7. Sacral nerve stimulation in children with constipation
  8. Adverse events with sacral nerve stimulation and constipation
  9. Discussion
  10. Disclosure
  11. References