Measuring Access to Justice: Transformation and Technicality in SDG 16.3
Abstract
Billions of people around the world live at the margins – pushed or kept out, often in silence, without adequate protection of the law. Denied healthcare, citizenship or fair pay, those unprotected by the law have problems that are both real and relentless, impacting their ability to reap the benefits of sustainable development. Despite this crushing reality, access to justice is a bedrock principle undergirding human rights. Despite its centrality, justice was not explicitly included in the Millennium Development Goals (MDGs). This omission was corrected when the SDGs were adopted with a stand-alone goal on justice. While Sustainable Development Goal (SDG) 16 was the result of years of political, strategic and scholarly work by human rights advocates, development practitioners and academics, its promise lies beyond the technocratic realms of development programming, by insisting that people's own experience of justice – and injustice – must remain at the center of efforts to assess progress toward a world where no one is ‘left behind’.
Billions of people around the world live at the margins – pushed or kept out, often in silence, without adequate protection of the law. Denied healthcare, citizenship or fair pay, those unprotected by the law have problems that are both real and relentless, impacting their ability to reap the benefits of sustainable development.
Despite this crushing reality, access to justice is a bedrock principle undergirding human rights (CLEP, 2008). The United Nations Development Programme (UNDP) has described access to justice as ‘a basic human right as well as an indispensable means to combat poverty, prevent and resolve conflicts’ (UNDP, 2004, p. 3). Under international human rights law, all people have the right to access justice without discrimination and states must take positive steps to make this promise real for all (CLEP, 2008; UNGA, 2012a). Despite its centrality, justice was not explicitly included in the Millennium Development Goals (MDGs) framework. This omission was corrected in September 2015 when the UN General Assembly adopted the Sustainable Development Goals (SDGs), through which the global community pledged to promote sustainable development and to ensure access to justice for all by 2030 (Goal 16) (UN, n.d.; UNGA, 2015). The inclusion of a stand-alone goal on justice was hard-won, and the framing of a target on access to justice was no easier. While SDG 16 was the result of years of political, strategic and scholarly work by human rights advocates, development practitioners and academics, its promise lies beyond the technocratic realms of development programming, by insisting that people's own experience of justice – and injustice – must remain at the center of efforts to assess progress toward a world where no one is ‘left behind’.
Through a mix of interviews with key experts and documentary research, this paper reflects on the process leading up to the adoption of the criminal justice indicators for SDG 16.3, while also looking ahead to the possibilities for inclusion of a more transformative civil justice indicator. The following section of the paper examines the development of Goal 16 and its target on access to justice. The third section provides an overview of the 16.3 indicator adoption process, describing important contestations that led to the cabining of access to justice through the privileging of criminal justice indicators and the discounting of civil justice metrics, a process that carried discursive and substantive consequences. The fourth section examines the dynamics that led to the rejection of a civil justice indicator, including data inertia resulting in the rejection of new methodologies and forms of data collection, institutional positioning, and the availability of resources. The next section unpacks the future potential for inclusion of an access to civil justice indicator, contending that what is needed is a global commitment to support a holistic indicator that captures legal needs and represents access to justice from the people's perspective. The final section concludes.
Advancing justice: The development of goal 16
In 2010, the General Assembly held a High-level Plenary Meeting on the MDGs to take stock of progress, catalyze political will to push through obstacles, and begin discussions about the development framework that would replace the MDGs in 2015 (UN, 2010). The framework sought to build upon the success of the MDGs through a broad and inclusive process, which at times led to competing UN bodies, NGO coalitions and streams of work (Kumar et al., 2016; Langford, 2016). The development of the SDGs proceeded roughly along two tracks: a political process explicitly focused on goals and targets; and a technical process devoted to indicators and statistics. These tracks often overlapped, with discussions on the technical side influencing the debates within the political process, and vice versa.
In 2011 the Secretary-General appointed a High-Level Panel to advise on the new development framework and a UN System Task Team (UNSTT), comprised of UN insiders (UNSTT, 2012). In June 2012 the Task Team delivered Realizing the Future We Want for All to the Secretary General. This report recognized that ‘the rule of law and human rights’ were left out of the MDGs despite being a ‘core value’ of the UN (UNSTT, 2012, p. 23).
A separate work stream focused on sustainable development was initiated in June 2012 when the UN Conference on Sustainable Development (Rio+20), concluded with an outcome document entitled The Future We Want. This document called for the creation of a set of sustainable development goals, with the rule of law and access to justice identified as core components (UN, 2012, para 8). In 2013, the General Assembly (GA) created the Open Working Group (OWG), a 70 member state body required by the Rio+20 outcome document, charged with drafting the official report on the SDGs. The GA also determined that the SDGs would be universal – applicable to the Global North as well as the South – and would focus on both poverty reduction and sustainable development (UNGA, 2013; Coumarelos et al., 2012; UNGA, 2012b). Over the next two years the OWG held numerous discussions on the importance of the rule of law and access to justice, which ultimately resulted in a proposed goal on peaceful, inclusive societies and effective justice institutions (SDGs, 2013, 2014b, 2014c). The deliberations were informed by formal contributions and proposals on issues of justice, governance, security, rule of law and corruption from UN agencies, which built their proposals on existing human rights standards and international treaties (UNGA, 2012a). Foreshadowing some of the technical debates to come, the Chairs noted that ‘there exist tools and indicators for measuring progress in the rule of law and governance, although there is not yet international consensus and there is a need for capacity building and assistance’ (OWG on SDGs, 2014a, p. 11). In its final session in 2014, the OWG presented Proposed Goal 16: ‘Achieve peaceful and inclusive societies, access to justice for all, and effective and capable institutions’, and Proposed Target 16.5: ‘by 2030 provide equal access for all to independent, effective, and responsive justice systems and promote the rule of law’ (OWG on SDGs, 2014d).
Simultaneous to the OWG sessions, a push for inclusion of justice as a stand-alone goal came from both inside and outside the UN system. The United Nations Office on Drugs and Crime (UNODC) argued for a separate goal on the rule of law, justice and security, setting forth proposed goals, targets and indicators (UNODC, 2013). A broad consultation with civil society, the private sector and the global ‘public’ was also held to elicit priorities for the new framework, culminating in the report, A Million Voices: The World We Want (United Nations Development Group, 2013). By the end of 2014, some 11 million people had engaged in consultations about their priorities for the post-2015 framework (TAP Network 2016). The World We Want called for ‘Accountable, inclusive institutions and access to justice’ with a move toward ‘an inclusive and people-centered approach to policy and development’ (United Nations Development Group 2013, p. 16). In addition, a global network of more than 300 legal empowerment and access to justice organizations mobilized for inclusion of justice in the post-2015 framework through advocacy initiatives at national and international levels and by piloting different approaches to monitoring and measuring justice (Namati, 2015a, 2015b).
In September 2015, world leaders unanimously adopted the 17 SDG goals and 169 targets suggested by OWG with minor modifications. SDG 16 was included as an explicit goal on access to justice for all (UNGA, 2015; Jandl, 2017).
Treasuring what we measure: existing data, technical expertise, resource inertia, and the 16.3 indicators
While the goals and targets for the post-2015 framework were being developed, the global indicator framework was also under construction. The double-track process was seen as necessary due in part to skepticism that specific goals not present in the MDGs – like justice – were truly measurable (UNDP and OHCHR, 2013; No. 5, interview 12 June 2018). A highly contested debate on the appropriate indicators for SDG 16.3 ensued, culminating in a decision that narrowed the capacious concept of access to justice through adoption of indicators capable of measuring only a very incomplete version of access to justice.
Indicators development process
In January 2013, the UNSTT created a Working Group on Monitoring and Indicators, whose role was to assemble lessons learned during the MDG era concerning quantitative target-setting and monitoring through indicators (UNSTT, 2013). In March 2013, the UN Statistical Commission (UNSC) called for a Friends of the Chair (FOC) group which included 22 National Statistical Offices (NSOs), UN agency statisticians and others to ‘develop broader measures of progress’ and to ‘ensure that a robust statistical measurement approach is incorporated from the outset’ (UNSC, 2013, p. 19). The FOC underlined the importance of a ‘data revolution’ to improve the quality of statistics while emphasizing that the inclusion of new data sources was possible so long as they could meet the standards required of ‘official statistics’ (FOC, 2013).
Meanwhile, the Working Group on Monitoring and Indicators collected and analyzed inputs from more than 60 UN bodies, releasing a report in July 2013 on Statistics and Indicators for the post-2015 development agenda (UNSTT, 2013). The report included criteria for indicator selection, recommending that indicators be ‘clear, concise and objectively measurable’ and ‘easy to understand, [using a] numerical scale for measurement’, and ‘capable of aggregation to represent global trends and regional trends’ (UNSTT, 2013, p. vi). A similar sentiment was advocated by UN agencies such as UNODC, which argued for indicators that were ‘well established, data are readily available and related methodological issues are largely resolved’ (UNODC, 2013, p. 10).
At its 46th Session in March 2015, the UNSC created two bodies that were central to the construction of indicators for SDG 16. First, the Commission created an Inter-Agency and Expert Group on SDG indicators (IAEG-SDG) (UNGA, 2015). The IAEG-SDG was composed of NSO representatives as members, and international agencies willing to support monitoring as observers and charged with developing global indicators to measure progress toward the SDGs (UNSC, 2015). At the same session, the Commission approved the creation of the Praia Group on Governance Statistics (Praia Group) to advance the capacity of national and regional bodies to measure governance, peace and the rule of law; this expert group was also influential in discussions about the SDG 16 indicators (UNDP, 2015).
Technical debates and power plays: the development of access to justice indicators
16.3.1 Proportion of those who have experienced a dispute in the past 12 months and who have accessed a fair formal, informal, alternative or traditional dispute mechanism
16.3.2. Percentage of total detainees who have been held in detention for more than 12 months while awaiting sentencing or final disposition of their case
The proposed indicator for 16.3.1 lacked any pedigree within the UN system. It had powerful backers, however, in the World Bank, the UN Peacebuilding Support Office of UNDP, and the UN Technical Support Team, as it had been developed during a June 2013 expert meeting in which these agencies were key participants. This set of agencies and international organizations saw access to civil justice as integral to achieving sustainable development (No. 7, interview 22 June 2018).
The agencies were asked to state their preference for one priority indicator per target, to categorize that indicator within one of three tiers, or to propose a new or modified indicator (IAEG-SDGs, 2015a, 2015b). Agencies were to choose Tier 1 if their proposed ‘indicator is conceptually clear, has an internationally established methodology and standards are available, and data are regularly produced by countries for at least 50 per cent of countries and of the population in every region where the indicator is relevant’; Tier 2 if their proposed ‘indicator is conceptually clear, has an internationally established methodology and standards are available, but data are not regularly produced by countries’ and Tier 3 if ‘no internationally established methodology or standards are yet available for the indicator, but methodology/standards are being (or will be) developed or tested’ (UNSD, 2015e). Groups such as the World Bank, PBSO and TST marked 16.3.1 as a Tier 2 indicator, referencing household surveys as the data source and noting that such data was available for 107 countries (IAEG-SDGs, 2015a; No. 7, interview 22 June 2018; Jandl, 2017).
However, UNODC, an agency viewed by states in the UNSC as the institutional experts on criminal justice, advocated significantly narrowing 16.3.1 (No. 5, interview 12 June 2018; No. 7, interview 22 June 2018). Framed in terms of selecting indicators that were ‘reliable and feasible’ to measure, UNDOC lobbied for the indicator to be limited to victims of violence and focused on those who reported the victimization to competent authorities (crime reporting rate) (No. 5, interview 12 June 2018). In advocating to become the custodian agency for 16.3 indicators, UNODC contended that the agency already tracked the relevant data in a number of countries and could begin producing numbers and analysis immediately (No. 5, interview 12 June 2018).
Custodian agencies play a significant role in the interpretation and harmonization of data: they are ‘responsible for compiling and verifying country data and metadata’, ‘developing international standards and recommending methodologies for monitoring’, ‘strengthening national monitoring and reporting capacity’ and filling data gaps through ‘estimates or adjust[ments of] the data’ (UN Water, n.d.).
The move by UNODC to narrow the scope of 16.3.1 and petition for custodian agency status was seen by some as a strategy to ensure ‘institutional protection’, whereas UNODC viewed it as an opportunity to be ‘relevant’ and ‘provide evidence to see if we are moving in the right direction’, with a hope that ‘funding will come’ (No. 5, interview 12 June 2018; No. 7, interview 22 June 2018). The agency classified the crime reporting rate indicator as Tier 2 and referenced victimization surveys as the primary data source and its collection of data on crime reporting rate through its annual data collection survey, the UN Survey of Crime Trends (UN-CTS) available in approximately 35 countries (IAEG-SDGs, 2015c).
As to 16.3.2, UNODC suggested modifying the indicator to ‘unsentenced detainees as a percentage of overall prison population’ and marked the indicator as Tier 1 referencing prison administration as the primary data source. UNODC noted that it had prison data available through UN-CTS on 114 countries and that the coverage could increase to 184 countries if data from research institutions and NGOs were included, signaling at least a limited embrace of third party sources (IAEG-SDGs, 2015b).
At the June 2015 IAEG-SDG meeting, the initial tier ratings, comments and counterproposals (where applicable) were presented (UNSD, 2015a). At the meeting, two discussion streams were created on the proposed indicators, one focused on ‘conceptual frameworks and indicator concepts and definitions’ and a second centered on ‘identifying interlinkages across Goals and targets’ (ECOSOC, 2015, para. 11).
16.3.1. Percentage of victims of violence in the previous 12 months who reported their victimization to competent authorities or other officially recognized conflict resolution mechanisms (also called crime reporting rate) (UNSD, 2015f).
UNODC's proposal for 16.3.2 was also accepted: to modify the indicator to focus on unsentenced detainees rather than those also awaiting final disposition (UNSD, 2015f).
much broader scope than just criminal justice. The Outcome Document calls for the global indicator framework to ‘preserve the political balance, integration and ambition’ of the agenda. To faithfully reflect these outcomes we should be suggesting an indicator (or indicators) that capture more fully the scope of the target, as well as the purpose of the SDGs overall. That is, an indicator that gets to the link between justice and sustainable development (IAEG-SDGs, 2015c, p. 280).
Regarding process, the World Bank, UNDP, PBSO and UNSG raised the point that the UNODC-designed 16.3.1 indicator was not included in earlier discussions, had never been subject to any Inter-Agency discussions over the preceding two years, and was not raised by UN agencies during the consultation process, which had included the NSOs. Specifically ‘[b]y putting forward an indicator outside of these processes we infer that it does not have the support of a wide range of stakeholders’ (IAEG-SDGs, 2015c, p. 281). In light of these concerns, the Virtual Network and TST submitted an alternative proposal that covered both civil and criminal disputes: ‘portion of those who have experienced a dispute in the past 12 months who have accessed a formal, informal, alternative or traditional dispute resolution mechanism and who feel it was just’ (UNSD, 2015c, p. 281).
did not start its work from scratch: the members of the network have collected a number of examples that already demonstrate both technical capacity and willingness on the part of national actors to measure and monitor issues related to Goal 16. Many countries already collect and report many of the administrative data proposed by the Virtual Network (UNDP, 2015, p. 17).
The Virtual Network underlined that for decades, national and international organizations had developed methodologies – which were often peer-reviewed – to monitor issues associated with peace, justice and institution building, and that such data was made available by groups including Transparency International, Global Financial Integrity, the World Justice Project (WJP), and Ushahidi, among others (No. 7, interview 22 June 2018; UNDP, 2015).
The Virtual Network emphasized that indicators should cover both criminal and civil justice and pushed for ‘an experiential survey indicator that was necessary to fully measure the concept of access to justice for all’ (UNDP, 2015, p. 32).
Similar arguments were raised by UNDP, the Rule of Law Division of the Commonwealth Secretariat, the European Commission, and the NSO of Cape Verde on behalf of African countries (UNSD, 2015c). The UNDP representative affirmed the need for a more inclusive indicator that took into consideration the lack of property rights of women and the fact that many gender-based violence cases go unreported (IAEG-SDGs, 2015a,). Civil society groups also argued that the UNODC-proposed indicator failed to include disputes beyond formal justice systems and lacked disaggregation by income and ethnicity – granularity needed to ‘leave no one behind’ (UNSD, 2015d). In response, civil society organizations proposed indicators that covered issues such as confidence in justice systems and public defenders provided through legal aid (UNSD, 2015d). A large number of groups expressed support for a broader indicator that included reference to general dispute resolution mechanisms as proposed by TST (UNSD, 2015d).
Fewer comments and objections were made about 16.3.2. Some pointed to the potential for perverse incentives, with the indicator possibly ‘incentiviz[ing] speedy but unjust trials,’ while another proposal suggested ‘compar[ing] crime reporting rates with conviction rates for crime’ (UNSD, 2015d). A group of chief statisticians from UN agencies working on SDG 16 came out in strong support of the proposed 16.3 indicators, suggesting that they should both be re-classified from Tier 2 to Tier 1 (UNSD, 2015c).
After the consultations, a revised list of indicators was presented at the IAEG-SDG's second meeting in Bangkok in October 2015. Based on a survey of IAEG-SDG members, the indicators were categorized as green, reserved for indicators that had been largely agreed upon; yellow, for those that could be resolved at the meeting, and grey, for indicators that warranted further discussion (UNSD, 2015e). 16.3.1 had been classified as yellow, but emerged as grey by the end of the meeting, while 16.3.2 started and finished as green (UNSD, 2015g). UNODC had significant influence on the final 16.3 indicators due in part to the agency's reputation as the institutional experts on criminal justice, and because UNODC had been the driving force behind the accepted revisions to the indicators (UNSD, 2015g; No. 3, interview 5 June 2018; No. 4, interview 5 June 2018).
Open consultations on the green and grey indicators were held in November and December 2015. Sixty substantive comments were submitted concerning 16.3.1, with the large majority pushing for inclusion of an indicator that measured access to dispute resolution mechanisms broadly (UNSD, 2015h). Despite significant commentary, 16.3.1 remained limited to criminal justice. 16.3.2 received minimal contestation, with most stakeholders supporting the indicator.
In November 2015, the IAEG-SDG submitted a revised list of proposed indicators to the UNSC which included all of the green and grey indicators (16.3.1 and 16.3.2 included). In March 2016, the IAEG-SDG report on SDG indicators was endorsed by the UNSC at its 47th session, which ‘agreed, as a practical starting point, with the proposed global indicator framework for the Goals and targets of the 2030 Agenda for Sustainable Development, subject to future technical refinement’ (ECOSOC, 2016, para. 2).
Measuring what we treasure: the omission of civil justice and the politics of knowledge
Chief among the lessons advanced by commentators on the MDG era was the warning that we must ‘measure what we treasure’, and not succumb to the pressure to ‘treasure what we measure’ simply because it was easily quantifiable (UN, 2015). There was near consensus that crucial elements of human development had been left out or distorted in the quest for quantifiable metrics (Fukuda-Parr et al., 2014). The debates over the post-2015 framework frequently included demands for indicators that were more fit for purpose. Despite the centrality of these issues, problematic dynamics observed in other contexts appeared. Nowhere was this more true than in relation to the access to justice indicators. This section will examine some of those dynamics, drawing on work concerning indicators as a tool for global governance (Davis et al., 2012a; Merry, 2016; Rosga and Satterthwaite, 2009).
Indicator is conceptually clear, has an internationally established methodology and standards are available, and data are regularly produced by countries for at least 50 per cent of countries and of the population in every region where the indicator is relevant (UNSDG, 2018).
We argued that the lack of a database should not restrict us. That we should first talk about what would be useful and then build a database. It was clear that we had a methodology as the Bank had used this methodology, other agencies had used other methodologies and had survey instruments and samplings. It was not an insurmountable hurdle to overcome. At that point there were [numerous] citizen surveys on justice needs. It wasn't something that was brand new, it wasn't incredibly common but it wasn't brand new. There was a method. We would have to build the global [data] set. During the MDGs, many of the indicators didn't have global [data] sets in the beginning. For example, the maternal mortality ratio (MMR) in 2000 didn't have a global [data] set; it was developed after the fact. The whole purpose of the SDG should give us the drive for new information (No. 7, interview 22 June 2018).
Despite the existence of methodologies to support a broader indicator, arguments rooted in official and available data had the effect of cabining the transformative potential of SDG 16.3 and shrunk the potential of the capacious concept of access to justice.
An examination of these justifications suggests that the preference for Tier 1 characteristics subtly shifted power back to entrenched players and officially endorsed forms of knowledge, at the expense of emerging players and forms of data gathering that could yield important insights from the perspective of rights-holders. This dynamic, in which otherwise substantive debates about what concepts, theories and ideas should be measured by a proposed indicator, is transformed into a technical discussion about measurability, data availability and uniformity, has been observed in other settings where indicators are used for global governance (Merry, 2016; Satterthwaite, 2011) This ‘rendering technical’ often hides power struggles and shuts down political debate over what matters most. This section examines the accepted global indicators against the standards embodied in the Tier 1 criteria, finding that ‘issues that recur as technical dilemmas’ in the 16.3 indicator debate ‘can be unpacked to reveal unresolved political debates’ at the heart of the development agenda (Satterthwaite, 2011, p. 874). In this case, the technical debates elided the importance of data about access to civil justice – data that could elucidate the ability of everyday people to enforce their rights.
‘Regularly produced by countries’: the focus on official data and broad coverage
The preference for widely used ‘official statistical indicators’ arose as a strong and persuasive rationale for the choice of criminal justice indicators and the exclusion of civil justice indicators. For example, the existence of ‘readily available data’ is what motivated the Group of African countries to change their position on 16.3.1, which had originally been supportive of a broader access to justice indicator proposal (No. 8, interview 28 May 2018). While not an explicit part of the ‘tier’ structure, the preference for official data is implicit in Tier 1's requirement that data are ‘regularly produced by countries for at least 50 per cent of countries and of the population in every region where the indicator is relevant’. This kind of regularity and coverage would generally be achievable only with respect to official statistics.
Further, since the indicator selection for monitoring of the SDG targets was vested with the UNSC and its IEAG-SDG, the members of such bodies dominated the indicator selection process. The UNSC's membership is drawn largely from NSOs. Unsurprisingly, this group advocated for the use of ‘official data coming from these [NSO] offices’, a move criticized by the policy and civil society members who were eager to utilize new sources of data (Jandl, 2017; Radermacher, 2013).
This emphasis could be seen as a reaction to the forceful call for a ‘data revolution’ by the High-Level Panel (HLP) at the outset of the SDG process (High Level Panel of Eminent Persons on the Post-2015 Development Agenda, 2013). While the HLP had been vague in its invocation of this revolution, a bold report followed from the Independent Expert Advisory Group on the Data Revolution for Sustainable Development (IEAG-Data), a group appointed by the Secretary-General. This IEAG-Data was very different from the IEAG-SDGs appointed by the UNSC: the IEAG-Data emphasized the twin challenges of ‘invisibility’ and ‘inequality,’ which it found together to necessitate ‘transformative actions’ related to data and knowledge, including major ‘improvements in how data is produced and used’ (Data Revolution Group, n.d.). Notably, these improvements were to include the broad use of innovative data-gathering and analytical techniques, such as the use of Big Data and the union of official and unofficial sources of data. Since knowledge is power, the data revolution threatened to move the locus of power outside of official statistical offices and into countless pockets of disbursed power/knowledge. The IAEG-SDG and the UNSC responded to this threat by returning to traditional statistical systems and reiterating the need for NSOs to be the arbiters of data for monitoring the SDGs.
in reality the discussions in the IAEG[-SDG] went beyond being purely technical discussions by statistical experts. Some observers voiced concerns that in some meetings of the IAEG[-SDG], it was representatives from the Permanent Missions who spoke on behalf of the countries, rather than the Statistical Offices who are the members of the IAEG[-SDG] (Jandl, 2017, p. 71).
The choice of UNODC as custodian agency for the 16.3 indicators cemented this move, since UNODC is an agency with a statistical mandate comprised of technical experts that collects and monitors member states’ justice statistics (UNODC Statistics and Survey Section, n.d.). The agency skews heavily toward criminal justice and routinely works directly with NSOs (UNODC Statistics n.d).; UNODC Statistics and Survey Section, n.d.). UNODC advocated to be identified as the Custodian Agency for 16.3, arguing in part that it had the needed experience and access to gather the relevant official data (No. 5, interview 12 June 2018; No. 7, interview 22 June 2018).
In the context of indicators for 16.3, the discussions tended to assert that relevant official statistics mainly consisted of administrative statistics concerning interactions with justice system institutions and criminal justice statistics gathered by official institutions – in part because these are the kinds of justice statistics that have been historically most readily available at the global level. Examining systems of civil justice would have required acceptance of non-official data – at least in the near term, as discussed below. Indicator 16.3.2 concerning the proportion of detainees who are unsentenced plainly reports data held by criminal justice agencies. It is axiomatic that governments that detain individuals must know and track their status, and UNODC has the data needed to construct this important metric for more than 140 countries (SDG16DI, 2017).
The situation was not as clear for indicator 16.3.1 which was placed in Tier 2. This indicator relies on data gathered by victimization surveys, which examine whether victims of crime have reported those crimes to the relevant authorities. While UNODC reportedly claimed during the debates that 81 countries had undertaken victimization studies (UNSDG, 2016), data publicly available through UNODC suggests that such data is available for somewhere between 33 and 37 countries, depending on which crime is used to index the reporting rate (UNODC, 2018). The recent SDG16 Data Initiative Report found that the data was available for only 37 countries with no increase since 2017 (SDG16DI, 2018). As one expert involved in the debates said, ‘if you take a close look’ at the crime reporting data, it is ‘quite patchy’ (No. 7, interview 22 June 2018). Because an insufficient number of countries regularly collect this data, UNODC is developing guidelines to aid NSOs (No. 5, interview 12 June 2018). It is possible that UNODC may have oversold the coverage rates for the crime reporting rate at the time of negotiations. Indeed, a more important factor than coverage in this discussion appears to have been the ‘official’ provenance of this data, since victimization surveys are usually undertaken by or with the close collaboration of NSOs (SDG16DI, 2017).
NSOs like to see themselves as apolitical but the work that they do is inherently political. The work of the IAEG[-SDG] is inherently political. The fact that they had to choose so few indicators is political. It's political when it comes to the engagement of the NSOs with each other. They may not want to call it negotiations, but that is what they had (No. 1, interview 5 July 2017).
An access to justice expert agreed, noting that the IAEG-SDG made decisions about what dimensions were important to measure even though this was treated as a methodological exercise (No. 2, interview 26 March 2018).
While many access to justice experts believed that broader indicators were both needed and available to supplement the criminal justice indicators, such data are often gathered by NGOs and were thus seen as suspect. Open Society Justice Initiative (OSJI) reports that of the more than 50 legal needs surveys (discussed below) that were conducted worldwide in the past 25 years, only a dozen or so were funded by a government agency (OSJI and OECD, 2018).
Despite the Tier 1 and 2 status of these global indicators, a review of reports submitted through the Voluntary National Review (VNR) process demonstrates that relatively few countries are reporting on either of the global indicators for target 16.3. Of the 64 VNR reports submitted in 2016 and 2017, only ten countries reported on one or the other of these indicators, and only three reported on both.
These low reporting rates, combined with the relatively widespread use of legal needs surveys, suggest that the confidence placed in the accepted global indicators may have been less about coverage and more about the desire to limit the production of knowledge concerning access to justice to official bodies rather than understanding access to justice from the people's perspective.
‘Internationally established methodology’: fear of new methods and sources of data
The data revolution's promise to embrace emerging sources of data such as Big Data, crowd-sourced data, and geo-located data (UNDP, 2015) failed to materialize in relation to access to justice. Global composite indicators like the World Bank's World Governance Indicators and the WJP'S Rule of Law Index were rejected because they purportedly involve complex data collection methods, increase the number of variables collected by official statistical systems, attribute arbitrary weights to sub-indicators, and are often not linked to specific policies making it difficult to monitor progress and offer policy recommendations (SDSN, 2015).
The issue of cost and capacity was also raised as a concern when reviewing the indicators, particularly the use of expensive new surveys (No. 1, interview 5 July 2017). There was an overall consensus that the SDGs should not be an open door to new mandates or require enormous amounts of new money. As a result, no additional funds or resources were on the horizon for data collection methodologies on civil justice even though indicators developed by the WJP and the World Bank contained promise and potential for large-scale study (No. 7, interview 22 June 2018). To the contrary, there was a strong push to adopt indicators that relied on existing or administrative data that would be easier and cheaper to access and analyze, including in monitoring complex targets like 16.3 (Jandl, 2017; No. 7, interview 22 June 2018). Along the same lines, on grounds of resource capacity, the IAEG-SDG was hesitant to expand the number of indicators due to the burden of reporting on each indicator (No. 1, interview 5 July 2017).
These preferences, and their power within the UN system, embody what Sally Engle Merry, observing how ‘indicator culture’ works in global governance, calls expertise and data inertia. Expertise inertia occurs when ‘insiders with skills and experience have a greater say than those without – a pattern that excludes the inexperienced and the powerless’. Data inertia arises because ‘the way categories are created and measured often depends on what data are available’ (Merry, 2016, pp. 6–7). Attending to such inertia can reveal lacunae and exclusions, since ‘[b]oth of these forms of inertia inhibit new approaches to measurement and tend to exclude inexperienced and resource-poor actors from having much influence on what is measured’ (Merry, 2016, p. 7). This dynamic played out in relation to the indicators for target 16.3, where a broad range of voices knowledgeable about access to justice were discounted in favor of the views of powerful repeat players, which won the day using arguments that took advantage of their institutional positioning and these forms of inertia to cement the rejection of a civil justice indicator and the adoption of two criminal justice-focused indicators.
‘Conceptual clarity’: the taming of ‘access to justice’
The indicators currently approved to monitor SDG 16.3 are conceptually clear from a technical viewpoint. The terms they use are not ambiguous and have a specific meaning in the context of criminal justice. While there are decisions to be made concerning which statistics to choose, a relatively small range of metrics have been used by governments across the globe to measure the crime reporting rate and the rate of unsentenced detention. However, at the broader conceptual level, there is less clarity, since these indicators are patently incapable of measuring progress toward access to justice on a variety of crucial dimensions.
Indicators often have embedded within them, or are placeholders for, a much further-reaching theory - which some might call an ideology - of what a good society is, or how governance should ideally be conducted to achieve the best possible approximation of a good society or a good policy.
In the case of SDG 16.3, the indicators evince a theory about the world in which access to justice is primarily concerned with the relationship between the state and the individual regarding criminal matters. Of course this relationship is crucially important: under human rights law, the coercive power of the state must be harnessed to protect the security of all equally, and because the state's carceral powers must not be arbitrary and must be subject to the rule of law. Thus, rising crime reporting rates and decreasing proportions of unsentenced detainees will usually signal improvements in access to criminal justice.
Indeed, the accepted global indicators can play an important role in efforts to make criminal justice systems more fair and responsive. 16.3.1 will help illuminate whether individuals who have experienced crime feel it is worth their while, safe, and wise to report crimes such as robberies, assaults or sexual assault to the authorities. Low reporting rates signal serious problems, including distrust of the police, discrimination within the justice system and barriers such as poverty, marginalization or reprisal for those who would report crimes absent those obstacles. As OSJI has noted, this kind of data ‘promote[s] understanding of crime and violence from the perspective of citizens as opposed to institutions,’ which is essential for human rights (OSJI and OECD, 2018, p. 2).
16.3.2 focuses on one of the most intractable problems facing administration of justice worldwide: the challenge of protecting against prolonged pre-trial/pre-adjudication detention. In many countries, the number of unsentenced detainees far surpasses the number of detainees who have been convicted, and not infrequently those awaiting trial serve more time in pre-trial detention than they would have been eligible for once convicted (OSJI, 2014). Ensuring that countries track this phenomenon will act as an important incentive in itself (OSJI, 2016). Disaggregation of data for both 16.3.1 and 16.3.2 will be especially helpful for identifying and confronting disparate impacts that are the hallmark of economic and political inequalities and embedded discrimination. In this way, criminal justice indicators have a ‘natural constituency’ made up of communities subject to over-incarceration, victims of crime in under-responsive justice systems and policy makers who use these statistics to measure reform efforts (No. 5, interview 12 Jun3 2018, No. 3, interview 5 June 2018).
[A]ccess to justice is broadly concerned with the ability of people to obtain just resolution of justiciable problems and enforce their rights, in compliance with human rights standards, if necessary, through impartial formal or informal institutions of justice and with appropriate legal support.
This concept of access to justice is significantly more complete than the rationale grounding the current indicators. Further, this concept is the one that best links justice to human development. As one expert involved in the drafting of the 2008 report of the Commission on the Legal Empowerment of the Poor explained, ‘Justice is a thread that runs through all the sustainable goals. Every SDG will require accessible justice services … not including civil justice was a missed opportunity’ (No. 6, interview 15 June 2018).
about a methodology, but it's also about how people need to be able to play a role in identifying what their needs are and what the solutions are – and that's hard for states to swallow. This is an empowerment perspective that is really hard for most states to wrap their arms around – legally, logistically and financially (No. 3, interview 5 June 2018).
If one or more additional indicators are not adopted to measure the missing elements of access to justice, the criminal justice indicators now used to monitor SDG 16.3 may have serious ‘knowledge effects’ like those identified by Fukuda-Parr et al. (2014, p. 2) in relation to the MDG indicators, in which ‘the indicators intended to reflect a concept effectively redefine it’. Further, though the 2030 Agenda is not a legal instrument, ‘it can be expected to eventuate the legalisation of some of the norms it sets out and strengthen those that are already codified elsewhere’ (Háa′k et al., 2016) . Redefining access to justice in such a cramped manner would signal an abject failure to learn the lessons of the MDG era and shut down the transformational potential of SDG 16.3.
The world inside a number: indicators for access to civil justice
At the time of writing, ‘access to civil justice’ is one of several dozen ‘proposed additional indicators’ under consideration for adoption by the IAEG-SDG in 2019. This indicator is intended to assess the essential ability of regular people to obtain resolution to everyday legal problems such as land disputes, evictions, divorce and labor grievances. Monitoring access to justice in these civil realms requires reliance on data beyond administrative or criminal justice statistics, including data recording people's views about their experience of broader systems of dispute resolution. This section will examine the availability of data for this proposed indicator, focusing on legal needs surveys (LNS) as a promising source for this vital data. Not only do LNS provide data now missing from the SDGs, they also redefine access to justice from the perspective of individuals seeking to enforce their rights, thus promising potentially democratizing knowledge effects. This reorientation is crucially needed to ensure that the SDGs can capture information calibrated to assess legal empowerment – the extent to which individuals can use the law to enforce their rights in a way that feels just.
For the past 25 years governments, NGOs, foundations and legal service providers in countries across the world have engaged in research efforts to understand people's ability to obtain civil justice through legal need surveys (Pleasence, 2016). Indeed, by ‘the end of 2018 legal needs studies will have been conducted in more than 100 countries from every region’ (OSJI, 2018). These household surveys aim to assist governments, civil society organizations, and multilateral institutions to better understand the nature and scope of common legal problems, the strategies people adopt to solve those problems, and the barriers they face in advancing their rights.
Legal needs surveys thus provide the ‘big picture’ of citizens’ efforts to access justice; a picture that cannot be obtained through any other means. They are therefore ideally suited to quantifying problem experiences across populations, mapping patterns of problem resolution behavior, and illuminating changes in patterns of experience and behavior over time. They are also suited to identifying obstacles to justice, from the citizen perspective; as well as providing insight into levels of legal capability (including legal understanding, awareness of services, legal confidence, resilience, etc.) and attitudes toward justice and the justice system … And crucially, they provide a citizen perspective on access to justice; a perspective unusual in the justice context (Pleasence, 2016, p. 4).
Although LNS have been designed differently across jurisdictions, there is significant methodological consistency in the surveys, which use ‘shared research questions concerning citizens’ experience of justiciable problems, the support they obtain, the processes that are involved in resolving problems and problem outcomes’ (OSJI and OECD, 2018, p. 4). The surveys have documented the ubiquitous nature of justice problems, evidence that poor and marginalized groups suffer from inequalities in accessing justice, and the impact that justice problems have on other social and economic difficulties (Pleasence, 2016, pp. 9–11). In places where these surveys have been carried out in conjunction with access to justice efforts, these findings have often catalyzed governments to develop more equitable justice policies.
The lack of a uniform definition of ‘legal need’ and the absence of harmonization in the survey methodologies employed to measure legal needs around the world currently limit the ability to compare the findings from different surveys (Coumarelos et al., 2006, p. 3; Curran and Noone, 2007). For example, it is difficult to draw firm conclusions about the relative incidence and resolution rates of specific legal issues in different jurisdictions. Legal problem categories are not necessarily identical in different countries (Coumarelos et al., 2006, p. 5). Further, although the incidence of legal problems would tend to be higher in the studies involving more disadvantaged samples, attitudinal differences in how respondents conceptualize disputes may be responsible for a lower incidence of legal problems in some settings (Genn and Paterson, 2001; Coumarelos et al., 2006). In China, Michelson found that political connections and regional area affected the experience of legal problems and the lodging of official complaints. He suggested that historical, economic and social contexts affect disputing behavior (Michelson, 2007a, 2007b). Similarly, Murayama noted that cultural and institutional factors have been proposed to explain the lower litigation rate in Japan compared to Western countries (Murayama, 2007).
These examples of incommensurability may seem to suggest that LNS data are not sufficiently comparable for global monitoring. While some of the incommensurability stems from the fact that these surveys have not been implemented on a global scale, some issues will remain even if a common methodology were adopted across borders. It should be noted that similar problems of commensurability exist in many global indicators due not only to differing attitudes, but also to the varied historical, economic, social and institutional contexts of different countries. The attention to these issues is welcome and the clarity about the limits of an indicator's ability to shed comparative light on a specific issue across the globe can be beneficial. Indeed, it may be that LNS are sufficiently harmonious to allow for broad cross-country comparisons but not so uniform that they flatten substantive and procedural differences in legal problems, resources and systems.
The need for careful interpretation of legal needs data may also be seen as a drawback. However, as Fukuda-Parr et al. (2014, p. 5) found in relation to the MDG indicators, there were many ‘unfortunate consequences of simplification,’ especially where people were seen as beneficiaries instead of agents of change. Using LNS data can never be reduced to a technocratic exercise, since significant interpretation will always be required when examining such data.
Other limits to legal needs data could be tackled through careful collaborative work at the global level, since these limits represent problems common to survey research. For example, some issues of commensurability relate to survey design. Legal needs surveys currently use variable units of measurement, different definitions and wording of each type of legal problem, different thresholds for the inclusion of legal problems, which influences the proportion of serious problems captured across studies, and variable reference periods, which can complicate cross-study comparisons by affecting the capture, observed impact and observed resolution of legal problems (Coumarelos et al., 2006, 2012). Some LNS also suffer from non-randomized sampling, poor response rates, and response bias (Coumarelos et al., 2012; Dale, 2007). These are all common – and surmountable – challenges that arise in cross-country surveys on many issues.
In recent years, OSJI has supported research on legal needs and LNS, with a focus on measuring access to civil justice and developing equitable justice policies including the scaling up of publicly funded legal services. OSJI recently partnered with the OECD to develop robust measurement tools for civil justice, particularly in the wake of SDG 16's commitment to inclusive institutions and access to justice for all. A detailed guidance paper developed toward this effort presents good practices for developing, implementing and using LNS, and provides methodological guidance for conducting such surveys, either as a stand-alone survey or as a module in more general surveys (OSJI and OECD, 2018).
an index score of civil justice mechanisms, determined by general population polling as well as surveys given to civil justice experts who practice in the country, collected by the World Justice Project. This index is a mix of perception and experience based questions. The World Justice Project (WJP) Rule of Law Index provides original, impartial data on how the rule of law is experienced by the general public … and measure[s] how the rule of law is experienced in practical, everyday situations by ordinary people around the world (SDG16DI, 2017).
The 2017–2018 WJP Rule of Law Index contains data on 113 countries and relied on more than 110,000 household surveys and 3,000 expert surveys (WJP, 2018). The WJP also released a legal needs module survey, Global Insights on Access to Justice, the ‘first public, cross-country dataset on access to civil justice’ in 45 countries around the world (WJP, n.d.). By the end of 2018, there will be more than 100 countries covered in this study.
Despite this comprehensive data set, adoption of the WJP index score seems unlikely at this stage given the opposition to unofficial data and index scores – both core features of this indicator. While there are certain drawbacks such as a limited sample size and design as compared to LNS, with the survey administered to about 1,000 households in three ‘major urban areas’ in each country rendering intra-country disaggregation and thus analysis of non-discrimination and equality very challenging, the index has a number of important advantages (OSJI and OECD, 2018).
First, because it was created by an NGO for use across the globe, it is standardized and allows for granular comparison in all of the counties where it has been administered. Questions in the survey module focus on whether people are aware of and can access civil remedies, legal services and advice; whether the civil justice system is discriminatory; whether the civil justice system is free of corruption; if there is improper government influence in civil justice mechanisms; whether there is unreasonable delay in the system; if remedies are enforced; and whether alternative dispute resolution mechanisms are accessible, impartial and effective (WJP, 2018).
The index is also quite comprehensive despite being relatively compact: it includes the essential elements of most LNS, and asks not only about individual experiences with the civil justice system, but also systemic issues such as corruption and accessibility. Critically, the WJP has shown that it is possible to collect data of this nature at a global scale. During the indicator development process, NSOs repeatedly inquired as to the existence of the relevant data, whether it was easily collectable, and the number of countries that have tested the indicator (No. 7, interview, 22 June 2018). Today WJP collects data in more than 100 countries using data-gathering modules that have been refined over years and across diverse geographies. The composite nature of the WJP Index, which one analyst characterized as a ‘good conversation starter’, lacks the granularity decision-makers need to know if policies are working. This could be overcome, however, by disaggregating the different measures within the WJP Index and using those which are most meaningful. The scope and quality of the index is an achievement that proves that is feasible to reliably and validly measure access to civil justice.
Drawing on LNS and the WJP Index surveys as a model, a relatively short and reliable module could be created for use with the major nationally representative surveys routinely carried out around the world (No. 7, interview 22 June 2018). This effort would be similar to what has been done in the water and sanitation sector, where a common module is used in the USAID-supported Demographic and Health Surveys, the UNICEF-supported Multiple Indicator Cluster Sample Surveys, and the World Bank-supported Living Standards Measurement Study. While methodological issues surely remain, wisdom could be gleaned from both the WJP staff and from countries that have been using LNS on a broad scale. For example, Argentina is implementing a LNS every two years, while South Africa has been using LNS as a tool for reforming rural legal services (OSJI, 2018).
Whichever approach is chosen, adding a global indicator to measure access to civil justice is crucial for ensuring that global monitoring of SDG 16.3 has transformative impact. In order to meet this pressing demand, a commitment of resources from donors is necessary to ensure the inclusion of justice from the people's perspective. Serious investments would enable agencies, such as UNDP or even WJP (if states are willing to accept a non-governmental agency as a custodian), with substantive knowledge and methodological know-how, to step forward as the potential custodian agency poised to provide the methodological guidance and data analysis needed to understand the lived realities of everyday people. Without such a measure, SDG 16.3 risks being severely diminished, and the official indicators could have unwanted knowledge effects over the decades to come.
Conclusions
The choice of SDG indicators has important signaling, knowledge and governance effects. These effects may be especially pronounced for issues that are newly on the global monitoring agenda. This is especially true for the core human rights issue of access to justice set out in SDG 16.3. While the official indicators for this target capture essential criminal justice phenomena, the exclusion of access to civil justice is so severe that it must be corrected. What's needed is the adoption of an indicator for which data can be gathered through a well-designed, inclusive survey module that assesses the ability of ordinary citizens to obtain resolutions to their everyday justice problems. This will require an embrace of data beyond traditional sources in order to meaningfully capture a holistic view of access to justice from the people's perspective. While many NSOs are well-equipped to take on this new task, there is no requirement that NSOs lead these efforts, so long as the surveys are executed in a methodologically rigorous manner. Indeed, for decades national and international organizations have developed and/or participated in methodologies aimed at monitoring legal needs. In some instances, the collection of data has been carried out by everyday citizens, individuals carefully trained on methodology and the underlying substance of the surveys to effectively capture the lived realities of accessing justice.
Whatever the approach chosen, adopting an indicator related to legal needs will ensure that policy-makers and development professionals have a better understanding of the legal capability of everyday citizens and where they are experiencing barriers. Crucially, this data will also allow the people to hold their own governments accountable for delivering justice – or failing to do so. Only once those challenges are targeted and measured will we be able to build equitable and dignified justice systems for all.
Notes
This paper benefited from support to the special issue project from: Friedrich-Ebert-Stiftung New York Office; UNDP; University of Oslo Centre for Environment and Development and the Environment; Julien J. Studley Grant to The New School Graduate Programs in International Affairs.
Biographies
Margaret L. Satterthwaite is Professor of Clinical Law, Faculty Director of the Bernstein Institute for Human Rights and the Center for Human Rights and Global Justice, and director of the Global Justice Clinic at NYU School of Law.
Sukti Dhital is a human rights lawyer and the Deputy Director of the Bernstein Institute for Human Rights at NYU School of Law. Previously she was the Executive Director and Co-Founder of Nazdeek, a legal empowerment organization committed to bringing access to justice closer to marginalized communities in India.
