Judge-Jury Agreement in Criminal Cases: The First Three Years of the Korean Jury System

Abstract

This study examined jury trials conducted during the first three years since the introduction of a new jury system in South Korea. Case information from all jury trials held during the time was collected and empirically analyzed with a focus on judge-jury agreement. The statistical analyses were guided by previous studies (Eisenberg et al. 2005; Spencer 2007). Results indicated that judges and juries agreed on the verdict 91.4 percent of the time (70.3 percent for conviction and 21.1 percent for acquittal). When they disagreed, juries had a greater tendency to acquit than did judges (7.4 percent and 1.2 percent, respectively). Neither evidentiary strength nor complexity had any systematic impact on agreement rates. Judges were more likely than juries to convict across all levels of evidentiary strength. The accuracy and error rates of jury verdicts were assessed by estimates of conditional probability.

I Introduction

In January 2008, South Korea instituted a jury system for the first time in the country's history (Act for Civil Participation in Criminal Trials 2007). The Korean jury system is similar to the U.S. equivalent in some respects: a panel of ordinary citizens is selected to serve as trier of fact and jurors deliberate without the presence of the judge before rendering a verdict. However, the Korean system has some unique features: jury trials are held only for serious criminal cases; each jury is composed of five, seven, or nine jurors in proportion to the seriousness of a crime; a unanimous verdict is required in principle, but if unanimity is not feasible, jurors can reach a verdict by a majority vote after consulting the presiding judge; if the jury finds the defendant guilty, jurors also make a sentencing recommendation in consultation with the judge. This new system grants Korean citizens an unprecedented opportunity to participate in legal decision making in serious criminal cases. It is, however, limited in a few ways: the defendant must express his or her intent to be tried by jury and, most importantly, the jury's verdict is merely advisory; only the judge's verdict is legally binding. The advisory nature of jury verdicts at present reflects the tentative status of the new jury trial system in Korea. The system's survival depends in part on how successful it is perceived to be during the first several years. If juries' legal decisions are generally considered sound and valid, it is much more likely that the jury system will continue to be an important part of the Korean criminal justice system.

At the heart of concerns and even skepticism about the validity of Korean jury decision making lies the problem of jury competence. Some Korean law professionals have voiced their doubts about the capacity of lay citizens to decide criminal cases in accordance with the facts and the law (e.g., Hwang 2010). They contend that jurors, who do not have legal education and training, are not sufficiently competent to decide according to the law. Critics often go a step further and argue that jury trials, which originated in common-law countries, are not appropriate to the Korean judicial system, which follows the civil-law tradition. Proponents of jury trials, on the other hand, believe that ordinary Korean citizens do have the capacity to serve competently as factfinders in criminal cases. Proponents also stress the social and political benefits of jury trials—specifically, that they will enhance the legitimacy and transparency of and public confidence in the trial process and the legal system as a whole (Ahn & Han 2005; Kim 2003).

This debate, although new to Korea, is hardly new to jury researchers in common-law countries, where the issue of jury competence has drawn a good deal of attention and discussion from legal scholars and practitioners for several decades. In particular, some researchers have taken an empirical approach to the issue. Two representative studies, for instance, have investigated how juries' verdicts differ from those of judges. The classic Chicago Jury Project (Kalven & Zeisel 1966) and, more recently, a replication of this project employing a new criminal case database assembled by the National Center for State Courts (Eisenberg et al. 2005) have revealed high rates of agreement between judges' and juries' decisions. Inspired and guided by these studies, we have drawn on actual records of all the jury trials conducted in Korea during the first three years of the new system (2008–2010) to examine the nature of jury decision making in Korea and, specifically, to compare rates of judge-jury agreement with those found in the United States.

Systematic empirical research on the issue of jury competence began with the pioneering work of Harry Kalven, Jr. and Hans Zeisel. In the 1950s they conducted a large-scale survey that included 555 judges across the United States who provided information about 3,576 criminal cases. The researchers asked judges how they would have decided the cases over which they presided if they had been in the position to make the verdict. The judges also provided detailed information about the cases, including a description of the case, the nature of the evidence, the jury's verdict, and reasons why the judge's verdict differed from the jury's (if it did). Kalven and Zeisel reported a high agreement rate between judges and juries: the judges indicated that they would have rendered the same verdicts as the jury did 78 percent of the time (Kalven & Zeisel 1966). Disagreement was asymmetric in that juries tended to acquit when judges would have convicted much more often than juries tended to convict when judges would have acquitted (19 percent and 3 percent, respectively). Kalven and Zeisel suggested that the judge-jury disagreement did not result from juries' misunderstanding of the evidence; instead, they attributed the disagreement mainly to different interpretations of the “beyond a reasonable doubt” standard, different evaluation of the evidence, jury sentiments toward the defendant, and the jury's disagreement with the law (Hans & Vidmar 1991).

Kalven and Zeisel's study made significant contributions to understanding jury competence and inspired other researchers to undertake empirical work on related themes (Chesterman et al. 2001; Eisenberg et al. 2005; Goodman-Delahunty et al. 2008; Heuer & Penrod 1994), but it has been criticized on a number of methodological grounds. For example, half the cases analyzed in their study were provided by only 15 percent of the judges who participated in the survey (Pennington & Hastie 1990). In addition to the problems of sampling representativeness, the source of the data was also arguably biased because all the survey responses came from judges. Only their views were considered in the attempt to explain judge-jury disagreement. In addition, the survey responses represented the judges' post hoc judgments and explanations made a considerable time after the cases were heard. Judges were unlikely to raise questions about the cases over which they presided and their recollections of case information may have been biased in a direction consistent with the jury's verdict.

More recently, a new empirical study, part of a large research project on hung juries (Hannaford-Agor et al. 2002), sought to replicate the findings from the Kalven and Zeisel study using the criminal case database at the National Center for State Courts (NCSC) (Eisenberg et al. 2005). Some methodological improvements are apparent in this study; for instance, the average response rate was a remarkably high 89 percent (Eisenberg et al. 2005:177) and multiple groups of people involved in the trials (i.e., judges, attorney, and jurors) participated in the survey. Overall, the study successfully replicated Kalven and Zeisel's major findings. Eisenberg and colleagues reported a judge-jury agreement rate of 75 percent, compared to the 78 percent found by Kalven and Zeisel (Eisenberg et al. 2005:181), and they found, as Kalven and Zeisel had, that juries acquitted when judges would have convicted much more often than juries convicted when judges would have acquitted (19 percent and 6 percent, respectively). Eisenberg and his colleagues (2005) conducted additional analyses to explain the judge-jury disagreement. By taking evidentiary strength as a moderator, they found that judges showed a greater tendency to acquit than did juries when judges regarded the evidence of guilt as weak, whereas judges had a greater tendency to convict than did juries when judges viewed the evidence of guilt as strong. In addition, judges tended to convict more than juries when the evidentiary strength was considered to be medium. The researchers also suggested that neither evidentiary nor legal complexity influenced rates of judge-jury disagreement.

Extending the empirical investigations of jury competence cross-nationally, the present study aims to report and analyze the results of jury trials in South Korea with a special focus on judge-jury agreement. In addition to replicating previous studies, we also made an effort to enhance methodological rigor. First, this study includes all the cases that were tried by Korean juries during the first three years since the introduction of the new jury system. No study has ever covered all the jury trials in a certain period of time. This avoids problems of sample representativeness and sampling adequacy. 1 Second, the relevant law in Korea facilitates the independence of judges' judgments from those of juries; because the jury verdict does not have a binding effect, the judge is obliged to make his or her own verdict on the same day that closing arguments are made (Act for Civil Participation in Criminal Trials 2007:Article 48). This helps avoid the cognitive biases to which judges in the previous research, who had to “remember” the details of the trial after a long lapse of time, may have been subject. Third, judges' responses in the former studies represented hypothetical verdicts (not actual verdicts), for which the judges were unlikely to be held accountable. In contrast, the judges' verdicts in the current study represent their actual, legally binding decisions. Korean law mandates that the judge make a verdict in jury trials independent of the jury's decision, since, as noted, the jury verdict is only advisory. In addition, if the judge's verdict differs from the jury's, the judge ought to provide explanations for the defendant (e.g., through an oral communication) while documenting in an official statement the reasons for rendering a different verdict. This requirement enables a more thorough examination of whether and how the jury's verdict differs from the judge's.

In addition to using rigorous methodologies, the present study has unique elements that are new to this area of research. First, we attempted to go beyond the legal judgments of the defendant's guilt or innocence to explore the extent of judge-jury agreement regarding the arguments of defendants, for instance, who plead guilty but contend that their reduced capacity at the time of the crime should mitigate the sentence. The analysis of this additional layer of judgments permits a more comprehensive understanding of how judges and juries differ in making legal decisions. Furthermore, we elaborated on how the number of cases should be counted. In fact, “cases” could be counted either by the number of defendants or by the number of charged crimes. We have created a set of rules to resolve this issue, which is described below.

II Overview of the Study

In the present study, we collected and analyzed data from jury trials held in South Korea from January 2008 through October 2010. During this period of time, a total of 321 jury trials were held (64 in 2008, 95 in 2009, and 162 in 2010). However, because we could not obtain access to trial information for the last two months of 2010, the current study analyzed data from 274 jury trials until the end of October 2010. These 274 trials yielded a total of 323 “cases” (as defined below) for the purposes of the study. We collected case data from several types of archival sources and then conducted a series of statistical analyses on the collected data to explore rates of agreement and disagreement between judges and juries, reasons for disagreement, and the estimated level of accuracy of judges' and juries' judgments.

III Method Used for Data Collection

A Cases and Counting Rules

We collected trial information about 323 “cases” from 274 criminal trials in which lay citizens participated as legal decisionmakers from January 2008 through October 2010. The major sources of the data are described below. Most of the materials are available online, whereas some are available only for research purposes. 2

We developed a set of rules for counting the number of cases so that we could perform more accurate and meaningful analyses of the data. First, each defendant who agreed to trial by jury constituted a separate “case.” Because multiple defendants may be involved in the same trial and different verdicts rendered for each, counting by defendant rather than by trial allows for more precise analysis.

Second, when a single defendant was charged with multiple crimes and the judge and jury agreed on the verdict as to one charge but disagreed as to the other, 3 or when judge and jury agreed as to both charges but convicted the defendant on one while acquitting on the other, the trial was coded as two “cases”: one of judge-jury agreement and one of disagreement, or two cases of agreement (one conviction and one acquittal), respectively.

Third, if judge and jury agreed as to the defendant's guilt but disagreed as to whether to accept the defendant's argument, for instance, that he or she had reduced mental capacity when committing the crime, this would also count as two “cases”: one of judge-jury agreement as to guilt, and one of judge-jury disagreement as to the mitigation argument. 4

Finally, there is the question of how to count a case in which the defendant confesses and pleads guilty to the offense(s). While in a common-law country this situation is most likely to lead to a plea bargain instead of a full jury trial, a jury trial can still be held (for sentencing) in South Korea because plea bargaining is not currently permitted. As mentioned above, it is a unique feature of the Korean jury trials. We decided to include such cases in the analyses although statistical analyses are conducted with and without the cases in which the defendant confessed.

B Materials and Dependent Measures

The major sources of our data included archival records and other materials, including ruling records, court statements, and a jury trial white paper (published by the court). Through these materials, we were able to obtain a good deal of the data needed for the study, such as basic information about each case, the key issues in dispute, jury information (e.g., number of jurors and their demographic information), juries' and judges' verdicts, whether the jury verdict was unanimous or majority, juries' and judges' acceptance of defendants' arguments, and the level of jurors' understanding of the trial materials. Evidentiary strength and evidentiary complexity were later assessed by the two independent raters (legally trained professionals) using the same archival records and materials. The raters, unaware of either verdict, were asked to rate the strength and the complexity of the prosecution evidence favoring conviction on a five-point scale. 5 The interrater reliability indexes were high (over 0.90). Information about the evidence favoring defendants was incomplete and mostly unavailable.

IV Results of Data Analyses

The collected case information was encoded into electronic data files and statistically analyzed to answer the questions raised regarding judge-jury agreement in legal decision making. Following the analyses of the collected data, probability models with latent classes were tested using the same data to estimate the accuracy of jury verdicts. The results of the analyses are presented below.

A Preliminary Analyses

1 Defendant's Dispute on Guilt

Among the 323 cases analyzed, the defendant pleaded not guilty in 256 cases (79.3 percent). In these cases, roughly four out of every five, a trial on guilt or innocence occurred. For the rest of the cases (20.7 percent), trial was held for purposes of sentencing only.

2 Number of Jurors

According to Korean law (Act for Civil Participation in Criminal Trials 2007:Article 13), the number of jurors can be five, seven, or nine in proportion to the seriousness of the crime. For instance, if conviction can lead to a death penalty or life imprisonment, as in the case of murder or manslaughter, a nine-person jury is used. For all other serious crimes that can be tried by jury (e.g., robbery, assault, rape, theft, etc.), seven-person juries are used. A five-person jury can be used when a defendant admits his or her guilt. Among the 323 cases analyzed, a nine-person jury was used for 124 cases, a seven-person jury for 162 cases, and a five-person jury for 37 cases. The average number of jurors across all the jury trials analyzed was 6.96. 6

3 Decision Rule

Under the current Korean law (Act for Civil Participation in Criminal Trials 2007:Article 46), in principle, jurors must unanimously agree on the verdict. However, if unanimity is not likely, jurors are allowed to reach a verdict by a majority vote after consulting the presiding judge (in order to avoid a hung jury). 7 The results of data analyses indicated that unanimous verdicts were reached in 265 cases (82.0 percent). In addition, the unanimity rate was higher for cases in which the judge and the jury arrived at the same verdict (83.1 percent) than for the cases in which they reached different verdicts (71.4 percent). The proportions of unanimity were 83.8 percent and 82.8 percent when both judges and juries acquitted and convicted, respectively. The proportion of unanimity was 70.8 percent when juries acquitted and judges convicted, while it was 75.0 percent when juries convicted and judges acquitted. Another preliminary analysis indicated that judge-jury agreement rates were 92.4 percent for the 265 cases in which unanimous verdicts were rendered and 86.2 percent for the 58 cases in which verdicts were made by the majority of jurors. We did not observe any systematic differences when the agreement rates in the unanimity and majority cases were further analyzed by evidentiary strength and evidentiary complexity.

4 Conviction/Acquittal Rates

Juries convicted less frequently than judges did. Juries convicted the defendant in 71.5 percent of all cases, while the judges convicted in 77.7 percent of the cases. In fact, the conviction rate in all jury trials was much lower than the conviction rate in bench trials during the same period (97.7 percent in 2008 and 96.8 percent in 2009). In addition, the conviction rates in Korea before the jury system was introduced were 98.7 percent in 2005, 98.4 percent in 2006, and 98.1 percent in 2007. 8

A simple comparison in conviction rates between jury trials and bench trials (before the introduction of the jury system) would lead to a misleading notion that juries are too lenient toward the defendants. In fact, critics of jury trials in Korea refer specifically to conviction rates when questioning the validity of juries' legal decisions.

However, this finding should be interpreted with caution; the data do not imply that defendants are more likely to be acquitted in jury trials than in bench trials. First, the vast majority of defendants in Korean bench trials plead guilty. The defendant's guilt is never in dispute for most criminal cases tried by the bench. By contrast, only about 20 percent of defendants in jury trials plead guilty. As a result, it is not surprising to observe a lower conviction rate in jury trials. In addition, when a defendant is charged with multiple offenses and found guilty of any of them, Korean courts typically count it as simply a conviction. Since a defendant's acquittal on the other charges is not reflected in the government statistics while our methodology for counting cases would capture it, the comparison of the conviction rates we found in jury trials to those reported by the government in bench trials may overstate the actual difference.

The conviction rate of approximately 80 percent found in our study is in stark contrast to the extremely high conviction rate in a neighboring country, Japan. In May 2009, Japan reimplemented a unique jury system, and the conviction rate in Japanese lay participation trials through October 2010 was reported to be over 99 percent; only five out of about 1,300 defendants who were tried by jury were acquitted (Fukurai 2011).

The conviction rate found in this study, however, is not unusual when compared with those in the United States as found in previous studies. The conviction rates from the Chicago Jury Project (Kalven & Zeisel 1966) and the NCSC hung jury study (Eisenberg et al. 2005) were even lower than in the present study. Because the vast majority of cases are settled by plea bargaining in the United States, the cases that remain to be tried by jury tend to be ones in which the defendant's guilt is seriously disputed; given that, it is not unreasonable to observe relatively low conviction rates. In the same vein, since defendants pleaded not guilty in about 80 percent of the cases in Korean jury trials, the conviction rate in Korean jury trials does not seem particularly low in comparison.

B Primary Analyses

1 Rates of Agreement on Guilt of the Defendant

The results indicated that the percentage of cases in which the judge and the jury reached the same verdict was 91.4 percent. Judges and juries agreed to convict in 70.3 percent of cases and to acquit in 21.1 percent (see Table 1). These findings are comparable to those in previous studies (Eisenberg et al. 2005; Kalven & Zeisel 1966). It should be noted again that jury verdicts are only advisory to the judge in Korea; judges can (and must) use their own discretion to decide without adopting the jury's verdict, and the judge overrules the jury's verdict in every case in which they disagree. This makes the very high agreement rate all the more striking. Of the remaining 8.6 percent of cases in which juries and judges disagreed about the defendant's guilt, juries acquitted when judges convicted 7.4 percent of the time; juries convicted when judges acquitted 1.2 percent of the time. This asymmetric pattern of disagreement also parallels the findings in previous studies.

Table 1. Judge-Jury Agreement on Guilt Judgment (N = 323)

	Jury Acquits	Jury Convicts
Note: The overall agreement rates are 91.4 percent when all cases are included and 89.1 percent when the cases in which defendants confessed are excluded.
Source: Korean Supreme Court Reports (2008–2010).
With all cases included
Judge acquits	68 (21.1%)	4 (1.2%)
Judge convicts	24 (7.4%)	227 (70.3%)
With cases with defendants' confession excluded
Judge acquits	68 (26.6%)	4 (1.5%)
Judge convicts	24 (9.4%)	160 (62.5%)

An additional set of computations was performed after excluding 67 cases in which the defendant pleaded guilty. Because these cases were included in the category in which both judges and juries convicted, excluding these cases did not change the case count for the other three conditions (see Table 1). It did, however, slightly change the rates of agreement: the overall percentage of agreement was 89.1 percent (62.5 percent for conviction and 26.6 percent for acquittal), and of the remaining 10.9 percent of cases in which judge and jury disagreed, juries acquitted when judges convicted 9.4 percent of the time; juries convicted when judges acquitted 1.5 percent of the time.

With the data presented above, we calculated an agreement rate that could occur by chance. When including all the cases, it was 0.619; 9 when excluding the cases with defendant confessions, it was 0.562. Because the agreement rates presented in this study were substantially higher (by 0.3) than these chance-level agreement rates, it is unlikely that the findings occurred by chance. In addition, Cohen's kappa coefficients, which indicate the level of agreement while taking into consideration the agreement occurring by chance, were 0.777 and 0.751, respectively. According to Fleiss (1981), a kappa coefficient over 0.75 is considered to indicate strong agreement. Thus, it seems apparent that the rates of judge-jury agreement we found did not occur by chance.

2 Rates of Agreement on Accepting the Defendant's Mitigation Claim

Judges and juries were in complete agreement regarding whether to accept the defendant's argument that his or her mental state at the time of the commission of the crime should be considered in mitigation of punishment. That is, for every case in which the judge accepted (or denied) the defendant's argument, so did the jury. In no case did they disagree on accepting the defendant's argument when they both convicted the defendant. As a result, the overall agreement rates of accepting the defendant's mitigation argument were the same as those presented above (91.4 percent when including all cases, 89.1 percent when excluding cases in which defendants pled guilty). Their breakdowns, however, differed. Specifically, when all cases were included, judges and juries accepted the defendant's argument 34.1 percent of the time, while they denied it 57.3 percent of the time (see Table 2). When excluding the cases in which the defendant confessed, they accepted the mitigation argument 35.6 percent of the time and denied it 53.5 percent of the time. The results suggest that both judges and juries concurrently tended to accept the defendant's mitigation claim while still finding the defendant guilty of the charged offense(s). The rates of disagreement were identical to those presented above.

Table 2. Judge-Jury Agreement on Accepting the Defendant's Mitigation Claim (N = 323)

	Jury Accepts	Jury Denies
Note: The overall agreement rates are 91.4 percent when all cases are included and 89.1 percent when the cases in which defendants confessed are excluded.
Source: Korean Supreme Court Reports (2008–2010).
With all cases included
Judge accepts	110 (34.1%)	4 (1.2%)
Judge denies	24 (7.4%)	185 (57.3%)
With cases with defendants' confession excluded
Judge accepts	91 (35.6%)	4 (1.5%)
Judge denies	24 (9.4%)	137 (53.5%)

3 Analysis of Judge-Jury Agreement by Evidentiary Strength

One further way to investigate judge-jury agreement is to break down the rates by evidentiary strength. The rates of agreement regarding (1) judgment of guilt and (2) acceptance of the defendant's mitigation claim were therefore analyzed as a function of the strength of the prosecution's evidence as assessed by independent raters.

As presented in Table 3, the agreement rates increased as evidentiary strength increased. Specifically, when the strength of the evidence was at the weakest level (Level 1), the percentage of agreement between judges and juries was 84.0 percent. As the level of evidentiary strength went up to the strongest level (Level 5), the agreement rate also generally rose (84.4 percent, 87.0 percent, 98.3 percent, and 97.9 percent, respectively). As reported by previous researchers (Hannaford-Agor et al. 2002; Kalven & Zeisel 1966), the agreement rates increased as evidentiary strength became stronger. One possibility is that the stronger the prosecution's evidence, the better able jurors were to understand and appreciate the weight of that evidence and to decide accordingly. As the evidence of conviction becomes stronger, there is less room for other factors to affect juries' judgment (Reskin & Visher 1986).

Table 3. Judge-Jury Agreement on Guilt Judgment by Evidentiary Strength (N = 323)

	Jury Acquits	Jury Convicts	Total
Note: The overall agreement rates are 84.0 percent, 84.4 percent, 87.0 percent, 98.3 percent, and 97.9 percent, respectively, as the strength of evidence favoring conviction goes from “weak” to “strong.”
Source: Korean Supreme Court Reports (2008–2010) and Jury Trial White Paper (2009).
Evidence of Conviction: Weak
Judge acquits	36 (72.0%)	2 (4.0%)	38 (76.0%)
Judge convicts	6 (12.0%)	6 (12.0%)	12 (24.0%)
Total	42 (84.0%)	8 (16.0%)	50 (100.0%)
Evidence of Conviction: Somewhat Weak
Judge acquits	26 (40.6%)	2 (3.1%)	28 (43.7%)
Judge convicts	8 (12.5%)	28 (43.8%)	36 (56.3%)
Total	34 (53.1%)	30 (46.9%)	64 (100.0%)
Evidence of Conviction: Medium
Judge acquits	6 (11.1%)	0 (0.0%)	6 (11.1%)
Judge convicts	7 (13.0%)	41 (75.9%)	48 (88.9%)
Total	13 (24.1%)	41 (75.9%)	54 (100.0%)
Evidence of Conviction: Somewhat Strong
Judge acquits	0 (0.0%)	0 (0.0%)	0 (0.0%)
Judge convicts	1 (1.7%)	58 (98.3%)	59 (100.0%)
Total	1 (1.7%)	58 (98.3%)	59 (100.0%)
Evidence of Conviction: Strong
Judge acquits	0 (0.0%)	0 (0.0%)	0 (0.0%)
Judge convicts	2 (2.1%)	94 (97.9%)	96 (100.0%)
Total	2 (2.1%)	94 (97.9%)	96 (100.0%)

In addition, both judges and juries demonstrated a similar tendency to convict the defendant as evidentiary strength became stronger (see Table 3). Although judges' conviction rates were consistently higher than those of the juries, both judges and juries were more likely to convict the defendant as the evidence of conviction became stronger, as presented in Figure 1 (24.0 percent, 56.3 percent, 88.9 percent, 100 percent, and 100 percent for judges; 16.0 percent, 46.9 percent, 75.9 percent, 98.3 percent, and 97.9 percent for juries). Indeed, when evidentiary strength was high (Levels 4 and 5), agreement rates approached 100 percent, and in every case of agreement at these levels, both judges and juries convicted. These findings seem to give at least some support to the validity of legal decisions by Korean juries. Although information about the strength of the evidence favoring defendants was not available, the strength of the prosecution's evidence apparently served as a key determinant for the juries' decisions, which in turn largely corresponded to the decisions that judges made.

Figure 1. Conviction rates of the judge and the jury in percentages as a function of strength of evidence for the prosecution (N = 323).

Source: Korean Supreme Court Reports (2008–2010) and Jury Trial White Paper (2009).

We also examined judge-jury agreement on accepting the defendant's argument (i.e., mitigation claim) as a function of evidentiary strength (see Table 4). The overall pattern was virtually identical to what was presented above, except that both judges and juries were more likely to accept the defendant's claim than they were to acquit (compared with the data in Table 3) for most levels of evidentiary strength. The rate of agreement between judges and juries at every level of evidence strength did not change.

Table 4. Judge-Jury Agreement on Accepting the Defendant's Argument by Evidentiary Strength (N = 323)

	Jury Accepts	Jury Denies	Total
Note: The overall agreement rates are 84.0 percent, 84.4 percent, 87.0 percent, 98.3 percent, 97.9 percent, respectively, as the strength of evidence favoring conviction goes from “weak” to “strong.”
Source: Korean Supreme Court Reports (2008–2010) and Jury Trial White Paper (2009).
Evidence of Conviction: Weak
Judge accepts	39 (78.0%)	2 (4.0%)	41 (82.0%)
Judge denies	6 (12.0%)	3 (6.0%)	9 (18.0%)
Total	45 (90.0%)	5 (10.0%)	50 (100.0%)
Evidence of Conviction: Somewhat Weak
Judge accepts	33 (51.6%)	2 (3.1%)	35 (54.7%)
Judge denies	8 (12.5%)	21 (32.8%)	29 (45.3%)
Total	41 (64.1%)	23 (35.9%)	64 (100.0%)
Evidence of Conviction: Medium
Judge accepts	11 (20.4%)	0 (0.0%)	11 (20.4%)
Judge denies	7 (13.0%)	36 (66.6%)	43 (79.6%)
Total	18 (33.4%)	36 (66.6%)	54 (100.0%)
Evidence of Conviction: Somewhat Strong
Judge accepts	7 (11.9%)	0 (0.0%)	7 (11.9%)
Judge denies	1 (1.7%)	51 (86.4%)	52 (88.1%)
Total	8 (13.6%)	51 (86.4%)	59 (100.0%)
Evidence of Conviction: Strong
Judge accepts	0 (0.0%)	0 (0.0%)	0 (0.0%)
Judge denies	2 (2.1%)	94 (97.9%)	96 (100.0%)
Total	2 (2.1%)	94 (97.9%)	96 (100.0%)

4 Analysis of Judge-Jury Agreement by Evidentiary Complexity

An additional analysis was conducted to determine if evidentiary complexity affected the rate of judge-jury agreement. The rationale for the analysis was that when evidentiary complexity was high, jurors might be less able to understand the evidence and consequently more likely to disagree with the judges. To test this possibility, two raters independently rated overall evidentiary complexity on a five-point scale using trial records and materials. The judge-jury agreement was then broken down by the level of evidentiary complexity.

The agreement rates were generally consistent across all the levels of evidentiary complexity. As the complexity level goes from Level 1 (low) to Level 5 (high), the agreement rates were 100.0 percent, 96.0 percent, 81.0 percent, 91.3 percent, and 93.5 percent, respectively. We did not find any indication that judges and juries tended to disagree more because of the overall complexity of the evidence. In fact, the agreement rate was relatively low when the evidentiary complexity was medium, a finding for which there seems to be no logical and theoretical explanation.

5 Estimation of Jury Accuracy

The analyses of actual jury trial results presented above would seem to be most important in understanding the nature of jury trials; however, we attempted to address the issue of jury verdict accuracy further by statistical estimation. Although the “correct” verdict cannot be known or observed in reality, the accuracy of jury verdicts can be estimated through statistical methods. Estimating jury accuracy appears to be especially important to this study as it may have significant implications for the public policy.

Statistical estimation of jury trial accuracy has been introduced in previous research (Gastwirth & Sinclair 1998, 2004; Spencer 2007). In particular, much of the estimating work in this article was done by importing statistical methods similar to what was presented in Spencer (2007:318–26), although the specific latent class model used in the present study was different, which is described below.

Four latent class models were tested using maximum likelihood estimation. Mplus 6.0 (Muthen & Muthen 1998–2010) was used in the analyses. Model 1 took three categorical variables—judges' verdicts, juries' verdicts, and evidentiary strength (recoded into three levels)—using all 323 cases. Model 2 was identical to Model 1 except that the cases with defendant confession were excluded. Model 3 was similar to Model 1, but added another variable (i.e., judgments from higher courts). Model 4 was the same as Model 3 except that the cases with defendant confession were excluded. Because all cases were included and analyzed in Models 3 and 4, one may take Models 1 and 2 as special cases of Models 3 and 4, respectively. The frequency distribution of cases for each model is presented in Tables S1 and S2 in the Online Appendix.

Probability estimates for four models are presented in Table S3 (see Online Appendix) and discussed below. In this table, we employed notations similar to those in Spencer (2007) to facilitate comparisons between the two. Specifically, the correct state for the case (not observed) was denoted by U, the judge by A, the jury by B, the higher courts by C, and evidentiary strength by D (after being recoded into three levels). Each of U, A, and B could take one of the two values; 0 for “not guilty” and 1 for “guilty.” C could take three values: 0 for “not guilty,” 1 for “guilty,” and 2 for “not reviewed by higher courts.” D could take three values (0, 1, and 2) for evidentiary strength.

For the conditional probabilities in Table S3, the notation will denote the probability that A = a. The probability that U = u is denoted by , for U = 0,1, and we have . The conditional probability that A = a given U = u is denoted by , and the conditional probability that U = u given A = a is denoted by . The overall probabilities of correct decision are and .

In the present study, we used a latent class model (presented below) to estimate the parameters. In this model, A and B represent observed variables, U represents unobserved latent class, and T represents the number of U. After entering the probabilities computed from the sample on the left term of the equation, we estimated conditional probabilities (as presented in Table S3) that satisfy the equation. The latent class model used in this study is as follows:

• 

The results of the analyses indicated that Models 1, 2, and 4 fitted well to the data (Model 1: L² = 5.68, df = 2, p > 0.05; Model 2: L² = 4.33, df = 2, p > 0.05; Model 4: L² = 33.79, df = 22, p > 0.05). However, Model 3 failed to fit to the data as indicated by the significance level (Model 3: L² = 42.82, df = 22, p < 0.01), although Model 3 was still judged to be acceptable.

Across four models, probability estimates for jury accuracy ranged from 0.90 to 0.96 (as denoted by the statistic, , in Table S3), while estimates for judge accuracy ranged from 0.94 to 0.99 (as denoted by the statistic, , in Table S3). Specifically, in Models 1 and 2, the accuracy estimates for juries' judgment were 0.95 and 0.96, which were slightly higher than those for judges' judgment. By contrast, in Models 3 and 4, the accuracy estimates for judges' judgment were higher than those for juries' judgment by 0.06 to 0.09. Such findings may be attributable to a relatively higher correlation between evidentiary strength and judges' decisions and a higher correlation between higher courts' decisions and judges' decisions.

Partially consistent with the findings in Spencer (2007), the results of the study showed that juries appeared to be more accurate than judges when the true verdict (as inferred by statistical methods) was not guilty, as indicated by greater values of relative to for Models 1 and 2. In turn, judges appeared to be more accurate than juries when the true verdict was guilty, as judges showed perfect accuracy by convicting the defendants; the corresponding statistic, , was 1.000 for all models, while ranged from 0.880 to 0.973.

The probability models also allowed us to estimate error rates in judges' and juries' judgments. When the true verdict was not guilty, the probability that juries convicted () was 0.056 for Models 1 and 2 (false conviction or “Type I error”). In contrast, when the correct verdict was guilty, the probability that juries acquitted () was 0.027 for Model 1 and 0.051 for Model 2 (false acquittal or “Type II error”). On the other hand, Type I error rates for judges () were 0.206 and 0.186 for Models 1 and 2, respectively. Interestingly, Type II error rates for judges () were invariably 0.000 for both models. These results suggest that while judges were making highly accurate decisions when the true verdict was guilty, they were more likely than juries to commit Type I errors when evidentiary strength was considered along with the judges' and juries' verdicts in the model, which replicates the results in Spencer (2007).

Furthermore, among the defendants who were found guilty by judges, the proportions of wrongful conviction () were 0.074, 0.089, 0.019, and 0.015 for Models 1, 2, 3, and 4, respectively. Such differences may suggest that wrongful conviction rates at the trial courts can be substantially improved at the higher courts. On the other hand, among the defendants found guilty by juries, the proportions of wrongful conviction () were 0.022 and 0.030 for Models 1 and 2. Simple comparisons between judges' and juries' wrongful conviction rates for the first two models indicated that the judges' rate could be lowered to 2 to 3 percent if jury verdicts had binding effects. However, it represents only a speculation based on statistical inferences.

When juries convicted, the post hoc probabilities that the defendants were actually guilty () were 0.978, 0.970, 0.978, and 0.973 for Models 1, 2, 3, and 4, respectively. However, when juries acquitted, the probabilities that the defendants were actually not guilty () were 0.932, 0.907, 0.778, and 0.764 for Models 1, 2, 3, and 4, respectively. These results suggest that whereas the accuracy of conviction decisions by juries was noticeably high and consistent across the models (97 to 98 percent across the board), the accuracy of their acquittal decisions was not as consistent across the models. On the other hand, when judges convicted, the post hoc probabilities that the defendants were actually guilty () were 0.926, 0.911, 0.981, and 0.985 for Models 1, 2, 3, and 4, respectively. These figures were moderately lower compared with the corresponding estimates for juries, particularly in Models 1 and 2.

6 Cases of Judge-Jury Disagreement

Among the 323 cases analyzed in this study, judges and juries disagreed in only 28 cases. As the number of disagreement cases was small, it would be premature to infer too much about the nature of the disagreements. Instead, we report some descriptive characteristics of the cases. The defendants in all 28 cases pled not guilty. The average number of jurors was 7.79, which was slightly higher than that for the entire group of cases. Of those 28 cases, juries acquitted (and judges convicted) in 24 cases. The average level of evidentiary strength favoring the prosecution was 2.3 out of 5, which was lower than the average for the entire set of cases (3.3). The average of evidentiary complexity was 3.1 out of 5. There was only one case for which the complexity was at the highest level (Level 5). Otherwise, there was no other case that required understanding complicated and difficult evidence, such as scientific evidence. The type of crimes varied among the cases: there were 10 cases of robbery, eight cases of homicide (murder or manslaughter), five cases of sexual assault, two cases of theft, two cases of (physical) assault, and one case of embezzlement.

One very intriguing and unexpected finding included a certain type of robbery (“quasi-robbery” if literally translated into English). Quasi-robbery 10 is a statutory offense, defined in the Criminal Law of Korea as follows: “When a larcener uses force or intimidation to resist recapture of the wealth, to avoid arrest, or to destroy the trace of the crime …” (Criminal Law of Korea, Article 335). Three out of the 28 cases concerned quasi-robberies. In all three cases, the defendants were only stealing at the beginning, apparently with no intent to rob the victims or use force (e.g., there was no indication that they brought any weapon to the site), but they happened to face the victims and eventually harmed them in the course of struggling to run away. According to the Criminal Law of Korea, a defendant charged with quasi-robbery, if convicted, is subject to the same amount of punishment as robbery or aggravated robbery—a minimum of three or five years in prison, respectively. In all three cases, the juries found the defendants not guilty of quasi-robbery, while the judges convicted. Furthermore, the juries questioned the legitimacy of Article 335 of the Criminal Law while voicing their concerns that the punishment was unreasonably harsh. These could have been the first (and provocative) cases of jury nullification in Korea, although Korean juries at present do not formally have the power to nullify the law. Nevertheless, the norms and values of the lay citizens of Korea will likely influence the practice of the law in various ways, and more such decisions by Korean juries could well affect future legislation on the subject.

V Summary and Conclusion

We have presented the results of analyses of a set of data from all jury trials conducted during the first three years (January 2008 through October 2010) since the implementation of a jury system for serious criminal cases in South Korea. Trial information from 323 cases (as counted by the rules set out in this study) was statistically analyzed, with a focus on rates of judge-jury agreement. The statistical analyses in this study were guided by previous studies (Eisenberg et al. 2005; Spencer 2007); we also used the results of those studies as a frame of reference to understand the outcomes of Korean jury trials.

Our findings appear to be importantly similar to those found in the previous research (Eisenberg et al. 2005; Kalven & Zeisel 1966; Spencer 2007). Results indicated a high level of judge-jury agreement on verdicts. When they disagreed, juries tended to be more lenient than judges. Judge-jury agreement increased slightly as the evidence of conviction became stronger, although judges were more likely than juries to convict at every level of evidentiary strength. In addition, evidentiary complexity did not affect the agreement rates in any meaningful manner.

Unfortunately, it is not yet clear why the agreement rate was particularly high in Korean jury trials. One possibility is that the independence between the judges' and juries' verdicts, although formally secured, might have been compromised in some ways. For instance, if the judge made the verdict while being affected by the jury's verdict in a way, the independence between the two judgments, from a methodological viewpoint, was violated. Alternatively, Korean jurors might have had an inkling of how the judge was inclined to decide based on judicial remarks or instructions and rendered their verdicts accordingly. Another possibility might have to do with the legal procedure in jury trials. Even if a defendant expresses his or her intent to have a jury trial, the court may still assign the case to the bench if the case is deemed inappropriate to be tried by jury. Because of this feature of the system, cases in which judges and juries were expected (by the court) to disagree might have been excluded from trial by jury, which would have boosted the agreement rate. These are, however, mere speculations and should be investigated in further studies.

In addition to the analyses of actual trial data, we tested four probability models to estimate the accuracy and the error rates of jury verdicts. The probability estimates for jury accuracy ranged from 0.90 to 0.96. Furthermore, estimates of conditional probability indicated that juries were found to be more accurate than judges when the correct verdict was not guilty (i.e., juries were less likely than judges to commit Type I error by falsely convicting innocent defendants), whereas judges were more accurate than juries when the correct verdict was guilty (i.e., juries were more likely than judges to commit Type II error by falsely acquitting guilty defendants).

It is remarkable to witness, over a span of nearly half a century, the consistency of findings regarding the patterns of judge-jury agreement. The classic jury study by Kalven and Zeisel (1966) has stood the test of time, being (at least partially) replicated in Eisenberg et al. (2005). The current study, which analyzed jury trial data from a country with very different historical, legal, political, and social contexts, yielded similar results. South Korea has long followed the civil-law tradition and Korean citizens had no opportunity to participate in making legal judgments until a few years ago. Taking the environmental differences into consideration, the robustness of the empirical findings appears to have significant implications for theory and research on jury decision making and jury systems. Questions still remain, however, about the specific reasons underlying the disagreement between judges and juries observed in this study. For instance, did Korean jurors interpret and apply the standard of “beyond a reasonable doubt” more strictly than judges did, as was often presumed in previous studies? Were Korean jurors more likely than judges to be swayed by sympathy toward the defendants? If so, under what conditions were they likely to be affected by such emotional factors? Further empirical research is needed to address these issues.

The findings from the study also suggest important policy implications for citizen participation in legal decision making in Korea. The high level of judge-jury agreement observed in this study should ease concerns about jury competence in making legal judgments according to the law. Although Korean jury verdicts do not have binding effects at present, the results of the study can help pave the way for judicial reforms that would grant lay citizens greater power in making legal decisions.

Two recent Korean Supreme Court cases are worth mentioning with regard to this issue. In one case, the Korean Supreme Court overturned a ruling by the appellate court and supported the decision from a lower court in a robbery and assault case (2009Do14065). This case was initially tried before a jury of nine citizens, who found the defendant not guilty of the charged crimes and rendered a unanimous verdict. The presiding judge adopted the verdict and ruled in favor of the defendant. The prosecution appealed the ruling to the appellate court and a bench trial ensued. Without an adequate investigation of new evidence for the case, the appellate court reversed the decision based on additional questioning of the victim only. The Supreme Court, however, overturned the appellate court and reinstated the verdict of acquittal. The Supreme Court ruled that the appellate court cannot reverse the decision from a lower court, unless a clear and convincing situation emerges through a new investigation of evidence, when (1) a criminal case is tried by a jury, (2) the jury returns a unanimous verdict that the defendant is not guilty, and (3) the bench adopts the verdict and rules accordingly. This ruling is of great importance for the status and influence of jury verdicts, particularly when one considers that those verdicts are not legally binding. In the second, more recent criminal case, the Supreme Court reconfirmed these restrictions on the appellate court's authority to reverse lower court decisions in jury trials (2010Do4450). Thus, although jury verdicts still play only an advisory role in general, this pair of Supreme Court rulings has established a new legal principle for the practice of criminal law in Korea that reflects a new respect for the role of the jury.

The Korean jury system is still in its infancy. The government will make an official evaluation of jury trials at the end of 2012, and the fate of the new system will be determined based on the results of the evaluation. Many legal professionals, harboring deep-rooted doubts about jury competence, have strongly opposed granting lay citizens the power to decide criminal cases. However, the empirical findings presented in this study appear to offer a more positive and promising story: Korean juries were able to understand the evidence and serve as legal factfinders reasonably well when compared with judges. Taken together with the recent Supreme Court cases, these findings provide some support for civil participation in deciding criminal cases. To be sure, even if the jury system settles in and becomes a part of the Korean judicial system, many procedural problems have yet to be solved: the legal effects of jury verdicts, the role of judges in guiding jury deliberation when jurors cannot reach a unanimous verdict, effective measures to prevent decision errors by juries, and so on. These and other issues call for further empirical research.

Footnotes

1. 1

   The problem of sampling adequacy still pertains in the later section in which probability models are tested. Although all the cases were analyzed, the number of cases in our analyses may not be sufficient because the population of cases is treated as having been generated by the probability models.

2. 2

   Basic information about jury trials (e.g., date, number of jurors, type of the charge(s), etc.) is available at the website of the Supreme Court of Korea (www.scourt.go.kr). Although the website has an English version, its scope is limited as the contents are selectively translated into English. Jury trial information is not available yet in the English version. In addition, some materials and documents are available for research purposes, such as a jury trial white paper published by the National Court Administration and ruling records and reports published by the Supreme Court. These materials and document are accessible only at the Court library (not available online) and do not have English translations.

3. 3

   There were 10 cases (of the 323 cases) in which a single defendant was charged with multiple crimes that resulted in different judgments. In eight of the 10 cases, juries found the defendant not guilty while judges found the defendant guilty.

4. 4

   For the 323 cases analyzed in this study, there was no case in which the judge and the jury disagreed on accepting the defendant's argument while they agreed on his or her guilt. Thus, the third counting rule may have little utility value for the present study. However, this counting rule is still important and we expect that it will be utilized in analyzing future cases.

5. 5

   Although the two raters were provided information about the evidence proffered during the trials as documented in archival records and materials, such information might not be the same as what had been presented during the actual trials, which represents a limitation of the current study. In addition, the legally trained professionals are likely to view the cases more like the judges do.

6. 6

   This average was calculated based on the number of jurors in actual jury trials conducted during the period of time (274 cases). If the number of cases was counted by the rules presented in this article (323 cases), the average number of jurors would be changed to 7.54.

7. 7

   It is still possible that jurors reach a unanimous vote after consulting the judge. Unfortunately, we were not able to obtain information about (1) the number of cases in which juries consulted the judge, although it is expected to be small and (2) the number of cases in which jurors initially split in their judgments but made a unanimous decision after consulting the judge. Such information was not available in the materials we used.

8. 8

   Such statistics were obtained from the website of the Supreme Court of Korea (http://www.scourt.go.kr).

9. 9

   These calculations are based on the proportions on the margin. For instance, for cases that include defendant confessions, 0.285 comes from adding the two percentages in the first column of the top panel in Table 1 (0.285 = 0.211 + 0.074) and 0.223 from adding the two in the first row (0.223 = 0.211 + 0.012). This leads to the following computation: (0.285 × 0.223 + 0.715 × 0.777) = 0.619. When excluding the cases with defendant confessions, the analogous calculation is 0.562 = (0.360 × 0.281 + 0.640 × 0.719).

10. 10

    Among the 323 cases analyzed in the present study, nine cases had to do with quasi-robbery. It represents less than 3 percent of the entire cases tried by jury.

Biographies

• Sangjoon Kim is Presiding Judge at Seoul High Court in South Korea;

• Jaihyun Park is Associate Professor in the Psychology Department at Baruch College-City University of New York;

• Kwangbai Park is Professor in the Psychology Department at Chungbuk National University in South Korea;

• Jin-Sup Eom is Lecturer in the Psychology Department at Chungbuk National University in South Korea.