Jose Pina-Sánchez¹, Melissa Hamilton² and Peter W. G. Tennant³

1 Introduction

A vast literature documents that offenders from different socio-demographic groups appear to be sentenced differently, suggesting a violation of the principle of equality under the law. Yet, research on sentencing disparities is well known to be fraught with methodological challenges (Baumer 2013; Wooldredge 1998). Chief amongst these is confounding bias, which arises from the difficulty ensuring that cases are truly comparable. Otherwise, disparities across socio-demographic groups cannot be confidently interpreted as unwarranted, as they may simply reflect differences in legitimate case characteristics that affect sentence severity (i.e. legal factors). This challenge stems partly from the impossibility to ethically randomise offenders by socio-demographic characteristics in real court settings.⁴ Consequently, most research in this area relies on ‘selection on observables’ designs. The problem is further compounded by two factors: i) even in jurisdictions where judicial discretion is tightly constrained, the set of legal factors that judges may legitimately consider is quite large — potentially infinite if we take the principle of individualisation at heart and accept that ‘every case is different’ (Cole 1997) — and ii) the official sentencing data made available to researchers through Sentencing Commissions has often focused on a few characteristics broadly defining the case, thereby omitting many of the legal factors considered by the judge.

The inability to control for all relevant legal factors has been recurrently highlighted as the main methodological limitation in both official government reports on sentencing disparities (Hopkins et al. 2016; Isaac 2020; United States Sentencing Commission 2012), and the wider academic literature. As Baumer (2013) notes, after interviewing 25 leading scholars in the field of race disparities in sentencing, four out of five identified ‘omitted variable bias’ as a problem having a major impact in their findings. Critics have long argued that apparent sentencing disparities across demographic groups are simply the result of group differences in the types of crimes committed (Halevy 1995; Wilbanks 1987). Their scepticism is supported by numerous studies showing that estimates of seemingly unwarranted disparities against particular socio-demographic groups are often reduced — or even disappear — once key legal factors are controlled for (Mitchell 2005; Pina-Sánchez et al. 2019). As a result, until recently the dominant modelling strategy amongst sentencing scholars has been to control for as many legal factors as possible. This advice has been stated explicitly in reviews of the literature (Baumer 2013; Pina-Sánchez and Linacre 2016; Zatz 1987) and has served as an inclusion criteria in more sophisticated meta-analyses (Ferguson and Smith 2024; Mitchell 2005).⁵

However, this view should be re-examined. In general, selecting controls simply based on their predictive power is poor modelling practice. At best, it risks overfitting, which reduces statistical power and can introduce multicollinearity. More importantly, ignoring the causal role of variables included in the model is likely to induce bias. A growing number of researchers are increasingly questioning such an approach, since many of the legal factors traditionally used as controls in sentencing research are themselves defined at the discretion of sentencers. Consequently, controlling for such factors may inadvertently adjust for the very judicial prejudice under investigation (Kurlychek and Johnson 2019; Lynch 2019), resulting in post-treatment bias. Specifically, in studies of race disparities in the U.S. federal courts, Hofer (2019) and Holmes and Feldmeyer (2024) describe a new disciplinary divide. Econometricians tend to favour more parsimonious models and avoid controlling for discretionarily defined legal factors (Fischman and Schanzenbach 2012; Starr and Rehavi 2013; Yang 2015), whereas criminologists typically control for all relevant legal factors. A key point of contention is whether to control for the offence base level or the presumptive sentence. Econometricians argue that the base level provides a more upstream and objective measure of criminality, while criminologists contend that judicial decisions shaping the presumptive sentence — such as reductions for minor participation in the offence or increases for obstruction of justice — are key legal factors that legitimately influence the final decision and should therefore be controlled for (Holmes and Feldmeyer 2024).

Both arguments are valid but mutually exclusive, leaving sentencing researchers in a ’pick your poison’ dilemma whose implications are seldom acknowledged. More troubling still, the challenge is even more complex than previously recognised: many commonly used controls in the sentencing literature can introduce additional forms of bias that have so far gone largely unnoticed. To clarify these mechanisms, we draw on directed acyclic graphs (DAGs), a transparent framework for expressing causal assumptions (Pearl 2009). DAGs serve two main purposes: i) to represent and communicate causal relationships explicitly, and ii) to establish the role of each variable within the causal process, thereby determining the appropriate set of variables to adjust for when estimating causal effects (Cinelli et al. 2022; VanderWeele and Staudt 2011). Although now widely used in fields such as public health and epidemiology (Tennant et al. 2021; Textor et al. 2016), DAGs have not yet been systematically applied to sentencing research.⁶

In the next section we provide a succinct introduction to DAGs so that unfamiliar readers can follow the arguments developed in the rest of the article. Specifically, in Section 3 we use DAGs to reconsider the causal role of explanatory variables frequently invoked in sentencing studies, grouping them according to how their inclusion or omission may introduce different types of bias. Building on this classification, in Section 4 we propose a modelling framework to strengthen the robustness of observational studies estimating unwarranted disparities attributable to direct discrimination in sentencing. We then illustrate the framework with two applications: an analysis of racial disparities among drug-trafficking offenders in U.S. federal courts (Section 5.1), and an analysis of gender disparities among shoplifting offenders in the magistrates’ courts of England and Wales (Section 5.2).

2 Representing Causal Paths with DAGs

DAGs are non-parametric causal diagrams that express assumed relationships amongst variables within a given context (Fox et al. 2024; Tennant et al. 2021). Each node represents a variable at a specific point in time, and directed arrows represent hypothesised causal links between them. Paths are formed between nodes — such as between an exposure X and an outcome Y — when they are connected by one or more arrows, and these paths may be open or closed. Open paths transmit statistical associations between variables on that path, closed paths do not.

A causal path is one in which all arrows flow in the same direction. For example, X→M→Y, where X here is the origin node, Y the terminal node, and M a chain node, because it has one incoming and one outgoing arrow. Causal paths are open and transmit causal associations unless an intermediate node is statistically controlled. For instance, controlling for the mediator M in X→M→Y closes the path and removes the causal association between X and Y via M.

A non-causal path is one in which the arrows do not all flow in the same direction. This can occur due to the presence of fork nodes, where two or more arrows diverge from a single node (e.g., C in X←C→Y), or collider nodes, where two or more arrows converge at a single node (e.g., Z in X→Z←Y). Paths that exclusively include chain or fork nodes are open and, when they connect the exposure and outcome they are known as confounding paths. Such paths may be closed by controlling for intermediate nodes, e.g. controlling for confounder C in X←C→Y would close the path and remove confounding bias between X and Y.

In contrast, paths that contain a collider are naturally closed but become open if the collider – or one of its descendants – is controlled for. For example, controlling for collider M in X→M←L→Y opens the path and introduces collider bias between X and Y. Closing this path would require controlling for another intermediate variable, e.g. controlling for mediator-outcome confounder L would close X→M←L→Y.

With this knowledge, DAGs can be used to estimate a particular causal effect from an association between two variables after appropriately conditioning on nodes to ensure that all desired causal paths are open and all non-causal paths are closed (Fox et al. 2024; Tennant et al. 2021). Figure 1 illustrates each of the scenarios described above. In the next section, we demonstrate how these structures appear in research on discrimination in sentencing and discuss their implications.

Figure 1. Representation of common causal and non-causal paths using DAGs.

3 How the Choice of Controls Affects Estimates of Discrimination in Sentencing

Researchers examining discrimination in sentencing often fail to explicitly state the causal assumptions underlying their statistical models. Figure 2 depicts the causal model typically implied in studies of this kind.⁷ It shows the central modelling challenge repeatedly highlighted in the literature. Namely, the need to control for all offence and offender characteristics that legally define the case, such as offence seriousness, the offender’s criminal history, or the entry of a guilty plea. The usual assumption is that once these case characteristics are controlled for, the estimand of interest — generally, the direct effect of the offender’s socio-demographic attribute (e.g., race, gender, nationality) on sentence severity — can be identified (panel B).

Without full adjustment (panel A), observed differences in sentencing across groups could simply reflect differences in the types of crimes committed. In other words, if legal factors considered by the judge are not fully controlled, the estimated disparities may partly capture ‘warranted’ rather than ‘unwarranted’ disparities (Ulmer 2018). Technically, failing to control for the main case characteristics leaves open the indirect path: offender’s socio-demographics → case characteristics → sentence, thereby biasing the effect of interest, offender’s socio-demographics → sentence. Notice how we draw a dashed — rather than a continuous — path from case characteristics to sentence in panel B of Figure 2 to indicate that such path is closed after controlling for case characteristics.

Figure 2. Approximation of the causal model implicitly invoked in the typical study on sentencing disparities (panel A). To identify the direct effect of the offender’s socio-demographic trait of interest on sentence severity, all relevant case characteristics must be controlled (panel B).

In what follows, we show how this oversimplified model of the sentencing process obscures important causal mechanisms and, as a result, overlooks additional sources of bias that commonly affect estimates of discrimination in sentencing. To develop this argument, we next expand our DAG to provide a more accurate representation of the sentencing process (Figure 3). We introduce two new nodes, deliberation and judicial prejudice, drawn as ellipses to indicate that these are unobserved (i.e. latent) variables.

Figure 3. A more accurate representation of the causal mechanisms underlying sentencing decisions (panel A). The effect of interest runs from judicial prejudice to sentence severity, mediated by the process of deliberation. Because both judicial prejudice and deliberation are unobserved, this causal effect is approximated by the estimated effect of the offender’s socio-demographic trait on sentence severity. Identifying that effect requires controlling for all relevant case characteristics (panel B).

We posit that the causal effect of interest is more accurately conceived as stemming from a hypothetical judicial prejudice that is triggered once the judge forms an initial impression of the offender’s socio-demographic group. This hypothetical prejudice then feeds into the process of deliberation, in which all case characteristics are weighed before the sentence is imposed. Because we do not observe judicial prejudice nor the deliberation process, the causal effect of interest is approximated by the estimated effect of the offender’s socio-demographic trait on sentence severity, conditional on all other determinant of sentencing disparities that are not attributable to judicial prejudice — represented in this simplified model by the case characteristics.

Figure 3 serves as our reference DAG for illustrating the different causal roles of the main explanatory variables typically included in studies of discrimination in sentencing. We group these variables into two categories of legal factors: pre-defined and judicially-defined case characteristics; and three categories of non-legal factors: non-legal offender characteristics, judge characteristics, and court characteristics.

3.1 Pre-Defined and Judicially-Defined Case Characteristics

An implicit assumption in conventional models of sentencing discrimination is that the case characteristics controlled for in the model represent legal factors exogenous to the judge, who simply weighs them in determining the appropriate sentence. This simplified view of sentencing — reducing it to an algorithmic processing of inputs into outputs — is not only inaccurate but also highly problematic.

Certain case characteristics can reasonably be viewed as pre-defined or exogenous inputs, at least from the judge’s perspective, because they represent objective legal facts whose presence is not subject to judicial discretion. We refer to these as pre-defined case characteristics. Examples include whether the offender entered a guilty plea, whether the offence was committed while on bail, or, in drug cases, the type and quantity of the controlled substance. These factors are typically established through evidence or statutory definitions and would be the same regardless of which judge hears the case.

However, as a growing number of scholars note (Kurlychek and Johnson 2019; Lynch 2019; Yang 2014), many case characteristics commonly treated as legal controls are in fact discretionarily determined by the judge and are therefore endogenous to the very decision-making process under study. For instance, deciding whether an offender’s expression of remorse is genuine — and thus warrants a reduced sentence — ultimately rests on the judge’s subjective judgment. This assessment is further complicated by the fact that cultural norms shape how remorse is expressed, creating ample room for misinterpretation (Bennett 2016; Johansen 2019; Rossmanith 2015). Consequently, when researchers control for ‘remorse’, they may not only be adjusting for a legitimately mitigating factor — acknowledgement of harm and a commitment to desist (Bandes 2016; Maslen 2015) — but also inadvertently controlling for judicial prejudice. The same concern extends to many other case characteristics whose presence or severity is determined through judicial discretion. Aggravating or mitigating circumstances such as whether a violent offence was premeditated or committed in self-defence, are prime examples. Because these factors are partly products of the judge’s deliberation, we refer to them as judicially-defined case characteristics.

Figure 4 illustrates this trade-off by presenting a more nuanced depiction of the sentencing process. It distinguishes two key points in judicial deliberation. The first occurs when the judge initially receives the case information (for example, through a pre-sentence report). The second follows the judge’s consideration of additional case characteristics that they deem relevant. As discussed, controlling for both pre-defined and judicially-defined case characteristics can help prevent confounding bias, because both sets of factors provide legitimate legal criteria for determining the appropriate sentence. Yet the latter group is determined only after the judge’s initial deliberation — precisely when any hypothetical judicial prejudice may have already taken effect. Adjusting for these variables can therefore introduce post-treatment bias by blocking part of the causal effect we seek to estimate. This dual causal role is reflected in Figure 4, where judicially defined case characteristics are shown in purple, combining the red used for confounders and the blue used for mediators.

Specifically, when we control for judicially-defined case characteristics (panel B) we risk blocking the path: judicial prejudice → initial deliberation → judicially-defined case characteristics → final deliberation → sentence. Conversely, if we do not control for them (panel A) we leave open the backdoor path: judicial prejudice ← socio-demographic trait → judicially-defined case characteristics → final deliberation → sentence.

Figure 4. The trade-off posed by judicially-defined case characteristics (e.g., whether the offender shows remorse or whether the offence was premeditated). Controlling for these factors (panel B) risks post-treatment bias by blocking the path: judicial prejudice → initial deliberation → judicially-defined case characteristics → final deliberation → sentence. Failing to control for them (panel A), however, risks confounding bias by leaving open the backdoor path: judicial prejudice ← socio-demographic trait → judicially-defined case characteristics → final deliberation → sentence.

In sum, we should always control for pre-defined case characteristics. By contrast, deciding whether to control for judicially-defined case characteristics entails a trade-off between confounding and post-treatment bias, one that renders estimates of sentencing discrimination ultimately unidentifiable.⁸ As we show in Section 4, recognising this limitation does not mean abandoning attempts to estimate discrimination in sentencing. Rather, it requires acknowledging that such estimates are inevitably shaped by model uncertainty. Moreover, although we have presented pre-defined and judicially-defined case characteristics as a simple dichotomy for clarity, in practice that boundary is often blurred and, as discussed in Section 5, highly context-specific.

3.2 Non-Legal Factors

Beyond the offender’s socio-demographic trait under analysis and the legal factors discussed above, a wide range of non-legal factors (i.e., variables not specified in the criminal code or invoked in case law) are also commonly used in models of sentencing disparities. We distinguish three broad groups of non-legal factors: i) offender characteristics, ii) judge characteristics, and iii) court and area characteristics.

A vast criminological literature documents substantial differences in crime involvement across socio-demographic groups. Most clearly, men are more likely than women to commit violent crimes (Steffensmeier et al. 2006), and amongst adults, criminal activity declines with age (Steffensmeier et al. 1989). Beyond these well-established patterns, numerous studies have reported differences in criminal involvement across many other socio-demographic dimensions, including race, nationality, employment status, and level of education. At the same time, these socio-economic traits are also repeatedly found to be associated with sentence severity (Bontrager et al. 2013; Mitchell 2005; Wermink et al. 2022). Accordingly, when estimating sentencing discrimination against a particular socio-demographic group, researchers should consider the causal role of other offender traits.

Figure 5 extends our standard model by adding a separate node for other socio-demographic characteristics. This refers to traits different from the one of interest, which we now call the focal socio-demographic trait. For example, when analysing racial disparities, the new node captures all other offender socio-demographic traits aside from race. This additional node plays a similar role to that of the focal trait: it can influence both case characteristics (reflecting the well-documented variations in criminal involvement across socio-demographic groups) and sentence severity through its own form of judicial prejudice (reflecting the sentencing disparities observed across multiple socio-demographic characteristics). In other words, if we ignore relevant socio-demographic characteristics that simultaneously influence both the likelihood of offending and the severity of sentencing, we risk attributing to the focal group trait disparities that actually stem from these other correlated factors.

Figure 5. Other offender socio-demographic characteristics must be controlled for to identify the specific disparity of interest (panel B). This is due to prejudice being unobserved, which conflates different forms of prejudice, but also as result of collider bias from controlling for case characteristics. Specifically, when we control for case characteristics (panel A) we open up the path: focal socio-demographic → case characteristics ← other offender’s characteristics → other prejudice → deliberation → sentence; thereby biasing the effect of interest.

Because both judicial prejudice against the focal socio-demographic trait and other forms of judicial prejudice are unobserved, failing to control for the offender’s other socio-demographic characteristics makes it impossible to disentangle their respective effects. Even if focal prejudice could be observed, omitting these additional characteristics would still induce a new form of bias, collider bias. This is a less intuitive but well-documented problem in studies that control for mediators (Richiardi et al. 2013). In the sentencing context, case characteristics act as a mediator of the effect of offender socio-demographics on sentence severity. Once we control for these case characteristics the two nodes representing socio-demographic traits in Figure 5 become associated, and we cannot separate their effects unless we also adjust for the offender’s other socio-demographic characteristics (panel B), or for any other variables that jointly affect both case characteristics and sentence severity. Technically, controlling for case characteristics opens the path: focal socio-demographic → case characteristics ←other offender’s characteristics → other prejudice → deliberation → sentence; biasing the effect of interest.

A different picture emerges when we consider judge characteristics. Although the empirical literature is not always consistent, numerous studies document substantial variation in sentencing across judges (Johnson 2006; Pina-Sánchez et al. 2019). The evidence base suggests that a judge’s gender, race and age can affect sentence severity (Cheng et al. 2023; Johnson 2014), and that these characteristics may interact with the offender’s socio-demographic traits (Cohen and Yang 2019). Figure 6 illustrates this mechanism: judge characteristics can influence the sentence both through the deliberation process (reflecting differences in punitiveness across judges) and through judicial prejudice. Because judicial prejudice itself is unobserved, and we infer its effect on sentence severity examining the observed socio-demographic trait of the offender, controlling for judge characteristics would partially remove the very effect that we aim to estimate (panel B). Accordingly, when seeking to estimate the direct effect of an offender’s socio-demographic attribute on sentence severity, judge characteristics should not be controlled for (panel A).

Figure 6. Judge characteristics influence sentencing both through prejudice and through the deliberation process. Because judicial prejudice is unobserved, controlling for judge characteristics would inadvertently net out differences in prejudice itself (panel B), preventing identification of the effect of interest.

Figure 7. Trade-off when deciding whether to control for courts or their characteristics. If courts are left uncontrolled (panel A), collider bias arises from controlling for the mediator case characteristics. Specifically, omitting court characteristics opens the path: case characteristics ← court characteristics → deliberation → sentence. If court characteristics are controlled (panel B), part of the effect of the offender’s socio-demographic trait on sentence severity - operating through judicial prejudice - is inadvertently explained away.

The setting becomes more complex when we consider court — or area — characteristics. A large body of research shows that factors such as local caseload volume, political voting patterns, or population density can shape both overall punitiveness and the magnitude of ethnic disparities across courts (Ulmer and Johnson 2004; Ulmer and Kramer 1996; Wu and Spohn 2010). These features interact with the criminal justice practitioners that inhabit them to create court cultures (Eisenstein et al. 1988) manifested in different levels of punitiveness and prejudice. However, unlike individual judges — who generally face similar caseloads, especially in jurisdictions with random case assignment — courts located in different areas may confront systematically different offence mixes. Those differences often reflect the underlying patterns of crime in the surrounding community and will be captured in the offenders’ case characteristics. Figure 7 captures this additional layer of complexity.⁹

As with other offender socio-demographic characteristics (Figure 5), the effect of courts on the mediator — case characteristics — creates collider bias (panel A) unless court characteristics are controlled for (panel B). This occurs because court characteristics act as a mediator-outcome confounder. Specifically, failing to control for them opens the path: case characteristics ← court characteristics → deliberation → sentence. This would justify the inclusion of court or area fixed effects (Rehavi and Starr 2014; Steffensmeier and Demuth, 2006), or alternatively, controlling for court-level characteristics shown to affect sentence severity, such as court caseload or the socio-demographic composition of the area (Caravelis et al. 2011; Ulmer and Johnson 2004).¹⁰ Yet doing so risks once again absorbing some of the hypothetical judicial prejudice we seek to estimate (panel B), much as when controlling for judge characteristics. Consequently, deciding whether to control for court or area characteristics entails a trade-off closely paralleling that discussed for judicially-defined case characteristics in Section 3.1.

4 Modelling Direct Discrimination in Sentencing

Building on our discussion of how both controlling and failing to control for legal and non-legal factors can introduce bias, this section outlines a modelling framework designed to help researchers minimise such bias, quantify uncertainty, and assess the robustness of their findings. We identify four key steps — not to be conceived as a prescriptive set of rules, but as a flexible guidelines that can be expanded to incorporate additional sources of model uncertainty, and adapted to different jurisdictions, court types, decision-makers, and stages of the sentencing process.

Step 1: Define the Target Estimand

Any causal modelling exercise should begin with a clear definition of the target estimand, i.e., the specific effect that we seek to estimate (Lundberg et al. 2021). Yet this is rarely done in the sentencing literature. In practice, the implicit research goal is often to detect discrimination in sentencing, although to avoid criticism researchers commonly frame their objective more modestly, as the estimation of unwarranted disparities. However, the concept of unwarranted disparities, as generally understood — “[...] sentencing differences between comparable, similar offenses and offenders that are not attributable to legally prescribed factors” (Ulmer 2018, 2) — lacks the necessary precision. For example, inter-court variability (i.e. when certain courts are systematically more punitive than others) constitutes a form of unwarranted disparity (Drápal 2020; Malin and Tanskanen 2024) but it does not necessarily imply discrimination against a specific socio-demographic group.

Landmark studies synthesising the evidence on this area of research (Mitchell 2005; Spohn 2000; United States Sentencing Commission 2004) have been clearer in their stated goal by invoking Blumstein et al. (1983) influential distinction between disparities and discrimination: “Disparity exists when ‘like cases’ with respect to case attributes – regardless of their legitimacy – are sentenced differently”, whereas discrimination “[...] exists when some case attribute that is objectionable (typically on moral or legal grounds) can be shown to be associated with sentence outcomes after all other relevant variables are adequately controlled” (Blumstein et al. 1983, 72). This is a useful distinction. However, once again, it is not precise enough. It does not acknowledge the possibility of judicially-defined legal factors (Figure 4), ignores that other offender socio-demographic traits could be biasing the association of interest (Figure 5), and conflates direct discrimination — judicial prejudice — with other forms of structural or indirect discrimination — such as more punitive courts processing a higher number of a particular socio-demographic group of offenders (Figure 7).

Hence, when the research goal is to explore the presence of direct discrimination in sentencing, we suggest adopting the estimand proposed in Section 3, which can be precisely defined as follows: The average causal effect of (judicial attitude towards socio-demographic group A) compared to (judicial attitude towards socio-demographic group B) on (sentence severity) in (population P). However, since judicial prejudice (i.e., differences in judicial attitudes towards different socio-demographic groups) is unobservable, the above can only be regarded as a theoretical estimand. In practice, we rely on a proxy estimand: The direct causal effect of (socio-demographic characteristic A) compared to (socio-demographic characteristic B) on (sentence severity) in (population P), not mediated through warranted differences in case characteristics.

Note also that, to define our estimand, we must link it to a clearly specified target population. Researchers often focus not on sentencing practices in a jurisdiction as a whole, but on specific time periods (e.g. before and after the introduction of new guidelines), offence types (e.g., drug offences), or court levels (e.g., magistrates’ courts). In such cases, clearly stating the target population is essential to avoid selection bias when generalising findings.

While some subpopulations are easy to conceptualise, others are more ambiguous. For instance, studies often focus on sentence length while excluding the custodial decision itself, potentially introducing selection bias (Bushway et al. 2007; Knox et al. 2020; Pina-Sánchez and Gosling 2020). Similarly, when sentencers are involved in both conviction and sentencing — as with magistrates in England and Wales — analyses limited to sentencing decisions must clarify that findings do not generalise to the broader decision-making process.

Another common mismatch between the analytical sample and the target population arises in jurisdictions where sentences are negotiated between the defence and prosecution through plea agreements — that is, when sentencing is not solely reserved to judges. As we will discuss in Step 3, clearly identifying the decision-makers under study is essential not only for the sake of transparency, but also because it affects how we classify legal factors as being subjectively defined.

Step 2: Operationalise Sentence Severity

The notion of discrimination in sentencing presupposes the ability to rank sentences according to their relative severity. However, sentencers have access to a wide range of disposal types, many of which are measured on different scales (e.g., fines in monetary units, prison sentences in days), complicating comparative analysis. This technical challenge has traditionally been addressed in two main ways: by treating sentence severity as a binary variable indicating whether a custodial sentence was imposed, or by analysing the duration of custodial sentences only (Bushway and Piehl 2001; Ostrom et al. 2008). Both approaches entail a substantial loss of information. The former cannot distinguish severity within each category (custody or non-custody), while focusing only on custodial sentences excludes a significant portion of sentencing decisions, potentially introducing selection bias.

Increasingly, researchers adopt a third strategy: analysing sentence length while coding non-custodial sentences as zeros. This approach allows for the inclusion of the full range of sentencing outcomes, thereby avoiding selection bias, while still enabling differentiation among custodial sentences. Several modelling strategies can be applied to such data, each resting on different assumptions. Given this paper’s focus on bias and uncertainty arising from the choice of control variables, we provide only a brief overview and refer readers to Bushway et al. (2007) for a more detailed discussion.

A key consideration is whether, in the jurisdiction under study, the length of a custodial sentence is determined separately from the disposal type, or whether both are decided simultaneously. In the former case, two-stage models such as Heckman selection models (Heckman 1976) or hurdle models (Hester and Hartman 2017) may be appropriate. In the latter, single-stage models such as Tobit regression (King et al. 2010) or negative binomial models (Petersen and Omori 2020), which treat non-custodial sentences as part of the sentence length distribution, may be more suitable.

An alternative strategy involves estimating the relative severity of each disposal type on a common scale (Irwin-Rogers and Perry 2015; Leclerc and Tremblay 2016; Pina-Sánchez and Gosling 2020). This approach has the unique advantage of distinguishing between non-custodial sentences, thereby maximising the use of available information. Its main drawback lies in the reliance on subjective assessments of severity, which introduces additional assumptions and affects the robustness of findings (Pina-Sánchez and Gosling 2022).

Step 3: List and Classify the Necessary Controls

Once sentence severity is operationalised and an appropriate outcome model selected, the next step is to identify the explanatory variables required to estimate the target estimand. This involves considering both legal and non-legal factors, and — crucially — distinguishing between variables that should always be controlled for (as doing so can only reduce bias), and those with a dual causal role, which may simultaneously reduce some forms of bias while introducing others.

The list of relevant legal factors can be derived from the criminal code or sentencing guidelines applicable in the jurisdiction under study. To ensure thoroughness, we recommend conducting separate analyses for different offence groups. This is because the relevance of legal factors and the way they are applied vary across offence types, and failing to account for this heterogeneity may lead to model misspecification (Hofer 2019). For example, aggravating factors such as targeting a vulnerable victim may carry more weight in violent or sexual offences than in property offences, while mitigating factors like the return of stolen goods are specific to the latter. Analysing multiple offence types simultaneously increases sample heterogeneity, which complicates the identification of the causal effect of interest.

Once the relevant legal factors are identified, they should be classified into two categories: pre-defined factors — those presented to the sentencer as part of the case file — and judicially-defined factors — those that may be included or omitted at the sentencer’s discretion (Figure 4). As discussed in Section 3.1, this distinction is highly context-dependent. For instance, in England and Wales, pre-trial detention is decided by magistrates. Thus, pre-trial detention (i.e., remand) should be considered a judicially-defined factor when analysing magistrates’ courts, but a pre-defined factor when focusing on Crown Court judges.

Similarly, U.S. research on sentencing discrimination involving plea agreements should expand the category of judicially-defined factors to include those shaped by prosecutorial discretion. This logic can be extended further to include probation and police officers if the research goal is to estimate discrimination across the broader criminal justice system. In essence, the number of discretionarily defined legal factors increases the further upstream the analysis moves in the decision-making process (Kurlychek and Johnson 2019).¹¹ Likewise, it becomes increasingly important to rely on samples that accurately reflect the wider population of interest in order to avoid introducing selection bias (Bushway et al. 2007; Knox et al. 2020).

While a comprehensive list of legal factors is essential to detect the presence of unwarranted disparities, it is insufficient for identifying direct discrimination. Researchers must also include offender socio-demographic traits that may be associated with both sentence severity and the trait of interest (Figure 5). Most critically, controls for gender and age should be included, alongside other traits such as citizenship, employment status, and educational attainment.

Additionally, court or area identifiers should be considered to help distinguish direct from indirect discrimination. However, as with judicially-defined factors, controlling for court characteristics may introduce post-treatment bias (Figure 7). Given this similarity, we refer to variables with such dual causal roles — namely, judicially-defined case characteristics and court or area characteristics — as endogenous controls, while all other variables in the control set are labelled exogenous controls.

Finally, judge characteristics should never be included, as doing so risks controlling away judicial prejudice (Figure 6).¹² More broadly, controls should not be added solely to improve model fit. Any expansion of the set of controls must be justified in causal terms: variables should be included only if they help reduce a specific form of bias affecting the estimation of direct discrimination in sentencing.

Step 4: Estimate the Model Uncertainty

Because some necessary controls are endogenous to the modelling process — and because controlling for all relevant legal and non-legal factors is ultimately impossible — the target estimand is, strictly speaking, unidentifiable. Nevertheless, we encourage researchers to acknowledge and, where possible, quantify both sources of model uncertainty. Doing so allows us to move beyond reporting point estimates that we know to be biased, and instead offer a bounded range of estimates within which the true effect is likely to lie.

One way to document the uncertainty associated with endogenous controls is to report point estimates for the focal socio-demographic trait across two nested models: one controlling only for the exogenous factors listed in Step 3, and another including both exogenous and endogenous controls. As discussed, both models are misspecified — the former omits relevant variables, while the latter may control for part of the effect we aim to estimate. However, comparing them allows us to assess the uncertainty associated to this trade-off and establish the direction of the bias affecting each of the two modelling strategies.

Since we only suspect — but cannot confirm — that endogenous factors introduce post-treatment bias (e.g., we hypothesise that offender gender may influence whether judges perceive remorse as genuine, but lack direct evidence), a more sophisticated approach may be warranted. Rather than comparing just two models, researchers can use a specification curve analysis (Simonsohn et al. 2020) to estimate a range of models across all — or a random subset of — possible combinations of endogenous controls. This enables us to provide not just an estimate of the bounds within which the true effect might lie, but by plotting the collection of estimates obtained under different sets of controls as a probability density function we could intuitively assess where the true effect might be more likely to lie.¹³

We also encourage researchers to acknowledge bias arising from failing to control for relevant factors listed in Step 3. Over the last decade, institutions such as the U.S. Sentencing Commission and the Sentencing Council for England and Wales have released increasingly detailed administrative and survey data, alleviating a key limitation affecting past sentencing research (Ulmer 2011; Zatz 1987). However, even in these jurisdictions gaps remain, especially regarding non-legal factors. Thus, even studies focusing on relatively homogeneous offence groups and controlling all available case characteristics (Hartley and Miller 2010; Holland and Prohaska 2021; Ulmer and Parker 2020) may still be missing key controls. Rather than addressing this limitation solely through qualitative caveats — typically placed at the end of the study — we advocate for the use of sensitivity analysis to quantify the potential bias (Lash et al. 2014). As far as we are aware, to date only two studies have applied sensitivity analysis to explore the problem of unobserved confounders in the context of sentencing research (Pina-Sánchez et al. 2024; Ward et al. 2016). Both assume a single unobserved confounder that increases sentence severity (i.e., an aggravating factor), and neither accounts for the variance already explained by observed factors. More sophisticated sensitivity analysis techniques can accommodate multiple unobserved confounders (Groenwold et al. 2016), or even model uncertainty arising from multiple sources simultaneously (Fox et al. 2021; Smith et al. 2021). These approaches could also address additional challenges in sentencing research, such as non-random missing data (Stockton et al. 2024), or misclassification of offender ethnicity.¹⁴

For researchers unfamiliar with sensitivity analysis, we recommend techniques that are easy to implement across studies. These include the ‘E-value’ proposed by VanderWeele and Ding (2017) for effect sizes expressed as odds ratios or risk ratios, and the ‘robustness value’ introduced by Cinelli and Hazlett (2020) for estimates derived from linear models. Rather than estimating the bias directly, these metrics indicate how strong the association between an unobserved confounder and both the socio-demographic trait of interest and sentence severity would need to be for the observed disparity to disappear entirely.

5 Applications

We now illustrate how the proposed modelling framework can be applied to investigate the presence of direct sentencing discrimination in two distinct contexts: i) against racial minority offenders charged with drug trafficking in U.S. federal courts, and ii) against male offenders charged with shoplifting and sentenced in magistrates’ courts in England and Wales.¹⁵ We present two examples, rather than one, to highlight how our proposed modelling framework should be tailored to the specific context under analysis — recognising, in particular, the nuances arising from the jurisdiction, type of court, and decision-makers involved.

5.1 Race Disparities in the U.S. Federal Courts

The analysis of race disparities in U.S. federal courts represents one of the most extensively studied topics in sentencing research. Meta-analyses have consistently found statistically significant racial disparities in sentencing outcomes (Ferguson and Smith 2024; Mitchell 2005), although the estimated effect sizes tend to be small and highly sensitive to the choice of controls. In this application, we seek to estimate the presence of direct sentencing discrimination against racial minority offenders convicted of drug trafficking. We focus on drug trafficking offences due to their high volume and relatively limited exposure to difficult-to-measure legal factors, such as offender dangerousness. Importantly, given that only 2.3% of drug cases in U.S. federal courts went to trial in 2023, our definition of direct discrimination encompasses sentencing decisions made by both judges and prosecutors.

The data are sourced from the U.S. Sentencing Commission.¹⁶ We analyse sentences imposed between fiscal years 2020 and 2023, yielding a sample of 72,943 drug trafficking cases. Fourteen variables used in the analysis exhibit item-level missingness, although the extent is minor, with the most affected variable missing only 0.7% of cases. To enable the specification of nested models, we imputed all missing values using multiple imputation via chained equations and predictive mean matching, implemented through the mice package in R (Van Buuren 2018), using the same variables included in the analysis as auxiliary data.¹⁷ Descriptive statistics for these variables — prior to imputation — are presented in Table 1.

Our outcome variable is prison sentence length, measured in months. Probation sentences are coded as 0, and prison sentences are capped at 470 months, consistent with the dataset’s coding scheme. The explanatory variables include offence and offender characteristics, as well as district identifiers.¹⁸ We identify five endogenous controls: districts, final offence seriousness level, final criminal history category, pre-trial detention status, and whether a departure was granted for substantial assistance to the prosecution.

Assistance to the prosecution is a relevant mitigating factor but is discretionarily defined by both prosecutors and judges. Pre-trial detention is more complex. In some cases, it reflects factors unrelated to legal severity — such as the offender’s ability to post bail.¹⁹ In other cases, it may serve as a proxy for offender dangerousness or risk of reoffending, and thus could be considered a relevant legal factor. However, even under that interpretation, it remains a judicially-defined factor, as pre-trial decisions are typically made by magistrate judges and, when appealed, by district court judges. The final offence seriousness and criminal history variables represent their most complete versions, incorporating discretionary judicial decisions. These include assessments of the offender’s role in the offence (e.g., organiser vs. courier), whether the offence targeted a vulnerable victim, and whether prior convictions involved physical force.

We employ negative binomial regression models, which are well-suited to the outcome variable’s distribution — right-skewed with a large concentration of zero values. This modelling choice also reflects our assumption that prosecutors and judges jointly determine both the disposal type and its magnitude, rather than through a two-stage process. Table 2 presents results from two nested models: Model 1 includes only exogenous controls, while Model 2 adds the endogenous controls identified above. For simplicity, we focus our analysis on disparities affecting black and Hispanic offenders, whose regression coefficients are highlighted in bold.

Both models indicate small but statistically significant disparities against black offenders. In contrast, estimates for Hispanic offenders vary widely depending on the set of controls included, to the extent that different model specifications yield contradictory conclusions — suggesting either disadvantage or advantage. This highlights a substantial degree of model uncertainty.

Figure 8 illustrates this uncertainty by showing the distribution of estimated disparities for black and Hispanic offenders across 31 unique combinations (excluding the empty set) of the five endogenous controls added to the exogenous control set from Model 1. For black offenders, the 95% interval of the distribution ranges from 1.04 to 1.12, corroborating the small but consistent disadvantage observed in Table 2. For Hispanic offenders, however, the interval spans from 0.82 to 1.18, indicating an extreme degree of model uncertainty and precluding any conclusive interpretation.

Figure 8. Model uncertainty associated with the inclusion of endogenous controls in estimating racial disparities. The left panel shows relatively low uncertainty for black offenders, with the entire distribution lying to the right of the vertical dashed line, indicating consistent disadvantage. The right panel shows substantial uncertainty for Hispanic offenders, with the distribution crossing the line, making any conclusion about discrimination highly speculative.

To further assess the robustness of the observed disparities against black offenders, we examine whether they could be explained by omitted legal factors, such as drug quantity or whether the offence resulted in death or serious bodily injury. We apply the ‘robustness value’ technique proposed by Cinelli and Hazlett (2020), which is designed for linear models. Accordingly, we replicate Models 1 and 2 using linear regression after log-transforming sentence length.²⁰ These models yield estimated sentence increases of 15.7% and 6.4% for black offenders, respectively. The corresponding robustness values are 0.041 and 0.023, indicating that an unobserved aggravating factor would need to explain more than 4.1% to 2.3% of the residual variance, and be more than 4.1% to 2.3% more prevalent among black than white offenders, for the observed disparities to be entirely spurious.

Given the relevance of unobserved legal factors such as drug quantity or bodily injury, we expect the first condition to be met. However, the second condition is more questionable. Rehavi and Starr (2014) indicate that in the last three years (2001 to 2003) when quantity of drugs seized at arrest where reliably reported by the Federal Prosecutors Database (EOUSA), no differences were found across racial groups. We are unaware of any studies documenting racial disparities in the prevalence of other aggravating factors such as serious bodily injury. Therefore, under the assumption that black offenders do not systematically traffic larger quantities of drugs or commit offences resulting in bodily injury more frequently than comparable white offenders, our findings may be interpreted as evidence of direct racial discrimination in the sentencing of drug trafficking offenders in U.S. federal courts.

5.2 Gender Disparities in the Magistrates’ Courts

Several studies have documented gender disparities in the Crown Court of England and Wales, with male offenders receiving substantially harsher sentences than their female counterparts (Hopkins et al. 2013; Pina-Sánchez and Harris 2020). However, it remains unclear whether similar disparities exist in the magistrates’ courts — the lower-tier courts where 92% of criminal cases are processed (Sturge 2021). This raises concerns about the external validity of findings based solely on Crown Court data. In this application, we use the dataset ‘Theft from a Shop or Stall’, published the Sentencing Council for England and Wales.²¹ Specifically, we analyse a sample of 2,116 convicted shoplifting offenders sentenced under the 2016 theft offences sentencing guidelines. The data were collected via self-completed questionnaires²² administered to magistrates and district judges across 81 magistrates’ courts.²³ A key strength of this dataset is that it is completed directly by sentencers, allowing for detailed coverage of the legal factors listed in the sentencing guidelines. However, this also means that the threshold for considering certain legal factors as pre-defined (exogenous) is harder to meet than if they had been reported by administrative staff, such as court clerks.

Our outcome variable is whether the offender received a custodial sentence (either suspended or immediate) or a non-custodial sentence (discharge, fine, or community order), modelled using binary logistic regression. While this is a coarse measure of sentence severity, its use is justified by the low prevalence of immediate custodial sentences for shoplifting offences in magistrates’ courts (18.7% of our sample), and by the absence of sentence length data for suspended custodial sentences. Cases transferred to the Crown Court were excluded, as their final sentencing outcomes are unknown. Similarly, we lack data on conviction decisions, which are also made by magistrates. This mismatch between our target estimand — gender-based direct discrimination in magistrates’ court — and our analytical sample introduces potential selection bias. However, we do not expect this bias to be substantial. In 2016 conviction rates for shoplifting offenders were similar across genders (92.2% for female offenders and 92.9% for their male counterparts), and only 0.7% of shoplifting cases were transferred from magistrates’ courts to the Crown Court (Ministry of Justice 2017).

The explanatory variables include offender characteristics and multiple indicators of culpability, harm, aggravating, and mitigating factors present in the case.²⁴ Although we aim to include all legal factors listed in the sentencing guidelines, some are too rare to meaningfully influence our estimates. Therefore, we exclude case characteristics present in fewer than 1% of cases.²⁵ Eight variables exhibit item-level missingness, with the highest being guilty plea status (15%). All missing values were imputed using the same multiple imputation approach described in the U.S. federal courts application. Descriptive statistics for the variables — prior to imputation — are reported in Table 3.

In spite of its level of detail, the Sentencing Council datasets have been criticised for omitting difficult-to-measure legal factors, such as offender dangerousness or potential for rehabilitation (Isaac 2020). In the context of shoplifting offenders, however, dangerousness is unlikely to be a major concern. Moreover, our model includes a proxy for rehabilitation potential — namely, whether the offender has taken steps to address an addiction. One potentially relevant legal factor that appears to be missing is assistance to the prosecution.²⁶ Nevertheless, given that shoplifting offences are typically low in severity and committed by individuals rather than organised groups, we do not expect this to be a common mitigating factor in our sample, nor one that would substantially influence our findings. The dataset is, however, notably sparse in terms of non-legal factors. In particular we lack information on offender socio-demographic characteristics beyond age, and have no data on the court or the geographical location where the sentence was imposed.

In spite of its level of detail, the Sentencing Council datasets have been criticised for omitting difficult-to-measure legal factors, such as offender dangerousness or potential for rehabilitation (Isaac 2020). In the context of shoplifting offenders, however, dangerousness is unlikely to be a major concern. Moreover, our model includes a proxy for rehabilitation potential — namely, whether the offender has taken steps to address an addiction. One potentially relevant legal factor that appears to be missing is assistance to the prosecution.²⁷ Nevertheless, given that shoplifting offences are typically low in severity and committed by individuals rather than organised groups, we do not expect this to be a common mitigating factor in our sample, nor one that would substantially influence our findings. The dataset is, however, notably sparse in terms of non-legal factors. In particular we lack information on offender socio-demographic characteristics beyond age, and have no data on the court or the geographical location where the sentence was imposed.

To classify controls as exogenous or endogenous, we consider whether each legal factor represents a factual case characteristic or one that could be discretionarily defined by magistrates. Among the variables listed in Table 3, the following are classified as exogenous controls: being subject to a banning order, total value of goods stolen, additional offences admitted, failure to comply with court orders, offence committed while on bail or licence, stealing goods to order,²⁸ voluntary return of stolen property, serious medical condition, and caring responsibilities.

Some of these — such as failure to comply with a court order or offending while on bail — could arguably be judicially-defined, given that magistrates also adjudicate convictions. However, since prior research indicates no meaningful gender disparities in conviction rates in magistrates’ courts, we treat these factors as pre-defined case characteristics. We make an exception for the aggravating factor number of previous convictions. This is because the sentencing guidelines indicate that only previous convictions considered relevant in relation to the offence committed should be counted, we therefore classify the number of previous convictions as judicially-defined, and therefore an endogenous control. All other legal factors listed in Table 3 are also classified as endogenous, as we cannot rule out the possibility that their presence in the case — or their inclusion in the survey form — was influenced by magistrate’s perceptions of the offender’s gender.

While the magnitude of the estimated gender disparity varies depending on the control set, both models (Table 4) consistently indicate more punitive sentencing for male offenders. Specifically, the odds ratios range from 1.38 to 1.54. When transformed into risk ratios,²⁹ this corresponds to a range from 1.26 to 1.37, indicating that male offenders are at least 26% more likely to receive a custodial sentence than female offenders.

As in the previous application, a more nuanced exploration of model uncertainty associated with the inclusion of endogenous factors can be obtained through a specification curve. In this case, we identified 20 case characteristics as judicially-defined, resulting in over one million possible model combinations. To simplify the analysis, we re-estimate the model using a random sample of 100 unique combinations. The results are presented in Figure 9, which shows a distribution of odds ratios similar to that observed when comparing Models 1 and 2. Specifically, 95% of the odds ratios fall within the range of 1.34 to 1.58.

Figure 9. Model uncertainty associated to the use of judicially-defined case characteristics in the estimation of gender disparities in magistrates’ courts. There is a large amount of variability in the distribution of odds ratios making it difficult to identify the specific effect size. However, the distribution remains consistently above 1, indicating a robust disadvantage for male offenders.

An additional benefit of empirically estimating model uncertainty is the ability to assess the influence of specific judicially-defined factors on the estimated disparity. Figure 9 reveals a bimodal distribution, driven primarily by two influential variables. The number of previous convictions substantially reduces the estimated gender disparity by 15.2 percentage points when included in the model, while the presence of injury to the victim increases the estimated disparity by 3.9 percentage points. These findings underscore the importance of carefully considering which judicially-defined factors are included in the model, as they can meaningfully alter the interpretation of sentencing disparities.

We now turn to explore the robustness of our findings to omitted relevant controls. We apply the ‘E-value’ technique proposed by (VanderWeele and Ding 2017), which is designed for discrete models. Specifically, we consider the most conservative of our estimates of gender disparities (Model 2) and acknowledge that our analysis does not control for offender characteristics beyond age (e.g., ethnicity, nationality, employment status, or education level) nor for court identifiers.

The E-value for the lower bound of our estimated risk ratio of receiving a custodial sentence — 1.26 against male offenders — is 1.84. This means that, for our most conservative estimate, for the observed gender disparities to be entirely explained away, an unobserved factor — or set of factors — would need to both increase the probability of receiving a custodial sentence by 84%, and be 84% more prevalent amongst male than female offenders. That represents a substantial threshold, particularly the second condition. We do not expect the socio-demographic profiles of male and female offenders to differ so markedly when conditioning on offence type. For instance, according to Ministry of Justice statistics, the proportion of male theft offenders from ethnic minority backgrounds is only 7% higher than that of female offenders Ministry of Justice (2017). Similarly, we find it unlikely that male and female offenders are distributed unevenly across courts.

Even if such differences in prevalence could be established, it remains doubtful that the unobserved factors would exert a sufficiently strong influence on sentencing outcomes to meet the required threshold. Existing evidence from England and Wales suggests that race- and deprivation-related disparities in sentencing are statistically significant but relatively modest in magnitude (Lymperopoulou 2024; Pina-Sánchez et al. 2024).

Therefore, we conclude that — conditional on: i) unobserved legal factors such as dangerousness or assistance to the prosecution not exerting a substantial influence on sentencing decisions for shoplifting offences, and ii) other socio-demographic traits beyond age not being strongly associated with offender gender — there is evidence of direct sentencing discrimination against male offenders in the magistrates’ courts. Crucially, however, we cannot generalise this finding to the entire decision-making process in magistrates’ courts, as our analysis is limited to sentencing decisions and does not include adjudications of guilt.

6 Discussion

For practical reasons, research on sentencing discrimination has largely relied on observational data, making it particularly sensitive to modelling assumptions. Traditionally, the prevailing approach has been to control for as many legal factors as possible, in order to isolate unwarranted disparities from legitimate sources of variation in sentence severity. More recently, scholars have raised concerns about the potential bias introduced by controlling for legal factors that are discretionarily defined by judges — such as mitigating factors like remorse or aggravating factors like premeditation — arguing that only legal factors exogenous to the judge should be included. The choice between these approaches often reflects disciplinary norms, with criminologists tending to favour the traditional strategy, while econometricians advocate for more parsimonious models (Hofer 2019).

In this article, we have shown that both approaches aim to reduce important types of bias, but by framing the issue as a binary choice, they fail to address each other’s concerns. Ultimately, both are flawed. Using causal diagrams, we illustrate how judicially-defined legal factors often play a dual causal role – acting simultaneously as confounders and mediators of the effect of judicial prejudice on sentence severity. For example, black offenders are less likely to plead guilty than white offenders (Metcalfe and Chiricos 2018), which correlates with expressions of genuine remorse (Guilfoyle and Pina-Sánchez 2024). Therefore, failing to control for remorse could lead to confounding bias and wrongly conclude that judges discriminate against black offenders (i.e., such racial disparities might simply be reflecting the sentence reduction warranted from a guilty plea and a genuine expression of remorse). However, the assessment of remorse is itself a subjective decision, potentially influenced by the same judicial prejudice under investigation (Johansen 2019). In that case, controlling for remorse could also control for judicial prejudice, leading to post-treatment bias and to underestimate the true extent of discrimination. This trade-off renders estimates of sentencing discrimination formally unidentifiable.

By conceptualising the causal mechanisms underlying the sentencing process using DAGs, we also highlighted additional sources of bias stemming from non-legal factors commonly used in sentencing research. Specifically, we show that: i) omitting key offender’s socio-demographic characteristics can lead to a less intuitive form of collider bias; ii) controlling for judge characteristics may introduce post-treatment bias; and iii) court or area characteristics can lead to collider bias if unobserved, or post-treatment bias if controlled for.

Beyond documenting these sources of bias, we have used the insights gained to develop a new modelling framework aimed at facilitating more robust and transparent estimation of direct discrimination in sentencing. A key contribution of this framework is its reframing of the trade-off posed by judicially-defined legal factors, and other endogenous non-legal factors like court characteristics. Rather than choosing between two flawed approaches — controlling for them or excluding them — we advocate for explicitly acknowledging the model uncertainty stemming from this trade-off. We demonstrate how this can be achieved through specification curve analysis, reporting the distribution of estimates derived from models using different combinations of endogenous controls. By embracing modelling uncertainty, researchers can reduce bias and avoid overconfident interpretations based on naïve point estimates.

To illustrate the framework, we conducted two applications using data from U.S. federal courts and magistrates’ courts in England and Wales. These examples underscore the importance of mapping modelling uncertainty, as both the presence and influence of judicially-defined and other endogenous controls vary significantly across contexts. For instance, while estimates of disparities against black offenders in U.S. federal courts appear relatively stable, estimates for Hispanic offenders vary dramatically depending on the control set. Had we followed the criminological approach, we would have concluded that Hispanic offenders receive 5% longer sentences than white offenders; had we followed the econometric approach, we would have concluded they receive 11% shorter sentences. The honest answer is that we do not know which is correct. Without further research into how judicial prejudice is embedded in the construction of legal factors, it is impossible to determine which modelling strategy is superior. Both are biased, and the extent of that bias is unknown. Therefore, without any further information, the best we can do is to report the interval of plausible estimates derived from different model specifications.

Clearly, a more transparent and thoughtful modelling framework for the estimation of discrimination in sentencing is needed. However, it is important to emphasise that the framework presented here should not be viewed as prescriptive set of rules. Rather, it should be understood as a flexible set of guidelines, to be adapted to the specific context under investigations, including: i) the legal framework governing the sentencing process; ii) the socio-demographic realities of the jurisdiction; and iii) the specific decisions and practitioners under analysis. Researchers must exercise discretion in determining which factors are relevant, the causal role they may play, the appropriateness of controlling for them, and the implications of failing to do so.

Similarly, the modelling framework proposed is not definitive, just a step forward in the path towards improving the transparency and robustness of research in sentencing disparities. While our framework focuses on the choice of controls, this is only one source of model uncertainty in sentencing research. Additional steps could be incorporated to: i) address missing data — still a widespread and largely overlooked issue in sentencing research (Stockton et al. 2024); ii) implement pre-registration protocols to reduce researcher bias (Baldwin et al. 2022) — we are only aware of three sentencing studies that have employed pre-registered analytical strategies (Ferguson and Smith 2024; Pina-Sánchez et al. 2024; Pina-Sánchez and Brunton-Smith 2025); iii) present effect sizes more intuitively — for example, replacing odds ratios with marginal effects expressed as probabilities (Mize et al. 2019); and iv) expand specification curve analyses to account for model uncertainty stemming from different, yet equally defensible, functional forms used to model the outcome variable.

To conclude, even under the most rigorous modelling strategy and with the most detailed dataset, the precise level of discrimination in sentencing can never be fully identified. Yet, this epistemic limitation should not lead to scientific nihilism or discourage efforts to investigate the issue. Discrimination in sentencing undermines the principle of equal treatment, which lies at the heart of justice. Moreover, perceptions of discrimination — whether grounded on reality or not — can have real-word consequences, eroding trust in public institutions and alienating segments of the population.

Thus, we must reject both the overconfidence that treats single-point estimates as definitive, and the social constructivist perspective that dismisses any attempt to quantify judicial discrimination. Instead, we advocate for a middle path: abandoning the pursuit of a singular estimate of judicial prejudice and focusing instead on documenting the uncertainty surrounding the estimation process. This approach offers a more realistic understanding of the problem, and – when applied carefully, and when the resulting evidence is sufficiently clear – can allow researchers to make confident claims about the presence or absence of discrimination in sentencing.

Funding

This research was funded by the Economic and Social Research Council (Grant/Award Number: ES/W00738X/1).

References

Abrams, David S., Marianne Bertrand, and Sendhil Mullainathan. 2012. Do Judges Vary in Their Treatment of Race? Journal of Legal Studies 41 (2): 347–383.

https://doi.org/10.1086/666006

Anderson, James M., Jeffrey R. Kling, and Kate Stith. 1999. Measuring Interjudge Sentencing Disparity: Before and After the Federal Sentencing Guidelines. The Journal of Law and Economics 42 (1): 271–308. https://doi.org/10.1086/467426

Baldwin, Jessie R., Jean-Baptiste Pingault, Tabea Schoeler, Hannah M. Sallis, and Marcus R. Munafò. 2022. Protecting against Researcher Bias in Secondary Data Analysis: Challenges and Potential Solutions. European Journal of Epidemiology 37 (1): 1–10.

https://doi.org/10.1007/s10654-021-00839-0

Bandes, Susan A. 2016. Remorse and Criminal Justice. Emotion Review 8 (1): 14–19.

https://doi.org/10.1177/1754073915601222

Baumer, Eric P. 2013. Reassessing and Redirecting Research on Race and Sentencing. Justice Quarterly 30 (2): 231–261. https://doi.org/10.1080/07418825.2012.682602

Belton, Ian, and Mandeep Dhami. 2023. The Role of Character-Based Personal Mitigation in Sentencing Judgments. Journal of Empirical Legal Studies 21 (1): 208–239.

https://doi.org/10.1111/jels.12376

Bennett, Christopher D. 2016. “The Role of Remorse in Criminal Justice.” In Oxford Handbook Online in Criminology and Criminal Justice, edited by Michael Tonry. Oxford: Oxford University Press. https://doi.org/10.1093/oxfordhb/9780199935383.013.37

Blumstein, Alfred, Jacqueline Cohen, Susan E. Martin, and Michael H. Tonry. 1983. Research on Sentencing: The Search for Reform. Vol. 1. Washington, DC: National Academy Press.

Bontrager, Stephanie, Kellee Barrick, and Elizabeth Stupi. 2013. Gender and Sentencing: A Meta-Analysis of Contemporary Research. Journal of Gender, Race & Justice 16: 349–372.

https://heinonline.org/HOL/P?h=hein.journals/jgrj16&i=371

Bushway, Shawn D., Brian D. Johnson, and Lee Ann Slocum. 2007. Is the Magic Still There? The Use of the Heckman Two-Step Correction for Selection Bias in Criminology. Journal of Quantitative Criminology 23 (2): 151–178. https://doi.org/10.1007/s10940-007-9024-4

Bushway, Shawn D., and Anne Morrison Piehl. 2001. Judging Judicial Discretion: Legal Factors and Racial Discrimination in Sentencing. Law & Society Review 35 (4): 733–764.

https://doi.org/10.2307/3185415

Caravelis, Cindy, Ted Chiricos, and William Bales. 2011. Static and Dynamic Indicators of Minority Threat in Sentencing Outcomes: A Multi-Level Analysis. Journal of Quantitative Criminology 27: 405–425. https://doi.org/10.1007/s10940-011-9130-1

Cheng, Kevin K., Sayaka Ri, and He Chengchen. 2023. Judges’ Characteristics and Sentencing in Hong Kong. Criminology & Criminal Justice 25 (4): 1244-1261. https://doi.org/10.1177/17488958231187215

Chu, Haitao, Zhaojie Wang, Stephen R. Cole, and Sander Greenland. 2006. Sensitivity Analysis of Misclassification: A Graphical and a Bayesian Approach. Annals of Epidemiology 16 (11): 834–841. https://doi.org/10.1016/j.annepidem.2006.04.001

Cinelli, Carlos, Andrew Forney, and Judea Pearl. 2022. A Crash Course in Good and Bad Controls. Sociological Methods & Research 53 (3), 1071–1104.

https://doi.org/10.1177/00491241221099552

Cinelli, Carlos, and Chad Hazlett. 2020. Making Sense of Sensitivity: Extending Omitted Variable Bias. Journal of the Royal Statistical Society: Series B 82 (1): 39–67.

https://doi.org/10.1111/rssb.12348

Cohen, Alma, and Crystal S. Yang. 2019. Judicial Politics and Sentencing Decisions. American Economic Journal: Economic Policy 11 (1): 160–191. https://doi.org/10.1257/pol.20170329

Cole, Kevin. 1997. The Empty Idea of Sentencing Disparity. Northwestern University Law Review 91 (4): 1336–1341.

Drápal, Jakub. 2020. Sentencing Disparities in the Czech Republic: Empirical Evidence from Postcommunist Europe. European Journal of Criminology 17 (2): 151–174.

https://doi.org/10.1177/1477370818773612

Eisenstein, James, Roy B. Flemming, and Peter F. Nardulli. 1988. The Contours of Justice: Communities and Their Courts. Boston: Little, Brown; Lanham, MD: University Press of America.

Engen, Rodney L., and Randy R. Gainey. 2000. Modelling the Effects of Legally Relevant and Extralegal Factors under Sentencing Guidelines: The Rules Have Changed. Criminology 38 (4): 1207–1230. https://doi.org/10.1111/j.1745-9125.2000.tb01419.x

Ferguson, Christopher J., and Sven Smith. 2024. Race, Class, and Criminal Adjudication: Is the US Criminal Justice System as Biased as Is Often Assumed? A Meta-Analytic Review. Aggression and Violent Behavior 75: 101905. https://doi.org/10.1016/j.avb.2023.101905

Fischman, Joshua B., and Max M. Schanzenbach. 2012. Racial Disparities under the Federal Sentencing Guidelines: The Role of Judicial Discretion and Mandatory Minimums. Journal of Empirical Legal Studies 9 (4): 729–764. https://doi.org/10.1111/j.1740-1461.2012.01266.x

Fox, Matthew, Onyebuchi A. Arah, Sonja Swanson, and Vivian Viallon. 2024. Causal Diagrams to Evaluate Sources of Bias. In Statistical Methods in Cancer Research Volume V: Bias Assessment in Case–Control and Cohort Studies for Hazard Identification. Lyon: International Agency for Research on Cancer.

Fox, Matthew P., Richard F. MacLehose, and Timothy L. Lash. 2021. Applying Quantitative Bias Analysis to Epidemiologic Data. New York: Springer.

Groenwold, Rolf H. H., Jonathan A. C. Sterne, Debbie A. Lawlor, Karel G. M. Moons, Arno W. Hoes, and Kate Tilling. 2016. Sensitivity Analysis for the Effects of Multiple Unmeasured Confounders. Annals of Epidemiology 26 (9): 605–611.

https://doi.org/10.1016/j.annepidem.2016.07.009

Guilfoyle, Eoin, and Jose Pina-Sánchez. 2024. Racially Determined Case Characteristics: Exploring Disparities in the Use of Sentencing Factors in England and Wales. British Journal of Criminology 65 (2): 241–260. https://doi.org/10.1093/bjc/azae039

Gustafson, Paul. 2003. Measurement Error and Misclassification in Statistics and Epidemiology: Impacts and Bayesian Adjustments. Boca Raton, FL: CRC Press.

https://doi.org/10.1201/9780203502761

Halevy, T. 1995. Racial Discrimination in Sentencing: A Study with Dubious Conclusions. Criminal Law Review: 267–271.

Hartley, Richard D., and J. Mitchell Miller. 2010. Crack-ing the Media Myth: Reconsidering Sentencing Severity for Cocaine Offenders by Drug Type. Criminal Justice Review 35 (1): 67–89. https://doi.org/10.1177/0734016809348359

Heckman, James. 1976. The Common Structure of Statistical Models of Truncation, Sample Selection and Limited Dependent Variables and a Simple Estimator for Such Models. Annals of Economic and Social Measurement 5: 475–492.

Hester, Rhys, and Todd Hartman. 2017. Conditional Race Disparities in Criminal Sentencing: A Test of the Liberation Hypothesis from a Non-Guidelines State. Journal of Quantitative Criminology 33: 77–100. https://doi.org/10.1007/s10940-016-9283-z

Hofer, Paul J. 2019. Federal Sentencing after Booker. Crime and Justice 48 (1): 137–186.

https://doi.org/10.1086/701712

Holland, Melanie M., and Ariane Prohaska. 2021. Gender Effects across Place: A Multilevel Investigation of Gender, Race/Ethnicity, and Region in Sentencing. Race and Justice 11 (1): 91–112. https://doi.org/10.1177/2153368718767495

Holmes, Bryan, and Ben Feldmeyer. 2024. Modeling Matters: Comparing the Presumptive Sentence versus Base Offense Level Approaches for Estimating Racial/Ethnic Effects on Federal Sentencing. Journal of Quantitative Criminology 40 (2): 395–420.

https://doi.org/10.1007/s10940-023-09573-0

Hopkins, Kathryn, Miriam Light, and Jorgen Lovbakke. 2013. Analysis of Gender as a Factor Associated with Custodial Sentences for Breach of a Court Order. London: Ministry of Justice.

Hopkins, Kathryn, Noah Uhrig, and Matthew Colahan. 2016. Associations between Ethnic Background and Being Sentenced to Prison in the Crown Court in England and Wales in 2015. Technical report. London: Ministry of Justice.

Irwin-Rogers, Keir, and Thomas Perry. 2015. Exploring the Impact of Sentencing Factors on Sentencing Domestic Burglary. In Sentencing Guidelines: Exploring Sentencing Practice in England and Wales, edited by J. Roberts, 213–239. Basingstoke: Palgrave.

https://doi.org/10.1057/9781137390400_11

Isaac, Amber 2020. Investigating the Association between an Offender’s Sex and Ethnicity and the Sentence Imposed at the Crown Court for Drug Offences. Technical report. London: Sentencing Council for England and Wales.

Johansen, Louise V. 2019. Impressed by Feelings: How Judges Perceive Defendants’ Emotional Expressions in Danish Courtrooms. Social & Legal Studies 28 (2): 250–269. https://doi.org/10.1177/0964663918764004

Johnson, Brian D. 2006. The Multilevel Context of Criminal Sentencing: Integrating Judge- and County-Level Influences. Criminology 44 (2): 259–298.

https://doi.org/10.1111/j.1745-9125.2006.00049.x

Johnson, Brian D. 2014. Judges on Trial: A Reexamination of Judicial Race and Gender Effects across Modes of Conviction. Criminal Justice Policy Review 25 (2): 159–184.

https://doi.org/10.1177/0887403412463933

Kemp, Steven, and Daniel Varona. 2023. Foreign and Dangerous? Unpacking the Role of Judges and Prosecutors in Sentencing Disparities in Spain. British Journal of Criminology 63: 984–1002. https://doi.org/10.1093/bjc/azac068

King, Ryan, Kecia R. Johnson, and Kelly McGeever. 2010. Demography of the Legal Profession and Racial Disparities in Sentencing. Law & Society Review 44 (1): 1–32.

https://doi.org/10.1111/j.1540-5893.2010.00394.x

Knox, Dean, Will Lowe, and Jonathan Mummolo. 2020. Administrative Records Mask Racially Biased Policing. American Political Science Review 114 (3): 619–637.

https://doi.org/10.1017/S0003055420000039

Kurlychek, Megan C., and Brian D. Johnson. 2019. Cumulative Disadvantage in the American Criminal Justice System. Annual Review of Criminology 2: 291–319.

https://doi.org/10.1146/annurev-criminol-011518-024815

Lash, Timothy L., Matthew P. Fox, Richard F. MacLehose, George Maldonado, Lawrence C. McCandless, and Sander Greenland. 2014. Good Practices for Quantitative Bias Analysis. International Journal of Epidemiology 43 (6): 1969–1985. https://doi.org/10.1093/ije/dyu149

Leclerc, Chloé, and Pierre Tremblay. 2016. Looking at Penalty Scales: How Judicial Actors and the General Public Judge Penal Severity. Canadian Journal of Criminology and Criminal Justice 58 (3): 354–384. https://doi.org/10.3138/cjccj.2014.E31

Lundberg, Ian, Rebecca Johnson, and Brandon M. Stewart. 2021. What Is Your Estimand? Defining the Target Quantity Connects Statistical Evidence to Theory. American Sociological Review 86 (3): 532–565. https://doi.org/10.1177/00031224211004187

Lymperopoulou, Kitty. 2024. Ethnic Inequalities in Sentencing: Evidence from the Crown Court in England and Wales. British Journal of Criminology 64 (5): 1189–1210.

https://doi.org/10.1093/bjc/azae005

Lynch, Mona. 2019. Focally Concerned about Focal Concerns: A Conceptual and Methodological Critique of Sentencing Disparities Research. Justice Quarterly 36 (7): 1148–1175. https://doi.org/10.1080/07418825.2019.1686163

Malin, Tiina, and Maiju Tanskanen. 2024. Exploring Sentencing Disparities in the Nordic Context: A Multilevel Analysis of Court- and Judge-Level Variation in Sentences of Aggravated Driving under the Influence in Finnish District Courts. Criminology & Criminal Justice: 17488958241270733. https://doi.org/10.1177/17488958241270733

Maslen, Hannah. 2015. Remorse, Penal Theory and Sentencing. London: Bloomsbury Publishing.

Metcalfe, Christi, and Ted Chiricos. 2018. Race, Plea, and Charge Reduction: An Assessment of Racial Disparities in the Plea Process. Justice Quarterly 35 (2): 223–253.

https://doi.org/10.1080/07418825.2017.1304564

Ministry of Justice. 2017. Criminal Justice System Statistics Quarterly: December 2016. London: Ministry of Justice.

Mitchell, Ojmarrh. 2005. A Meta-Analysis of Race and Sentencing Research: Explaining the Inconsistencies. Journal of Quantitative Criminology 21 (4): 439–466.

https://doi.org/10.1007/s10940-005-7362-7

Mize, Trenton D., Long Doan, and J. Scott Long. 2019. A General Framework for Comparing Predictors and Marginal Effects across Models. Sociological Methodology 49 (1): 152–189. https://doi.org/10.1177/0081175019852763

Ostrom, Brian J., Charles W. Ostrom, Roger A. Hanson, and Matthew Kleiman. 2008. Assessing Consistency and Fairness in Sentencing: A Comparative Study in Three States. Washington, DC: National Institute of Justice.

Pearl, Judea. 2009. Causality. Cambridge: Cambridge University Press.

Petersen, Nick, and Marisa Omori. 2020. Is the Process the Only Punishment? Racial–Ethnic Disparities in Lower-Level Courts. Law & Policy 42 (1): 56–77.

https://doi.org/10.1111/lapo.12140

Pina-Sánchez, Jose, Sara Geneletti, Ana Veiga, Ana Morales, and Eoin Guilfoyle. 2024. Can Ethnic Disparities in Sentencing Be Taken as Evidence of Judicial Discrimination? Journal of Legal Research Methodology 3 (1): 54–82. https://doi.org/10.19164/jlrm.v3i1.1394

Pina-Sánchez, Jose, and John Paul Gosling. 2020. Tackling Selection Bias in Sentencing Data Analysis: A New Approach Based on a Scale of Severity. Quality & Quantity 54: 1047–1073. https://doi.org/10.1007/s11135-020-00973-z

Pina-Sánchez, Jose, Diana Grech, Ian Brunton-Smith, and Dimitrios Sferopoulos. 2019. Exploring the Origin of Sentencing Disparities in the Crown Court: Using Text Mining Techniques to Differentiate between Court and Judge Disparities. Social Science Research 84. https://doi.org/10.1016/j.ssresearch.2019.102343

Pina-Sánchez, Jose, and Lyndon Harris. 2020. Sentencing Gender? Investigating the Extent and Origin of Sentencing Gender Disparities in the Crown Court. Criminal Law Review 1: 3–28.

Pina-Sánchez, Jose, and Robin Linacre. 2016. Refining the Measurement of Consistency in Sentencing: A Methodological Review. International Journal of Law, Crime and Justice 44: 68–87. https://doi.org/10.1016/j.ijlcj.2015.06.001

Pina-Sánchez, Jose, Julian V. Roberts, and Dimitrios Sferopoulos. 2019. Does the Crown Court Discriminate against Muslim-Named Offenders? A Novel Investigation Based on Text Mining Techniques. British Journal of Criminology 59 (3): 718–736.

https://doi.org/10.1093/bjc/azy062

Pina-Sánchez, Jose, and Ian Brunton-Smith. 2025. What Is the External Validity of Sentencing Research? A Multi-Level Meta-Analysis of Race and Gender Disparities. SocArXiv.

https://doi.org/10.31235/osf.io/5d2bq_v3

Pina-Sánchez, Jose, and John Paul Gosling. 2022. Enhancing the Measurement of Sentence Severity through Expert Knowledge Elicitation. Journal of Legal Research Methodology 2 (1): 26–45. https://doi.org/10.19164/jlrm.v2i1.1241

Pina-Sánchez, Jose, Ana Morales, Eoin Guilfoyle, Ana Veiga, and Sara Geneletti. 2024. Testing the Interrelationship between Area Deprivation and Ethnic Disparities in Sentencing. Analyses of Social Issues and Public Policy. https://doi.org/10.1111/asap.12446

Rehavi, M. Marit, and Sonja B. Starr. 2014. Racial Disparity in Federal Criminal Sentences. Journal of Political Economy 122 (6): 1320–1354. https://doi.org/10.1086/677255

Richiardi, Lorenzo, Rino Bellocco, and Daniela Zugna. 2013. Mediation Analysis in Epidemiology: Methods, Interpretation and Bias. International Journal of Epidemiology 42 (5): 1511–1519. https://doi.org/10.1093/ije/dyt127

Rossmanith, Kate. 2015. Affect and the Judicial Assessment of Offenders: Feeling and Judging Remorse. Body & Society 21 (2): 167–193. https://doi.org/10.1177/1357034X14558073

Ruhland, Ebony, L., and Bryan Holmes. 2023. An Examination of Sentencing Outcomes in Rural and Urban Locations. American Journal of Criminal Justice 48: 701–722.

https://doi.org/10.1007/s12103-022-09678-5

Sentencing Council. 2015. Annex B: Quality and Methodology Note. Technical report. London: Sentencing Council.

Simonsohn, Uri, Joseph P. Simmons, and Leif D. Nelson. 2020. Specification Curve Analysis. Nature Human Behaviour 4 (11): 1208–1214. https://doi.org/10.1038/s41562-020-0912-z

Smith, Louisa H., Maya B. Mathur, and Tyler J. VanderWeele. 2021. Multiple-Bias Sensitivity Analysis Using Bounds. Epidemiology 32 (5): 625–634.

DOI: 10.1097/EDE.0000000000001380

Spohn, Cassia. 2000. Thirty Years of Sentencing Reform: The Quest for a Racially Neutral Sentencing Process. Criminal Justice 3 (1): 427–501.

Starr, Sonja, B., and M. Marit Rehavi. 2013. Mandatory Sentencing and Racial Disparity: Assessing the Role of Prosecutors and the Effects of Booker. Yale Law Journal 123 (1): 2–80.

Steffensmeier, Darrell, and Stephen Demuth. 2006. Does Gender Modify the Effects of Race–Ethnicity on Criminal Sanctioning? Sentences for Male and Female White, Black, and Hispanic Defendants. Journal of Quantitative Criminology 22: 241–261.

https://doi.org/10.1007/s10940-006-9010-2

Steffensmeier, Darrell, Hua Zhong, Jeff Ackerman, Jennifer Schwartz, and Suzanne Agha. 2006. Gender Gap Trends for Violent Crimes, 1980 to 2003: A UCR-NCVS Comparison. Feminist Criminology 1 (1): 72–98. https://doi.org/10.1177/1557085105283953

Steffensmeier, Darrell J., Emilie Andersen Allan, Miles D. Harer, and Cathy Streifel. 1989. Age and the Distribution of Crime. American Journal of Sociology 94 (4): 803–831.

https://doi.org/10.1086/229069

Stockton, Benjamin, C. Clare Strange, and Ofer Harel. 2024. Now You See It, Now You Don’t: A Simulation and Illustration of the Importance of Treating Incomplete Data in Estimating Race Effects in Sentencing. Journal of Quantitative Criminology 40: 563–590.

https://doi.org/10.1007/s10940-023-09577-w

Sturge, Georgina 2021. Court Statistics for England and Wales. Technical report. London: Ministry of Justice.

Tennant, Peter W. G., Eleanor J. Murray, Kellyn F. Arnold, Laurie Berrie, Matthew P. Fox, Sarah C. Gadd, Wendy J. Harrison, Claire Keeble, Lynsie R. Ranker, Johannes Textor, Georgia D. Tomova, Mark S. Gilthorpe, and George T. H. Ellison. 2021. Use of Directed Acyclic Graphs (DAGs) to Identify Confounders in Applied Health Research: Review and Recommendations. International Journal of Epidemiology 50 (2): 620–632.

https://doi.org/10.1093/ije/dyaa213

Textor, Johannes, Benito Van der Zander, Mark S. Gilthorpe, Maciej Liśkiewicz, and George T. H. Ellison. 2016. Robust Causal Inference Using Directed Acyclic Graphs: The R Package Dagitty. International Journal of Epidemiology 45 (6): 1887–1894.

https://doi.org/10.1093/ije/dyw341

Ulmer, Jeffery. 2018. Race, Ethnicity, and Sentencing. Oxford Research Encyclopedia of Criminology. Oxford: Oxford University Press.

https://doi.org/10.1093/acrefore/9780190264079.013.262

Ulmer, Jeffery T. 2000. The Rules Have Changed—So Proceed with Caution: A Comment on Engen and Gainey’s Method for Modeling Sentencing Outcomes under Guidelines. Criminology 38 (4): 1231–1243. https://doi.org/10.1111/j.1745-9125.2000.tb01420.x

Ulmer, Jeffery T. 2011. Recent Developments and New Directions in Sentencing Research. Justice Quarterly 29 (1): 1–40. https://doi.org/10.1080/07418825.2011.624115

Ulmer, Jeffery T., and Brian Johnson. 2004. Sentencing in Context: A Multilevel Analysis. Criminology 42: 137–177. https://doi.org/10.1111/j.1745-9125.2004.tb00516.x

Ulmer, Jeffery T., and John H. Kramer. 1996. Court Communities under Sentencing Guidelines: Dilemmas of Formal Rationality and Sentencing Disparity. Criminology 34 (3): 289–464. https://doi.org/10.1111/j.1745-9125.1996.tb01212.x

Ulmer, Jeffery T., and Brandy R. Parker. 2020. Federal Sentencing of Hispanic Defendants in Changing Immigrant Destinations. Justice Quarterly 37 (3): 541–570.

https://doi.org/10.1080/07418825.2019.1621362

United States Sentencing Commission. 2004. Fifteen Years of Guidelines Sentencing. Washington, DC: United States Sentencing Commission.

United States Sentencing Commission. 2012. Report to the Congress: Continuing Impact of United States v. Booker on Federal Sentencing. Washington, DC: United States Sentencing Commission.

Van Buuren, Stef. 2018. Flexible Imputation of Missing Data. Boca Raton, FL: CRC Press.

https://doi.org/10.1201/9780429492259

VanderWeele, Tyler J., and Peng Ding. 2017. Sensitivity Analysis in Observational Research: Introducing the E-Value. Annals of Internal Medicine 167 (4): 268–274.

https://doi.org/10.7326/M16-2607

VanderWeele, Tyler J., and Nancy Staudt. 2011. Causal Diagrams for Empirical Legal Research: A Methodology for Identifying Causation, Avoiding Bias and Interpreting Results. Law, Probability and Risk 10 (4): 329–354. https://doi.org/10.1093/lpr/mgr019

Ward, J. T., Richard D. Hartley, and Rob Tillyer. 2016. Unpacking Gender and Racial/Ethnic Biases in the Federal Sentencing of Drug Offenders: A Causal Mediation Approach. Journal of Criminal Justice 46: 196–206. https://doi.org/10.1016/j.jcrimjus.2016.05.008

Wermink, Hilde, Michael T. Light, and Alicja P. Krubnik. 2022. Pretrial Detention and Incarceration Decisions for Foreign Nationals: A Mixed-Methods Approach. European Journal on Criminal Policy and Research 28 (3): 367–380. https://doi.org/10.1007/s10610-022-09519-5

Wilbanks, William. 1987. The Myth of a Racist Criminal Justice System. Monterey, CA: Brooks/Cole.

Wooldredge, John D. 1998. Analytical Rigor in Studies of Disparities in Criminal Case Processing. Journal of Quantitative Criminology 14: 155–179.

https://doi.org/10.1023/A:1023076104837

Wu, Jawjeong, and Cassia Spohn. 2010. Interdistrict Disparity in Sentencing in Three U.S. District Courts. Crime & Delinquency 56 (2): 290-322.

https://doi.org/10.1177/001112870730999

Yang, Crystal S. 2014. Have Interjudge Sentencing Disparities Increased in an Advisory Guidelines Regime? Evidence from Booker. NYU Law Review 89: 1268–1342.

Yang, Crystal S. 2015. Free at Last? Judicial Discretion and Racial Disparities in Federal Sentencing. Journal of Legal Studies 44 (1): 75–111.

Zatz, Marjorie S. 1987. The Changing Forms of Racial/Ethnic Biases in Sentencing. Journal of Research in Crime and Delinquency 24: 69–92.

https://doi.org/10.1177/0022427887024001005

Zhang, Jun, and Kai F. Yu. 1998. What’s the Relative Risk? A Method of Correcting the Odds Ratio in Cohort Studies of Common Outcomes. Journal of the American Medical Association 280 (19): 1690–1691. doi:10.1001/jama.280.19.1690