# data[].attributes.headline
Definición de Plataformas de Aprendizaje Adaptativo

# data[].attributes.subject
Definición de Plataformas de Aprendizaje Adaptativo

# data[].attributes.body
Las plataformas de aprendizaje adaptativo son sistemas educativos impulsados por Inteligencia Artificial (IA) que personalizan la experiencia de aprendizaje. Analizan datos del estudiante (como sus respuestas, tiempo de resolución, patrones de clic y errores comunes) para ajustar dinámicamente la dificultad, el tipo de contenido y la ruta de aprendizaje en tiempo real.

Cita:

Baker, R. S., & Inventado, P. S. (2014). Educational Data Mining and Learning Analytics. In J. A. Larusson & B. White (Eds.), Learning Analytics: From Research to Practice (pp. 61-75). Springer, New York, NY.
(Este trabajo fundacional describe cómo los datos educativos se utilizan para adaptar y personalizar el aprendizaje).

# data[].attributes.headline
En Contexto Educativo ¿Qué es sesgo algorítmico? 

# data[].attributes.subject
En Contexto Educativo ¿Qué es sesgo algorítmico? 

# data[].attributes.body
El sesgo algorítmico surge cuando estas plataformas, de manera sistemática e injusta, producen resultados desventajosos para un grupo de estudiantes respecto a otro, en este caso, por razones de género.

Cita:

O'Neil, C. (2016). Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Crown Publishing Group.
(O'Neil define los "sesgos algorítmicos" como procesos que perpetúan la desigualdad de manera sistémica, especialmente en contextos sensibles como la educación).

# data[].attributes.body
Los algoritmos se "entrenan" con grandes conjuntos de datos históricos. Si estos datos reflejan estereotipos de género existentes (ej: que los chicos son mejores en matemáticas o que las chicas son más aplicadas en lenguaje), la IA aprenderá y reproducirá esos patrones.

Cita:

Bolukbasi, T., Chang, K. W., Zou, J. Y., Saligrama, V., & Kalai, A. T. (2016). Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. Advances in Neural Information Processing Systems, 29.
(Este estudio seminal muestra cómo los sesgos de género en los datos de entrenamiento se reflejan en los modelos de IA).

# data[].attributes.body
Las métricas que se eligen para medir el "éxito" o la "habilidad" pueden estar viciadas. Por ejemplo, si se prioriza la velocidad sobre la precisión, se puede penalizar involuntariamente a estudiantes más reflexivas.

Cita:

Bucher, T. (2018). If...Then: Algorithmic Power and Politics. Oxford University Press.
(Bucher analiza cómo las métricas y los criterios de diseño en los algoritmos pueden incorporar valores y prejuicios humanos).

# data[].attributes.body
El sistema crea un ciclo de retroalimentación. Si comienza a recomendar contenido de matemáticas más avanzado a los chicos porque el algoritmo asume que son mejores, ellos se volverán efectivamente mejores, reforzando la predicción inicial y enterrando el sesgo más profundamente.

Cita:

Ensign, D., Friedler, S. A., Neville, S., Scheidegger, C., & Venkatasubramanian, S. (2018). Runaway Feedback Loops in Predictive Policing. Proceedings of the 1st Conference on Fairness, Accountability and Transparency, 160-171.
(Aunque se centra en otro ámbito, el artículo describe perfectamente el fenómeno de los "bucles de retroalimentación" que amplifican los sesgos iniciales en sistemas algorítmicos).

# data[].attributes.headline
¿Qué consecuencias impactan en los/las estudiantes?

# data[].attributes.subject
¿Qué consecuencias impactan en los/las estudiantes?

# data[].attributes.body
Consecuencias para las estudiantes:

Direccionamiento Vocacional: Pueden ser guiadas lejos de áreas STEM (Ciencia, Tecnología, Ingeniería y Matemáticas) debido a recomendaciones basadas en estereotipos.

Techos de Aprendizaje: El algoritmo podría no ofrecerles desafíos de mayor dificultad, limitando inadvertidamente su potencial.

Refuerzo de Estereotipos: La plataforma, al tratarlas de manera diferente, envía el mensaje subliminal de que sus capacidades son inherentemente distintas.

Cita:

UNESCO. (2019). I'd Blush if I Could: Closing Gender Divides in Digital Skills through Education. Equals Global Partnership.
(Este informe documenta exhaustivamente cómo los sesgos de género en la tecnología y la educación pueden desviar a las mujeres y niñas de las carreras STEM y reforzar estereotipos dañinos).

# data[].attributes.body
En esencia, lejos de ser herramientas neutrales, estas plataformas corren el riesgo de automatizar y amplificar la discriminación preexistente, convirtiéndose en un nuevo desafío para lograr una educación equitativa.

Cita:

Eubanks, V. (2018). Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor. St. Martin's Press.
(Eubanks argumenta de manera convincente que los sistemas automatizados a menudo institutionalizan y amplifican las desigualdades preexistentes en lugar de eliminarlas).

Benjamin, R. (2019). Race After Technology: Abolitionist Tools for the New Jim Code. Polity Press.
(El concepto de "New Jim Code" de Benjamin se aplica aquí para describir cómo la tecnología puede codificar la discriminación, incluido el sesgo de género en la educación).

# data[].attributes.headline
IA en corto - Ep 4 - Sesgos algorítmicos

# data[].attributes.subject
IA en corto - Ep 4 - Sesgos algorítmicos

# data[].attributes.body
¿Sabías que las computadoras pueden replicar prejuicios y sesgos a la hora de tomar una decisión?

En este episodio, exploramos cómo los algoritmos, a pesar de su aparente neutralidad, pueden verse afectados por sesgos inherentes. Descubrirás cómo estos sesgos pueden impactar nuestras vidas diarias, desde recomendaciones en línea hasta decisiones cruciales.

Tipo de fuente: Video Youtube de CENIA (Centro Nacional de inteligencia Artificial)

# data[].attributes.attachment_link.authors[].name
CENIA - Centro Nacional de Inteligencia Artificial

# data[].attributes.attachment_link.title
IA en corto - Ep 4 - Sesgos algorítmicos

# data[].attributes.body
En este artículo, revisamos el sesgo algorítmico en la educación, discutimos las causas de ese sesgo y revisamos la literatura empírica sobre las formas específicas en que se sabe que el sesgo algorítmico se ha manifestado en la educación. Si bien otros trabajos recientes han revisado las definiciones matemáticas de equidad y ampliado los enfoques algorítmicos para reducir el sesgo, nuestra revisión se centra en cambio en solidificar la comprensión actual de los impactos concretos del sesgo algorítmico en la educación —qué grupos se sabe que se ven afectados y qué etapas y agentes están implicados en el desarrollo y la implementación de algoritmos educativos. Discutimos perspectivas teóricas y formales sobre el sesgo algorítmico, conectamos esas perspectivas con el proceso de aprendizaje automático y revisamos métricas para evaluar el sesgo. A continuación, revisamos la evidencia sobre el sesgo algorítmico en la educación,comenzando con las categorías más estudiadas de raza/etnia, género y nacionalidad, y pasando a la evidencia disponible de sesgo hacia categorías menos estudiadas, como el estatus socioeconómico, la discapacidad y el estatus relacionado con el ejército

Tipo de fuente: cita bibliográfica o paper científico

Baker, R.S., Hawn, A. Algorithmic Bias in Education. Int J Artif Intell Educ 32, 1052–1092 (2022). https://doi.org/10.1007/s40593-021-00285-9

# data[].attributes.attachment_link.readable_text
Algorithmic Bias in Education

Abstract

In this paper, we review algorithmic bias in education, discussing the causes of that bias and reviewing the empirical literature on the specific ways that algorithmic bias is known to have manifested in education. While other recent work has reviewed mathematical definitions of fairness and expanded algorithmic approaches to reducing bias, our review focuses instead on solidifying the current understanding of the concrete impacts of algorithmic bias in education-which groups are known to be impacted and which stages and agents in the development and deployment of educational algorithms are implicated. We discuss theoretical and formal perspectives on algorithmic bias, connect those perspectives to the machine learning pipeline, and review metrics for assessing bias. Next, we review the evidence around algorithmic bias in education, beginning with the most heavily-studied categories of race/ethnicity, gender, and nationality, and moving to the available evidence of bias for less-studied categories, such as socioeconomic status, disability, and military-connected status. Acknowledging the gaps in what has been studied, we propose a framework for moving from unknown bias to known bias and from fairness to equity. We discuss obstacles to addressing these challenges and propose four areas of effort for mitigating and resolving the problems of algorithmic bias in AIED systems and other educational technology.

Similar content being viewed by others

Article Open access

Article

Discover the latest articles, books and news in related subjects, suggested using machine learning.

Introduction

Today, algorithms influence our lives in a wide variety of ways. Computer algorithms that make decisions and predictions are often viewed as inherently fair and objective (Lee, 2018). But in recent years, a competing perspective has emerged -- the perspective that algorithms often encode the biases of their developers or the surrounding society, producing predictions or inferences that are clearly discriminatory towards specific groups. Examples of algorithmic bias cross contexts, from criminal justice (Angwin et al., 2016), to medicine (O'Reilly-Shah et al., 2020), to computer vision (Klare et al., 2012), to hiring (Garcia, 2016). These limitations appear -- and are particularly salient -- for high-stakes decisions such as predicting recidivism (Angwin et al., 2016) or administering anaesthesia (O'Reilly-Shah et al., 2020).

This limitation is found in educational algorithms as well, an increasing number of papers assert (e.g. Bridgeman et al., 2009, 2012; Ocumpaugh et al., 2014; Yudelson et al., 2014; Kai et al., 2017; Hu & Rangwala, 2020; Yu et al., 2020). The problem of bias in educational testing has been documented since the 1960s and anticipated many aspects of the modern literature on algorithmic bias and fairness (see review and discussion in Hutchinson & Mitchell, 2019). In recent years, algorithms have become applied in educational practices at scale for a range of applications, often high-stakes, including dropout prediction (Christie et al., 2019; Milliron et al., 2014), automated essay scoring (Ramineni & Williamson, 2013), graduate admissions (Waters & Miikkulainen, 2014), and knowledge inference (Ritter et al., 2016). Academics have been warning about possible uneven effectiveness and lack of generalizability across populations in educational algorithms for several years (e.g. Bridgeman et al., 2009; Ocumpaugh et al., 2014). This concern became very salient to the general public in the 2020 UK GCSE and A-Level grading controversy (H. Smith, 2020), where a set of formulas was developed by the national qualifications regulator (by hand rather than by an automated algorithm) to assign predicted examination grades based on teacher predictions -- the algorithm assigned poorer grades to students in state-funded schools and better grades (even better than teacher prediction) to students in smaller independent schools.

In this paper, we review algorithmic bias in education, discussing theoretical work on the root causes of algorithmic bias, and reviewing the existing empirical literature on the specific ways that algorithmic bias is known to have manifested in education. In doing so, we distinguish ourselves from more algorithmically-focused reviews of algorithmic bias in education (e.g. Kizilcec & Lee, 2021). There is a great deal of merit to reviewing mathematical definitions of fairness and algorithmic approaches to reducing bias. Our review focuses instead on understanding exactly who appears to be impacted, and the impact played by the context surrounding the algorithms themselves. We focus in particular on biases emerging from how variables are operationalized and which data sets are used. This review is also distinct from broader discussions of how artificially intelligent technologies can be biased (e.g. Holstein & Doroudi, in press), including in the processes of the design of these technologies, focusing on the narrower issue of bias in the algorithms used to assess and make decisions.

Our review starts in Section 2 with a discussion of theoretical and formal perspectives on algorithmic bias. Within that, we define algorithmic bias, and connect it to different perspectives on the machine learning pipeline. We review metrics for assessing bias and taxonomies on forms bias, and consider the essential link between two forms of bias in data collection -- situating key challenges around algorithmic bias not in the algorithms themselves, but in the data input to those algorithms.

In Section 3, we review the evidence around algorithmic bias in education. We start by reviewing the evidence around the most heavily-studied categories (race/ethnicity, gender, and nationality). We then review and discuss the evidence for algorithmic bias in less-studied categories. We then discuss the (very large) gaps in what has been studied, with a focus on the America-centric bias of most of the research on algorithmic bias in education.

Finally, in Section 4, we propose a framework for moving from unknown bias to known bias to fairness. We discuss the obstacles to addressing algorithmic bias in education, and propose four directions for the field of AIED to move in, that can help to mitigate and resolve the problem of algorithmic bias, for the benefit of the students our community serves.

Theoretical and Formal Perspectives on Algorithmic bias

Defining Algorithmic bias

Discussions of algorithmic bias in education have been complicated by overlapping meanings of the term bias (Crawford, 2017; Blodgett et al., 2020). For example, a recent survey of the uses of the term bias in natural language processing across 146 papers found several areas for clarification in how authors define and write about bias, from a lack of explanation about how exactly systems were biased to confusion as to the eventual harms these forms of bias might cause (Blodgett et al., 2020). While not attempting to reconcile these conflicts, we will briefly discuss some issues of definition for algorithmic bias before proposing a more limited working definition we use in our survey.

Algorithmic bias in Emerging Use

The term algorithmic bias has been applied to an array of examples of unfairness in automated systems, only some of which seem to fit statistical or technical definitions of bias. In response, as examples have appeared in popular use, researchers have attempted to describe criteria for bias. Mitchell et al. ( 2021) summarizes "popular media" usage of the term biased as describing cases where "a model's predictive performance (however defined) unjustifiably differs across disadvantaged groups along social axes such as race, gender, and class" (p.1). Similarly, Gardner et al. ( 2019) use bias as "inequitable prediction across identity groups" (p. 228). Applying a broader definition, Suresh and Guttag ( 2020), refer to biases as possible sources of harm throughout the machine learning process, including "unintended or potentially harmful" properties of the data that lead to "unwanted or societally unfavorable outcome[s]" (p. 1-2).

Other authors favor the use of unfair over biased, preserving bias for its statistical meaning and using fair/unfair for its social/moral implications. Mehrabi et al. ( 2019) describe fairness in the context of algorithmic decision-making as "the absence of any prejudice or favoritism toward an individual or a group based on their inherent or acquired characteristics. An unfair algorithm is one whose decisions are skewed towards a particular group of people" (p. 1). Similarly, while acknowledging that algorithmic bias, in its popular sense, refers to socially objectionable "demographic disparities," Barocas et al. ( 2019) avoid this popular usage of bias altogether, instead using demographic disparity and discrimination to refer to the negative impacts of applying some models, while maintaining bias in its statistical sense as systematic error in either data or model estimates. Friedman and Nissenbaum's ( 1996) definition foreshadows these distinctions, claiming that biased computer systems " systematically and unfairly discriminate against individuals or groups of individuals in favor of others [emphasis in original]" (p. 332).

Overall a range of "biases" appear across definitions, from the statistical biases of measurement and error to imbalances in how well a model performs across groups, to systematic skew in results, to disparate impacts and discrimination as model results are interpreted and applied.

As the nature and impacts of algorithmic bias continue to emerge, Blodgett et al.'s ( 2020) suggestions for clarity in description of bias are timely, particularly in education, where fewer cases of algorithmic bias impacting real-world outcomes have been reported, compared with criminal justice or healthcare. The authors suggest including descriptions of the kinds of system behaviors found to be harmful, how and why the system behaviors are harmful, who is harmed, and the normative reasons for making these judgements.

Research outside of education has put a spotlight on some of the very real harms that can result from algorithmic bias. Such harms have been categorized broadly into allocative and representational forms (Crawford, 2017; Suresh & Guttag, 2020). Allocative harms result from the withholding of some opportunity or resource from specific groups or the unfair distribution of a good across groups. Some examples include gender and racial bias in ad delivery for housing and employment (Ali et al., 2019; Benner et al., 2019); gender bias in assigning credit limits (Knight, 2019; Telford, 2019); racial bias in sentencing decisions (Angwin et al., 2016), racial bias in identifying patients for additional health care (Obermeyer et al., 2019), and -- in education -- bias in standardized testing and its resulting impact on high stakes admission decisions (Dorans, 2010; Santelices & Wilson, 2010). Representational harms, on the other hand, manifest as the systematic representation of some group in a negative light, or in a lack of positive representation (Crawford, 2017). Multiple forms of representational harm have been uncovered in recent years. Work by Sweeney ( 2013) identifies representational harms of denigration and stereotyping, where the word "criminal" was more frequently returned in online ads after searches for black-identifying first names. Kay et al. ( 2015) describe other representational harms of recognition and under-representation, with work finding that image search results for male-dominated professions displayed a higher proportion of males than the proportions suggested by the data from U.S. Bureau of Labor and Statistics.

As can be seen, there are a range of ways that algorithmic bias can be defined and viewed. In the current article, we focus on studying algorithmic bias in terms of situations where model performance is substantially better or worse across mutually exclusive groups (i.e. Gardner et al., 2019; Mehrabi et al., 2019; Mitchell et al., 2021). Other forms of algorithmic bias can also be highly problematic, but -- as we discuss below -- the published research in education thus far has focused on this form of algorithmic bias. In this review, we also focus on the bias in algorithms, not in the broader design of the learning or educational systems that use these algorithms. Bias can emerge in the design of learning activities, leading to differential impact for different populations (Finkelstein et al., 2013), but this is a much broader topic than the bias that comes from algorithms.

Bias against Whom?

Researchers have also considered a range of groups which have been, or might be, impacted by algorithmic bias. Many of these groups have been defined by characteristics protected by law, in various countries. In the United Kingdom, for instance, the Equality Act 2010 merged over a hundred disparate pieces of legislation into a single legal framework, unifying protections against discrimination on the basis of sex, race, ethnicity, disability, religion, age, national origin, sexual orientation, and gender identity. In the United States, the same categories are protected by a combination of different legislation, commission rulings, and court rulings, dating back to the Civil Rights Act of 1964. Similar laws afford protections in the European Union and most other countries around the world, though differing somewhat in which groups are protected and how groups are defined.

However, while critical for social equity, looking for bias only under the lamppost of nationally-protected classes (categories often with their own complicated histories) may be leaving unexamined serious impacts on other, under-investigated, groups of people. Depending on the social context of specific algorithmic systems, other researchers have suggested additional characteristics vulnerable to bias, such as urbanicity (Ocumpaugh et al., 2014), military-connected status (Baker et al., 2020), or speed of learning (Doroudi & Brunskill, 2019). Section 3.2 reviews the limited education research into algorithmic bias associated with other groups.

While recognizing that many of the implications and harms of algorithmic bias currently fall outside of legal purview, legal frameworks used to decide which classes of people merit protection from discrimination may be helpful in assessing the unknown risks that algorithmic bias poses to as yet unidentified groups (Soundarajan & Clausen, 2018). Past discriminatory patterns against individuals, as well as the degree that class membership can be changed, may be useful in focusing efforts in bias mitigation. Judicial standards may also be helpful, particularly in medicine and education, in determining whether or not discrimination is justified by a compelling interest. Machine-learning applications that diagnose medical conditions or generate Early Warning Indicators (EWIs) for high school dropout could potentially satisfy criteria justifying discrimination when the benefits of an application outweigh possible harms, when individuals are aware of algorithmic decisions and are able to opt out or modify them, and when interventions are narrowly tailored to the relevant features of a targeted class (Soundarajan & Clausen, 2018).

A Note on the Term "Algorithmic"

In this review, most examples of algorithmic bias are drawn from cases where large datasets of past examples were used to train predictive models using algorithmic or statistical methods. In other work, the term algorithmic in this context has been used to mean automated decision-making systems in general, with machine-learning products and processes playing an increasing role in such automated systems. While this article uses algorithmic to refer more specifically to processes where models were created by applying automated algorithms to data, the recent A-Level grading controversies in the UK make clear that bias and discrimination can result in cases where models and automation were coded by hand as well. While most of the cases that we review involve automated algorithms rather than manually-created models, many of the themes we discuss are relevant to manually-created models as well.

Origins of bias and Harm in the Machine Learning Pipeline

Researchers in education have recently begun drawing attention to existing cases of algorithmic bias in educational technologies, as well as to the possible increase, through adoption of artificial intelligence and machine learning, of such bias and its harmful impacts. As a starting place in understanding the origins of algorithmic bias, Mitchell et al. ( 2021) make a helpful distinction between statistical and societal forms of bias, where statistical bias encompasses sampling bias and measurement error and societal bias refers to "concerns about objectionable social structures that are represented in the data" (p. 4). In a model used in practice, either of these forms of bias might contribute to overall algorithmic bias and possible real-world discrimination and harms. Several authors working at the intersection of computer science and the social sciences have laid the groundwork for this examination, expanding the statistical/societal distinction to describe the kinds of bias that can arise at each stage of the machine learning lifecycle and categorizing the possible harms and impacts arising from that bias.

In the context of fairness and bias, process descriptions of machine learning range from the simple to the complex. Some collapse the process into broad stages, such as Barocas et al. ( 2019) and Kizilcec and Lee ( 2021), who view the process as going from measurement to model learning to action, or Mehrabi et al. ( 2019), who considers the process as going from user interaction to data to algorithm. Others, such as Suresh and Guttag ( 2020), have conceptualized machine learning in a finer-grained way and have considered the possibility of bias at each of the following stages: data collection, data preparation, model development, model evaluation, model post-processing, and model deployment.

There have also been efforts in collaboration between industry and academia to develop process maps to support decision-making around bias in organizational contexts. One such process, illustrated in Fig. 1, includes additional stages for Task Definition prior to Dataset Construction, along with separate stages for a Testing Process prior to Deployment, and ongoing Feedback from users (Cramer et al., 2019). The authors contend that explicitly considering each of these stages is critical in an organizational response to mitigating bias.

Descriptions of the Machine Learning Process and Possible Sources of Bias

More exhaustively, Suresh and Guttag ( 2020) categorize bias into Historical bias, Representation bias, Measurement bias, Aggregation bias, Evaluation bias, and Deployment bias and map these forms of bias to stages in the machine learning process (See Fig. 1 for a complete mapping). Historical bias involves mismatches between the world as it is and values for how the world should be, causing models that replicate decisions made in the world to be biased. An example in education might be including student demographics as predictors of grades and generating a model that achieves better performance by using those features to predict lower grades for some students -- i.e. if a student is in group X, they get 5 points lower on their final grade (i.e. Wolff et al., 2013). Such a model achieves better accuracy at the cost of potentially perpetuating bias. Bias of this nature can also arise even when demographics are not explicitly encoded in models, for instance when a proxy for a demographic variable is unintentionally included as a predictor. This form of bias is surprisingly common in education -- in a recent survey of the role of demographics in the Educational Data Mining (EDM) community, Paquette et al. ( 2020) found that roughly half of papers including demographics in analyses used at least one demographic attribute as a predictive feature within the model, without incorporating demographics into model testing or validation.

Representational bias is when a group that is under-sampled in the training data is predicted more poorly; for example, Anderson et al. ( 2019) find that the college graduation prediction they develop works poorly for indigenous learners, who comprise a very small proportion of the overall sample. Measurement bias occurs when choosing variables, for example when variables do not have construct validity for what they are intended to represent, and biases in the variables chosen cause unequal prediction across groups. For example, a model predicting school violence may be biased if the labels of which students engage in violence are created by a process that involves prejudice - e.g. the same violent behavior by two students is documented and punished for members of one race but not for members of another race (Bireda, 2002). As Holstein and Doroudi ( in press) note, disparities in access to learning technologies can in turn create representational biases, lowering the quality of learning technologies for students who already have less access to them.

Moving past the data collection and preparation phases of machine learning and into model development, Suresh and Guttag ( 2020) discuss how aggregation bias occurs when distinct populations are combined in the same model and the resulting model does not work for some -- or all -- groups of learners. This is seen in Ocumpaugh et al. ( 2014), where detectors of student emotion trained on a combination of urban, rural, and suburban students function more poorly for all three groups than detectors trained on individual groups. Evaluation bias occurs when the test sets used to evaluate a model do not represent the eventual population where the model will be applied. As reviewed in Paquette et al. ( 2020), many models in educational data mining are developed on non-representative populations, and many papers do not even report what populations the models were tested on, making detection of evaluation bias quite difficult. Finally, deployment bias involves a model being used in inappropriate ways -- being designed for one purpose and then used for a different purpose, such as using a model designed to identify student disengagement for formative purposes to assign participation grades.

While raising overlapping concerns with Suresh and Guttag ( 2020), Olteanu et al. ( 2019) highlight the potential for bias specifically in the analysis of social data, such as user-generated content, online behavior and networks. To the degree that education algorithms leverage data from social software functions, they may be vulnerable to some of the same biases as analysis of social media in other contexts. The authors frame bias in a statistical sense, as a threat to the validity of research conclusions, where proof/disproof of hypotheses is undermined by biases that threaten the internal, external, or construct validity of the research findings. However, they note how examples of statistical bias may relate to the systemic, discriminatory biases which are often thought of in connection to algorithmic bias.

Bias of the forms discussed above has the potential to manifest in a range of educational applications. Algorithmic bias has been documented in situations ranging from at-risk prediction for high school or college dropout (Anderson et al., 2019), at-risk prediction for failing a course (Hu & Rangwala, 2020; Lee & Kizilcec, 2020), automated essay scoring (Bridgeman et al., 2009, 2012), assessment of spoken language proficiency (Wang et al., 2018), and even the detection of student emotion (Ocumpaugh et al., 2014). There has been documentation of algorithmic bias impacting educational algorithms in terms of student race, ethnicity, nationality, gender, native language, urbanicity, parental educational background, socioeconomic status, and whether a student has a parent in the military. We review the literature on these specific findings in Section 3 below.

Figure 1 summarizes several of the machine learning life-cycles mentioned previously alongside sources of bias that might arise at each stage. Where possible the broad categories of measurement, learning, and action on the vertical axis align with taxonomies from Barocas et al. ( 2019) and Kizilcec and Lee ( 2021). Sources of bias are drawn from multiple papers (Cramer et al., 2019; Mehrabi et al., 2019; Olteanu et al., 2019; Mitchell et al., 2021; Paullada et al., 2020; Suresh & Guttag, 2020), indicated by a bracketed number within the figure.

Alternative schema for the Origins of bias

While several attempts have been made at locating sources of bias within the machine learning pipeline (See Fig. 1), other researchers have argued for locating algorithmic bias, not only at a stage within this process, but also as the product of the social interactions surrounding the production and use of an algorithm. Drawing from sociocultural activity theory, Ferrero and Barujel (2019) describe an algorithm as the artifact of an activity system, locating bias within the connected parts of that system, where subjects make decisions within a context of objects, rules, community, and division of labor. Applying this alternative framework, they identify biases as either theoretical, methodological, interpretive, due to decontextualization, or due to data training. The authors' description of bias as the artifact of specific decisions by particular agents draws attention to the fact that identifying a stage at which bias is generated goes only part of the way towards mitigating that bias. The activity system view elaborates other possible pathways for eliminating bias by grounding the temporal stages of a process model in the decisions of subjects working with organizational constraints. Ferrero and Barujel's (2019) activity framing also encourages an examination of algorithmic bias as it develops across contexts. As commonly happens in education, one organization, perhaps commercial, develops a predictive algorithm, while another, a school or district, applies it (see discussion of this issue also in Holstein & Doroudi, in press). Locating machine learning in a similarly broad context, Dieterle et al. ( under review) describe algorithmic bias as one in a larger series of AI-driven divides in educational technology. The authors describe how a digital divide in access to online learning leads to divides in the representativeness of data across populations, in how algorithms are developed, in how data are interpreted in schools, and eventually in how civic society is impacted.

The increasing quantity of research and journalism over the last decade, empirically describing cases of algorithmic bias and their far-reaching harms, has galvanized public awareness and prompted extensive academic and industry research into the ways that algorithmic bias can be more effectively identified, mitigated, and its harms reduced.

Formal Fairness and its Application (in a Messy World)

Several researchers (Barocas et al., 2019; Kizilcec & Lee, 2021; Verma & Rubin, 2018) divide mathematical formalizations of fairness into high-level categories of statistical, similarity-based, and causal definitions. Statistical measures examine fairness through the lens of the confusion matrix, calculating relationships and ratios between predicted or true, positive or negative values in relation to membership in a sensitive group. Many of these formalized criteria for fairness connect back to three possible relationships of conditional probability across three main variables: the prediction, the true outcome, and membership in the relevant group. For example, the criteria of independence, requires simply that the algorithm's decision or score be independent of membership in the group under consideration, with specific measures including statistical parity, group fairness, and demographic parity (Barocas et al., 2019; Kizilcec & Lee, 2021). Judged against this criterion, an independent (and therefore fair) algorithm might predict that equal percentages of students belonging to different demographic groups will fail a college course or that urban and rural students will have equal knowledge of farming technologies. Independence can be useful in some contexts, particularly when there are laws or regulations requiring that individuals be admitted in equal proportion of racial or gender categories (Chouldechova, 2017; Makhlouf et al., 2020). However, this criterion has its limitations in not considering true outcomes in relation to predicted values; depending on context, this definition of fairness may correct for the historical biases impacting real-world data, or may produce undesirable results until those historical biases have been addressed.

Two other criteria of statistical fairness, separation and sufficiency, consider group membership and algorithmic predictions, while also incorporating the true outcome of the predicted variable. Generally, separation requires that an algorithm's performance be fair across groups, or stated another way, that correct and incorrect predictions are distributed equally in relation to the groups under consideration. This criterion has generated several attempts at precise formulation, from equal opportunity/equalized odds (Hardt et al., 2016) to slicing analysis (Gardner et al., 2019) to predictive parity (Chouldechova, 2017). The sufficiency criterion asks the slightly different question of whether true outcomes are independent of group membership, while taking into account the predicted values. Satisfying s ufficiency would require, for example, that of all students predicted to drop out in some educational setting, the same proportion are correctly predicted across sensitive groups (Kizilcec & Lee, 2021).

Along with this expanded set of formalized metrics and their clarifications of algorithmic bias has come the recognition that applying fairness measures in practice reveals its own range of obstacles. Specifically, technical obstacles to the use of fairness metrics manifest in several "impossibility" results (Chouldechova, 2017; Kleinberg et al., 2017; Berk et al., 2018; Loukina et al., 2019; Lee & Kizilcec, 2020, Darlington, 1971), where satisfaction of one statistical criterion of fairness makes "impossible" satisfaction of another. Kleinberg et al. ( 2017), for example, describe how the fairness criteria of calibration, balance for the positive class, and balance in the negative class cannot be simultaneously satisfied, except in special cases of perfect prediction and equal base rates, where both groups have an equal proportion of members in the positive class. Calibration here refers to the criterion that the predicted probability of a group to achieve the target variable should match the overall proportion of actual positive instances in the same group. Chouldechova ( 2017) comes to similar conclusions when analyzing the COMPAS dataset (Angwin et al., 2016) in light of competing fairness perspectives of calibration, predictive parity, and the balance of false predictions across groups.

Education-specific analyses have pointed out similar trade-offs in automated scoring for language proficiency exams (Loukina et al., 2019) and predictions of above-median grades in required college courses (Lee & Kizilcec, 2020). As Kleinberg et al. ( 2017) point out, varied rules for fairness provide slightly different answers to the same general question: are an algorithm's predictions equally effective across groups? Nonetheless, when the presence of the target variable is imbalanced across groups, attempts to satisfy criteria of calibration or predictive parity result in an imbalance in the rate of false positives or false negatives, leading to disparate impacts across groups (Chouldechova, 2017).

In historical perspective, Hutchinson and Mitchell ( 2019) point out that many formalized definitions of fairness have precedents from the 1960s and 1970s as researchers worked to minimize bias in standardized testing. By the mid-1970s, however, these same researchers recognized persistent confusion in the use of fairness metrics in practice, both among researchers and in communication to the general public (Cole & Zieky, 2001). Questions were raised about applying different fairness metrics in contexts of competing goals and values (Hunter & Schmidt, 1976). Other work noted that formal definitions often disagreed, that it was challenging to find a principled way to select between metrics, and broached the possibility that efforts towards the formalization of fairness might ultimately distract from more direct efforts to address societal problems of equity and justice (Petersen & Novick, 1976).

As these authors suggest, it is important for data scientists and AI researchers to push the boundaries of investigation beyond a canonical understanding of algorithms to an understanding that includes algorithmic interventions and application in context (Green & Viljoen, 2020). Towards that end, it may also be highly important to also focus on identifying and mitigating bias in the earlier stages of the machine learning lifecycle, the stages of data collection and data preparation. There has been extensive ML research addressing the later stages of the ML pipeline: model development, model evaluation, model postprocessing. There may also be considerable gains towards fairness to be found by examining and mitigating bias from the upstream portions of the cycle.

Representational and Measurement Biases: The Key Role for Data Collection

Much of the work studying algorithmic bias, both in education and beyond, has focused on how algorithms and metrics can be used to assess and remove algorithmic bias (see review in Kizilcec & Lee, 2021). Clearly, there is an important role for both of these types of work. Finding better metrics to assess algorithmic bias -- and pushing the field to use them -- can help to catch many forms of algorithmic bias. Algorithms that attempt to ensure fairness, or at least reduce the likelihood or degree of bias, can play an important role in improving outcomes for students who might otherwise be disadvantaged.

However, attempts to address biases through adjustments to algorithms -- even if the biases are identified at this point -- may be ineffective if we have not collected the right data. Both representational and measurement bias, to use the terminology from Suresh and Guttag ( 2020), can prevent methods further down the pipeline from being able to detect or resolve bias.

In terms of representational bias, if we do not collect data from the right sample of learners, we cannot expect our models to work on all learners. For example, if we collect training data only from suburban upper middle-class children, we should not expect our model to work for urban lower-income students (we may get lucky -- the model may indeed function effectively -- but we lack a basis for believing this).

A simple proportional data set may be insufficient -- there may be groups of learners for whom insufficient data has been collected to develop and validate a model. Even when we collect a seemingly complete data set -- every learner in a university we are developing a learning analytics model for -- this may still be insufficient. For example, Anderson et al. ( 2019) noted that their sample of all undergraduates in their university only had 44 learners of indigenous descent, a sample that was too small for their models to work reliably. In cases like this one, we may want to group data from multiple universities together so that there are enough learners of a group we want to ensure our models work for. In general, we may need to over-sample (in terms of actual data collection rather than the machine learning technique) learners from less common groups to be confident that we have enough data for each group of learners for whom the model may function less effectively.

And, all too often, we are unable to sample proportionally, particularly when sampling across institutions. As Baker ( 2019) notes, it is often much harder to collect data in some schools than others. Some urban school districts have extensive protections and limitations on research that make it far more costly and difficult to conduct research or collect data involving their students. Some rural school districts are sufficiently remote to make it highly costly and difficult to conduct studies in person. Some schools and even universities are too poor (or bureaucratic) to have high-quality data systems. Some organizations are simply more welcoming to researchers than others.

Measurement bias can also be a significant challenge that improved metrics or algorithms cannot entirely address. Although Suresh and Guttag ( 2020) refer to measurement biases in both training labels and predictor variables, the most concerning measurement biases involve training labels. If a training label is biased for some specific population -- for instance, if Black students are more likely to be labeled as engaging in school violence than White students, even for the same behavior -- then it is difficult to determine whether an algorithm works equally well for both groups, or indeed to find any way to be confident that the algorithm's functioning is not biased. The training label's bias may even come from the student themself, if -- for instance -- we are predicting a self-report variable that is prone to biases in student responses due to factors such as confidence, cultural interpretation, or stereotype threat (Tempelaar et al., 2020). In these cases, finding an alternate variable to predict -- one not as impacted by bias -- may be the best alternative.

In other cases, the measurement bias may emerge when human coders label data that has already been collected, a step in the pipeline which it is easier to exert control over. Sociocultural factors may impact the reliability of human labeling, for instance -- Okur et al. ( 2018) find that American coders labeling the emotions of Turkish students from facial expressions have systematic biases in their interpretation. These problems can go beyond just inaccuracy -- for example, racial biases can enter into value judgments (Kraiger & Ford, 1985).

In the case where a predictor is biased, it may end up serving as a proxy for a group label, in which case it may be best to discard it from consideration. Alternatively, if it seems to have predictive power after controlling for group, it may be possible to create a distilled variable that contains this predictor's variance after partialing out the group variable.

Population Factors and Student Modeling

What Have we Learned about Algorithmic bias in Education from Looking at the most Commonly Studied Demographic Categories?

The majority of research on actual algorithmic bias in education (as opposed to theoretical algorithmic bias in education) has looked at three categories: race/ethnicity, nationality (comparing learners' current national locations), and gender. In this section, we review this literature and investigate what the field has learned from these investigations. Table 1 lists the studies included in this section and the demographic categories they each address.

Table 1 Studies of algorithmic bias involving the most commonly studied demographic categories

Race/Ethnicity

Kai et al. ( 2017) differentiated performance between African-American and White students in a model predicting student retention in an online college program. They found that a JRip decision tree model performed much more equitably than a J48 decision tree model. Hu and Rangwala ( 2020) investigated a range of algorithms for predicting if a student is at risk of failing a course, finding that their models generally perform worse for African-American students, but that this result is inconsistent across university courses. Anderson et al. ( 2019) differentiated performance between students in different racial/ethnic groups in a model predicting six-year college graduation, across five different algorithms. They found that the algorithms generally had higher false positive rates for White students and higher false negative rates for Latino students. In contrast to these results, Christie et al. ( 2019) found only very minor differences in the quality of a school dropout model by race, possibly because they included proxies for race in their predictors. Lee and Kizilcec ( 2020) compare an unmodified algorithm to an equity-corrected algorithm, finding that the unmodified algorithm for predicting course grade performs worse for students in underrepresented racial and ethnic groups than for White and Asian students, but that their correction improves several indicators of fairness. Yu et al. ( 2020) studied prediction of undergraduate course grades and average GPA and found that if race data was included in models, students of several racial backgrounds were inaccurately predicted to perform worse than other students, but that this effect dissipated if only clickstream and survey data were included. Yu et al. ( 2021) studied prediction of college dropout, finding that their models have worse true negative rates and better recall for students who are not White or Asian, and also worse accuracy if the student is studying in person. In the case of automated essay scoring, the E-Rater system was reported to inaccurately give 11th grade Hispanic and Asian-American students significantly higher scores than human essay raters, while being more accurate for White and African American students (Bridgeman et al., 2009). This effect did not replicate with followup studies of GRE students using a later version of E-Rater; instead, for some types of essays, E-Rater gave African American students substantially lower scores than human raters did (Bridgeman et al., 2012; Ramineni & Williamson, 2018).

Despite the strong historical underrepresentation of indigenous learners worldwide (James et al., 2008), very little attention has been paid to indigenous learners across these studies. Across these studies, only Anderson et al. ( 2019) included indigenous learners as one of the categories studied for algorithmic bias; with only 44 indigenous learners in a sample of over 14,000 learners, model performance was very unstable for this group of learners. This omission likely reflects the overall non-inclusion of indigenous learners in the contexts studied. While this does not necessarily indicate intentional omission on the part of the researchers, it indicates a systematic bias towards conducting this research in contexts where indigenous learners are underrepresented.

Nationality (Current National Location)

In the case of automated essay scoring, the E-Rater system was reported to inaccurately give Chinese and Korean students significantly higher scores than human essay raters on a test of foreign language proficiency, while being more accurate for students of other nationalities (Bridgeman et al., 2009). A replication involving a later version of E-Rater was again found to give Chinese students higher scores than human essay raters (Bridgeman et al., 2012). This study also found that speakers of Arabic and Hindi were given lower scores (Bridgeman et al., 2012). E-Rater was also reported to correlate more poorly and bias upwards in terms of GRE essay scores for Chinese students, despite correlating comparably across 14 other countries (Bridgeman et al., 2009). The SpeechRater system for evaluating communicative competence in English was found to have substantial differences in accuracy for students of different nationalities, with performance particularly low for native speakers of German and Telugu. SpeechRater's evaluations were also found to systematically bias upwards for Chinese students and downwards for German students (Wang et al., 2018).

Ogan et al. ( 2015) built models predicting student learning gains from a mixture of their behaviors related to help-seeking. Models built using data from learners in the Philippines, Costa Rica, and the United States were each more accurate on students from their own countries than for students from other countries.

Li et al. ( 2021) used a large nationally representative dataset collected in 65 countries to predict student achievement on a standardized examination from variables related to student background. They found that a model trained on data from the United States was highly accurate for students from other economically developed countries but less accurate for students from less economically developed countries.

Gender

Kai et al. ( 2017) differentiated performance between male and female students in a model predicting student retention in an online college program. They found that performance was very good for both groups, and that a JRip decision tree model performed more equitably than a J48 decision tree model, but that the JRip model still had moderately better performance for female students than male students. Hu and Rangwala ( 2020) investigated a range of algorithms for predicting if a student is at risk of failing a course, finding that their models generally perform worse for male students, but that this result is inconsistent across university courses. Anderson et al. ( 2019) differentiated performance between male and female students in a model predicting six-year college graduation, across five different algorithms. They found that the algorithms generally had higher false negative rates for male students. Gardner et al. ( 2019) studied MOOC dropout prediction and found that several algorithms studied performed worse for female students than male students. Curiously, they found that this pattern was attenuated for courses with 50-80% male students (but -- again unexpectedly -- worse when there were fewer than 45% male students). Riazy et al. ( 2020) investigate whether course outcome prediction is impacted by whether a student is male or female. The differences in prediction quality found were very small -- on the order of a percentage point -- and the differences in overall proportion of predicted pass between groups were generally also fairly small and were inconsistent in direction between algorithms. Similarly, Christie et al. ( 2019) found only very minor differences in the quality of a school dropout model by gender. Lee and Kizilcec ( 2020) compare an unmodified algorithm to an equity-corrected algorithm, finding that the unmodified algorithm for predicting course grade performs worse for male students than for female students, but that their correction improves several indicators of fairness. Yu et al. ( 2020) studied prediction of undergraduate course grades and average GPA and found that female students were generally inaccurately predicted to perform better than male students. Yu et al. ( 2021) studied prediction of college dropout, finding that their models have somewhat worse true negative rates for male students, but somewhat better recall for male students taking courses in-person, regardless of whether protected attributes are included in the models. In the case of automated essay scoring, the E-Rater system was reported to be comparably accurate for male and female students for both 11th grade essays and foreign-language proficiency examinations (Bridgeman et al., 2009). This lack of difference replicated in a second set of studies on a later version of E-Rater (Bridgeman et al., 2012).

Notably, despite the relatively large number of studies differentiating male and female students, no study that we are aware of has explicitly looked at algorithmic bias in terms of non-binary or transgender learners (or any category within the space of LBGTQ identities, for that matter).

The Limited Research on Other "Populations" or "Subpopulations"

In the previous section, we investigated the research on three relatively more widely-studied demographic categories: race, current national location, and male/female. These three categories are likely selected due to a combination of being relatively easy to study, and a perception of their importance. These three categories are clearly important, and the research that has been conducted has established the importance of attending to algorithmic bias in education more generally. While we do not wish to dispute or diminish the importance of these categories in any way, we also would caution that an over-dependence on these categories -- or on traditionally-measured categories in general -- may miss algorithmic bias in other categories. In this section, we discuss the more limited research that has been seen into algorithmic bias for other categories of learners.

In doing so, we must establish some sort of selection criterion for which differences should be treated as categories for which algorithmic bias may occur. Clearly, there may be differences in detection accuracy that are not evidence of bias, per-se. For instance, perhaps detectors of student knowledge are less accurate for students who "game the system", intentionally misusing the system to obtain correct answers without learning (e.g. Johns & Woolf, 2006; Wu et al., 2017). This difference in accuracy is not bias, per-se, but a result of the intentional choice of the learner to defeat the model.

We propose a non-malleability test for whether a category should be considered in terms of algorithmic bias. A student can choose whether or not to game the system -- and we may be able to design learning experiences that are less often gamed (Baker et al., 2006; Xia et al., 2020). However, the designer of a learning system or an instructor cannot change or influence a student's race, national location, or gender. Similarly, Soundarajan and Clausen ( 2018) describe how the legal definition of a protected class rests in part on its "immutability," the degree to which an individual can move in and out of the suggested class.

This test suggests other categories that may be relevant: where a student lives (in a finer-grained fashion than national location); national origin; socioeconomic status; native language; disabilities; age; parental educational background; and parent work that affects student mobility (i.e. migrant work or high-risk jobs such as in the military). In the following section, we investigate the research into some of these categories. Table 2 lists the studies included in this section and the populations they address.

Table 2 Studies of algorithmic bias involving less frequently studied populations and subpopulations

Native Language and Dialect

Differences in model reliability based on a learner's native language have been studied in the context of educational applications of natural language processing, particularly automated essay scoring. Naismith et al. ( 2018) find that a common measure of lexical sophistication is often developed on word lists that are more appropriate for native speakers than second-language learners; while the method is effective at differentiating between learners at different levels of proficiency, there are often systematic differences in ratings between learners in different countries (i.e. Arabic-speaking learners are often rated lower than Chinese-speaking learners with comparable language proficiency). They also found evidence that biases differ between the corpora used by different testing organizations. Interestingly, Loukina et al. ( 2019) find that when conducting automated essay scoring among essays written by individuals from six different countries, training nation-specific models actually leads to different skews between groups, increasing algorithmic bias compared to training on all groups together.

Disabilities

Urbanicity

A small number of studies have investigated algorithmic bias involving student urbanicity. Ocumpaugh et al. ( 2014) studied the effectiveness of interaction-based automated detectors of student confusion, frustration, boredom, and engaged concentration, trained on urban, suburban, rural, and a combined population. They found that the detectors were generally more effective for new students within the population they were trained on than for other populations; a detector developed on a combined population was more effective for urban and suburban students than for rural students. The authors noted that this was not simply due to race or socioeconomic status, since the suburban and rural students had a similar racial profile, and the urban and rural students had a similar socioeconomic profile.

Contrastingly, however, Samei et al. ( 2015) tested detectors that recognized attributes of student classroom questions, looking at whether models trained on data from urban learners worked in non-urban settings, and vice-versa. They found no degradation in model performance.

Parental Educational Background

Socioeconomic Status

International Students

Yu et al. ( 2020) studied prediction of undergraduate course grades and average GPA and found that international students were inaccurately predicted to perform worse than other students if personal background was included in the model, but that if only clickstream and survey data were included, the models became fairer for these students.

Military-Connected Status

Baker et al. ( 2020) studied models predicting high school graduation and SAT score, training on students who were not military-connected (a student with a parent or close family member in the military) and testing on students who were military-connected, and vice-versa. They found evidence for moderate degradation in model quality when models were used in this fashion.

Summary and Discussion

Another factor that is quickly apparent in looking across studies is the idiosyncracy of the categories that have been studied. Three core categories have received most of the attention from researchers: race/ethnicity (but not indigenous learners), gender (but not non-binary or transgendered students), and nationality (for a small number of nationalities; in terms of learners' current locations). A handful of other categories have been discussed in one or two papers. The list of categories that have been studied seems idiosyncratic. To some degree it is based on convenience -- U.S. census categories are relatively likely to be collected, and a learner's current national location is likely to be known. To some degree it is based on the categorizations that are societally or politically important. To some degree it is based on the biases in what students even make it into the samples -- this may explain to a large degree why indigenous learners are omitted.

Even within the categories that have been relatively heavily studied, there is still considerable idiosyncrasy in the contexts where these categories are studied. For instance, despite the existence of large multi-national datasets involving MOOCs and considerable recent research using MOOCs as a context for conducting research on the differences between learners in different countries (Reich, 2015; Kizilcec & Brooks, 2017), MOOCs have not yet become a widely-used context for studying algorithmic biases involving national difference. Many of the findings discussed above were inconsistent across different studies. It is not yet clear whether this is simply due to noise and random factors, or whether some differences matter more in specific contexts than in other contexts. Fully understanding not only which categories matter, but what their characteristic manifestations are in different contexts, will need to wait until a much larger number of studies have occurred, conducted across a range of contexts.

It is not immediately obvious why some categories have been studied and other categories have not been studied. However, there seem to be effects showing up for a range of groups, suggesting that algorithmic bias likely impacts other groups as well. A broader range of groups need to be more explicitly studied. For instance, children of migrant workers experience many of the same challenges that military-connected students do, such as high personal mobility and concerns about the safety of family members abroad, but have not been studied. Religious minorities have not been studied. Age has not been studied as a factor in undergraduate courses, graduate courses, or professional learning.

Even when groups have been studied, they are often considered in an overly simplified fashion. Large and highly internally diverse groups are currently treated as a single entity. Why are Latinx learners or Spanish-speaking learners treated as monolithic groups, given the large amounts of variance in each of these categories? Why are all learners from China -- a highly-diverse country -- treated as being the same? Why are all Asian-Americans treated the same, from Japanese-Americans to Cambodian-Americans to Indian-Americans? The categories we typically work with come from political distinctions (Strmic-Pawl et al., 2018), not cultural or scientific distinctions. While detailed sub-division of groups may be impractical in small data sets, educational data sets used in AIED increasingly scale to tens or hundreds of thousands of learners. As such, many of the larger data sets studied in the field would be large enough to analyze differences between groups currently labeled together, if more fine-grained labels had been collected. Some distinctions may be infeasible to make in smaller data sets (true even for widely-used census categories), but documenting the differences between groups labeled together, where possible, can determine which labels are generally too coarse-grained, and what contexts and applications it is important to sub-divide them for.

The astute reader -- or at least a reader who has not grown up in the United States -- will also note the intense American focus of research discussed here. What about Cockneys, Northerners (UK), Travellers, or for that matter the difference between Irish, Scottish, Welsh and English learners? What about learners living in different European countries? What about urban and rural learners in Brazil or China? What about Pakistani descendants living in the UK, Kurdish descendants living in Scandinavia, ethnic minorities in Russia, and members of the Portuguese diaspora? What about linguistic minorities in Spain, France, and Italy? The American focus of the research presented here is a reflection of where this research is currently occurring. Almost every single paper we were able to find that studies algorithmic bias in a specific group of learners, based on an externally identifiable non-malleable characteristic, for a specific algorithm, involved authors working in the United States. Fascinatingly, relatively few of these authors were born in the United States. It is imperative that this research extend beyond the United States.

From Unknown bias to Known bias; from Fairness to Equity

In the previous sections, we outlined the published literature on evidence for specific algorithmic biases in education. As the summary of that literature indicates, there is not just evidence for algorithmic bias -- there is evidence for several algorithmic biases. But this evidence is highly sparse -- in several cases, there is only a handful -- or even just one -- cases where a specific algorithmic bias has been documented.

We can posit a progression in a field's efforts to address bias. First, must come unknown bias-- we do not know that a problem exists, or perhaps we do not know its extent or its exact manifestations. When the field does the research to better understand the problem, over time it moves to a stage of known bias -- we have a working understanding (likely imperfect) of how serious a problem is and what situations it emerges in. It is difficult to address a problem until it is reasonably well-understood, otherwise our efforts may be directed to the wrong manifestations of the problem, or to its symptoms rather than its causes. Even if we cannot discover all biases, or fully understand a bias we are aware of, we can do much better at discovering and understanding bias than the current state of the field (shown in section 3).

But, eventually, by understanding a specific bias and engaging in efforts to fix it (perhaps theoretically-driven, perhaps more trial-and-error), we can get to a point where we come closer to fairness. Even if it is mathematically impossible to simultaneously optimize for all definitions of fairness (see discussion in section 2.3), we again can improve considerably on the current state of the field, where most algorithms do not even consider fairness, even informally.

Fairer algorithms can be a step towards designing for equity -- creating a better world, with equal opportunity for all (see discussion in Holstein & Doroudi, in press). Equity for all does not imply ignoring the inequities that specific groups have suffered, often multi-generationally. Working towards equity necessarily implies focusing on the biggest problems -- the biggest failures of fairness and the biggest inequities -- first. But it also depends on understanding all the places that inequity hides in the shadows, the inequities that we may fail to see because of the biases baked into our assumptions and societal narratives. It is imperative on all of us to look for evidence of unknown or poorly-understood biases, and not simply to assume that the problems widely known today are the only problems worth tackling. If we analyze the attention given to algorithmic bias in the last few years, particularly outside of education, we see an intense focus on fairness -- developing metrics and algorithms for fairness. But it will be hard to achieve fairness if we cannot move from unknown biases to known biases. Perhaps the strongest message of this paper is how little we as a field know today about algorithmic bias in education, and how much more we have to learn.

What Obstacles Stand in the Way of Research into Unknown Algorithmic Biases?

Given the importance of revealing and understanding unknown biases, the next question is -- what can we do to make more progress and faster progress in this area?

Unknown biases come in multiple types. One type is a bias that is completely unknown -- we do not know that algorithmic biases exist for a specific group. There are likely to be several such biases out there waiting for our field to discover them -- given the aforementioned sparsity of the data on algorithmic bias in education, it seems highly improbable that we have discovered all important biases. This first type of unknown bias is very concerning. Without understanding which groups are at risk of algorithmic bias, even very well-intentioned attempts to resolve algorithmic bias may miss at-risk groups of students.

For example, systems like The Generalizer (Tipton, 2014) are often used to select samples for large-scale studies. The Generalizer has the goal of helping researchers select populations that are representative, by examining the degree to which the sample selected is representative of the target population in terms of pre-selected variables. However, those variables encode a perspective on which demographic and group variables matter; not all of the variables that appear to be associated with algorithmic bias in education (according to our review) are available in The Generalizer.

However, a second type of unknown bias is equally serious -- cases where we know that bias exists, in general, but not how it manifests itself or where it occurs. We would argue, based on the evidence provided in section 3, that this is true even for very well-known problems such as racism and sexism. The scattered and sometimes inconsistent evidence on algorithmic bias, even for these groups, suggests that our field still has a great deal to learn.

Perhaps the first and seemingly most obvious step is to collect more data on group membership, when data is already being collected. As Paquette et al. ( 2020) note, most research in educational data mining publication venues does not even mention learner demographics at an aggregate level. If we do not collect data on group membership, we cannot analyze our data sets for algorithmic bias.

However, even this simple step imposes challenges -- both concrete and abstract. First of all, reasonable concerns about privacy raise the possibility of risk stemming from collecting this type of data. For instance, it may be possible to re-identify a student from their demographic data (cf. Benitez & Malin, 2010); a classroom may only have one female indigenous student, and therefore reporting this information creates a serious privacy risk. There are methods that can be used to reduce this risk, discussed in the next section, but the risk is hard to entirely eliminate in a small data set.

Second, some forms of group data may be restricted in some contexts, due to legal or regulatory requirements. For example, many institutional review boards (or similar regulatory organizations in other countries) consider demographic data to be higher-risk and therefore create greater hurdles for collecting this data, creating an incentive for researchers to ignore issues of algorithmic bias. So too, differing protections on data on student disability status between countries have an impact on whether researchers are able to investigate algorithmic bias (or even design) related to student cognitive disabilities.

Several other challenges prevent the deeper research into unknown biases for groups known to be at risk of algorithmic bias in general. One is economic. There is a strong commercial disincentive for the developers of learning systems to reveal the algorithmic biases in their systems. Just collecting the data necessary to investigate algorithmic biases presents risks. Even in cases where the privacy risk for any specific student seems logically very small, there are many who will enthusiastically critique an apparent data risk, particularly if it involves a company or individual disliked for other reasons. Going further and publishing evidence that one's system has algorithmic biases -- even in an effort to fix those biases -- risks public critique by journalists, community members, and academics more eager to decry bias than fix it. It can potentially even risk lawsuits. As such, there is a strong "keeping one's head down" incentive not to collect or analyze group data.

Simultaneously, educational effectiveness is often treated as universal -- for example, clearinghouses such as the What Works Clearinghouse and Evidence for ESSA treat curricula as having evidence for efficacy or not having evidence for efficacy, rather than effective or ineffective for specific contexts or groups of learners. As such, in many cases, publicly investigating algorithmic bias currently may present more risk for commercial learning system developers than benefit. Fortunately, recently some school districts have begun to ask for evidence not only that a system works in general but also that it works for students similar to their current students (Rauf, 2020). This may create a countervailing incentive for developers to collect data and investigate algorithmic bias in their systems, or to partner with external researchers in doing so.

Another important possibility is that some algorithmic bias in education may be intersectional (Crenshaw, 1991) -- i.e. creating a specific impact based on multiple group memberships. For example, Black female students may be impacted differently than Black male or non-Black female students. There has been insufficient research into intersectionality in educational work on algorithmic bias, and indeed in algorithmic bias more generally (Cabrera et al., 2019). While there may not always be enough data to study intersectionality, where it is possible, it is an important next step.

Ultimately, there is a lot of work to be done to understand algorithmic biases in education that are currently unknown or incompletely understood. Algorithmic, metrics-based work to expose bias at downstream locations of the ML pipeline will have limited impact on the equity of educational opportunity and outcomes for students, without having the right data in place. This will require extensive effort to re-engineer incentives, eliminate barriers, and mitigate challenges that prevent this work. Only by addressing these challenges can we achieve the necessary steps of expanding the distribution of data collection and research efforts to populations being impacted by algorithmic bias. Only by addressing these challenges can we move towards fully.

understanding how population-specific factors (both well-known demographic factors and less-understood factors) impact the effectiveness of educational algorithmics and the interventions and system responses that build upon them.

In the next section, we take up the issue of how these obstacles -- and other obstacles to resolving algorithmic bias -- can be more effectively addressed.

How Should Researchers and Educators Address these Obstacles?

There are a number of steps that can be taken to address these obstacles. While no one of these steps can by itself eliminate algorithmic bias -- it is not clear that algorithmic bias can be entirely eliminated -- any one of these steps has the potential to drive progress in reducing the degree to which bias is present in the field's algorithms (for a summary of recommendations see Table 3).

Table 3 General strategies and specific recommendations for reducing algorithmic bias in education

Improving Data Collection

Our first area of recommendation -- likely unsurprising to anyone who has read through this paper until this point -- is to improve data collection. As our review above notes, the majority of the work on algorithmic bias, including in education, has focused on the stages of the process where an algorithm is developed and/or evaluated. However, it is difficult for any algorithmic or metric-based approach to make a positive impact if appropriate data is not available to the algorithm.

Towards getting the right data in place, we recommend that researchers collect extensive demographic data on learners whenever possible. It is impossible to evaluate if a model is fair towards a specific group of learners if we do not know which learners are in that group. These efforts should work to collect traditional census categories, absent in many data sets used in our field (Paquette et al., 2020), but this is a minimum -- as our review has shown, many other categories are associated with algorithmic bias. It is too early to list out all categories that should be included in an effort of this nature -- there are too many unknown unknowns, and more research is needed. However, it is not too early to focus more energy on our known unknowns and collect data about student membership in identity categories that already appear to be associated with algorithmic bias.

Based on the evidence thus far, we strongly recommend that whenever possible researchers collect data on gender, race, ethnicity, and national origin. There is not yet sufficient evidence about disability status, dialect, socioeconomic status, urbanicity, native language (and second-language learner status), national region, parental education background, military-connectedness, or migrant work to make broad recommendations, but that is due to the lack of research on how algorithmic bias manifests for these aspects of identity. We suspect that -- given research -- these categories will emerge as important to collect data on as well, and therefore recommend that researchers collect data on these categories as well, whenever possible.

Of course, as discussed in section 4.1, our recommendation to collect richer identity data does have the drawback of increasing risk around privacy and increased regulatory challenges. Approaches such as data obfuscation (Bakken et al., 2004), providing researchers the ability to use but not view variables (Gardner et al., 2018), legal agreements around data re-identification (ASSISTments Project, 2014), can mitigate these risks to a degree. Encouraging regulators and institutional review boards (or other privacy officers) to balance the risks of privacy violations with the risks of algorithmic bias will also be highly important. Once an algorithm development project has collected data on learners' identities, a next step will be to identify gaps in the data set's representativeness and address those gaps through additional data collection. As such, the field needs to do additional work to create practices for making sure training sets are representative and to address underrepresentation of key groups in data sets. A key step in this process will be to determine how much data is actually needed for subpopulations within the data set, in order to have reasonably high confidence that the model will work for new data in these subpopulations. There are not yet methods for selecting sample sizes for most types of machine learning that have the reliability of statistical power analysis (Kraemer & Blasey, 2015), but methods such as adversarial filtering -- which down-sample overrepresented groups rather than collecting new data for underrepresented groups (Le Bras et al., 2020) -- may provide insights that can eventually lead to guidelines on minimum data set sizes for underrepresented groups. Of course, in increasing efforts to collect data from underrepresented learners, it will be important not to impose undue burdens on learners (or teachers and administrators) in underrepresented contexts (Chicago Beyond, 2019).

Another important area where data collection can be improved is in avoiding (or reducing) bias during the labeling process. Both researcher-generated labels (such as classroom observations of student engagement) and labels coming from external sources (such as school data on violence or grades) can themselves be biased. As discussed in section 2.4, any label based on a human's judgment -- even self-report -- is at risk of being biased in this fashion. Therefore, it may be beneficial to replace subjective judgments with other training labels, where possible. However, the solution is not to abandon subjective judgment training labels entirely. Indeed, many constructs -- such as twenty-first-century skills and affect -- can currently only be detected based on subjective judgment training labels. Okur et al.'s work (2018) suggests that members of the group being labeled will provide more accurate and less biased training labels than members of other group. Where possible, data labeling should therefore be conducted by members of the group being studied. If this is infeasible, evidence for biases in the training data can be addressed by creating a distilled variable that partials out the effects of group variables.

Improve Tools and Resources

The last several years have seen a considerable amount of interest in different algorithms and metrics for addressing algorithmic bias, both in education and more broadly. However, the ideas represented in these papers have not emerged into widespread use within educational research and development around algorithms. Part of the challenge is that applying many of the algorithms and metrics that are available requires a developer to either implement these algorithms themselves, or to use often poorly-documented and buggy software packages.

Better software tools would therefore be a useful step towards increasing uptake of best practices. Hajian et al. ( 2016) make a distinction between tools for algorithms that increase fairness/reduce bias and tools that evaluate if bias is present. There is an increasingly large number of algorithms for increasing fairness, and research is proceeding quickly in this area. It is not yet clear which algorithms will be most effective and useful in education. It might therefore be premature to create standard tools for algorithms, especially considering the high amounts of effort needed to create scalable, production-quality algorithm packages.

However, the situation for metrics for evaluating if an algorithm is biased is somewhat different. Metrics are typically considerably less complex than algorithms and are therefore more straightforward to implement. And having several metrics available facilitates evaluation and decision-making around algorithms. As Mitchell et al. ( 2021) note, considering a range of metrics makes it possible to consider the trade-offs around different algorithms and ways of applying them, as well as making assumptions explicit. Therefore, it is both feasible and desirable -- right now -- to create packages in Python and R that can be used by researchers to obtain a standard set of metrics around algorithmic bias and fairness. Efforts are already underway to create domain-general packages for algorithmic bias/fairness metrics, and their developers welcome external contributions (Bellamy et al., 2019). Thus, we call for an effort to build on these domain-general packages to create educational algorithm focused packages. This effort would focus on providing functionality for researchers and developers to quickly obtain a standard set of metrics.

A second potentially useful resource would be the creation of "reference data sets" for use by researchers interested in algorithmic aspects of fairness, within education. Reference data sets have been an essential part of progress in a range of domains, from computer vision (Soomro et al., 2012), to computational linguistics (Bird et al., 2008), and -- in educational research -- for refining models that infer student knowledge (Selent et al., 2016; Stamper & Pardos, 2016). Researchers of algorithmic bias in other domains have recommended the creation and curation of high-quality reference data sets (Paullada et al., 2020). A reference data set can be used to compare algorithms to reduce algorithmic bias, to compare metrics for evaluating algorithmic bias, and as a model for future data set developers to compare their own efforts to. Within education, reference data sets should contain a wide variety of demographic variables (as detailed in the previous sub-section), have a variety of label variables across data sets (both macro-outcomes such as school dropout or course failure, and finer-grained variables such as student affect), and should have clear evidence for algorithmic bias.

Openness and Incentive Structures

A key step towards fixing algorithmic bias will be to shed greater light on where it is occurring -- reporting it and making the community aware of where the problems are. Given the incentives against openly reporting algorithmic bias, discussed above, we need steps to re-align incentives in favor of reporting algorithmic bias.

One such step would be for scientific journal editors and conference committees to adopt guidelines for analyzing algorithmic bias in education. Under these guidelines, all papers that report the use of prediction algorithms on authentic data sets collected after a selected year (announced in advance) would be required to conduct analyses to investigate whether algorithmic bias is present. Getting the details of this requirement right (which analyses? which metrics? which groups? how to apply this standard evenly worldwide? how to avoid a rush of papers to the journals which do not adopt standards?) would require concerted effort among many stakeholders, but would exert a powerful influence on practice in academia and the commercial organizations that value the reputational and other benefits of academic publishing.

This effort could be coordinated with ongoing efforts at school districts and local education agencies to create standards for demonstrating effectiveness for the local student population (Rauf, 2020), and could be combined with an effort to convince organizations such as the What Works Clearinghouse and Evidence for ESSA to consider evidence for generalizability instead of treating educational effectiveness as universal. In these efforts, more targeted to the general public, it might be worthwhile for requirements to include straightforward evidence on whether there is algorithmic bias, as well as more specialized technical demonstrations.

Beyond this, these same organizations could move towards creating guidelines and expectations for opening access to data sets and algorithms for inspection and critique, as is seen in the biomedical field and in journals like Science and Nature. However, companies have legitimate reasons to want to protect their intellectual property, developed at considerable cost and effort. One compromise might be for education agencies to require companies to allow access to the algorithms to specific designated research scientists who would conduct algorithmic bias reviews, the practice used by governmental agencies in demonstrating the effectiveness of new medicines (e.g. Ciociola et al., 2014).

It may be possible to drive greater openness and attention to these problems via positive incentives as well as the more negative pressure of guidelines. For instance, journal editors and conference organizers in education could create special issues and workshops that provide a forum and an opportunity to publish emerging research around algorithmic bias. These forums would be particularly beneficial if they created an opportunity for preliminary work around unknown unknowns -- creating a space for first publication of evidence for algorithmic bias impacting groups that have not been previously studied.

Another form of positive incentive is funding. Funders, both governmental and non-governmental, could create funding streams (or possibilities within existing funding streams) for research on specific, concrete forms of algorithmic bias in education. It also may be possible for non-profit and advocacy organizations to create awards that recognize accomplishments in revealing and addressing algorithmic biases, such as a public award for the commercial organization that has done the most this year to address inequities stemming from algorithmic bias.

Broaden the Community

One of the key ways to reduce algorithmic bias in education is to broaden the community of people who are working on solving these problems. Educational researchers and data scientists do not need to solve these problems on their own and will be less successful if they try to. Algorithmic bias in education ultimately impacts society in general, and involving teachers, school leaders, parents, students, employers, policymakers, and community organizers in the process of thinking about educational algorithms has several potential benefits.

There have been calls to broaden the discussion around the design and use of algorithms (Cramer et al., 2019; Mitchell et al., 2021), including in education (Holstein & Doroudi, in press), but these calls have not been followed up on to the degree necessary, either in educational domains or in the consideration of algorithmic bias more broadly. As Olteanu et al. ( 2019) state, "The interpretation and assessment of results are too often done by data experts, not by domain experts." Members of communities being affected can always do a better job of advocating for their perspective than well-meaning outsiders, and -- indeed -- well-meaning outsiders often fail to fully understand the issues at hand or the constraints on a successful solution (L. T. Smith, 2013). As such, involving community members in the entire process of algorithmic development and use can support more accurate interpretations of data, produce better hypotheses for the causes of the phenomena being observed, and lead to designs and interventions more likely to be found acceptable by the teachers and school leaders implementing them and the students and families who are affected. Fuller inclusion of the people who will use and be impacted by algorithms may also lead to better designs for the implementation and use of algorithms, avoiding cases where unfairness and bias stem not from an algorithm itself, but from how it is used.

These efforts are unlikely to be fully successful unless algorithms and their properties are effectively communicated to these community members. Community members -- and non-data scientists in general -- cannot provide useful recommendation and oversight if they are not given the appropriate information on algorithms and their properties. It is not just a matter of making the information available, it must also be understandable (Howley, 2018). Hence, progress in explainable artificial intelligence and interpretable artificial intelligence (Doran et al., 2018) and in research on explanation methods that make complex algorithms understandable (Zhou et al., 2020) will be essential to addressing the challenges surrounding algorithmic bias. Modern explainable AI methods make it possible to see the impact of a specific variable on prediction (Lundberg et al., 2020), making it easier to see not just where the biases are, but what variables might be causing the biases, towards developing better, less-biased algorithms. These efforts will also depend on bringing in a wider diversity of voices through the entire process of developing an algorithm and using it in practice. Inclusion cannot simply be asking for opinions at one stage in the process. Doing so may be perceived as an insincere attempt to obtain buy-in rather than an attempt to improve the algorithm, and those judgments may not be wrong. As noted by M. K. Lee et al. ( 2019), not only the outcome, but the process that derives the outcome determines perceptions of bias. A full and open inclusion of community members may suggest better variables to use, better consideration of trade-offs, and even may be able to mitigate the negative impacts of a flawed algorithm through choosing better ways to use its outputs and better interventions (see discussion in Mitchell et al., 2021). These partners may also be able to identify additional sources of bias in a learning technology beyond just the algorithm. As noted in several articles (i.e. Arroyo et al., 2013; Baker et al., 2019; Finkelstein et al., 2013; Mayfield et al., 2019; Melis et al., 2009; Woolf et al., 2010), considerable bias in education can come from how a learning interaction is designed.

Finally, in terms of broadening voices, it is essentially to bring a broader collection of voices not just into academic discussion of algorithmic bias, but into industrial work around educational algorithms as well. Corporate voices have been prominent in discussions of algorithmic bias in broader computer science discourse (Blodgett et al., 2020; Cramer et al., 2019; Gebru et al., 2018; Mitchell et al., 2021), but have been relatively less prominent in education. Building capacity in the developers of learning systems that use educational algorithms, including the inclusion of diverse voices and ideas in these organizations, will be essential to reducing algorithmic bias across the educational landscape.

Conclusion

In this article, we have reviewed the contemporary problem of algorithmic bias in education. We contextualize this within the broad scope of literature on algorithmic bias, across sectors, but focus on the specific manifestations of this problem in education. We discuss key theoretical perspectives on algorithmic bias and then turn to reviewing the still very sparse literature on how specific groups of learners are being impacted by algorithmic bias. This review reveals that there are a great number of "known unknowns" in the study of algorithmic bias in education -- areas where we know that our current knowledge is insufficient -- but also exposes the possibility that there may also be many "unknown unknowns" -- groups that we do not even realize are being severely impacted. Foremost among these "unknown unknowns" is the lack of empirical work on how algorithmic bias impact specific groups of learners outside the United States, at a finer grain-size than the national level.

This calls for an ambitious program of research, across our field, to study the known unknowns and reveal the unknown unknowns. We discuss the challenges currently present to the successful completion of this program of research, and offer potential solutions to these challenges, focusing on four potential areas of solution: improved data collection, improved tools and resources, increasing openness and improving incentive structures, and broadening the community of stakeholders who are included in our field's efforts to address algorithmic bias.

We do not pretend that these steps will at last resolve the deep, fundamental inequities in educational opportunity and practice around the world. We hope for a world with fewer unknown biases, where known biases are better documented, where known biases have been reduced in magnitude and impact, and where the world is more equitable. Getting there will be difficult. However, by working together to reduce and find ways to mitigate algorithmic bias, we can at minimum reduce the risk of worsening the situation further -- and we may be able to make a concrete positive difference.

References

Anderson, H., Boodhwani, A., & Baker, R. S. (2019). Assessing the fairness of graduation predictions. Proceedings of the 12th International Conference on Educational Data Mining, 488-491.

Baker, R. S. (2019). Challenges for the future of educational data mining: The Baker learning analytics prizes. Journal of Educational Data Mining, 11(1), 1-17. https://doi.org/10.5281/zenodo.3554745

Article MathSciNet Google Scholar

Baker, R. S., Walker, E., Ogan, A., & Madaio, M. (2019). Culture in computer-based learning systems: Challenges and opportunities. Computer-Based Learning in Context, 1(1), 1-13.

Google Scholar

Baker, R. S., Berning, A., & Gowda, S. M. (2020). Differentiating military-connected and non-military-connected students: Predictors of graduation and SAT score. EdArXiv. https://doi.org/10.35542/osf.io/cetxj . Accessed 1 Oct 2021.

Barocas, S., Hardt, M., & Narayanan, A. (2019). Fairness and Machine Learning. fairmlbook.org. http://www.fairmlbook.org. Accessed 1 Oct 2021.

Berk, R., Heidari, H., Jabbari, S., Kearns, M., & Roth, A. (2018). Fairness in criminal justice risk assessments: The state of the art. Sociological Methods & Research, 50(1), 3-44. https://doi.org/10.1177/0049124118782533

Article MathSciNet Google Scholar

Bird, S., Dale, R., Dorr, B. J., Gibson, B., Joseph, M. T., Kan, M. Y., Lee, D., Powley, B., Radev, D. R., & Tan, Y. F. (2008). No Title. Proceedings of the 6th International Conference on Language Resources and Evaluation, 1755-1759.

Bireda, M. R. (2002). Eliminating racial profiling in school discipline: Cultures in conflict. Scarecrow Press.

Google Scholar

Bridgeman, B., Trapani, C., & Attali, Y. (2009). Considering fairness and validity in evaluating automated scoring [Paper presentation]. Annual Meeting of the National Council on Measurement in Education (NCME), United States.

Christie, S. T., Jarratt, D. C., Olson, L. A., & Taijala, T. T. (2019). Machine-learned school dropout early warning at scale. Proceedings of The 12th International Conference on Educational Data Mining (EDM 2019), 726-731.

Dieterle, E., Dede, C., & Walker, M. (under review). The cyclical ethical effects of using artificial intelligence in education. Manuscript under review.

Green, B., & Hu, L. (2018, July 10-15). The Myth in the Methodology: Towards a Recontextualization of Fairness in Machine Learning [Conference presentation]. The Debates Workshop at the 35th International Conference on Machine Learning, Stockholm, Sweden.

Hanna, A., Denton, E., Smart, A., & Smith-Loud, J. (2020). Towards a critical race methodology in algorithmic fairness. In Proceedings of the 2020 conference on fairness, accountability, and transparency (pp. 501-512).

Hardt, M., Price, E., & Srebro, N. (2016). Equality of Opportunity in Supervised Learning. Proceedings of the 30th International Conference on Neural Information Processing Systems, 3323-3331.

Hellström, T., Dignum, V., & Bensch, S. (2020). Bias in Machine Learning -- What is it Good for? In A. Saffiotti, L. Serafini, & P. Lukowicz (Eds.), Proceedings of the First International Workshop on New Foundations for Human-Centered AI (NeHuAI) co-located with 24th European Conference on Artificial Intelligence (ECAI 2020) (pp. 3-10). RWTH Aachen University.

Holstein, K. & Doroudi, S. (in press). Equity and artificial intelligence in education: Will "AIEd" Amplify or Alleviate Inequities in Education? Invited chapter in Porayska-Pomsta, K. & Holmes, W. (Eds.), Ethics in AIED: Who Cares? Data, algorithms, equity and biases in educational contexts. Routledge Press.

Howley, I. (2018) If an algorithm is openly accessible, and no one can understand it, is it actually open? In Artificial Intelligence in Education Workshop on Ethics in AIED 2018.

Hu, Q., & Rangwala, H. (2020). Towards Fair Educational Data Mining: A Case Study on Detecting At-risk Students. Proceedings of The 13th International Conference on Educational Data Mining (EDM 2020), 431-437.

Jiang, J., Wang, R., Wang, M., Gao, K., Nguyen, D. D., & Wei, G.-W. (2020). Boosting tree-assisted multitask deep learning for small scientific datasets. Journal of Chemical Information and Modeling, 60(3), 1235-1244.

Johns, J., & Woolf, B. (2006). A Dynamic Mixture Model to Detect Student Motivation and Proficiency. Proceedings of the 21st National Conference on Artificial Intelligence, 1, 163-168.

Kai, S., Andres, J. M. L. ., Paquette, L., Baker, R. S. ., Molnar, K., Watkins, H., & Moore, M. (2017). Predicting Student Retention from Behavior in an Online Orientation Course. Proceedings of the 10th International Conference on Educational Data Mining, 250-255.

Kizilcec, R. F., & Brooks, C. (2017). Diverse big data and randomized field experiments in MOOCs. Handbook of Learning Analytics, 211-222.

Kizilcec, R. F., & Lee, H. (2021). Algorithmic Fairness in Education. Algorithmic Fairness in Education. In W. Holmes & K. Porayska-Pomsta (Eds.), Ethics in Artificial Intelligence in Education. Abingdon-on-Thames, UK: Taylor & Francis, in press.

Klingler, S., Wampfler, R., Käser, T., Solenthaler, B., & Gross, M. (2017). Efficient Feature Embeddings for Student Classification with Variational Auto-Encoders. Proceedings of the 10th International Conference on Educational Data Mining, 72-79.

Knight, W. (2019). The Apple Card Didn't "See" Gender-and That's the Problem. Wired.

Le Bras, R., Swayamdipta, S., Bhagavatula, C., Zellers, R., Peters, M., Sabharwal, A., & Choi, Y. (2020). Adversarial filters of dataset biases. Proceedings of the 37th International Conference on Machine Learning, 119, 1078-1088.

Li, X., Song, D., Han, M., Zhang, Y., & Kizilcec, R. F. (2021). On the limits of algorithmic prediction across the globe. arXiv preprint arXiv:2103.15212.

Loukina, A., Madnani, N., & Zechner, K. (2019). The many dimensions of algorithmic fairness in educational applications. Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, 1-10.

Milliron, M. D., Malcolm, L., & Kil, D. (2014). Insight and action analytics: Three case studies to consider. Research & Practice in Assessment, 9, 70-89.

Google Scholar

Naismith, B., Han, N.-R., Juffs, A., Hill, B., & Zheng, D. (2018). Accurate Measurement of Lexical Sophistication with Reference to ESL Learner Data. Proceedings of 11th International Conference on Educational Data Mining, 259-265.

Samei, B., Olney, A. M., Kelly, S., Nystrand, M., D'Mello, S., Blanchard, N., & Graesser, A. (2015). Modeling Classroom Discourse: Do Models That Predict Dialogic Instruction Properties Generalize across Populations? Proceedings of the 8th International Conference on Educational Data Mining, 444-447.

Soundarajan, S., & Clausen, D. L. (2018). Equal Protection Under the Algorithm: A Legal-Inspired Framework for Identifying Discrimination in Machine Learning. Proceedings of the 35th International Conference on Machine Learning.

Tatman, R. (2017). Gender and dialect Bias in YouTube's automatic captions. Proceedings of the First Workshop on Ethics in Natural Language Processing, 53-59.

Yu, R., Li, Q., Fischer, C., Doroudi, S., & Xu, D. (2020). Towards Accurate and Fair Prediction of College Success: Evaluating Different Sources of Student Data. Proceedings of The 13th International Conference on Educational Data Mining (EDM 2020), 292-301.

Yu, R., Lee, H., & Kizilcec, R. F. (2021). Should college dropout prediction models include protected attributes?. In Proceedings of the Eighth ACM Conference on Learning@ Scale (pp. 91-100).

Yudelson, M. V., Fancsali, S. E., Ritter, S., Berman, S. R., Nixon, T., & Joshi, A. (2014). Better Data Beat Big Data. Proceedings of the 7th International Conference on Educational Data Mining, 205-208.

Zhou, T., Sheng, H., & Howley, I. (2020). Assessing post-hoc Explainability of the BKT algorithm. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (pp. 407-413).

Download references

Funding

Schmidt Futures and University of Pennsylvania. All opinions and perspectives presented in this article are the opinions of the authors and do not necessarily represent the funders' opinions or perspectives.

Author information

Authors and Affiliations

Graduate School of Education, University of Pennsylvania, Philadelphia, PA, 19104, USA

Ryan S. Baker & Aaron Hawn

Authors

1. Ryan S. Baker

2. Aaron Hawn

Corresponding author

Correspondence to Ryan S. Baker.

Ethics declarations

Ethics Approval

Not applicable.

Consent to Participate

Not applicable.

Consent for Publication

Not applicable.

Conflicts of Interest/Conflicting Interests

None to report.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

About this article

Cite this article

Baker, R.S., Hawn, A. Algorithmic Bias in Education. Int J Artif Intell Educ 32, 1052-1092 (2022). https://doi.org/10.1007/s40593-021-00285-9
Download citation

Accepted:

Published:

Issue Date:

DOI: https://doi.org/10.1007/s40593-021-00285-9

Share this article

Anyone you share the following link with will be able to read this content:

Get shareable link

Provided by the Springer Nature SharedIt content-sharing initiative

Keywords

# data[].attributes.attachment_link.title
Algorithmic Bias in Education - International Journal of Artificial Intelligence in Education

# data[].attributes.attachment_link.description
In this paper, we review algorithmic bias in education, discussing the causes of that bias and reviewing the empirical literature on the specific ways that algorithmic bias is known to have manifested in education.

# data[].attributes.headline
Investigación del sesgo algorítmico en el seguimiento del progreso de los estudiantes

# data[].attributes.subject
Investigación del sesgo algorítmico en el seguimiento del progreso de los estudiantes

# data[].attributes.body
Esta investigación investiga el sesgo en IA algoritmos utilizados para monitorear el progreso de los estudiantes, centrándose específicamente en los sesgos relacionados con la edad, la discapacidad y el género. El estudio está motivado por incidentes como la controversia sobre las calificaciones de nivel A en el Reino Unido, que demostró las implicaciones en el mundo real de los algoritmos sesgados.

Tipo de fuente: cita bibliográfica o paper científico

Jamiu Adekunle Idowu a, Adriano Soares Koshiyama b, Philip Treleaven a

a Department of Computer Science, University College London, UK

b Holistic AI, London, UK Received 29 January 2024, Revised 16 July 2024, Accepted 17 July 2024, Available online 18 July 2024, Version of Record 20 July 2024.

# data[].attributes.headline
Los sesgos algorítmicos en el proceso de enseñanza aprendizaje: Definición, problemática y sugerencias.

# data[].attributes.subject
Los sesgos algorítmicos en el proceso de enseñanza aprendizaje: Definición, problemática y sugerencias.

# data[].attributes.body
Tipo de fuente: Artículo de divulgación en Linkedin

# data[].attributes.attachment_link.title
Los sesgos algorítmicos en el proceso de enseñanza aprendizaje: Definición, problemática y sugerencias.

# data[].attributes.attachment_link.description
“La solución a la mayoría de problemas es comúnmente encontrada en la misma fuente que los creó: la mente humana.” Yuval Noah Harari Con esta frase, el historiador Israelí y autor de best sellers como SAPIENS, DE ANIMALES A DIOSES, nos dice que la mente humana es una fuente incansable de invenciones