# data[].attributes.headline
1. The Intergenerational Transmission of Teaching

# data[].attributes.subject
1. The Intergenerational Transmission of Teaching

# data[].attributes.body
First Published: October 14, 2020.
Paradigm: positivism.
Reason: It uses quantitative method to verify the veracity of its result and didn't explore the underlying mechanisms well.

# data[].attributes.attachment_link.readable_text
Parental influences, particularly parents' occupations, may influence individuals' entry into the teaching profession. This mechanism may contribute to the relatively static demographic composition of the teaching force over time. We assess the role of parental influences on occupational choice by testing whether the children of teachers are disproportionately likely to become teachers themselves and whether the intergenerational transmission of teaching varies by race or sex. Overall, children whose mothers are teachers are seven percentage points more likely to enter teaching than children of nonteachers. The transmission of teaching from mother to child is about the same for White children and for Black daughters; however, transmission rates for Hispanic daughters are even larger while those for Black sons are near zero.

References

Altonji, J. G., Bharadwaj, P., Lange, F. (2009). Constructing AFQT scores that are comparable across the NLSY79 and the NLSY97. https://harris.uchicago.edu/files/aftq.pdf
Google Scholar

Dolton, P. J. (2020). Teacher supply. In Bradley, S., Green, C. (Eds.), The economics of education: A comprehensive overview (pp. 391-402). Elsevier. https://doi.org/10.1016/B978-0-12-815391-8.00028-8
Google Scholar

Haveman, R., Wolfe, B. (1995). The determinants of children's attainments: A review of methods and findings. Journal of Economic Literature, 33(4), 1829-1878.
Google Scholar

Hout, M., Janus, A. (2011). Educational mobility in the United States since the 1930s. In Duncan, G. J., Murnane, R. J. (Eds.), Whither opportunity? Rising inequality, schools, and children's life chances (pp. 165-185). Russell Sage.
Google Scholar

James, J., Wyckoff, J. (2020). Teacher labor markets: An overview. In Bradley, S., Green, C. (Eds.), The economics of education: A comprehensive overview (pp. 355-370). Elsevier. https://doi.org/10.1016/B978-0-12-815391-8.00026-4
Google Scholar

Loeb, S., Myung, J. (2020). Economic approaches to teacher recruitment and retention. In Bradley, S., Green, C. (Eds.), The economics of education (pp. 403-414). Academic Press. https://doi.org/10.1016/B978-0-12-815391-8.00029-X
Google Scholar

Sawchuk, S. (2015). Steep drops seen in teacher-prep enrollment numbers. Education Digest, 80(7), 9.
Google Scholar

Solon, G. (1992). Intergenerational income mobility in the United States. American Economic Review, 82(3), 393-408.
Google Scholar

Watt, H. M., Richardson, P. W. (2012). An introduction to teaching motivations in different countries: Comparisons using the FIT-Choice scale. Asia-Pacific Journal of Teacher Education, 40(3), 185-197. https://doi.org/10.1080/1359866X.2012.700049
Google Scholar

# data[].attributes.attachment_link.title
The Intergenerational Transmission of Teaching - Alberto Jacinto, Seth Gershenson, 2020

# data[].attributes.attachment_link.description
Parental influences, particularly parents' occupations, may influence individuals' entry into the teaching profession. This mechanism may contribute to the relatively static demographic composition of the teaching force over time. We assess the role of parental influences on occupational choice by testing whether the children of teachers are disproportionately likely to become teachers themselves and whether the intergenerational transmission of teaching varies by race or sex.

# data[].attributes.headline
2. Remembering and Revisiting Pedagogies of the Home

# data[].attributes.subject
2. Remembering and Revisiting Pedagogies of the Home

# data[].attributes.body
First Published: September 3, 2020.
Paradigm: interpretivism.
Reason: It regards subjectivity as a source of knowing and uses the interview method to explain and analyses the data collected.

# data[].attributes.attachment_link.readable_text
Almost two decades after Delgado Bernal's theorization of pedagogies of the home, this article examines pedagogies of the home of four Chicana/o college-educated families to understand the role of parent engagement not only in the college choice processes but also in college completion and graduate school enrollment. Using Chicana feminisms to inform educational oral histories, four Chicana/o parent-child dyads were interviewed. The findings suggest that among Chicana/o college-educated families the (re)making of home, (re)covering tensions, and (re)claiming and (re)learning of cultural knowledge were the pedagogies of the home that were embraced by two successive generations of college completers. Complexities, contradictions, and nuances among Chicana/o college-educated families add to the theorization of pedagogies of the home.

References

Alexander, M. J. (2013). Remembering this bridge, remembering ourselves: Yearning, memory, and desire. In Anzaldúa, G., Keating, A. (Eds.), This bridge we call home: Radical visions for transformation (pp. 95-117). Routledge.
Google Scholar

Alvarez-Galván, M. E. (2001). Chicanas with doctorate degrees: Factors that contribute to their educational persistence [Doctoral dissertation, University of Southern California]. University of Southern California Digital Library. http://digitallibrary.usc.edu/cdm/ref/collection/p15799coll30/id/329060
Google Scholar

Anzaldúa, G. (2002). Preface: (Un)natural bridges, (un)safe spaces. In Anzaldúa, G., Keating, A. (Eds.), This bridge we call home: Radical visions for transformation (pp. 1-5). Routledge.
Google Scholar

Anzaldúa, G. (2009). The new mestiza nation. In Keating, A. (Ed.), The Gloria Anzaldúa reader (pp. 203-216). Duke University Press.
Google Scholar

Auerbach, C., Silverstein, L. B. (2003). Qualitative data: An introduction to coding and analysis. New York University Press.
Google Scholar

Garcia, N. M. (2019). Pa'lante, siempre pa'lante: Pedagogies of the home among Puerto Rican college educated families. International Journal of Qualitative Studies in Education, 32(6), 576-590. https://doi.org/10.1080/09518398.2019.1609116
Google Scholar

Garcia, N. M., Salinas, C., Cisneros, J. (Eds.). (in press). Studying Latinx/a/o students in higher education: A critical analysis of concepts, theory, and methodologies. Routledge.
Google Scholar

Hossler, D., Gallagher, K. S. (1987). Studying student college choice: A three-phase model and the implications for policymakers. College and University, 62(3), 207-221.
Google Scholar

Kiyama, J. M., Rios-Aguilar, C. (Eds.). (2017). Funds of knowledge in higher education: Honoring students' cultural experiences and resources as strengths. Routledge. https://doi.org/10.4324/9781315447322
Google Scholar

Matias, C. E. (2016). "Mommy, is being brown bad?" Critical race parenting in a post-race era. Race & Pedagogy Journal: Teaching and Learning for Justice, 1(3), Article 1.
Google Scholar

Moraga, C., Anzaldúa, G. (1983). This bridge called my back: Writings by radical women of color (2nd ed.). Kitchen Table, Women of Color Press.
Google Scholar

Pérez, E. (1998). Irigaray's female symbolic in the making of Chicana lesbian sitios y lenguas (sites and discourses). In Trujillo, C. (Ed.), Living Chicana theory (pp. 87-101). Third Woman Press.
Google Scholar

Perna, L. W. (2006). Studying college access and choice: A proposed conceptual model. In Smart, J. C. (Ed.), Higher education: Handbook of theory and research (pp. 99-157). Springer. https://doi.org/10.1007/1-4020-4512-3_3
Google Scholar

Solorzano, D. G., Yosso, T. J. (2001). From racial stereotyping and deficit discourse toward a critical race theory in teacher education. Multicultural Education, 9(1), 2-8.
Google Scholar

Villenas, S. (2010). Thinking Latina/o education with and from Chicana/Latina feminist cultural studies: Emerging pathways-Decolonial possibilities. In Leonardo, Z. (Ed.), Handbook of cultural politics and education (Vol. 4, pp. 451-476). Brill/Sense. https://doi.org/10.1163/9789460911774_022
Google Scholar

Yosso, T. J. (2005). Whose culture has capital? A critical race theory discussion of community cultural wealth. Race Ethnicity and Education, 8(1), 69-91. https://doi.org/10.1080/1361332052000341006
Google Scholar

# data[].attributes.attachment_link.title
Remembering and Revisiting Pedagogies of the Home - Nichole M. Garcia, Dolores Delgado Bernal, 2020

# data[].attributes.attachment_link.description
Almost two decades after Delgado Bernal's theorization of pedagogies of the home, this article examines pedagogies of the home of four Chicana/o college-educated families to understand the role of parent engagement not only in the college choice processes but also in college completion and graduate school enrollment.

# data[].attributes.headline
3. School Effects Revisited: The Size, Stability, and Persistence of Middle Schools’ Effects on Academic Outcomes

# data[].attributes.subject
3. School Effects Revisited: The Size, Stability, and Persistence of Middle Schools’ Effects on Academic Outcomes

# data[].attributes.body
First Published: August 28, 2020.
Paradigm: positivism.
Reason: It makes four hypotheses and uses quantitative method to testify them.

# data[].attributes.attachment_link.readable_text
Since the early 2000s, educational evaluation research has primarily centered on teachers', rather than schools', contributions to students' academic outcomes due to concerns that estimates of the latter were smaller, less stable, and more prone to measurement error. We argue that this disparity should be reduced. Using administrative data from three cohorts of Massachusetts public school students ( N = 123,261) and two-level models, we estimate middle schools' value-added effects on eighth-grade and 10th-grade math scores and, importantly, a non-test score outcome: 4-year college enrollment. Comparing our results to teacher-centered studies, we find that school effects (encompassing both teaching- and nonteaching-related factors) are initially smaller but nearly as stable and perhaps more persistent than are individual teacher effects. Our study motivates future research estimating the long-term effects of both teachers and schools on a wide range of outcomes.

References

Chetty, R., Friedman, J. N., Rockoff, J. E. (2014a). Measuring the impacts of teachers I: Evaluating bias in teacher value-added estimates. American Economic Review, 104(9), 2593-2632. https://doi.org/10.1257/aer.104.9.2593
Google Scholar

Chetty, R., Friedman, J. N., Rockoff, J. E. (2014b). Measuring the impacts of teachers II: Teacher value-added and student outcomes in adulthood. American Economic Review, 104(9), 2633-2679. https://doi.org/10.1257/aer.104.9.2633
Google Scholar

Jencks, C., Mayer, S. E. (1990). The social consequences of growing up in a poor neighborhood. In Lynn, L., McGeary, M. G. H. (Eds.), Inner-city poverty in the United States (pp. 111-186). National Academies Press.
Google Scholar

Konstantopoulos, S. (2006). Trends of school effects on student achievement: Evidence from NLS: 72, HSB: 82, and NELS: 92. Teachers College Record, 108(12), 2550-2581.
Google Scholar

Konstantopoulos, S. (2007a). A comment on variance decomposition and nesting effects in two- and three-level designs (IZA DP No. 3178). IZA-Institute of Labor Economics. http://ftp.iza.org/dp3178.pdf
Google Scholar

Konstantopoulos, S. (2007b). How long do teacher effects persist? (WP-07-12). Institute for Policy Research Northwestern University. https://www.ipr.northwestern.edu/documents/working-papers/2007/IPR-WP-07-12.pdf
Google Scholar

Raudenbush, S. W., Bryk, A. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Sage.
Google Scholar

Von Secker, C. E., Lissitz, R. W. (1999). Estimating the impact of instructional practices on student achievement in science. Journal of Research in Science Teaching, 36(10), 1110-1126. https://doi.org/10.1002/(SICI)1098-2736(199912)36:10<1110::AID-TEA4>3.0.CO;2-T
Google Scholar

Xu, Z., Swanlund, A. (2013). Estimating teacher contributions to student learning: The role of the school. American Institutes of Research.
Google Scholar

# data[].attributes.attachment_link.title
School Effects Revisited: The Size, Stability, and Persistence of Middle Schools' Effects on Academic Outcomes - Tracey Lloyd, Jared N. Schachner, 2020

# data[].attributes.attachment_link.description
Since the early 2000s, educational evaluation research has primarily centered on teachers', rather than schools', contributions to students' academic outcomes due to concerns that estimates of the latter were smaller, less stable, and more prone to measurement error. We argue that this disparity should be reduced.

# data[].attributes.headline
1. Learning Attribute Hierarchies From Data: Two Exploratory Approaches

# data[].attributes.subject
1. Learning Attribute Hierarchies From Data: Two Exploratory Approaches

# data[].attributes.body
paradigim: Positivism
reason:
In this article, the author studied two exploratory approaches to identify attribute hierarchies from data, the LVS approach and the RLCM approach.

They developed hypotheses then try to verify the hypothesis.

# data[].attributes.attachment_link.readable_text
In cognitive diagnostic assessment, multiple fine-grained attributes are measured simultaneously. Attribute hierarchies are considered important structural features of cognitive diagnostic models (CDMs) that provide useful information about the nature of attributes. Templin and Bradshaw first introduced a hierarchical diagnostic classification model (HDCM) that directly takes into account attribute hierarchies, and hence, HDCM is nested within more general CDMs. They also formulated an empirically driven hypothesis test to statistically test one hypothesized link (between two attributes) at a time. However, their likelihood ratio test statistic does not have a known reference distribution, so it is cumbersome to perform hypothesis testing at scale. In this article, we studied two exploratory approaches that could learn the attribute hierarchies directly from data, namely, the latent variable selection (LVS) approach and the regularized latent class modeling (RLCM) approach. An identification constraint was proposed for the LVS approach. Simulation results revealed that both approaches could successfully identify different types of attribute hierarchies, when the underlying CDM is either the deterministic input noisy and gate model or the saturated log-linear CDM. The LVS approach outperformed the RLCM approach, especially when the total number of attributes increases.

References

Briggs, D., Alonzo, A. (2009). Building a learning progression as a cognitive model [Paper presentation]. Annual Meeting of the National Council on Measurement in Education, San Diego, CA, United States.
Google Scholar

Cai, Y., Tu, D., Ding, S. (2018). Theorems and methods of a complete Q matrix with attribute hierarchies under restricted Q-matrix design. Frontiers in Psychology, 9, 1413.
Google Scholar

Chiu, C. Y., Douglas, J. A., Li, X. (2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74(4), 633.
Google Scholar

Gierl, M. J., Cui, Y., Hunka, S. (2007a). Using connectionist models to evaluate examinees' response patterns on tests. [Paper presentation]. Annual Meeting of the National Council on Measurement in Education, Chicago, IL, United States.
Google Scholar

Gierl, M. J., Leighton, J. P., Hunka, S. M. (2007b). Using the attribute hierarchy method to make diagnostic inferences about examinees' cognitive skills. In Leighton, J. P., Gierl, M. J. (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 242-274). Cambridge University Press.
Google Scholar

Gu, Y., Xu, G. (2019a). Learning attribute patterns in high-dimensional structured latent attribute models. https://arxiv.org/abs/1904.04378
Google Scholar

Henson, R. A., Templin, J. L., Willse, J. T. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74(2), 191.
Google Scholar

Leighton, J., Gierl, M. (2007). Cognitive diagnostic assessment for education: Theory and applications. Cambridge University Press.
Google Scholar

Morphew, J., Mestre, J., Kang, H., Chang, H., Fabry, G. (2018). Using computer adaptive testing to assess physics proficiency and improve exam performance in an introductory physics course. Physical Review Physics Education Research, 12. https://doi.org/10.1103/PhysRevPhysEducRes.14.020110
Google Scholar

Nichols, P., Chipman, S., Brennan, R. (1995). Cognitively diagnostic assessment. New York: Routledge.
Google Scholar

Popham, W. J. (2008). Formative assessment: Seven stepping-stones to success. Principal Leadership, 9(1), 16-20.
Google Scholar

Reckase, M. D. (2009). Multidimensional item response theory models. In Multidimensional item response theory (pp. 79-112). Springer.
Google Scholar

Tatsuoka, K. K. (1990). Toward an integration of item-response theory and cognitive error diagnosis. In Fredrickson, N., Glaser, R. L., Lesgold, A. M., Shafto, M. G. (Eds.), Diagnostic monitoring of skills and knowledge acquisition (pp. 453-488). Hillsdale: Erlbaum.
Google Scholar

Tatsuoka, K. K. (2009). Cognitive assessment: An introduction to the rule space method. New York: Routledge.
Google Scholar

Xu, G., Shang, Z. (2018). Identifying latent structures in restricted latent class models. Journal of the American Statistical Association, 113(523), 1284-1295.
Google Scholar

# data[].attributes.attachment_link.title
Learning Attribute Hierarchies From Data: Two Exploratory Approaches - Chun Wang, Jing Lu, 2020

# data[].attributes.attachment_link.description
In cognitive diagnostic assessment, multiple fine-grained attributes are measured simultaneously. Attribute hierarchies are considered important structural features of cognitive diagnostic models (CDMs) that provide useful information about the nature of attributes.

# data[].attributes.headline
2. On the Treatment of Missing Data in Background Questionnaires in Educational Large-Scale Assessments: An Evaluation of Different Procedures

# data[].attributes.subject
2. On the Treatment of Missing Data in Background Questionnaires in Educational Large-Scale Assessments: An Evaluation of Different Procedures

# data[].attributes.body
paradigm: positivism
Reason:
support hupotheses and then verify.

# data[].attributes.attachment_link.readable_text
In cognitive diagnostic assessment, multiple fine-grained attributes are measured simultaneously. Attribute hierarchies are considered important structural features of cognitive diagnostic models (CDMs) that provide useful information about the nature of attributes. Templin and Bradshaw first introduced a hierarchical diagnostic classification model (HDCM) that directly takes into account attribute hierarchies, and hence, HDCM is nested within more general CDMs. They also formulated an empirically driven hypothesis test to statistically test one hypothesized link (between two attributes) at a time. However, their likelihood ratio test statistic does not have a known reference distribution, so it is cumbersome to perform hypothesis testing at scale. In this article, we studied two exploratory approaches that could learn the attribute hierarchies directly from data, namely, the latent variable selection (LVS) approach and the regularized latent class modeling (RLCM) approach. An identification constraint was proposed for the LVS approach. Simulation results revealed that both approaches could successfully identify different types of attribute hierarchies, when the underlying CDM is either the deterministic input noisy and gate model or the saturated log-linear CDM. The LVS approach outperformed the RLCM approach, especially when the total number of attributes increases.

References

Briggs, D., Alonzo, A. (2009). Building a learning progression as a cognitive model [Paper presentation]. Annual Meeting of the National Council on Measurement in Education, San Diego, CA, United States.
Google Scholar

Cai, Y., Tu, D., Ding, S. (2018). Theorems and methods of a complete Q matrix with attribute hierarchies under restricted Q-matrix design. Frontiers in Psychology, 9, 1413.
Google Scholar

Chiu, C. Y., Douglas, J. A., Li, X. (2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74(4), 633.
Google Scholar

Gierl, M. J., Cui, Y., Hunka, S. (2007a). Using connectionist models to evaluate examinees' response patterns on tests. [Paper presentation]. Annual Meeting of the National Council on Measurement in Education, Chicago, IL, United States.
Google Scholar

Gierl, M. J., Leighton, J. P., Hunka, S. M. (2007b). Using the attribute hierarchy method to make diagnostic inferences about examinees' cognitive skills. In Leighton, J. P., Gierl, M. J. (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 242-274). Cambridge University Press.
Google Scholar

Gu, Y., Xu, G. (2019a). Learning attribute patterns in high-dimensional structured latent attribute models. https://arxiv.org/abs/1904.04378
Google Scholar

Henson, R. A., Templin, J. L., Willse, J. T. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74(2), 191.
Google Scholar

Leighton, J., Gierl, M. (2007). Cognitive diagnostic assessment for education: Theory and applications. Cambridge University Press.
Google Scholar

Morphew, J., Mestre, J., Kang, H., Chang, H., Fabry, G. (2018). Using computer adaptive testing to assess physics proficiency and improve exam performance in an introductory physics course. Physical Review Physics Education Research, 12. https://doi.org/10.1103/PhysRevPhysEducRes.14.020110
Google Scholar

Nichols, P., Chipman, S., Brennan, R. (1995). Cognitively diagnostic assessment. New York: Routledge.
Google Scholar

Popham, W. J. (2008). Formative assessment: Seven stepping-stones to success. Principal Leadership, 9(1), 16-20.
Google Scholar

Reckase, M. D. (2009). Multidimensional item response theory models. In Multidimensional item response theory (pp. 79-112). Springer.
Google Scholar

Tatsuoka, K. K. (1990). Toward an integration of item-response theory and cognitive error diagnosis. In Fredrickson, N., Glaser, R. L., Lesgold, A. M., Shafto, M. G. (Eds.), Diagnostic monitoring of skills and knowledge acquisition (pp. 453-488). Hillsdale: Erlbaum.
Google Scholar

Tatsuoka, K. K. (2009). Cognitive assessment: An introduction to the rule space method. New York: Routledge.
Google Scholar

Xu, G., Shang, Z. (2018). Identifying latent structures in restricted latent class models. Journal of the American Statistical Association, 113(523), 1284-1295.
Google Scholar

# data[].attributes.attachment_link.title
Learning Attribute Hierarchies From Data: Two Exploratory Approaches - Chun Wang, Jing Lu, 2020

# data[].attributes.attachment_link.description
In cognitive diagnostic assessment, multiple fine-grained attributes are measured simultaneously. Attribute hierarchies are considered important structural features of cognitive diagnostic models (CDMs) that provide useful information about the nature of attributes.

# data[].attributes.headline
4. Competing for Bachelor’s Degrees: Are Community Colleges Cutting Into the Market Share of 4-Year Institutions?

# data[].attributes.subject
4. Competing for Bachelor’s Degrees: Are Community Colleges Cutting Into the Market Share of 4-Year Institutions?

# data[].attributes.body
Paradigm: positivism.
Reason: It uses second data and empirical strategy as stated in the article. And hopes future work to generalize what have been found in it.

# data[].attributes.attachment_link.readable_text
To address local workforce needs and expand access to affordable bachelor's degrees, some states allow community colleges to offer bachelor's degree programs. Despite concerns that community college baccalaureate (CCB) programs will duplicate efforts and cut into the market share of nearby 4-year institutions, extant literature has yet to examine the impact of CCB adoption on bachelor's degree program enrollment and bachelor's degree production at 4-year institutions. Using program-level data, our findings show that local CCB degree programs have a negative effect on overall bachelor's degree enrollment and bachelor's degree production at 4-year institutions, but this effect is concentrated primarily within for-profit 4-year institutions. This study represents the first comprehensive evaluation of the impact of CCB degree programs on neighboring 4-year institutions.

References

Abadie, A., Athey, S., Imbens, G. W., Wooldridge, J. (2017). When should you adjust standard errors for clustering? (Working Paper No. w24003). National Bureau of Economic Research. https://doi.org/10.3386/w24003
Google Scholar

Cook, T. D., Campbell, D. T. (1986). The causal assumptions of quasi-experimental practice. Synthese, 68(July), 14 1-180. https://doi.org/10.1007/BF00413970
Google Scholar

Essink, S. E. (2013). The community college baccalaureate: A mixed methods study of implementation and best practices [Doctoral dissertation, University of Nebraska-Lincoln]. Educational Administration: Theses, Dissertations, and Student Research. https://digitalcommons.unl.edu/cehsedaddiss/152
Google Scholar

Florida Senate . (2003a). K-20 Education Code 1004.65 Community Colleges: definition, mission, and responsibilities.
Google Scholar

Florida Senate . (2003b). Site-determined baccalaureate degree access.
Google Scholar

McKee, J. V. (2001). Factors and issues surrounding development of one community college baccalaureate degree program [Doctoral dissertation, Oregon State University]. [email protected] https://ir.library.oregonstate.edu/concern/graduate_thesis_or_dissertations/73666676h
Google Scholar

National Student Clearinghouse Research Center . (2019). Tracking transfer: Measures of effectiveness in helping community college students to complete bachelor's degrees (signature report No. 13).
Google Scholar

Person, A. E., Rosenbaum, J. E., Deil-Amen, R. (2006). Student planning and information problems in different college structures. Teachers College Record, 108(3), 374-396.
Google Scholar

Shapiro, D., Dundar, A., Huie, F., Wakhungu, P., Yuan, X., Nathan, A., Hwang, Y. A. (2017, April). A national view of student attainment rates by race and ethnicity (Signature Report No. 12b). National Student Clearinghouse Research Center. https://nscresearchcenter.org/wp-content/uploads/Signature12-RaceEthnicity.pdf
Google Scholar

Shapiro, D., Dundar, A., Wakhungu, P. K., Yuan, X., Nathan, A., Hwang, Y. (2016, September). Time to degree: A national view of the time enrolled and elapsed for associate and bachelor's degree earners (Signature Report No. 11). National Student Clearinghouse Research Center. https://nscresearchcenter.org/wp-content/uploads/SignatureReport11.pdf
Google Scholar

Zumeta, W., Breneman, D. W., Callan, P. M., Finney, J. E. (2012). Financing American higher education in the era of globalization. Harvard Education Press.
Google Scholar

# data[].attributes.attachment_link.title
Competing for Bachelor's Degrees: Are Community Colleges Cutting Into the Market Share of 4-Year Institutions? - Dennis A. Kramer, Justin C. Ortagus, Jacqueline Donovan, 2020

# data[].attributes.attachment_link.description
To address local workforce needs and expand access to affordable bachelor's degrees, some states allow community colleges to offer bachelor's degree programs. Despite concerns that community college baccalaureate (CCB) programs will duplicate efforts and cut into the market share of nearby 4-year institutions, extant literature has yet to examine the impact of CCB adoption on bachelor's degree program enrollment and bachelor's degree production at 4-year institutions.

# data[].attributes.headline
3. Aggregate-Level Test-Scale Linking: A New Solution for an Old Problem?

# data[].attributes.subject
3. Aggregate-Level Test-Scale Linking: A New Solution for an Old Problem?

# data[].attributes.body
paradigm"
positivism
The author used data and quantitative method to verify whether a new solution for an old problem.

# data[].attributes.attachment_link.readable_text
The Reardon, Kalogrides, and Ho article on validation methods for aggregate-level test scale linking is an attempt to validate a district-level scale aligning procedure that appears to be a new solution to an old problem. Their aligning procedure uses the National Assessment of Educational Progress (NAEP) scale to piece together a patchwork of data structures from different tests of different constructs obtained under different administration conditions and used in different ways by different states. In this article, we critique their linking and validation efforts. Our critique has three components. First, we review the recommendations for linking state assessments to NAEP from several studies and commentaries to provide background from which to interpret Reardon et al.'s validation attempts. Second, we provide a replication of the Reardon et al. empirical validations of its proposed linking procedure to demonstrate that correlations between district means on two test scores can be high even when (1) the constructs being measured by the tests are different and (2) the district-level means estimated using the Reardon et al. linking approach can differ substantially from actual district-level means. Then, we suggest additional checks for construct similarity and subpopulation invariance from other concordance studies that could be used to assess whether the inferences made by Reardon et al. are warranted. Finally, until such checks are made, we urge cautious use of the results of the Reardon et al. results.

References

Dorans, N. J. (2015, March). Uncommon measures revisited [Invited presentation]. Association of Test Publishers, Palm Springs, CA.
Google Scholar

Dorans, N. J. (2020). Uncommon measures revisited (ETS Research Report No. RR-20-04). Educational Testing Service.
Google Scholar

The Educational Opportunity Project at Stanford University . (2019). Average test scores. https://edopportunity.org/explorer/#/map/us/districts/avg/ses/all/3.65/41.65/-95.97
Google Scholar

Feuer, M. J., Holland, P. W., Green, B. F., Bertenthal, M. W., Hemphill, F. C. (Eds.). (1999). Uncommon measures: Equivalence and linkage among educational tests (Report of the Committee on Equivalency and Linkage of Educational Tests, National Research Council). National Academy Press.
Google Scholar

Ho, A. D., Haertel, E. H. (2007). (Over)-interpreting mappings of state performance standards onto the NAEP scale. Paper commissioned by the Council of Chief State School Officers, Washington, DC.
Google Scholar

Holland, P. W., Dorans, N. J. (2006). Linking and equating. In Brennan, R. L. (Ed.), Educational measurement (4th ed., pp. 187-220). American Council on Education/Prager.
Google Scholar

Liu, J., Dorans, N. J., Moses, T. P. (2010, May). Evaluating the subpopulation sensitivity of the ACT-SAT concordances [Paper Presentation]. Updating ACT/SAT concordances, at the 2010 annual meeting of the National Council on Measurement in Education, Denver, CO.
Google Scholar

McLaughlin, D. H. (2000). Protecting State NAEP Trends from Changes in SD/LEP Inclusion Rates (Report to the National Institute of Statistical Sciences). American Institutes for Research.
Google Scholar

McLaughlin, D. H., Bandeira de Mello, V. (2002, April). Comparison of state elementary school mathematics achievement standards using NAEP 2000 [Paper presentation]. American Educational Research Association Annual Meeting, New Orleans, LA.
Google Scholar

Reardon, S. F., Kalogrides, D., Ho, A. D. (2019). Validation methods for aggregate-level test scale linking: A case study mapping school district distributions to a common scale. Journal of Educational and Behavioral Statistics. Advance online publication. https://doi.org/10.3102/1076998619874089
Google Scholar

Reardon, S. F., Shear, B. R., Castellano, K. E., Ho, A. D. (2017). Using heteroskedastic ordered probit models to recover moments of continuous test score distributions from coarsened data. Journal of Educational and Behavioral Statistics, 42(1), 3-45. |
Google Scholar SAGE Journals

Santayana, G. (1905). The life of reason, reason in common sense. Scribner's.
Google Scholar

Thissen, D. (2016). Commentary on the assessment of the fairness of comparisons under divergent measurement conditions. In Dorans, N. J., Cook, L. L. (Eds.), NCME application of educational assessment and measurement, Volume 3: Fairness in educational assessment and measurement (pp. 203-214). Routledge.
Google Scholar

Williams, V., Rosa, K. R., McLeod, L. D., Thissen, D., Sanford, E. E. (1998). Projecting to the NAEP scale: Results from the North Carolina End-of-Grade Testing Program. Journal of Educational Measurement, 35(4), 277-296.
Google Scholar

# data[].attributes.attachment_link.title
Aggregate-Level Test-Scale Linking: A New Solution for an Old Problem? - Tim Moses, Neil J. Dorans, 2020

# data[].attributes.attachment_link.description
The Reardon, Kalogrides, and Ho article on validation methods for aggregate-level test scale linking is an attempt to validate a district-level scale aligning procedure that appears to be a new solution to an old problem.

# data[].attributes.headline
5. Can Center-Based Care Reduce Summer Slowdown Prior to Kindergarten? Exploring Variation by Family Income, Race/Ethnicity, and Dual Language Learner Status

# data[].attributes.subject
5. Can Center-Based Care Reduce Summer Slowdown Prior to Kindergarten? Exploring Variation by Family Income, Race/Ethnicity, and Dual Language Learner Status

# data[].attributes.body
Paradigm:
Reason: It uses quantitative method to add to the early childhood and summer learning literatures.

# data[].attributes.attachment_link.readable_text
This study examines growth in language and math skills during the summer before kindergarten; considers variation by family income, race/ethnicity, and dual language learner status; and tests whether summer center-based care sustains preschool gains. Growth in skills slowed during summer for all children, but the patterns varied by domain and group. Non-White and dual language learner students showed the largest drop-off in language skills during summer. Lower-income students demonstrated slower summer growth in math skills than their higher-income peers. Students enrolled in summer center-based care had faster growth in math skills than those who did not attend care. Yet lower-income students who attended center-based care showed slower growth in language skills during summer than similar nonattenders. Implications are discussed.

References

Atteberry, A., McEachin, A. (2016). School's out: Summer learning loss across grade levels and school contexts in the United States today. In Alexander, K., Pitcock, S., Boulay, M. C. (Eds.), The summer slide: What we know and can do about summer learning loss (pp. 35-54). Teachers College Press.
Google Scholar

Bronfenbrenner, U., Morris, P. A. (1998). The ecology of developmental processes. In Lerner, R. M. (Ed.), Handbook of child psychology: Vol. 1. Theoretical models of human development (5th ed., pp. 993-1028). Wiley.
Google Scholar

Burchinal, M., Magnuson, K., Powell, D., Hong, S. S. (2015). Early childcare and education. In Bornstein, M. H., Leventhal, T. (Eds.), Handbook of child psychology and developmental science: Vol. 4. Ecological settings and processes (pp. 223-267). Wiley. https://doi.org/10.1002/9781118963418.childpsy406
Google Scholar

Condliffe, B. (2016). Selecting summer: How elementary school parents make choices about summer programs. In Alexander, K., Pitcock, S., Boulay, M. C. (Eds.), The summer slide: What we know and can do about summer learning loss (pp. 146-160). Teachers College Press.
Google Scholar

McEachin, A., Augustine, C. H., McCombs, J. (2018). Effective summer programming: What educators and policymakers should know. American Educator, 42(1), 10-11.
Google Scholar

Roskos, K. A. (2017). Play and literacy in early childhood: Research from multiple perspectives. Routledge.
Google Scholar

Singer, J. D., Willett, J. B. (2003). Applied longitudinal data analysis: Modeling change and event occurrence. Oxford University Press.
Google Scholar

Woodcock, R. W., Mather, N., McGrew, K. S., Wendling, B. J. (2001). Woodcock-Johnson III tests of cognitive abilities. Riverside.
Google Scholar

Yoshikawa, H., Weiland, C., Brooks-Gunn, J., Burchinal, M., Espinosa, L. M., Gormley, W. T., Ludwig, J., Magnuson, K. A., Phillips, D., Zaslow, M. (2013). Investing in our future: The evidence base on preschool education. Society for Research in Child Development & Foundation for Child Development. https://www.srcd.org/sites/default/files/file-attachments/mb_2013_10_16_investing_in_children.pdf
Google Scholar

# data[].attributes.attachment_link.title
Can Center-Based Care Reduce Summer Slowdown Prior to Kindergarten? Exploring Variation by Family Income, Race/Ethnicity, and Dual Language Learner Status - Meghan P. McCormick, Mirjana Pralica, Paola Guerrero-Rosada, Christina Weiland, JoAnn Hsueh, Barbara Condliffe, Jason Sachs, Catherine Snow, 2020

# data[].attributes.attachment_link.description
This study examines growth in language and math skills during the summer before kindergarten; considers variation by family income, race/ethnicity, and dual language learner status; and tests whether summer center-based care sustains preschool gains. Growth in skills slowed during summer for all children, but the patterns varied by domain and group.

# data[].attributes.headline
6. The Effects of Providing and Receiving Peer Feedback on Writing Performance and Learning of Secondary School Students

# data[].attributes.subject
6. The Effects of Providing and Receiving Peer Feedback on Writing Performance and Learning of Secondary School Students

# data[].attributes.body
Paradigm: positivism.
Reason: It uses quantitative method to answer questions asked before the experiment to test unique associations between the reviewing behaviors, the revision behaviors, and learning outcomes, separately for high-level and low-level aspects of writing.

# data[].attributes.headline
1.The Impact of Project Creation on Learning Mathematics in aTransdisciplinary Setting

# data[].attributes.subject
1.The Impact of Project Creation on Learning Mathematics in aTransdisciplinary Setting

# data[].attributes.body
First Published: May 15, 2020
Paradigm：interpretivism
Reason：The article says:" The writing is based on an investigation that was carried out with high school students from Puerto Rico. From a qualitative paradigm with a research design in action, it was explored how students in higher grades resolve situations within their environment. " So the author used a qualitative paradigm with suvery.

# data[].attributes.attachment_link.provider_name
The Impact of Project Creation on Learning Mathematics in a Transdisciplinary Setting

# data[].attributes.attachment_link.readable_text
Glenda Collazo Rivera
Keywords: Project-based learning, transdisciplinary practice, situations in context, learning strategy.
Anderson, J. R., Reder, L. M., Simon, H. A. (1996). Situated learning and education. Educational Research, 25(5), 5-11. https://doi.org/10.2307/1176775

Bales, R. F. (1999). Social interaction systems: Theory and measurement. https://doi.org/10.1037//1089-2699.4.2.199

Baptiste, I. (2001). Qualitative data analysis: Common phases, strategic difference. Forum Qualitative Social Research, 2(3), Art.22. http://www.qualitative-research.net/index.php/fqs/article/view/917/2003.

Bature, I. J., & Atweh, B. (2019). Collaboration: A collective bargain for achieving quality mathematics classroom practice. International Journal of Educational Methodology, 5(3), 347-361. https://doi.org/10.12973/ijem.5.3.347

Benitez Mojica, D., & Londono Millan, N. (2009). Situaciones problematicas en contexto en el aprendizaje del Calculo [Problematic situations in context in Calculus learning]. El calculo y su ensenanza/ The calculation and its teaching, 1(3), 33-43. https://mattec.matedu.cinvestav.mx/el_calculo/index.php?vol=1&index_web=7&index_mgzne

Bilgin, I., Karakuyu, Y., & Ay, Y. (2015). The effect of project based learning on undergrate students' achievement and self-efficacy beliefs towards science teaching. Euroasia Journal of Mathematics, Science and Technology Education, 11(3), 469-477.

Blandez, J.A. (2000). La formacion permanente y la investigacion educativa en el marco contextual de la L.O.G.S.E. [Ongoing training and educational research in the contextual framework of the L.O.G.S.E.]. In J. B. Angel (Ed.), La investigacion en accion: Un reto para el profesorado: Guia p ractica para grupos de trabajo, seminarios y equipos de investigacion [Research in action: A challenge for teachers: Practical guide for work groups, seminars and research teams]. (pp. 17-27). INDE.

Bogdan, R. C., & Biklen, S. K. (1998). Qualitative research for education: An introduction to theory and methods. Allyn & Bacon.

Calik-Uzun, S., Kul, U., & Celik, S. (2019). The impact of instructional technology and material design course on pre-service teachers. International Journal of Educational Methodology, 5(3), 451-463. https://doi.org/10.12973/ijem.5.3451

Changtong, N., Maneejak, N., Yasri, P. (2020). Approaches for Implementing STEM (Science, Technology, Engineering & Mathematics) Activities among Middle School Students in Thailand. International Journal of Educational Methodology, 6(1), 185-198. https://doi.org/10.12973/ijem.6.1.185

Coleman, H., & Unrau, Y. A. (2005). Analyzing qualitative data. In R. M. Grinnell & Y. A. Unrau (Eds.), Social work: Research and evaluation; Quantitative and qualitative approaches (7th ed., pp. 403-420). Oxford University Press.

Collazo, G. (2016). Resolucion de situaciones matematicas en contexto por estudiantes de nivel superior (Grados 10-12) [Resolution of mathematical situations in context by higher level students (Grades 10-12)] [Unpublished doctoral dissertation]. University of Puerto Rico..

Corbin, J., & Strauss, A. (2008). Basic of qualitative research: Techniques and procedures for developing grounded theory. Sage.

Departamento de Educacion de Puerto Rico. (2016). Lista de clasificacion de escuelas segun el plan de flexibilidad [School classification list according to the flexibility plan]. Autor.

Dewey, J. (1985). Democracia y escuela [Democracy and school]. Eumo.

Dunbar, R. (1998). The social brain hypothesis. Evolutionary Anthropology, 6(5), 178-189. https://doi.org/10.1002/(SICI)1520-6505(1998)6:5<178::AID-EVAN5>3.0.CO;2-8

English, L.D. (2007). Cognitive psychology and mathematics education: Reflections on the past and the future. ResearchGate. https://www.researchgate.net/publication/27476502

Freudenthal, H. (1973). Mathematics as an educational task. Reidel.

Freudenthal, H. (1991) Revisiting mathematics education: China lectures. Kluwer.

Gravemeijer, K. (1994). Educational development and developmental research in mathematics education. Journal for Research in Mathematics Education, 25(5), 443-471. https://doi.org/10.2307/749485

Hernandez, R., Fernandez, C., & Baptista, P. (2006). Metodologia de la investigacion [Methodology of investigation] (4th ed.). McGraw.

Knoll, M. (1997). The project method: Its vocational education origin and international development. Journal of Industrial Teacher Education, 34(3), 59-80.

Korman, H. (1986). The focus group Sensig. Suny at Stony Brook.

Kreijns, K., Kirschner, P., & Vermeulen, M. (2013) Social Aspects of CSCL Environments: A Research Framework, Educational Psychologist, 48(4), 229-242

Lopez, A.B., Franco-Mariscal, A.J., & Ramos, E.E. (2015). Ensenar quimica en el contexto de problemas y situaciones de la vida diaria relacionados a la salud [Teaching chemistry in the context of health problems and situations of daily life]. Chemical Education at EduQ/ Educacio Quimica a EduQ, 20, 40-47. https://doi.org/10.2436/20.2003.02.150

Lucca Irizarry, N., & Berrios Rivera, R. (2003). Investigacion cualitativa en educacion y ciencias sociales [Qualitative research in education and social science s]. Publicaciones Puertorriquenas.

Moran, S., & John-Steiner, V. (2003). Creativity in the making: Vygotsky's contemporary contribution to the dialectic of creativity & development doctrine & development. Creativity and Development, Oxford Scholarship Online. https://doi.org/10.1093/acprof: oso.9780195149005.003.0003.

National Council of Teachers of Mathematics. (2000). Principles and standards for schools mathematics. Author.

Pujol Cunill, F. (2017). El Aprendizaje Basado en Proyectos y el Aprendizaje por Descubrimiento Guiada como estrategias didacticas en Biologia y Geologia de 4 de ESO [Project-Based Learning and Guided Discovery Learning as teaching strategies in Biology and Geology at ESO 4] [Unpublished master's thesis]. International University of La Rioja.

Quality Improvement Agency for Lifelong Learning. (2008). Guidance on teaching and assessment at Key Stage 4: Teaching and learning programme. Northern Ireland Curriculum.

Quintero, A. (2010). Matematicas con sentido aprendizaje y ensenanza [ Mathematics with a meaningful learning and teaching]. Editorial Universidad de Puerto Rico.

Puerto Rico Math and Science Partnership (2009) Memorias de las comunidades de practica en investigacion en accion de AlACiMa [Memories of research communities of practice in AlACiMa action]. http://alacima.uprrp.edu/Nuevo-Portal/wp content/uploads/MemoriasInvestigacion.pdf

Richland, L., Stigler, J., & Holyoak, K. (2012). Teaching the conceptual structure of mathematics. Educational Psychologist, 47(3), 189-203.

Saxe, G. B, De Kirby, K., Kang, B., & Schneider, M. L. (2015). Studying cognition though time in a classroom community. The interplay between "Everyday" and "Scientific Concepts" Human Development, 58(1), 5-44. https://doi.org/10.1159/000371560

Sierpinska, A., & Lerman, S. (1996). Epistemologies of mathematics and of mathematics education. International Handbook of Mathematics Education (pp. 827-876). Springer.

Streefland, L. (1991). Fraction in realistic mathematics education, a paradigm of development research. Kluwer Academic Publisher.

Stringer, E. (2008). Action research in education (2nd ed.). Pearson.

Suryanarayanan, S., Kleinman, D. L., Gratton, C., Toth, A., Guedot, C., Groves, R., Piechowski, J., Moore, B., Hagadorn, D., Kauth, D., Swan, H., & Celley, M. (2018). Collaboration matters: Honey bee health as a transdisciplinary model for understanding real-world complexity. BioScience, 68(12), 990-995. https://doi.org/10.1093/biosci/biy118

Velazquez, L. M., & Figarella, F. V. (2012). La problematizacion en el aprendizaje: Tres e strategias para la creacion de un curriculo autentico [Problematization in learning: Three strategies for creating an authentic curriculum]. Isla Negra.

Vygotsky, L. S. (1981). The genesis of higher mental functions. In J. V. Wertsch (Ed.), The concept of activity in Soviet psychology (pp. 144-188). Sharpe.

# data[].attributes.attachment_link.title
The Impact of Project Creation on Learning Mathematics in a Transdisciplinary Setting

# data[].attributes.attachment_link.description
Glenda Collazo Rivera Keywords: Project-based learning, transdisciplinary practice, situations in context, learning strategy. Anderson, J. R., Reder, L. M., Simon, H. A. (1996). Situated learning and education. Educational Research, 25(5), 5-11. https://doi.org/10.2307/1176775 Bales, R. F. (1999). Social interaction systems: Theory and measurement. https://doi.org/10.1037//1089-2699.4.2.199 Baptiste, I. (2001). Qualitative data analysis: Common phases, strategic difference.

# data[].attributes.headline
7. School’s Out: The Role of Summers in Understanding Achievement Disparities 

# data[].attributes.subject
7. School’s Out: The Role of Summers in Understanding Achievement Disparities 

# data[].attributes.body
Paradigm: positivism.
Reason: It uses longitudinal quantitative method to explore the role of summers.

# data[].attributes.attachment_link.readable_text
Summer learning loss (SLL) is a familiar and much-studied phenomenon, yet new concerns that measurement artifacts may have distorted canonical SLL findings create a need to revisit basic research on SLL. Though race/ethnicity and socioeconomic status only account for about 4% of the variance in SLL, nearly all prior work focuses on these factors. We zoom out to the full spread of differential SLL and its contribution to students' positions in the eighth-grade achievement distribution. Using a large, longitudinal NWEA data set, we document dramatic variability in SLL. While some students actually maintain their school-year learning rate, others lose nearly all their school-year progress. Moreover, decrements are not randomly distributed-52% of students lose ground in all 5 consecutive years (English language arts).

References

Benson, J., Borman, G. (2010). Family, neighborhood, and school settings across seasons: When do socioeconomic context and racial composition matter for the reading achievement growth of young children? Teachers College Record, 112(5), 1338-1390.
Google Scholar

Duncan, G. J., Magnuson, K. (2011). The nature and impact of early achievement skills, attention skills, and behavior problems. In Duncan, G. J., Murnane, R. J. (Eds.), Whither opportunity: Rising inequality, schools, and children's life chances (pp. 47-70). Russell Sage Foundation.
Google Scholar

Heyns, B. (1978). Summer learning and the effects of schooling. Academic Press.
Google Scholar

Kaushal, N., Magnuson, K., Waldfogel, J. (2011). How is family income related to investments in children's learning? In Duncan, G. J., Murnane, R. J. (Eds.), Whither opportunity: Rising inequality, schools, and children's life chances (pp. 187-205). Russell Sage Foundation.
Google Scholar

McCombs, J. S., Pane, J. F., Augustine, C. H., Schwartz, H. L., Martorell, P., Zakaras, L. (2015). First outcomes from the National Summer Learning Study (RB-9819-WF). RAND Corporation. https://doi.org/10.7249/RB9819
Google Scholar

Raudenbush, S. W., Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (Vol. 1). Sage.
Google Scholar

Reardon, S. F. (2008). Thirteen ways of looking at the black-white test score gap (Working Paper No. 2008-08). Stanford University.
Google Scholar

Thum, Y. M., Hauser, C. H. (2015). NWEA 2015 MAP norms for student and school achievement status and growth. Northwest Evaluation Association. https://www.nwea.org/content/uploads/2018/01/2015-MAP-Norms-for-Student-and-School-Achievement-Status-and-Growth.pdf
Google Scholar

von Hippel, P. T., Workman, J., Downey, D. B. (2018). Inequality in reading and math skills forms mainly before kindergarten: A replication, and partial correction, of "Are Schools the Great Equalizer?" Sociology of Education, 91(4), 323-357. https://doi.org/10.1177/0038040718801760
Google Scholar

# data[].attributes.attachment_link.title
School's Out: The Role of Summers in Understanding Achievement Disparities - Allison Atteberry, Andrew McEachin, 2020

# data[].attributes.attachment_link.description
Summer learning loss (SLL) is a familiar and much-studied phenomenon, yet new concerns that measurement artifacts may have distorted canonical SLL findings create a need to revisit basic research on SLL. Though race/ethnicity and socioeconomic status only account for about 4% of the variance in SLL, nearly all prior work focuses on these factors.

# data[].attributes.headline
4. Validation Methods for Aggregate-Level Test Scale Linking: A Case Study Mapping School District Test Score Distributions to a Common Scale

# data[].attributes.subject
4. Validation Methods for Aggregate-Level Test Scale Linking: A Case Study Mapping School District Test Score Distributions to a Common Scale

# data[].attributes.body
post-positivism
reason:
To quantify the total amount of error in cross-state TUDA comparisons by falsification.

# data[].attributes.attachment_link.readable_text
Linking score scales across different tests is considered speculative and fraught, even at the aggregate level. We introduce and illustrate validation methods for aggregate linkages, using the challenge of linking U.S. school district average test scores across states as a motivating example. We show that aggregate linkages can be validated both directly and indirectly under certain conditions such as when the scores for at least some target units (districts) are available on a common test (e.g., the National Assessment of Educational Progress). We introduce precision-adjusted random effects models to estimate linking error, for populations and for subpopulations, for averages and for progress over time. These models allow us to distinguish linking error from sampling variability and illustrate how linking error plays a larger role in aggregates with smaller sample sizes. Assuming that target districts generalize to the full population of districts, we can show that standard errors for district means are generally less than .2 standard deviation units, leading to reliabilities above .7 for roughly 90% of districts. We also show how sources of imprecision and linking error contribute to both within- and between-state district comparisons within versus between states. This approach is applicable whenever the essential counterfactual question-"what would means/variance/progress for the aggregate units be, had students taken the other test?"-can be answered directly for at least some of the units.

References

Braun, H., Qian, J. (2007). An enhanced method for mapping state standards onto the NAEP scale. In Dorans, N. J., Pommerich, M., Holland, P. W. (Eds.), Linking and aligning scores and scales. Statistics for social and behavioral sciences (pp. 313-338). New York, NY: Springer.
Google Scholar

Gewertz, C. (2015). PARCC restructures, allows states to customize test. Education Week. Retrieved from http://blogs.edweek.org/edweek/curriculum/2015/11/parcc_allows_states_to_customize_test.html
Google Scholar

Haertel, E. H. (1991). Report on TRP analyses of issues concerning within-age versus cross-age scales for the National Assessment of Educational progress. Washington, DC: National Center for Educational Statistics. Retrieved from https://files.eric.ed.gov/fulltext/ED404367.pdf
Google Scholar

Kolen, M., Brennan, R. (2014). Test equating, scaling, and linking. New York, NY: Springer-Verlag.
Google Scholar

McClellan, C. A., Donoghue, J. R., Gladkova, L., Xu, X. (2005). Cross-grade scales in NAEP: Research and real-life experience. Longitudinal Modeling of Student Achievement. Maryland Assessment Research Center for Education Success. College Park, MD: University of Maryland. Retrieved from http://marces.org/conference/Longitudinal/Mcclellan.ppt
Google Scholar

Raudenbush, S. W., Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.
Google Scholar

Reardon, S. F., Ho, A. D. (2015). Practical issues in estimating achievement gaps from coarsened data. Journal of Educational and Behavioral Statistics, 40(2): 158-189. |
Google Scholar SAGE Journals

Thissen, D. (2007). Linking assessments based on aggregate reporting: Background and issues. In Dorans, N. J., Pommerich, M., Holland, P. W. (Eds.), Linking and aligning scores and scales (pp. 287-312). New York, NY: Springer.
Google Scholar

U.S. Department of Education . (2009). Race to the top executive summary. Washington, DC. Retrieved from https://www2.ed.gov/programs/racetothetop/executive-summary.pdf
Google Scholar

U.S. Department of Education . (2015). ED Facts Submission System User Guide V11.2 (SY 2014-2015). Washington, DC: ED Facts. Retrieved from http://www.ed.gov/EDFacts
Google Scholar

U.S. Department of Education . (n.d.). NAEP data explorer. Washington, DC: National Center for Education Statistics, Institute of Education Sciences.
Google Scholar

Wilson, R. E., Bowers, M. J., Hyde, R. L. (2011). Special investigation into test tampering in Atlanta's school system. Retrieved from https://ia801000.us.archive.org/32/items/215252-special-investigation-into-test-tampering-in/215252-special-investigation-into-test-tampering-in.pdf
Google Scholar

# data[].attributes.attachment_link.title
Validation Methods for Aggregate-Level Test Scale Linking: A Case Study Mapping School District Test Score Distributions to a Common Scale - Sean F. Reardon, Demetra Kalogrides, Andrew D. Ho, 2019

# data[].attributes.attachment_link.description
Linking score scales across different tests is considered speculative and fraught, even at the aggregate level. We introduce and illustrate validation methods for aggregate linkages, using the challenge of linking U.S. school district average test scores across states as a motivating example.

# data[].attributes.headline
2.Facilitating collaborative reflective inquiry amongst teachers: What do we currently know?

# data[].attributes.subject
2.Facilitating collaborative reflective inquiry amongst teachers: What do we currently know?

# data[].attributes.body
Published: Tuesday 15 September 2020, 14:00 – 15:00
Paradigm：positivism
Reason：With RPI study the author engages in a meta-narrative literature review, covering a range of disciplines (including education, medicine, and psychology), in an attempt to fill these knowledge gaps. Findings indicate that there are no existing interventions designed to foster RPI that have been rigorously evaluated.

# data[].attributes.attachment_link.readable_text
Collaborative teacher learning centred on research evidence is thought to improve teaching practice and student outcomes.

Key to such learning is Reflective Professional Inquiry (RPI); seen as vital if practitioners are to engage effectively with new knowledge and ideas. Yet RPI is under-conceptualized and little is known about how to facilitate effective RPI.

With this presentation I will present the findings of a meta-narrative literature review, covering a range of disciplines (including education, medicine, and psychology), in an attempt to fill these knowledge gaps. Findings indicate that there are no existing interventions designed to foster RPI that have been rigorously evaluated. Consequently, there are no approaches that could be employed as part of collaborative teacher learning, with the expectation that practice or student outcomes will subsequently improve. The study does however provide clues for what any RPI-related intervention might need to consider moving forward.

Professor Chris Brown, Professor in Education, Director of Research, Durham University.

The Networked School Leader: How to improve teaching and student outcomes using learning networks by Chris Brown.

The Research-Informed Teaching Revolution: a handbook for the 21st century teacher Edited by Chris Brown, Jane Flood and Graham Handscomb.

Networks for Learning: Effective Collaboration for Teacher, School and System Improvement Edited by Chris Brown and Cindy Poortman.

# data[].attributes.attachment_link.title
Facilitating collaborative reflective inquiry amongst teachers: what do we currently know?

# data[].attributes.attachment_link.description
Collaborative teacher learning centred on research evidence is thought to improve teaching practice and student outcomes. Key to such learning is Reflective Professional Inquiry (RPI); seen as vital if practitioners are to engage effectively with new knowledge and ideas. Yet RPI is under-conceptualized and little is known about how to facilitate effective RPI.

# data[].attributes.headline
3.Demonstrating the potential of text mining for analyzing school inspection reports: a sentiment analysis of 17,000 Ofsted documents

# data[].attributes.subject
3.Demonstrating the potential of text mining for analyzing school inspection reports: a sentiment analysis of 17,000 Ofsted documents

# data[].attributes.body
Published online: 17 Sep 2020
Paradigm: positivism
Reason: In doing so, we hope to demonstrate the efficiency with which text mining approaches can provide representative analysis of very large volumes of inspection reports, making them a useful complement to smaller-scale, manual analyses.

# data[].attributes.attachment_link.readable_text
ABSTRACT
Many national education systems incorporate a central inspectorate tasked with visiting, evaluating and reporting on the performance of schools. The judgements produced by inspectors often play a part in the way that schools are held to account and constitute an important source of data in their own right. Inspection reports are therefore of great interest to researchers. However, the sheer quantity of inspection reports produced by national school inspectorates creates challenges for analysts. We demonstrate the use of text mining - automated processing and analysis of unstructured textual data - to analyse the complete corpus of school inspection reports released by the English national schools inspectorate since the turn of the century. More precisely, we report the results of a sentiment analysis, comparing the tone of inspection reports across the different grades awarded in each inspection and across different Chief Inspectors. In doing so, we hope to demonstrate the efficiency with which text mining approaches can provide representative analysis of very large volumes of inspection reports, making them a useful complement to smaller-scale, manual analyses. Resources and references are provided for researchers looking to use text mining techniques.

Disclosure statement

No potential conflict of interest was reported by the author(s).

# data[].attributes.attachment_link.title
Demonstrating the potential of text mining for analyzing school inspection reports: a sentiment analysis of 17,000 Ofsted documents

# data[].attributes.attachment_link.description
ABSTRACT Many national education systems incorporate a central inspectorate tasked with visiting, evaluating and reporting on the performance of schools. The judgements produced by inspectors often play a part in the way that schools are held to account and constitute an important source of data in their own right.

# data[].attributes.headline
8. Can a Positive School Climate Promote Student Attendance? Evidence From New York City

# data[].attributes.subject
8. Can a Positive School Climate Promote Student Attendance? Evidence From New York City

# data[].attributes.body
Paradigm: positivism.
Reason: The researcher estimated the correlation first and then testified by analyzing two annual surveys.

# data[].attributes.attachment_link.readable_text
Nearly 15% of American students are chronically absent from school. To address absenteeism, many states have recently made chronic absence a core component of their school accountability plans. Scholars have theorized that a positive school climate can promote student attendance, but empirical support for this idea is lacking. In this study, the relationship between four student-reported measures of school climate and student attendance are investigated by analyzing two annual school climate surveys (N = 823,753) from New York City. Results indicate small associations among the four measures of perceived school climate and student attendance. Furthermore, school-level changes in perceived school climate between middle and high school were only marginally associated with student attendance.

References

Alspaugh, J. W. (1998). The relationship of school-to-school transitions and school size to high school dropout rates. High School Journal, 81(3), 154-160.
Google Scholar

Cohen, J., McCabe, L., Michelli, N. M., Pickeral, T. (2009). School climate: Research, policy, practice, and teacher education. Teachers College Record, 111(1), 180-213.
Google Scholar

Cordes, S., Leardo, M., Rick, C., Shwartz, A. E. (2019). Can school buses drive down (chronic) absenteeism? In Gottfried, M. A., Hutt, E. L. (Eds.). Absent from school: Understanding and addressing student absenteeism (pp. 121-136). Harvard Education Press.
Google Scholar

Cortiella, C., Boundy, K. B. (2018, April). Students with disabilities and chronic absenteeism (NCEO Brief No. 15). University of Minnesota, National Center on Educational Outcomes.
Google Scholar

Goodman, J. (2014). Flaking out: Student absences and snow days as disruptions of instructional time (No. w20221). National Bureau of Economic Research.
Google Scholar

Hilbe, J. M. (2011). Negative binomial regression. Cambridge University Press.
Google Scholar

Sheldon, S. B., Epstein, J. L. (2004). Getting students to school: Using family and community involvement to reduce chronic absenteeism. School Community Journal, 14(2), 39-56.
Google Scholar

Thompson, R. A., Goodvin, R. (2016). Social support and developmental psychopathology. In Cicchetti, D. (Ed.), Developmental psychopathology: Vol. 4. Risk, resilience, and intervention (3rd ed., pp. 86-135). Wiley.
Google Scholar

Williams, E., Richman, S. (2007). The first year of high school: A quick stats fact sheet. National High School Center. https://files.eric.ed.gov/fulltext/ED501073.pdf
Google Scholar

# data[].attributes.attachment_link.title
Can a Positive School Climate Promote Student Attendance? Evidence From New York City - Daniel Hamlin, 2020

# data[].attributes.attachment_link.description
Nearly 15% of American students are chronically absent from school. To address absenteeism, many states have recently made chronic absence a core component of their school accountability plans. Scholars have theorized that a positive school climate can promote student attendance, but empirical support for this idea is lacking.

# data[].attributes.headline
4.Motivation of higher education faculty: Theoretical approaches, empirical evidence, and future directions

# data[].attributes.subject
4.Motivation of higher education faculty: Theoretical approaches, empirical evidence, and future directions

# data[].attributes.body
Paradigm: interpretivism
Reason: The author explores and elaborate on four questions to inspire future research on faculty motivation.

# data[].attributes.attachment_link.readable_text
Highlights

*

Four questions to inspire future research on faculty motivation are investigated:

*

Why we should be concerned with the motivations of higher education faculty?

*

Why is research on this topic still rather underdeveloped?

*

Which motivation theories apply well to faculty and how they align with one another?

*

What should thorough international research on faculty motivation entail?

Abstract

We explore and elaborate on four questions to inspire future research on faculty motivation. The first question asks why we should be concerned with the motivations of higher education faculty in the first place, particularly in regard to studying them empirically? Moreover, if research on higher education faculty motivation is important, why is it still rather underdeveloped? Building on this, considering the plethora of motivation frameworks, we wondered which theories apply well to faculty members and how they align with one another? Finally, what should thorough international research on faculty motivation entail? This question encompasses a search for solutions on what could be done to broaden the scope of this important line research.

Recommended articles Citing articles (0)

# data[].attributes.attachment_link.title
Motivation of higher education faculty: Theoretical approaches, empirical evidence, and future directions

# data[].attributes.attachment_link.description
Four questions to inspire future research on faculty motivation are investigated: * Why we should be concerned with the motivations of higher education faculty? * Why is research on this topic still rather underdeveloped? * Which motivation theories apply well to faculty and how they align with one another?

# data[].attributes.headline
5.The Significance of Motivations and Selected Effects of Student Employment in the Course of Studies: A Case of an Economics Study Program in Poland

# data[].attributes.subject
5.The Significance of Motivations and Selected Effects of Student Employment in the Course of Studies: A Case of an Economics Study Program in Poland

# data[].attributes.body
First Published: January 15, 2020
Paradigm: positivism
Reason: It refers to areas that have not yet been investigated by other researchers. The data used in the article come from the survey conducted by the author at the Faculty of Economics, at the University of Economics in Katowice through 2014-2017.

# data[].attributes.attachment_link.provider_name
The Significance of Motivations and Selected Effects of Student Employment in the Course of Studies: A Case of an Economics Study Program in Poland

# data[].attributes.attachment_link.readable_text
Izabela Ostoj
Keywords: Student employment, higher education, full-time studies, Poland.
Bachmann, A., & Boes, S. (2014). Private transfers and college student's decision to work. Economics of Education Review, 42, 34-42.

Ballesteros-Regana, C., Soles-Rojas, C., Hervas-Gomes, C., & Diaz-Noguera, M. D. (2019). Improving the quality of teaching internships with the help of the platforms. European Journal of Educational Research, 8(4), 1101-1112.

Baum S. (2010). Student work and the financial aid system. In L.W. Perna (Ed.), Understanding the working college student. New research and its implications for policy and practice (pp. 3-22). Sterling, VA: Stylus Publishing LLC.

Bayram, A. (2019), The reflection on neoliberal economic policies on education: privatization of education in Turkey. European Journal of Educational Research, 7(2), 341-347.

Beerkens, M., Magi, E., & Lill, L. (2011). University studies as a side job: causes and consequences of massive student employment in Estonia. Higher Education, 61(6 ), 679-692.

Body, K. M. D., Bonnal, L., & Giret J. F. (2014). Does student employment really impact academic achievement? The case of France. Applied Economics, 46(25), 3061-3073.

Callender, C. (2008). The impact of term-time employment on higher education students' academic attainment and achievement. Journal of Education Policy, 23(4), 359-377.

Dobbs, R. L., Sun, J. Y., & Roberts, P. B. (2008). Human capital and screening theories: Implications for Human Resource development. Advances in Developing Human Resources, 10(6), 788-801.

Grobelna, A. (2017). The perception of job-related motivators when choosing a career in the tourism and hospitality industry - a comparative study between Polish and Spanish students. International Journal of Management and Economics, 53(2),84-106.

Hauschildt, K., Vogtle, E. M., & Gwosc, C. (2018). Social and economic conditions of student life in Europe. Eurostudent VI 2016-2018. Bielefeld, Germany:W. Bertelsman Verlag GmbH & Co. KG.

Hovdhaugen, E. (2015). Working while studying: the impact of term-time employment on dropout rates. Journal of Education and Work, 28(6), 631-651.

IPSS & e-Dialog Sp. z o.o. (2012). Raport z badan II edycji projektu Uczelnia przyjazna pracodawcom [Survey report in the second edition of the Employer-Friendly University project]. Warsaw, Poland: Institute of Work and Social Affairs.

Jacob, M., Gerth, M., & Weiss, F. (2018). Student employment: social differentials and field-specific developments in higher education. Journal of Education and Work, 31(1), 87-108.

Jarecki, W. (2010). Praca i dochody studentow [Work and income of students]. Social Policy /Polityka Spoleczna, 1, 21-25.

Jelonek M., Antosz P., & Balcerzak-Raczynska A. (2014). Przyszłe kadry polskiej gospodarki [ Future workforce of Poland's economy]. Warsaw, Poland: Polish Agency for Enterprise Development/PARP.

Khelifi, S. (2019). Interplay between politics and institution in higher education reform. European Journal of Educational Research, 8(3), 671-681.

Kosi, T., Nastav, B., & Susteric, J. (2013). Does student employment deteriorate academic performance? The case of Slovenia. Croatian Journal of Social Policy/ Revija za Socijalnu Politiku, 20(3), 253-274.

Neill, C. (2015). Rising student employment: The role of tuition fees. Education Economics, 23(1), 101-121.

Ormiston, R. (2016). Does high school employment develop marketable skills? Journal of Labor Research, 37(1), 53-68.

Park, Y., & Sprung, J. M. (2013). Work-school conflict and health outcomes: beneficial resources for working college students. Journal of Occupational Health Psychology, 18(4), 384-394.

Perna, L. W. (2010). Conclusions and recommendations for policy, practice and future research. In L.W. Perna (Ed.), Understanding the working college student. New research and its implications for policy and practice (pp. 283-308). Sterling, VA: Stylus Publishing LLC.

Pinto, V. (2010). Students employment and social inequalities in higher education. Social Science Research Acts/Actes de la Recherche en Sciences Sociales, 183, 58-71.

Riggert, S. C., Boyle, M., Petrosco, J. M., Ash, D., & Rude-Parkins, C. (2006). Student employment and higher education: empiricism and contradiction. Review of Educational Research, 76(1), 63-92.

Ruhm, C. J. (1997). Is high school employment consumption or investment? Journal of Labor Economics, 15(4), 735-776.

Sanchez-Gelabert A., Figueroa M., & Elias M. (2017). Working whilst studying in higher education: the impact on the economic crisis on academic and labour market success. European Journal of Education, 52(2), 232-245.

Scott-Clayton, J. (2012). What explains trends in labour supply among us undergraduates? National Tax Journal, 65(1), 181-210.

Scott-Clayton, J., & Minaya, V. (2014). NBER Working Paper Series No. 20329. Should student employment be subsidized? Conditional Counterfactuals and the Outcomes of Work-Study Participation. Retrieved from NBER database.

Spence, M. (1973). Job Market Signaling. Quarterly Journal of Economics, 87(3), 355-374.

Statistics Poland (2019). Higher education and their finances in 2018. Gdansk, Poland: Statistical Office.

Stephenson, S. P. J. (1981). In-school labor force status and post-school wage rates of young men. Applied Economics, 13, 279-302.

Triventi M. (2014). Does working during higher education affect students' academic progression? Economics of Education Review, 41, 1-13.

Trolian T. L., Jach E. A., & Snyder K. F. (2018). Connecting college and work: examining the relationship between students' college employment experiences and their professional and career attitudes. Journal of Education and Work, 31(4), 366-380.

Tur-Sinai A., Romanov D., & Zussman N. (2017). The true effect of students' employment on the duration of studies. Applied Economics, 49(33), 3328-3340.

Yanbarisova D. M. (2015). The effects of student employment on academic performance in Tatarstan higher education institutions. Russian Education and Society, 57(6), 459-482.

# data[].attributes.attachment_link.title
The Significance of Motivations and Selected Effects of Student Employment in the Course of Studies: A Case of an Economics Study Program in Poland

# data[].attributes.attachment_link.description
Izabela Ostoj Keywords: Student employment, higher education, full-time studies, Poland. Bachmann, A., & Boes, S. (2014). Private transfers and college student's decision to work. Economics of Education Review, 42, 34-42. Ballesteros-Regana, C., Soles-Rojas, C., Hervas-Gomes, C., & Diaz-Noguera, M. D. (2019).

# data[].attributes.headline
5.Flipped Classroom Educational Model (2010-2019): A Bibliometric Study

# data[].attributes.subject
5.Flipped Classroom Educational Model (2010-2019): A Bibliometric Study

# data[].attributes.body
First Published: October 15, 2020
Paradigm: positivism
Reason: The research was conducted using a 4-stage systematic mapping method.

# data[].attributes.headline
9. “I’m a Teacher, I’m Gonna Always Protect You”: Understanding Black Educators’ Protection of Black Children

# data[].attributes.subject
9. “I’m a Teacher, I’m Gonna Always Protect You”: Understanding Black Educators’ Protection of Black Children

# data[].attributes.body
Paradigm: Interpretivism.
Reason:It uses comparative case studies, interviews, and observational data to understand black educators' conceptualizations of protection and the mechanisms of protection they enact.

# data[].attributes.attachment_link.readable_text
Abstract

Many Black educators in the United States demonstrate a political clarity about white supremacy and the racialized harm it cultivates in and out of schools. We highlight the perspectives of some of these educators and ask, (1) How do they articulate the need to protect Black children? and (2) What mechanisms of protection do they enact in their classrooms and schools? Through further elaborating the politicized caring framework, our analyses show how Black educators disrupt the racialized harm produced within schools to instead (re)position Black students as children worthy of protection via caring relationships, alternative discipline policies, and other interpersonal and institutional mechanisms. This study has implications for teaching, teacher education, and how the "work" of teachers is conceptualized and researched.

Introduction

The white1 supremacist foundations of American society ( Baptiste, 2015; King, 2011) have always rendered Black children socially vulnerable to physical and psychological violence. This through line of vulnerability is repeatedly made visible in narratives by formerly enslaved persons like Frederick Douglass, in the horrific murder of Emmitt Till, and in each homegoing mural or t-shirt, hashtag (e.g., #SayHerName, #TonyRobinson), or viral video that concurrently humanizes and makes a spectacle of Black children's lives and bodies. These vulnerabilities are the subject of national dialogues (e.g., #BlackLivesMatter) and urgent conversations among many Black adults about how to protect Black children. These conversations are grounded in Black adults' embodied understandings of how white supremacy often dictates the psychosocial and physical safety of the Black body, especially of children2 ( Kendi, 2016; Phillip, 2018).

Assaults on Black children, especially in schools (e.g., Vance County Middle School in North Carolina, Spring Valley High School in South Carolina, East Middle School or Success Academy in New York, Ozen High School in Texas), disturbingly document interpersonal and institutional racism in action, where adults who are charged with educating and supporting children are instead perpetrators of uncategorizable forms of violence against them. These assaults on Black children in their schools show how spaces presumedly designed to be "safe" for children and benign actions, like walking or talking, get quickly distorted into opportunities to use extreme or lethal force (e.g., Fantz et al., 2015; Greenwell, 2012). These highly publicized and lesser known assaults necessitate that we consider the questions, "What do we mean by 'safe'?,""Which spaces are 'safe'?," and "For whom is a place 'safe'?"

Many Black educators3 in the United States, themselves once Black children and K-12 students, recognize this double-edged sword of "safety" and continually place themselves on the frontlines-discursively and physically-by denouncing the racialized harm and neglect Black children experience (e.g., Beauboeuf-Lafontant, 1999; Dixson, 2003; Walker, 1996). Racialized harm refers to the perpetual racism Black children encounter as their Black bodies get racially perceived and interacted with in their day-to-day lives in and out of school. A part of the legacy of these educators is a sociopolitical consciousness or political clarity (Madkins & McKinney de Royston, 2019) about white supremacy and the violence it cultivates in and out of schools (e.g., Foster, 1990; Milner, 2006; Walker, 2009a, 2009b). This protection involves serving as active allies, advocates, or co-conspirators ( Christens & Kirshner, 2011; Dixson, 2011; Love, 2019a, 2019b) alongside Black children, including enacting culturally and politically relevant pedagogies (e.g., Beauboeuf-Lafontant, 1999; Ladson-Billings, 2005).

As heartbreakingly demonstrated by the Black teacher in the Spring Valley video ( Ford et al., 2015) or by Black educators whose actions problematically maintain the status quo, value "respectability" over the dignity of the child, and/or are rooted in cultural deficit, culture of poverty, or other racist internalizations, acts of protection are not the birthright of every Black educator. Similarly, there are non-Black educators who consciously and diligently enact means of protection for Black children. In focusing on a few educators among a cadre of Black educators across the United States who think and operate outside of white supremacist and assimilationist frameworks, our aim is to elucidate how they articulate the need to protect Black children and the practices they enact every day that reflect these concerns.

The empirical literature that documents Black educators and their distinct politicized relationships with Black students (e.g., Foster, 1990, 1998; Walker, 1996) is frequently retrospective and situated in the segregated South. This scholarship helpfully articulates strategies Black educators have used to ensure Black students' success, yet there are few empirical examples of how 21st-century Black educators think about and attend to Black children's racialized needs in the current racial context ( Dumas & ross, 2016). This requires understanding why many Black educators intentionally enact protective stances towards Black students, and what this means for their work in schools. Here, we highlight the perspectives of Black educators who engage in this risky work of caring for and protecting Black children. To examine the contours of this work, we ask, (1) How do Black educators articulate the need to protect Black children? and (2) What mechanisms of protection do they enact in their classrooms and schools?

Our aim with this analysis is to further theorize 21st-century Black educators' discourses of care and protection and to offer empirical accounts of this protection in action. In this article, we first contextualize this risky work as part of a broader literature that documents how race and racism attack Black childhood, the importance of positive interpersonal connections within schools for Black children, and the distinct, relational work of Black educators. We then articulate how the framework of politicized caring drives our analysis, emphasizing how protection (from physical, psychological, symbolic, to other interpersonal and institutionalized forms of racialized harm) is an undertheorized aspect of caring. In the Methods section we describe how interviews are used to draw out Black educators' articulations of and mechanisms for protecting Black children and Black parents' and students' perceptions about them, and the importance of using observations to analyze how such acts of protection are enacted. Findings demonstrate that these Black educators consciously engage in creating caring relationships and safety nets to protect Black children from racialized harm. Finally we consider the implications of this study for teaching, teacher education, and how we conceptualize the "work" of teachers.

Literature Review

The Erasure of Black Childhood

Stereotypes about intellectual capacity, personalities, activities, and investment in education create a pervasive culture of suspicion and fear around Black children ( Carter et al., 2014; Dumas, 2014; Okonofua & Eberhardt, 2015). These stereotypes are palpable in the lowered expectations of teachers, implicit and explicit racial tracking in schools, disproportionate and harsher disciplinary practices (e.g., removal from class, suspension, expulsion) experienced by Black children, and variances in Black students' perceptions of school climate and safety ( Carter et al., 2014; Ferguson, 2000; Lacoe, 2013). Ferguson's (2000) ethnographic study, Bad Boys, tellingly documented how such a culture of fear gets enacted via school discipline policies and practices that adultify and criminalize Black children's actions.

This adultification and criminalization of Black children's bodies and actions in schools perpetually deny them "any access to childhood humanity" ( Dancy, 2014, p. 49). Such actions link today's schooling practices to a legacy of anti-Black endeavors (e.g., human enslavement) in the United States that steal Black youth's childhoods through "arbitrary authority, punishment, and separation" ( King, 2011, p. xxi). Having an embodied understanding of what this continual denial of Black childhood means-that protections designed for and given to children are generally withheld from Black children-Black educators offer counternarratives to protect Black children. Moreover, they call out the racialized forms of harm Black children experience to argue that it is Black students who need protection ( McKinney de Royston et al., 2017). This honoring of Black students as children acknowledges their vulnerabilities as such and the role adults can play as developmental partners. This literature suggests that Black adult-child partnerships in schools, in part, hinges upon this affirmation of Black children's humanity and childhood by signaling to them that they are worthy of protection.

Perceptions of Safety and Teacher-Student Relationships

Honoring the humanity and childhood of Black children is imperative. They, like other children, are savvy observers and sense-makers of social dynamics and tensions. Children's perceptions of safety-socially, emotionally, intellectually, physically, and otherwise-affect their well-being, learning, and academic performance ( Astor et al., 2010; Devine & Cohen, 2007; Lacoe, 2013; Thapa et al., 2013). Lacoe's (2013) longitudinal analysis of survey data from over 340,000 children in nearly 700 middle schools across New York City found that Black and Latinx students are "more likely to report feeling unsafe at school than white or Asian peers who attend the same schools and share the same classrooms" (p. 5), with these feelings being related to students' perceptions about disparities in disciplinary fairness, school disorder, racial tension, and academic performance.

Thapa et al.'s (2013) review of school climate literature echoed these findings, noting that it is common for Black and Latinx students to not feel physically and emotionally safe in schools. Making explicit the link between students' racialized perceptions of safety and interpersonal and institutional factors, they argue, "In schools without supportive norms, structures, and relationships, students are more likely to experience violence, peer victimization, and punitive disciplinary actions" (p. 360). Such research problematizes what safety means and the racialized tenor of safety, namely, who gets to feel safe and benefits from "safety measures in schools" like metal detectors, police officers, and zero-tolerance policies.

The Relational Work of Black Educators

Acknowledging the physical, intellectual, psychological, emotional, and relational aspects of safety is salient to understanding Black children's racialized experiences in schools. Their experiences of racialized harm present a challenge to U.S. schools to facilitate the belonging and intellectual thriving of Black students ( Williams et al., 2016). One critical source of insight should be Black educators, who have a legacy of intentionally fostering supportive, race-conscious relationships with Black children ( Ginwright, 2010; McKinney de Royston et al., 2017; Milner & Tenore, 2010). For example, Black segregated schools (cf. Sowell, 1974; Walker, 1996, 2009a, 2009b) are rightfully lauded for their strong interpersonal mechanisms, such as teacher-student relationships ( Foster, 1990, 1998; Lomotey, 1989, 2003; Tillman, 2004; Walker, 1996, 2000). Postdesegregation schools designed for Black children by Black educators have been found to have similar mechanisms (cf. Lee, 1992; Lomotey, 1992; McKinney de Royston, 2011; J. E. Morris, 1999). Supportive Black educator-student relationships have also been found in all-Black classrooms within racially heterogeneous school environments (e.g., McKinney de Royston et al., 2017) as well as between Black educators and Black students that interact within racially heterogeneous classrooms and schools ( Ladson-Billings, 1995, 2005; Milner, 2006)

This literature makes clear that Black educator-Black student relationships are historically and politically situated mechanisms of interpersonal care that support the well-being, academic achievement and long-term success of Black students (e.g., Anderson, 1988; Beauboeuf-Lafontant, 1999; Dixson, 2003; Foster, 1998; Irvine, 1989; Ladson-Billings, 2005; Tillman, 2004; Walker, 1996). This relational work is often rooted in Black educators' clarity about the historical and political landscape they and their students work, worship, and otherwise live within daily. Such clarity guides Black educators' intention to protect and defend the humanity of Black children and the "sense of urgency" to their relational work ( Milner, 2006, p. 94). Moreover, this political clarity shapes Black educators' understanding of racialized disparities and experiences in schools as having little or nothing to do with the inability or disinterest of individual or groups of Black children or families. Instead, they understand these as by-products of systemic racism in (e.g., the school to prison nexus; Meiners, 2011) and out of schools (e.g., intergenerational poverty, violence, stress, and lack of access to health care). Their relational work seeks to disrupt and counteract how race and racism get operationalized upon Black children's bodies to signal to Black children their inherent and infinite capacity and goodness.

The relational work of Black educators is designed to support the well-being and success of Black students-such as holding high expectations for them-and serve these students in ways that go beyond those considered "typical" for teachers ( Foster, 1990; Irvine & Fraser, 1998; Ladson-Billings, 1997; Ware, 2006). These latter efforts may include offering students clothing, food, or transportation; talking about students' personal or home life; and supporting students' positive self-images and/or challenging essentialized conceptions of Blackness, including culturally grounded examples in their lessons, and other practices that do not disentangle the pedagogical from the political and the personal ( Milner & Tenore, 2010; ross et al., 2016). These enactments by Black educators' counter the hidden curriculum4 ( Giroux & Penna, 1979) in schools and work to protect Black children against the racialized trauma it can cause.

Theoretical Framework

To further understand how Black educators who work outside of white supremacist and assimilationist frameworks conceptualize the notion of protection and enact it to protect Black children from racialized harm, we draw upon the framework of politicized caring ( McKinney de Royston et al., 2017). This framework was developed through studying Black educators who were concerned with the vulnerabilities of Black children and how they experience racialized neglect in schools. Politicized caring draws upon theories of caring in education, interpersonal and institutional care in Black learning spaces, culturally relevant teaching, and Black feminist thought. Below we explain how these theories inform this framework and how further developing it to include protection is useful for the present analysis.

In the United States, care and caring are viewed as important, if not necessary, elements of schools and education. Scholars have theorized how care and caring are tangible and intangible aspects of education that get communicated and enacted pedagogically and through specific teacher-student interactions (e.g., Noblit et al., 1995; Noddings, 1992, 2012) and are important for children's learning and schooling experiences. Scholars have challenged color-evasive (i.e., perspectives that minimize or ignore issues of race and/or racism) theories that focus on ethics of care and caring relations with little or no attention to inequalities and power dynamics in schools, especially those experienced by racially minoritized children ( Ladson-Billings, 1997; McKinney de Royston et al., 2017; Roberts, 2010; Valenzuela, 1999; Ware, 2006). A major critique of such approaches is that when they do not yield the desired effect, the "problem" is attributed to a minoritized student's lack of motivation, grit (i.e., passion and perseverance to achieve long-term goals; Duckworth, 2016), or capacity rather than to structural, institutional, or cultural conditions of schools and schooling ( García & Guerra, 2004; Love, 2019a, 2019b; Patterson et al., 2008; Valencia & Solórzano, 1997). This is despite successive studies showing that race is "central to what schools do and how they do it" ( Patterson et al., 2008, p. 98) and that racism gets perpetuated through a lack of institutional and interpersonal care for Black children ( McKinney de Royston et al., 2017; Roberts, 2010; Walker, 1996).

Valenzuela (1999) asked, "What does it mean to care about children from marginalized communities, given the political and social context in which education takes place?" (p. xvi). This question suggests that theories of care and caring cannot exist apart from historical and structural analyses of how schools are set up and how educators view their students and the world ( Patterson et al., 2008). Indeed, scholars drawing upon Black feminist thought challenge the political and racial neutrality of education, pedagogy, and theories of care and instead acknowledge systemic forces as ongoing sources of everyday societal inequity and oppression (e.g., Beauboeuf-Lafontant, 2002; Collins, 1990; hooks, 1994; Ladson-Billings, 1997, 2005; McKinney de Royston et al., 2017; Walker, 1996; Ware, 2006). These scholars theorize how the caring, culturally and politically relevant approaches of Black educators reflect their awareness about Black children's racialized lives and schooling experiences.

Bringing the aforementioned theories into contact with each other led to the development of the framework of politicized care and its four dimensions: political clarity, communal bonds, potential affirming, and developmentally appropriate. The dimensions are described in detail elsewhere ( McKinney de Royston et al., 2017); here we focus on the dimension of political clarity and its relation to care or caring. Political clarity reflects an educator's sociopolitical consciousness about the historical and institutional nature of oppression-from systemic acts of racialized violence to microaggressions-that frequently shape Black students' lived experiences in and out of schools. Political clarity explains how a Black educator's caring for Black children becomes politicized-that is, a politics gets enacted-through their social interaction with them precisely because those educators' understand Black children as jeopardized and in need of care due to (1) the categorical nature of racial stereotypes ( Carter et al., 2014; Milner, 2012) ( Beauboeuf-Lafontant, 1999) and (2) the complicated connections between interpersonal and structural forms of racism that limit the opportunities for Black children to survive and thrive educationally, emotionally, psychologically, and physically.

Linking political clarity with protection makes clear why and how Black educators' adopt protective institutional and pedagogical stances toward safeguarding Black children from the by-products of systemic racism and white supremacy inside and outside of schools. Elaborating politicized caring to explicitly attend to the notion of protection expands caring beyond notions of nurturing to encompass strategic acts of shielding and defending Black children from racialized harm (physical, psychological symbolic, interpersonal, and institutionalized, and otherwise). Within schools, politicized caring has been depicted as involving conscious acts-verbal or otherwise-that nurture Black children. Adding in a lens of protection acknowledges how and when Black educators may also be repositioning Black students as children who are worthy of compassion and reclaiming Black childhood as a developmental period in the life course that is defined by the ability to play, make mistakes, and grow. These complementary conscious acts of nurturing and protecting counter the pervasive positioning of Black children as problems to reposition them as children made vulnerable by a school structure (and social order) that mislabels them as smaller versions of the adult monsters or criminals it presumes they will become. Protection means actively (re)positioning Black children as worthy of the expectations and opportunities of any child, and shielding students from racialized harms. Thus, a robust rendering of politicized caring links together the twin notions of nurturing and protecting.

Methods

The data presented in this article come from a larger study situated in Oakland, California,5 a city that has grappled with racially disparate experiences and outcomes for racially, economically, and linguistically minoritized students. In response, the school district developed a number of reforms and invited our research team to examine how initiatives relevant for Black students were understood and taken up in schools. Using an approach of community nomination by stakeholders (e.g., school administrators, teachers, families, school alumni) in the district, data collection was focused on "successful" schools that had large populations of Black student populations and where the sociopolitical contexts outside of schools was mentioned by multiple participants as affecting activities insides of schools. We chose this focus as unnuanced characterizations of "unsuccessful" schools are overrepresented and continue a troubling pattern of research, particularly on schools are run by Black educators and serve Black children, that Lightfoot (1986) admonished as engaging in a "search for pathology" (p. 11). Instead, we honor those schools stakeholders identified as "successful" and seek to complexify our understanding of the work of contemporary Black educators. We thus defined success based on an inclusive school climate for Black children as determined by cstakeholders and/or normative academic measures (e.g., Academic Performance Indicator scores). Several elementary, middle, and high schools and K-8 schools were recommended via community nominations6 ( Foster, 1991; Ladson-Billings, 1995) from district officials, parents, and community organizations. Based on their willingness to participate, seven sites were selected: two elementary schools, three middle schools, and two high schools.

Study Design

The broader study sought to develop comparative case studies ( Bartlett & Vavrus, 2016) within and across focal schools to understand the pedagogical, relational, and institutional characteristics that supported the relative success of Black students. We focus on data from one elementary school, two middle schools, and one high school (see Table 1 for demographics at time of data collection). These sites were selected from the data corpus because we were able to gain the greatest degree of prolonged access to these schools and the educators and classrooms within them, as well as to stakeholders, like parents. This access afforded a more comprehensive data set compared to other sites. Each focal school was in a highly underserved area, and during data collection school community members referred to what was happening outside of each school as a way to talk about what needed to happen inside of it. These comments demonstrated a level of sociopolitical awareness emblematic of these sites that we discuss later. After preliminary analysis demonstrated some consistent characteristics across levels of schooling, we chose to foreground an elementary school, middle school, and a high school to show how Black educators' conceptualizations and mechanisms of protection index an awareness about the needs of Black children from elementary to high school.

Our case is delimited at the level of the racialized notion of protection, as it is produced by Black teachers in this study and can be traced across those teachers and across the focal sites ( Bartlett & Vavrus, 2018). The cases presented here illustrate how the concept of protection is an underexamined aspect of many Black educators' politicized caring approach. In analyzing Black educators' protective consciousness and acts, we highlight the multifaceted nature of politicized caring conceptually and empirically. Our analysis thus focuses on the following questions: (1) How do Black educators articulate the need to protect Black students? and (2) What are the mechanisms of protection these educators enact in their classrooms and in schools?

Data Sources

Data for this study consist of observations of classrooms and semistructured interviews with Black educators at the focal schools, which are supplemented with limited anecdotal evidence from interviews with Black parents and Black students (see Table 2).

Data Sources From Focal Schools

Observational Data

The observational data were collected from Spring 2014 to Spring 2015 at the four focal school sites. Researchers took ethnographic fieldnotes in classrooms and common areas, such as hallways, school cafeteria, recreational spaces inside and outside of the schools, and the main office. These observations describe the school contexts, learning settings, key participants, and in the case of classrooms, the structure and content of the class being observed ( DeWalt & DeWalt, 2011; Emerson et al., 1995; see Supplemental Appendix A, available in the online version of the journal). Researchers paid particular attention to pedagogical practices, such as the discourse or interactions related to student positioning and identity, discipline practices, teacher-student and student-student relationships, equity-focused practices, and quality of the academic content being taught (i.e., level of cognitive demand on students). Due to variation in school schedules and access, different numbers of observations were conducted across sites. Press articles, the district website, and each school's website were used to understand the history of each school and its surrounding areas.

Interviews

Semistructured interviews were conducted with any parents, students, and educators at the focal schools who signed consent forms and were available for interviews. Parents were recruited via letters home and snowball sampling, while students were recruited in their classes, via recommendation by educators and parents, and snowball sampling. Although snowball sampling can identify potential participants that meet selection criteria and provide in-depth understandings, this method can produce bias by limiting the diverse perspectives generated by the resulting nonrandom sample ( Penrod et al., 2003; Ritchie et al., 2014; Valerio et al., 2016). We tried to reduce bias by using other recruitment strategies (e.g., letters home to families), using references as links to identify potential participants, and documenting the referral chain to keep track of potential participants ( Atkinson & Flint, 2001; Penrod et al., 2003; Ritchie et al., 2014).

To better understand students' and parents' experiences and perspectives on the school's relative success, researchers conducted semistructured interviews that each lasted 45 to 60 minutes. A semistructured interview format allowed researchers to reorder questions based on participant responses and probe participants for clarity and/or elaboration ( Åstedt-Kurki & Heikkinen, 1994; Bogdan & Biklen, 2007; Dearnley, 2005; Krauss et al., 2009; see Supplemental Appendices B-F, available in the online version of the journal). Individual interviews were conducted with students in Grades 6 to 12, and due to scheduling constraints a few focus groups were conducted. To facilitate participation and elicit young children's experiences, researchers conducted focus groups with elementary students during the school day ( Morgan et al., 2002; Ritchie et al., 2014).

Educators were recruited via (1) community nominations ( Foster, 1991; Ladson-Billings, 1995); (2) staff, grade level, or departmental meetings; and (3) recommendations from school administrators and other teachers. Since the larger study focused on schools where Black students were experiencing success academically and/or where there was a positive social and relational climate for Black students, community members often directed our team toward educators who met those similar criteria or supported the school's success. This included administrators, counselors, coaches, instructional aides, teachers, security officers, and after-school staff. In this article, we focus on the voices of Black teachers and administrators who were interviewed at focal schools. Interviews were conducted individually, or in pairs when requested by teachers, lasting 45 to 60 minutes each.

Analysis Process

Our analysis began with open coding the data sources to document Black educators' ideas about and acts of protection as they articulated or enacted them. This involved each individual research team member reading through interview transcripts and fieldnotes and identifying broad thematic areas. In discussing our respective themes, we discussed points of overlap and divergence in order to refine recurring themes into specific codes and subcodes. Codes were developed with the intent of reflecting the voices and sentiments of the educators in the study ( Creswell & Poth, 2018; Miles et al., 2014; Saldaña, 2016; Saldaña & Omasta, 2018). The open coding of fieldnotes resulted in codes such as teacher-student communication and/or interactions, student-student communication and/or interactions, opportunities for engaging in discussion, academic rigor, attending to students' everyday and lived experiences, or use of strategies to engage and support various learners. Likewise, from the interview data the codes that we developed included race/racialization, discipline, care, teacher-student relationships, and protection/safety. Through a process of refinement, these codes were discussed and analyzed for presence and prevalence within and across data sources and focal sites to capture similarities and differences as well as to hone the definition and scope of codes and subcodes themselves. Looking across the focal sites and sources of data (interviews and observations) led to deeper analysis into broad patterns and context or participant specific codes. Once codes were mutually agreed upon by the entire research team and our application of codes was calibrated, everyone engaged in coding the data corpus using the Dedoose® software. Simultaneously, subgroups of researchers (assigned to focal schools) explored the themes and patterns at each school. Within each subgroup, three to five researchers coded each set of fieldnotes, wrote analytic memos, and discussed codes prior to research group meetings ( Maxwell, 2013; Saldaña & Omasta, 2018). This iterative and interactive process of refinement was accompanied by our going between the data and conceptual frameworks that could guide our analysis and capture the complexity and richness of the educators and school environments involved in the study ( DeWalt & DeWalt, 2011; Maxwell, 2013).

Across data sources, codes for protection and safe/safety often co-occurred with notions of caring, prompting further analysis into how these ideas were expressed within classrooms and in interviews with educators, parents, and classroom teachers. This analysis brought to the fore that during interviews Black educators across the focal sites nearly always linked notions of caring with protection and that they perceived racialized harm as a systemic issue that was manifested via acts of physical harm (e.g., police brutality) and symbolic harm (e.g., the reproduction of stereotypes). Interviews and fieldnotes were then reanalyzed to confirm the co-occurrence of caring and protection and establish if and how Black educators discussed or evoked these notions in their interviews discursively or pedagogically in their classrooms. To establish validity of our codes and preliminary analyses, member checks were conducted through presenting tentative findings to members of each school community. These initial analyses were further elaborated through additional data collection and engagement with the focal sites (e.g., volunteering and working at the schools, fostering school-university relationships) and ongoing discussions about the data ( Creswell & Poth, 2018).

In this study we privilege the voices of Black educators in order to understand their conceptualizations of protection and the mechanisms of protection they enact. Our analyses here are thus based on the content and qualities of their articulations of protection in interviews and as documented through observation rather than on static frequencies of individual codes (e.g., teacher-student relationships, protection, positioning and identity, discipline practices) that cannot be understood outside of their context. Attempts at quantifying this type of qualitative phenomenon can obfuscate and overshadow the importance of understanding how phenomenon were communicated, experienced, or made sense of by participants. Quantifying qualitative experiences runs the risk of legitimizing frequency over situational significance. For example, it may matter differently or perhaps less if a Black teacher constantly repositions Black children as deserving of protection during a typical classroom lesson (thus resulting in a high frequency of acts of protection) but did not intervene on one of those children's behalf when an issue arose between that child and another educator who made the child feel unsafe or wrongfully treated (a low-frequency occurrence of arguably important significance). In the next section we discuss the findings that came out of our situated analyses of Black educators' articulations and actions.

Findings

This section is separated into two parts: The first analyzes interview data to show how present-day Black educators conceptualized protection as a central concern, how they articulated the forms of racialized harm from which Black children need protection, and how they identified their Black students as worthy of protection. The second section focuses on how these educators used or created institutional mechanisms of protection inside of their schools. We demonstrate, through analyzing classroom observations and interviews, how Black educators consciously recognize Black students as vulnerable and in need of protection in two ways: institutionally, through the structures they create, and interpersonally, through the ways they interact with them.

Conceptualizing Protection: An Awareness of Racialized Harm

Mr. Coles was a science teacher turned administrator at North Pineville Middle School (NPMS). During his interview, he responded to a question about how race was talked about at NPMS by pointing out the race-consciousness there and the need for it in schools:

You know, its [race] just embedded in the system [at NPMS]. . . . It's in everything that we do. From the way we walk, to how we talk, it, it just is. I don't know how to explain it. It [race] has to be recognized. For a teacher to say "I see everyone the same." . . . you mean . . . you don't see color? . . . that's a problem cause everyone's not the same.

Mr. Coles stressed the importance of a race consciousness by educators and identified a color-evasive pedagogical approach (i.e., "don't see color) as a "problem" that needs to be addressed. Educators who take up color-evasive ideologies downplay or ignore the existence of race and see issues of race and racism as artifacts of the past ( Bonilla-Silva & Dietrich, 2011; Neville et al., 2005). In so doing, educators ignore students' racialized identities and their sociopolitical realities. He also articulated that an awareness about race is "in everything we do" and struggled to pinpoint its exact manifestations because it is so ingrained that "it just is." Implicit in Mr. Coles's argument is a racialized awareness that the work of teaching occurs within a racialized context and that students have distinct sociopolitical realities as racialized persons, which teachers need to pay attention to in learning environments. In this view, teachers who employ a color-evasive pedagogy could reinscribe the racialized harm students experience outside of schools inside their classrooms. Color-evasiveness, therefore, is not only an impossibility but also a form of racialized harm.

Other teachers made similar connections to how it is problematic for teachers to ignore race and racism as they discussed their own racial positionality and subjectivity. Mr. Saunders, a middle school teacher at Molly Williams Academy (MWA), offered the following:

I always identify myself as a Black man, and I feel like because I do that, I feel like kids internalize that as well and see themselves as Black. I've also had instructors tell me that . . . "Oh, well we feel like, you know, kind of like the Martin Luther King philosophy like, you know, people are not judging me [the student] by the color of my skin, but more the content of my character." You know, I would like to think . . . like that, [it] would be great if that was the case, but that's not the case. I try to be as realistic as possible with my kids. Like you should be judged by the content of your character and you shouldn't just have to look at yourself like a Black kid, but you know, society has another view.

Mr. Saunders highlighted color-evasiveness as missed opportunities for educators to preemptively acknowledge their subjectivity with their students and racially socialize Black students into what it means to be Black in the United States. He suggests that such perspectives are inappropriate in a society that "has another view" (i.e., one that essentialized Black children) and acknowledges the prevalence of deficit views of Black students and their families across schooling environments and the sociopolitical climate ( Bonilla-Silva & Dietrich, 2011; Kendi, 2016; Souto-Manning, 2019).

Earlier in his interview, Mr. Saunders articulated that both color-evasive perspectives and racist "views" have symbolic and physical consequences for contemporary Black children and that different forms of racialized harms are interrelated. After discussing the killing of Michael Brown7-a physical harm-Mr. Saunders goes on to talk about stereotypes within schools-a symbolic harm. He said, "Race and ethnicity . . . play out in the classroom . . . [and] a teacher's bias can gravely affect a student's achievement." In connecting Michael Brown's murder with teacher biases, Mr. Saunders suggested that a teacher's failure to acknowledge race and racism ignores the reality that Black children may be subjected to physical harm at any time. Likewise, teachers' failure to acknowledge the racialized realities and forms of oppression their students experience can reproduce symbolic harm through unchecked biases or stereotypes. Like Mr. Coles, Mr. Saunders alludes to the racialized harm Black children endure in today's society and the potential color-evasive approaches have to further inflict harm.

Mr. Franklin, a teacher at Bay Prep High School, argues that recognizing that Black children may experience racialized harm is not enough-educators must also make Black students aware of this harm. Relaying a conversation he had with a Black male student about an incident in school, Mr. Franklin warned,

"What would've happened if I wasn't the one that caught you [the student], if a cop caught you? . . . they're looking for people like you. It's open season on Black and Brown men, you know this. So if you had gotten caught . . . then it would've turned into a whole other thing, right?" So we kinda talked about that and he started coming to class.

Mr. Franklin underscores that his understanding of racialized harm was based upon his awareness of the systemic and racialized conditions that influence the lives of Black children and how their actions are perceived. His comments reflect a developmental awareness about the need to racially socialize Black children into understanding how their actions will be perceived and the potential consequences of those actions, even if he does not endorse those consequences. His recount of this conversation with his student reflects how he conceptualized the physical forms of racialized harm this student could not afford to ignore. It also demonstrates how he discursively positioned minoritized male youth's need for physical and symbolic protection and his engagement in that work. In sharing how this student took up this race-conscious approach, he alluded to a potential link between race consciousness, a racially socializing approach, and students' willingness to engage in school.

Other teachers' interviews likewise reflect their understanding of Black students as children made vulnerable by the racialized conditions of a society by locating anti-Black racism as an indelible feature of the present, not a relic of the past. For instance, Ms. Bailey at NPMS noted how her school's leadership was keenly aware of the racialized harm being done to Black communities by the penal system. Ms. Bailey referenced assembly topics that highlighted important social realities, like the school-to-prison pipeline (see Díaz, 2019; M. W. Morris, 2016; Winn, 2011), and praised the school leadership's transparency in discussing these realities.

Ms. Bailey reflected on how systemically racialized conditions manifest within the lives of students and drew connections between racialized harm and her work as a teacher. She stated,

[Students are] dealing with [their] dad being locked up, and [their] mom is gone, and it's hard. And if someone is not prepared to deal with that and have empathy for students who have to deal with that, you can't teach in this kind of environment. So, the kids, my kids, come with so much trauma. Like for the second marking period [fall to winter], I was mentally drained. I would cry a lot, just having to call home and hear what students had to deal with at home, and then figuring out why him or her are short tempered or acting this way [ sic]. You never know what these kids are coming with. It's hard.

Ms. Bailey's comments indicate her awareness of the conditions in which her predominately Black students lived and learned, as well as the trauma that may have been associated with these realities. Acknowledging the potential traumatization, Ms. Bailey offered a counternarrative (i.e., that there is nothing "wrong" with students themselves) and instead located the challenges in the situations students may have faced in their neighborhoods or homes. She also highlighted the potential impact of these situations on their engagement in school. This quote demonstrates Ms. Bailey's care for her students, without students' experiences or behaviors diminishing their potential in her eyes. She discussed why it is important to consider what is going on outside of school for her students, and how she used that information to "figure out" why they may be feeling or acting a certain way. Like many teachers in our study, Ms. Bailey's care for her students is expressed by viewing them as worthy and in need of protection, emphasizing the need to be aware of students' racialized realities, and advocating for educators to be intentional in addressing these realities in their work.

An administrator at Bay Prep High School similarly reflected upon what it means to be conscious about Black students' lives. When asked what Bay Prep does well in relation to the Black students at her school, she responded,

Just being able to kinda understand where, what our kids come from, and what they need without being judgmental, or looking at them with pity. . . .They don't want people to feel sorry for them. . . . And [we] just kinda give them the tools to deal with it.

In calling for educators to understand students' realities outside of schools, she cautioned that "understanding" is not "being judgmental" or viewing students "with pity." Like Ms. Bailey, this administrator evidenced a racialized awareness and care for who their students were and the racialized harm or trauma they experienced. They refuted societal critiques of Black children or families not valuing education or as being immoral, pathological, or intellectually inadequate. Rather, these educators analyzed the societal conditions and challenges their students grappled with outside of school and reflected on how to protect them from that racialized harm within schools without regarding the individual student with pity or contempt.

Teachers also argued that the lack of cultural alignment in schools perpetuated racialized harm. For example, when asked to identify "some of the primary challenges that Black students face at this school," Mr. Saunders at MWA stated,

The basic challenges that Black students experience at every single school across the nation, the assimilation of having to curtail your own feelings and your own beliefs and your own way of life and kind of catering it toward a structure that isn't necessarily befitting of the world you come from, you know? I feel like African American families have a different way of being able to show love and different ways of being able to show respect to each other that may not necessarily fit into a classroom. I feel like specifically when you're dealing with the Caucasian teachers, who may have a very traditional way of teaching, even if it does include speaking and being able to pair share, or getting up and being able to move around, there [are] still certain structures within it that doesn't necessarily fit with some African American boys and girls.

Mr. Saunders expressed that most schools do not share or value certain cultural norms and practices that may be common in Black students' lives. This cultural mismatch may require some Black students to "curtail" or suppress their feelings and "way of life" in classrooms, resulting in limited opportunities at school to be who they are-asymbolic form of racialized harm.

In sum, Black educators articulated an awareness of the systemic, racialized structures and conditions, as well as the interpersonal interactions in and out of schools that facilitate physical and symbolic racialized harm for Black students. They used these understandings to guide their work in schools and disrupt how Black children have been cast as damaged and deficient.

Creating Protective Structures in Schools

Through analyzing how Black educators conceptualized the need to protect their students it became clear that they enacted that protection through pedagogical and school-level structures. Making students feel "safe,""protected,""cared for," and "at home" emerged consistently as a key frame from educator interviews and discussions across sites. Below we examine how the pedagogical and institutional mechanisms of protection they created positioned students as worthy and in need of protection. We also explore how those structures fostered symbolically "safe" and protective school environments for Black students, and employed alternative disciplinary practices and positive positioning of students' identities.

Creating a Symbolically Protective Environment

The race consciousness of the Black educators in this study influenced how they attended to their Black students' socioemotional needs and sought to protect their well-being. Describing what it is like to be a Black student at Porter Elementary School, one administrator stated,

I think they find this place to be home. . . . I feel like they're very comfortable here at Porter. First, it's in their community, second, they see a lot people who look like them . . . I feel like all of them feel like, "Hey, I can come here and people care about me." . . . I feel like the experience here is like you're comin' home, this is this my Porter family [emphasis added], this is my Porter home [emphasis added].

In presenting Porter as a familial environment for Black students, this administrator highlighted three symbolically protective and racialized features of Porter that can go unnoticed without careful attention to how race and racism operate in racialized harm. First, because the school is in their community, students likely had a degree of awareness about how to manage their physical location and the surrounding environment. Students may not have had this awareness in other schools or neighborhoods they were less familiar with, where they knew fewer people, or where they were less likely to be known within the surrounding community. As Trayvon Martin's murder tragically evidences, Black children do not always have the privilege to not be known in a community or to operate in communities where the majority of the community do not "look like them."

Second, the racial demographics of the school served as a symbolically protective feature where students saw "people who look like them" and may not have experienced the racialized harm of isolation, stereotyping, being a "representative" of the race, or being racially profiled ( Haddix, 2012). Third, the administrator argued that students perceived that "people care about" them at Porter, a contrast with the lack of care Black students often experience in schools ( Irvine & Fraser, 1998; McKinney de Royston et al., 2017). Later, the administrator indicated the importance of feeling comfortable (i.e., "at home") while at school when he shared that former students, now middle and high schoolers, came back to visit and lamented that they had to leave Porter. His statements suggest the presence of symbolically protective features at Porter.

Ms. Bailey at NPMS described how she viewed her Black students as vulnerable and recognized their need for care and protection, and how that manifested in her pedagogy:

I feel like there are some really good things in this space, and [students] really try to live up to the expectations that we have of them. . . . I feel like when I am in front of them, my hope and desire is for them to know that they are cared for. And I want them to know I will not accept any excuses. Yes, I understand what happens to you outside these walls is hard, but what I want to give you is to move you so you can better yourself.

Ms. Bailey's comments reflect her awareness about her Black students' lived realities and adversities outside of school, and how this awareness neither altered how she viewed the academic potential of her students nor decreased the high expectations she held for them. Ms. Bailey argued that her pedagogical stance involved symbolically protecting students from potential harm they encountered outside of schools and from the racialized harm of lowering expectations for students' achievement and learning. In doing this, she identified her students as vulnerable children worthy of protection alongside pedagogically identifying them as students who are capable of academic success.

In the above quotes, Black educators demonstrated the affective characteristics of culturally relevant warm demanders ( Irvine & Fraser, 1998) or educators "who provide a tough-minded, no-nonsense, structured, and disciplined classroom environment for kids whom society has psychologically and physically abandoned" ( Irvine & Fraser, 1998, p. 56). Moreover, their remarks reflect the collective racialized consciousness many Black educators have toward the systemic conditions of their students' lives and the kinds of racialized harm they encounter or are likely to encounter if not protected. Like Ms. Baily above, across interviews educators articulated how they utilized their interactions with Black students to symbolically nurture and protect them from the racialized harm and bigotry of societal conditions and related experiences in schools.

Creating "Safe" School Environments

Harkening back to the comments by the administrator at Porter who suggests that familial school environments can be symbolically protective for Black students, Black students and educators remarked that such relationality and welcoming climate at a school can provide Black students with a physical sense of safety. One middle school student at NPMS described how her sense of safety was linked to her school's climate and community. She shared,

I like this kinda family relationship that we have [at the school] . . . we all know each other. We all somewhat get along together. We all know about each other. And it's like family here and you feel safe. You don't have to worry about fighting. It's different from my last school.

This student contended that for her the familial type of relationship in the school helped her feel safe and known, and that she knew everyone else as well. She also suggested that this family relationship at her current school created a kind of physical safety for her that, unlike at her past school, meant that she did not have to "worry about fighting" on a day-to-day basis. The importance of this sense of safety, especially from potentially violent situations like fights within school, cannot be understated for any student, much less for those students who may have been or are currently exposed to violence at school or elsewhere in their lives.

Due to targeted housing policies that create and sustain economic and racial segregation in Black neighborhoods across the country and specifically within Oakland ( Rothstein, 2017), the neighborhoods surrounding the focal school sites, particularly the elementary and middle schools, experienced high rates of poverty and violence. Students were aware of these conditions and spoke about how these conditions made them feel, including how they affected their feelings about physical safety when they were at their schools. In a focus group at Porter, young students contrasted their experience with violence in the neighborhoods around their school with the environment inside the school. One second grader shared,

I don't feel safe outside because there's a lot of shooting . . . inside I feel safe because it's a big building and we can go anywhere. We could run up the stairs because there's a lot of space and we can go all the way over there [to hide] if it's [people shooting] over here.

Other students expressed their fears about walking home or playing outside because of gun violence in the neighborhood around the school. Students, their parents, and the educators at the focal schools were all keenly aware of the violence in the neighborhoods around their schools and had concerns about students' physical safety based on a school's location. Without criminalizing or pathologizing communities or people, the Black educators in this study explained the daily realities for some of their students and how they as educators sought to offer Black students a sense of physical and symbolic safety and a departure from the harm they may encounter given the by-products of systemic racialized inequalities such as poverty and violence.

The focal educators argued that one way to offer Black students a physical and symbolic sense of safety involved maintaining a clean and "orderly" school environment that would serve as an "oasis" or safety net for students and avoided reproducing-visually or otherwise-the racialized harm that some may experience because of systemic poverty and violence in the surrounding neighborhood.8 Black educators suggested that aesthetic and environmental features, such as a school building and grounds being clean and orderly, would allow students to feel cared for and physically safe once at school in ways that might reduce the children's stress levels and the racialized harm they experienced by living in underresourced communities. Expressing this view, one administrator at NPMS argued,

If the kids come to a place that's clean, that they feel like, "Alright I like this now 'cuz it's safe" . . . then it gives them a sense to come, but it also makes them feel like they can relax . . . 'cause I don't know where they're coming from, we don't know the hassle. . . . If at least they can come and breathe deep and be at peace for a good six to eight hours, then we're helping them out. . . . To me that gives an umbrella, a bubble of security around the kids. It gives them a place where they can take out some of the nonsense, some of the things that stress their life and more focus [ sic] on getting better academically.

Like other educators across our sample, this administrator affirmed that a well-maintained school can serve as a form of care and protection because it allowed students to feel cared for and to "be at peace." While at school, students could feel a "bubble of security" around them where concerns about their physical safety were lessened and allowed them to instead focus on being children and on their academics. This positioning of Black children as in need of a place of "peace" and that honors Black childhood recognizes them as inherently capable and willing to engage in school and in learning but potentially in need of a protective space in which they can experience psychological safety and "relax" enough to focus on their academics. This framing provides a stark counternarrative to discourses about Black students and their families as unable, unwilling, or disinterested in engaging or succeeding in school to instead acknowledge the systemic realities they may encounter inside and outside of school.

The sense of physical and symbolic safety discussed in this section does not mean that Black educators could and can always protect Black children from physical and related forms of harm outside of schools, although there were incidences when this did occur. However, going to a place where Black children felt physically safe and likewise experienced less psychological and emotional trauma or harm was itself a form of protection from the racialized harm inflicted by systemic violence and poverty.

Unconventional Structures to Meet Students' Needs

In talking about developing Black children's sense of physical and symbolic safety, the focal Black educators in this study articulated the importance of Black students feeling not only protected but also deserving and worthy of protection. As discussed previously, the Black educators in our study attended to their Black students in this way interpersonally and through how they maintained the school environment. Black educators also protected Black children by employing nonconventional structures or programs that affected students' physical, socioemotional, and academic well-being. The data suggest that educators viewed students' physical, socioemotional, and academic well-being as tightly linked together, and they sought to support students in all of these domains. For example, at MWA, a food pantry regularly distributed food to students and families. MWA, like other schools in our study, had a health clinic with full-time staff for students and their families, offering dental, vision, mental health, and medical services as part of its full-service community school (FSCS) model9 (see McKinney de Royston & Madkins, 2019, for a review of FSCSs in our study) that operated in collaboration with other service-oriented agencies.

Another way that educators acknowledged students as in being need of protection and tried to support their feeling worthy of being protected was through after-school and extracurricular programming designed to meet students' needs. Most of our focal schools offered elective courses or after-school programs focused on students' racialized and gendered experiences, such as the Manhood Development Program (MDP). MDP was initially developed for African American male students in Grades 6 to 12 and was later extended into several elementary schools. MDP has been effective for supporting African American male students' academic performance, engagement, and navigation of schools.

The MDP program sought to position students as worthy and in need of protection and as academically capable. This also occurred at schools that did not have MDP, albeit through other means of bolstering student academic success. For instance, at MWA one such structure was the large whiteboard in the front office that listed the homework assignments each teacher assigned for the day. This homework board informed students and families about the day's assignments and kept them in the loop if they were absent or missed a class period. Rather than penalizing students and families for missing something, MWA created a structure that facilitated students' and families' ongoing awareness and engagement with the academic expectations of individual teachers and classes. In addition to long-standing school offerings (e.g., tutorial support, arts and crafts), MWA hosted an after-school program, The Genius Hour, where students engaged in learning tasks based on their interests and were continually positioned as experts and geniuses.

At NPMS, another academically focused protective structure was the annual science fair. The entire student body was required to participate in the fair, and the science and engineering teachers focused a considerable amount of instructional time on preparing students to develop the content of their presentation and presentation skills. In class periods leading up to the science fair, students shared their ideas, rehearsed their science fair presentations in small groups, and participated in mock question-and-answer sessions. During this time, students received targeted feedback from their instructors and peers verbally and in written form. By allowing students to engage in mock science fair presentations and offering feedback, teachers at NPMS helped students improve their presentations, facilitated the development of students' academic identities through positive positioning of them as scientists and constructors of knowledge, and modeled how scientists share their findings (Madkins & McKinney de Royston, 2019).

The science fair at NPMS was more than an academic event. NPMS educators specifically designed it to attend to the multiple needs of their Black students and families. To encourage families to attend, it was held in the evening and began with a free, buffet-style dinner. The list of invited judges-almost all of whom identified as Black-included individuals from local STEM-focused nonprofit organizations, community members, and a member of our research team (the second author) to provide students with role models from minoritized communities ( Felder Sumter, 2016). Students were required to have at least one adult in their lives (e.g., family member, after-school personnel, neighbor) serve as a judge for their projects; this score was factored into the student's final science fair project grade. The science fair included performances by the school band, science activities for young children, and a step show (a form of African American dance; see Malone, 1996) from the feeder elementary schools, the NPMS step team, and members of local chapters of traditionally Black fraternities and sororities. Representatives from the NPMS Family Resource Center, medical and mental health services providers, and other community-based organizations had vendor tables offering families information about free, local services; the school's food pantry hours and location; and summer programs. Although it is increasingly common for school personnel or science fair organizers to invite community members and professionals to serve as science fair judges ( Austin Regional Science Festival, 2019; Hunt, 2016), the NPMS format was atypical of most school science fairs because it was about recognizing students and their families as willing and capable partners in the intellectual life of the school and offering supports for their physical and mental well-being. These school-level structures served as a source of protection for Black students.

Alternative Disciplinary Views and Practices as Protection

In the previous section, we explored interpersonal-level supports and institutional-level structures and supports that school sites implemented to simultaneously frame Black students as worthy of protection and as academically capable. We now explore how these same schools reconceptualized discipline and enacted alternative disciplinary practices in an effort to protect their students from racialized harm. An extensive body of literature has captured how Black students continue to be disproportionately and unfairly disciplined in comparison to white and Asian students ( Ferguson, 2000; Gregory et al., 2010; E. W. Morris, 2007; Wallace et al., 2008). Given their understandings of these students' plight, many Black teachers were insistent upon not viewing their Black students as in need of discipline; instead, they often utilized other strategies when handling student behavioral issues.

Ms. Johnson, a teacher at Bay Prep High School, reflected this alternative view and enactment of discipline. She stated,

When a student gets in trouble, it's a different type of conversation, it's not, "We're so disappointed, I hate that I have to suspend you but . . .," It's like, "What are you thinking? This hurts me when I have to do this. It hurt me to see what you did." . . . and for them to hear someone say-because that's kind of a positive-negative-"I'm affected by what you did, even though it was negative. I believe in you" . . . It still screams, " I believe in you." It makes a difference [emphasis added].

Here, Ms. Johnson acknowledges students' racialized realities and illustrates a perspective that attends to Black students as in need of protection, not discipline. Saying, "I believe in you," to a student reflects Ms. Johnson's attempts to respond to the student's actions and acknowledge the student's humanity as not being limited to those actions.

Parents also discussed the importance of disciplinary practices and how alternative discipline practice reflected educators' conscious protection of Black students. Michelle, a parent who had children enrolled at NPMS during two different administrations, offered a comparative perspective. When asked to compare her "current experiences as a parent" to her experiences when her older daughter attended NPMS, Michelle stated the following:

I did like the old principal; she did seem to care and make changes. . . . but the dynamic of the school totally changed when they got Mr. Simpson and those guys up in here. They start caring more for children, and when I say carin', I think the other principal cared, but she did not have the staff to help her care [emphasis added]. They were interested in suspending them and getting the bad person out of here instead of seeing what's going on with this kid. I think that Mr. Coles and Mr. Simpson look more into the person and figure out if they have a program that can help a child succeed. They want everyone to succeed.

Michelle described a new trend in the current school culture with respect to discipline: where Black children were viewed as in need of protection from a disciplinary system. Mr. Simpson and Mr. Coles, the new administrators, did not suspend or give punitive treatments to their Black students as an immediate response. Instead, they acknowledged the student's humanity and attempted to identify additional resources and services the student might need. Michelle argued that under the new administration, no student was expendable and that the school offered programs attending to students' needs and academic success. And while both administrations cared for students, she argued that Mr. Simpson and Mr. Coles had a team of educators who developed and implemented structures grounded in care, and aimed at protecting students.

An administrator at MWA, Dr. Thomas, embodied how Black educators troubled the practices around discipline in caring and protective ways. She could easily be read as a staunch disciplinarian given her no-nonsense demeanor and firm tone, yet students provided a more balanced perspective of her. They shared that Dr. Thomas was a "she don't play that" type of principal but at the same time referred to her as a "softy" in a focus group interview. This sentiment especially came through when the group of students discussed how Dr. Thomas interacted with them in her office after they had gotten into trouble. When going to the principal's office, the students found they were not met by the rigid disciplinarian they expected but instead encountered a caring and protective educator who valued students' perspectives and employed disciplinary restraint. According to students, Dr. Thomas resisted relying on the dominant disciplinary practices imposed upon Black students (e.g., multiple days of in- or out-of-school suspension) and took intentional measures to address student behavior issues in ways that allowed them to still feel valued as students. During a focus group, three of seven Black male middle school students interviewed shared the following,

Yeah, she'll be like, "Stop running in the hallways!" or if she knows your name, she'll say your name till you stop running. And if you're wandering around the hallways she'll tell you to go to class.

She can be loving too cause I was in the office last year getting suspended and I went to her office and she wasn't as hard as you'd think. Like she wasn't real, real, real tough. She a softy a little bit. And she explained what she was going to do, why she was going to do it and then I was supposed to get a five-day suspension but I only got one [day]. But she give you [a chance] though [ sic].

I would like to add . . . she can be [soft] sometimes . . . it depends on what side you get her on . . . when she call your name on the intercom (sic), like when I got in trouble, and I went in there . . . she gives you a chance. Like just before you get sent home, she ask you multiple times [emphasis added], "Do you have a justification of why you did this?" Multiple times [emphasis added]-and if you have one she let up [ sic].

Dr. Thomas, like many of the other teachers in this study, resisted traditional disciplinary responses. While the principal's office can often be read as the punishing room-particularly for Black students-Dr. Thomas made it a point to use that space as a place to lighten up her typical demeanor as a warm demander ( Irvine & Fraser, 1998) that characterized her interactions with students on the school grounds. She allowed her office and one-on-one encounters with students in challenging situations to be a humanizing experience as opposed to a punitive one. The acts of protection these Black educators enacted were reflective of an awareness of not only the unfair disciplinary practices surrounding Black children in schools but also the empathy they have for their students because of the racialized harm students encounter outside of schools.

Similarly, two parents reflected on how Black educators at Porter Elementary School employed care and protection during the disciplinary encounters they observed between students they referred to as "bad, really bad"10 and administrators. The parents, Rochelle and Stephanie, shared that the level of patience and care these educators demonstrated with students was reminiscent of what one experiences from a parent. When to describe teachers and administrators' relationships with students, Rochelle and Stephanie said,

I think just caring. Like, even just Ms. Mustafa, even sometimes with the bad, really bad students where we're just like, "Whoa!" . . . you'd never know in the way that she treated them that this was like a really, you know, troubled kid 'cause she speaks to them like she does everyone else. She's still kind to them, they still participate.

Gives them chances and stuff and like all of that.

So, she's just kind to them.

Kind and caring, okay.

I was gonna say that I love how they say, "These are my babies."

I love when they say that cause it's like he [an administrator] really loves these kids.

These parents took comfort in the teachers identifying with the students as if they were their own children. The school leaders and teachers were invested in the students, and when disciplinary infractions occurred, they handled it in ways that did not compromise the students' dignity. The educators in our study viewed Black students as in need of forms of protection that did not negate the real issues that may be affecting their behavior, academic performance, and emotions but held them accountable to high expectations and to having a degree of personal agency.

Positive Positioning as a Form of Protection

Across our classroom observations, we documented how many Black educators, through their interactions with Black students, were intentional about wielding care as a form of protection. In the following excerpt from an observation in a sixth-grade classroom at MWA, Ms. Marshall interacts with one of her students, Chicha, who was having some behavioral challenges. Ms. Marshall engages with Chicha in an affirming manner and demands her engagement in class, avoiding positioning her as a problem student or viewing her with a deficit orientation:

Fifth period starts at 1:30 p.m. When the students come in they begin Sustained Silent Reading11 (SSR). While most of the students are at their desks reading, others have gone over to the bookshelf to get reading material. One little girl lingers. She's a small, dark-brown skinned girl with one big puffy ponytail on top. I noticed this student following Ms. Marshall around on another day as she did rounds to the other middle school classrooms. It appeared as though this young girl had gotten in trouble and Ms. Marshall made her shadow her until she got herself together to go back to class. The dynamic between the two of them did not seem tense even though the student seemed upset, but her anger didn't seem to be directly at the teacher, whom she was trailing behind. Ms. Marshall says to her, "Come on baby girl, pick a book so you can get to reading." The student says, "Okay," unaffected by Ms. Marshall's comment, and keeps looking through the books. Ms. Marshall then says, "Chicha, don't pick up another book." Chicha can tell Ms. Marshall meant business, because she takes the book she currently had in her hand and made her way over to her seat. Ms. Marshall smiles at her as she skips over to her seat, tracing her with her eyes, to let her know she's watching.

All the students are reading. Some are sitting or lying on the floor. It's pretty quiet and you can hear pages turning, some students shuffling their feet or readjusting their chairs. Occasionally students will talk to one another at a whisper, but it isn't long before they're met with a look of warning or their names being called just above a whisper. Ms. Marshall seems to be relaxed at handling the class. When SSR is over students pull out their [Google] Chromebooks® and begin working on their narratives. As they're doing this, Ms. Marshall is pulling up different students' documents on the projector through Google Docs and checking their progress and making comments as she goes in and out of different students documents online. "Ok girl, I see you, ahead of the middle column," she said looking over at a young Latina student, noting her progress on her narrative. They exchange smiles, and some students giggle at Ms. Marshall's use of casual/hip language with their classmate.

As the students are working, Ms. Marshall instructs them to go back to the beginning of their narratives. Chicha is not paying attention, or at least she isn't making eye contact, "tracking the speaker," which Ms. Marshall requires of all the students. She affectionately, but firmly states, "Lil Chocolate . . . Lil Chocolate," to get her attention. Chicha then gets it, and begins watching Ms. Marshall. "Thank You," she says with a playful smile. . . . After reading through Chicha's story, Ms. Marshall comments on the details she gives in her opening and says, "I'm witchu boo!" Moving on to the next student she says, "Alright thanks, Chocolate."

She then goes to Eduardo, the student she referred to as Paw Paw earlier, and begins reading his narrative. "My dad, my brother, and I . . ." She then turns to the class with a smile, "Who are the characters?! Look at him, setting up the context." She then highlights another line, "The living room felt cold and dismal" . . . She comments, "Look at them words," as a means of noting his use of vivid language. Chicha is trying to get Ms. Marshall's attention. Ms. Marshall gives her a look of suspicion. "Ms. Marshall, come here, girl," she says with a smile on her face. Ms. Marshall goes over and the student asks her for help with something about her story (November 13, 2014; researcher's inferences italicized).

This excerpt illustrates several key aspects of Ms. Marshall's interactions with students: the affectionate naming of students (e.g., Chocolate, Paw Paw, boo), how she reframed encounters that could have become disciplinary moments in ways that demonstrate her high expectations but give students opportunities to meet those expectations, and the affirming of student success and effort. Ms. Marshall's discourse with students demonstrates an investment in their identities on a personal level that connects them to the instructional approach. The way in which students, especially Chicha, respond to Ms. Marshall's comments and seek her out for support demonstrate how students both positively assess and feel safe to participate in this relationship.

Ms. Johnson, at Bay Prep, voiced her perspective on her work as an educator:

And I could not do this if I didn't love them. I call them my sweeties, I actually use those terms. . . And because I think if you start using other terms, then you're gonna start thinking of them in those other maybe negative terms, and that's not who they are. . . . [My] thing to them is always, heads up. Heads up. I'm not saying to them, "Have a nice weekend!" That might come out as a second but my first one is heads up . . . because when they go out there, you know there's no protection [emphasis added] and I would tell them, "I'm a teacher, I'm gonna always protect you," to be able to endure some of the adversities you'll face outside of this classroom.

Ms. Johnson highlighted the need to call her high schoolers by terms of endearment, like "sweetie." She did so in order to recruit her Black students back into their childhoods, thus protecting students from (or undoing) the damage of negative stereotypes they face daily.

Discussion

This article examines how Black educators articulate and enact a need to protect Black students from racialized harm. Continuing the legacy of those that precede them (e.g., Dixson, 2011; Ladson-Billings, 2004; Walker, 2009a), many of today's Black educators see education as a vehicle for racial liberation and justice and their role as educators as inherently political because of the racialized discourses and practices that stretch into and beyond schools ( Ginwright, 2010; Roberts, 2010). These contemporary educators work alongside and on behalf of Black children because they recognize the racialized harm that happens in and out of schools as well as that these children may not manage these experiences-be they singular instances or repeated acts of subtle or overt racism-in ways that are privileged within schools.

As our analyses demonstrate, these Black educators' political clarity-that is, their enactment of a sociopolitical consciousness ( Beauboeuf-LaFontant, 1999; Freire, 1970/2005; Ladson-Billings, 1995)-about the racialized realities of Black children's lives reflects their awareness that Black children navigate racialized harm, that is, symbolic and physical violence of biases and stereotypes, within schools just as they do in U.S. society writ large. Our analyses focus on Black educators, in part, because within our sample it was these educators who articulated their political clarity and were the most prevalent protectors of Black students. The racialized harm they reference included institutionalized racism as well as interpersonal acts of racialized harm such as daily racial microaggressions (i.e., acts of racism that are no less harmful despite their lack of intentionality or visibility to the perpetuator ( Huber & Solórzano, 2015)). This includes some teachers holding a color-evasive lens that erases Black children's lived experiences to Black children being stereotyped or institutionally ignored or disregarded. A lack of political clarity about the racialized realities of Black children further exacerbates the complex trauma Black children may experience and the types of coping, oppositional strategies, or resistant attitudes and behavior they may adopt in response ( Dumas, 2014; E. W. Morris, 2007; Wun, 2014).

By contrast, the Black educators illustrated in this study held a nuanced and evolving clarity about how race and racism function egregiously and subtly, in society and in schools. They enacted this political clarity in many ways, from reframing students' actions as opportunities to engage rather than punish to calling out the ways that Black children are adultified ( Ferguson, 2000), criminalized, and inducted into the school-to-prison nexus ( Meiners, 2011; Stovall, 2016). Their constant reference to their students as "our babies" was a verbal declaration of their kindred investment in Black children's well-being and success, in Black childhood, and in a positioning of Black children as children and as worthy learners in need of protection. These educators repeatedly displayed a praxis that consistently reframed Black students as children who are made vulnerable by the racialized conditions of society, rather than by the child's moral, pathological, or intellectual inadequacies.

We argue that these Black educators' positioning of Black students as in need and worthy of protection is extraordinary. It is extraordinary not because these educators are superhuman, "magical," or "uniquely qualified" ( Baldridge, 2017; Bristol & Mentor, 2018, p. 231) but because the ways they think about and care for Black children's lives is uncommon, even in a society in which Black-child-life is visibly threatened. We do not suggest that all Black educators everywhere articulate a race-conscious awareness around protection or enact mechanisms of protection, nor that they are the only ones to do so. Caring for and protecting each and every child should necessarily be the work of every educator, particularly those aiming to be culturally and politically relevant. It also has to be done by non-Black teachers who are the majority of teachers in most schools ( Ingersoll et al., 2018). Likewise, we do not essentialize Black educators as monolithically benign toward Black students-sadly, like other educators they can and do perpetuate racialized harm against Black children.

This research does suggest that teaching, as an personal endeavor and a profession, requires a racialized and politicized awareness that must be cultivated in teacher education programs and as teachers grow in their practice as professionals. It is imperative that teacher educators support preservice and in-service teachers to continually develop, refine, and enact an asset-based political clarity about their students and the communities and families they represent. This can be fostered by teachers sharing and critically reflecting on their own and their students' experiences and their teaching practices in relation to their social and political views and interests ( Diemer & Li, 2011; Jackson & Knight-Manuel, 2019; Kirkland, 2014; Ngo et al., 2017; Varghese & Snyder, 2018; Watts & Hipolito-Delgado, 2015).

The voices of Black educators featured here also challenges us, as educational researchers and educators, to consider what the term safe means, and when and which spaces are considered safe and for whom? The Black educators in our study sought to create safe spaces in schools because they perceived there to be generally a lack of safety in the spaces Black children moved through. This assumes that Black students felt unsafe in these other spaces, and that schools have the potential to be constructed as spaces in which Black students could possibly feel safe. A lingering question is what current Black educators and others mean by "safe" or "humanizing" spaces and if their definitions and enactments align with those of Black children. Safe or humanizing for whom, in what way, toward what ends? Is safety simply the lack of dehumanizing environments or experiences or something else? What types of protection might Black children be interested in or what kinds of shifts in schooling are they asking for? If children answer these questions, are we willing and able-as a society, as adults, as educators-to create those kinds of educational institutions and realities? Further research can help us understand what Black children identify as safe and if this is a role schools can and should play.

A notable tension in our data warranting further discussion is how Black students' need for protection is discussed in relation to the surrounding neighborhoods of the schools. This is a complicated conversation because it is easy to locate violence and trauma within individual acts (i.e., a shooting, violence in the home) or persons rather than consider how these acts are systematically orchestrated by the aggressive and institutionalized neglect and disenfranchisement minoritized communities continue to experience ( Rothstein, 2017; Shapiro, 2017). Some of the Black educators in this study attended to this, though their discourse at times centered on the micro level of violence and harm. We must trouble this discourse, not as an indictment of those who call our attention to micro-level issues but as an acknowledgment of the varied systemic and structural forms of racialized harm Black children face. We must resist convenient analyses that pathologize Black children, parents, and communities yet fail to implicate structures designed to disenfranchise and dehumanize them.

At the heart of these questions is perhaps an even more fundamental one for educators and researchers: What should be the role of schools in contemporary society? How do we respond to the systemic vulnerability of Black children given the physical and psychological violence anti-Blackness cultivates in schools? Grappling with these questions requires us to consider whether how we organize schools, curricula, assessments, discipline, in-school services, and classrooms, meets our own epistemic criteria for education, pedagogy, and humanity. In an era when public education, as a human right and a social good, is deeply challenged, these questions are vital. Notably, our findings demonstrate that such questions are intrinsic to and nonnegotiable for the everyday consciousness and practices of many Black educators.

Notes

Supplemental material is available for this article in the online version of the journal.

1
Like many Black scholars, in this article we capitalize the term "Black" but not "white" when referring to individuals, groups of people, or communities. Dumas (2016) explains, "In my work, I have decided to capitalize Black when referencing Black people, organizations, and cultural products. Here, Black is understood as a self-determined name of a racialized social group that shares a specific set of histories, cultural processes, and imagined and performed kinships" (p. 12-13). Likewise, the term white "is not capitalized in my work because it is nothing but a social construct, and does not describe a group with a set of common experiences or kinship outside of acts of colonization and terror" (p. 12-13).

2
The term children is used throughout to avoid terms like youth that can obscure childhood. The term student is used only when referencing children in schools or when mirroring language used in cited text.

3
In this article, the term educator refers to all staff, instructional or otherwise, who support children within schools. The term teacher will be used to refer to instructional staff specifically.

4
Giroux & Penna (1979, p. 38) define the hidden curriculum as "those social processes in all sociopolitical institutions including the classroom which militate against the creation of a democratic, social education."

5
The Oakland Unified School District asked for the city and district to be named in the research conducted on its reform initiatives during the period studied. Doing so creates challenges to maintaining the confidentiality and anonymity of specific participants, namely, educators who may be well-known community members and long-time residents.We have thus omitted highly identifiable information about the educators such as exact age, number of years of teaching experience, as well as their personal histories.

6
A sampling method where the researcher obtains names of potential participants through direct contact with community members via organizations, individuals, churches, or periodicals. This method was coined by Foster (1991) and has been taken up in educational research, most notably by Ladson-Billings (1995).

7
Michael Brown, an unarmed, Black teenager was shot and killed by a white police officer in Ferguson, Missouri, on August 9, 2014 (see Kvetenadze, 2018; Powell, 2016).

8
This comment underscores the commitment to the safety of the physical environment but also potentially stereotypes the city and particular areas of the city in problematic ways.

9
FSCSs are increasingly popular in urban districts with the goal of better connecting local and school communities. FSCSs offer services on school campuses, like job training; mental health, medical, and dental services; food banks; and crisis intervention ( Oakes et al., 2017; Sanders et al., 2018).

10
Since the interviewer did not probe for clarity of what the parent meant by "bad, really bad," we cannot infer additional meaning.

11
During SSR, students read a book of their choice typically for 15 to 30 minutes, with the aim of improving students' reading comprehension levels and literature appreciation.

References

Austin Regional Science Festival . (2019). http://www.sciencefest.org/
Google Scholar

Beauboeuf-Lafontant, T. (1999). A movement against and beyond boundaries. Teachers College Record, 100(4), 702-723. |
Google Scholar Crossref | ISI

Dancy, T. E. (2014). The adultification of Black boys: What educational settings can learn from Trayvon Martin. In Fasching-Varner, K. J., Reynolds, R. E., Albert, K. A., Martin, L. L. (Eds.), Trayvon Martin, race, and American justice: Writing wrong (pp. 49-55). Springer. |
Google Scholar Crossref

Felder Sumter, T . (2016, February). Questioning the impact of role models. ASBMBToday. https://www.asbmb.org/asbmb-today/opinions/020116/questioning-the-impact-of-role-models
Google Scholar

Huber, L. P., Solórzano, D. G. (2015). Racial microaggressions: What they are, what they are not, and why they matter (Latino Policy and Issues Policy Brief No. 30). https://www.academia.edu/18296103/Racial_Microaggressions_What_They_Are_What_They_Are_Not_and_Why_They_Matter
Google Scholar

Hunt, G. (2016, February). Why be a science fair judge. ASBMBToday, 44. https://www.asbmb.org/Asbmb.Web/media/files/atoday/ASBMBToday-2016-02.pdf
Google Scholar

Lee, C. D. (1992). Profile of an independent Black institution: African-centered education at work. Journal of Negro Education, 61(2), 160-177. |
Google Scholar Crossref

Lomotey, K. (1992). Independent Black institutions: African-centered education models. Journal of Negro Education, 61(4), 455-462. |
Google Scholar Crossref

Meiners, E. R. (2011). Ending the school-to-prison pipeline/building abolition futures. The Urban Review, 43(4), Article 547. |
Google Scholar Crossref

Morris, J. E. (1999). A pillar of strength: An African American school's communal bonds with families and community since Brown. Urban Education, 33, 584-605. |
Google Scholar SAGE Journals | ISI

Noblit, G. W., Rogers, D. L., McCadden, B. M. (1995). In the meantime: The possibilities of caring. Phi Delta Kappan, 76(9), 680-685. |
Google Scholar ISI

Wun, C. (2014). The anti-Black order of No Child Left Behind: Using Lacanian psychoanalysis and critical race theory to examine NCLB. Educational Philosophy and Theory, 46(5), 462-474. https://doi.org/10.1080/00131857.2012.732011
Google Scholar

# data[].attributes.attachment_link.title
"I'm a Teacher, I'm Gonna Always Protect You": Understanding Black Educators' Protection of Black Children - Maxine McKinney de Royston, Tia C. Madkins, Jarvis R. Givens, Na'ilah Suad Nasir, 2020

# data[].attributes.attachment_link.description
Abstract Many Black educators in the United States demonstrate a political clarity about white supremacy and the racialized harm it cultivates in and out of schools. We highlight the perspectives of some of these educators and ask, (1) How do they articulate the need to protect Black children?

# data[].attributes.headline
10. Academic Tracking of English Learners With Disabilities in Middle School

# data[].attributes.subject
10. Academic Tracking of English Learners With Disabilities in Middle School

# data[].attributes.body
Paradigm: interpretivism.
Reason: This qualitative embedded case study provided an initial account of these students’ curricular access.

# data[].attributes.attachment_link.readable_text
Despite increased attention to the academic progress of English learners (ELs) with disabilities as a result of the Every Student Succeeds Act of 2015, research has yet to investigate the educational opportunities of these students in secondary grades. This qualitative embedded case study examined the curricular access of 10 ELs with disabilities in middle school. Utilizing deficit thinking for its theoretical underpinnings, the analysis illuminated that ELs with disabilities were consistently placed in lower academic tracks through a number of mutually reinforcing institutional and perceptual factors. The findings have exigent implications for expanding opportunities to learn of ELs with disabilities through reform to placement criteria and provision of special education and linguistic support across a range of academic tracks.

References

Collier, C. (2011). Seven steps to separating difference from disability. Corwin. https://doi.org/10.4135/9781452219424
Google Scholar

Crockett, J. B., Kauffman, J. M. (2013). The least restrictive environment: Its origins and interpretations in special education. Lawrence Erlbaum. https://doi.org/10.4324/9781410603722
Google Scholar

Every Student Succeeds Act, Pub. L. No. 114-95, 161 Stat. 1177 (2015).
Google Scholar

Faulkner, V. N., Crossland, C. L., Stiff, L. V. (2013). Predicting eighth-grade algebra students with individualized education programs. Exceptional Children, 79(3), 329-345.
Google Scholar

Federation for Children with Special Needs . (n.d.). Frequently asked questions about access to the general curriculum. https://fcsn.org/rtsc/wp-content/uploads/sites/2/2013/11/FAQs-Access-to-General-Curriculum.pdf
Google Scholar

Goodley, D. (2014). Dis/ability studies: Theorising disablism and ableism. Routledge.
Google Scholar

Individuals with Disabilities Education Act of 1975, 20 U.S.C. § 1400 (1975).
Google Scholar

Jackson, L., Fitzpatrick, H., Alazemi, B., Rude, H. (2018). Shifting gears: Re-framing the international discussion about inclusive education. Journal of International Special Needs Education, 21(2), 11-22.
Google Scholar

Kuntz, K. J., McLaughlin, T. F., Howard, V. F. (2001). A comparison of cooperative learning and small group individualized instruction for math in a self contained classroom for elementary students with disabilities. Educational Research Quarterly, 24(3), 41-56.
Google Scholar

Muller, C., Riegle-Crumb, C., Schiller, K. S., Wilkinson, L., Frank, K. A. (2010). Race and academic achievement in racially diverse high schools: Opportunity and stratification. Teachers College Record, 112(4), 1038-1063.
Google Scholar

National Council on Disability . (2018). IDEA series: The segregation of students with disabilities.
Google Scholar

Pennsylvania Department of Education . (n.d.). Summary of Pennsylvania's reporting standards. https://www.stateboard.education.pa.gov/Documents/Research%20Reports%20and%20Studies/SummaryofPAPerformanceStandards1.pdf
Google Scholar

Thorius, K. A. K., Tan, P. (2016). Expanding analysis of educational debt: Considering intersections of race and ability. In Connor, D. J., Ferri, B. A., Annamma, S. A. (Eds.), DisCri: Disability studies and critical race theory in education (pp. 87-97). Teachers College Press.
Google Scholar

Valencia, R. R. (2010). Dismantling contemporary deficit thinking: Educational thought and practice. Routledge. https://doi.org/10.4324/9780203853214-5
Google Scholar

Zehler, A., Fleischman, H. L., Hostock, P. J., Stephenson, T. D., Pendzick, M. L., Sapru, S. (2003). Descriptive study of services to LEP students in and ELP students with disabilities. http://ncela.net/files/rcd/BE021199/special_ed4.pdf
Google Scholar

# data[].attributes.attachment_link.title
Academic Tracking of English Learners With Disabilities in Middle School - Sara E. N. Kangas, Megan Cook, 2020

# data[].attributes.attachment_link.description
Despite increased attention to the academic progress of English learners (ELs) with disabilities as a result of the Every Student Succeeds Act of 2015, research has yet to investigate the educational opportunities of these students in secondary grades. This qualitative embedded case study examined the curricular access of 10 ELs with disabilities in middle school.

# data[].attributes.headline
5. Design Considerations in Multisite Randomized Trials Probing Moderated Treatment Effects

# data[].attributes.subject
5. Design Considerations in Multisite Randomized Trials Probing Moderated Treatment Effects

# data[].attributes.body
Positivism
the authors validate the formulas for calculating statistical power and the minimum detectable effect size difference with simulations, probe its sensitivity to model assumptions, execute the formulas in accessible software, demonstrate an application, and provide suggestions in designing MRTs probing moderated treatment effects.

# data[].attributes.attachment_link.readable_text
Recent efforts by a broad range of societies and funding agencies have emphasized rigorous study design as an important lever for improving the quality of evidence produced by impact evaluations (e.g., U.S. Department of Education & National Science Foundation, 2013). According to the Common Guidelines (2013), a joint report released by the National Science Foundation and the Institute of Education Sciences, designs which randomly assign units to conditions are the most rigorous designs and have the potential to yield the highest quality of evidence. Random assignment may occur at the individual level, as is the case of a multisite randomized trial (MRT) in which individuals (students) are randomly assigned to condition within sites (schools). Random assignment may also occur at the cluster level, as is the case of a cluster randomized trial (CRT), in which clusters (schools) are randomly assigned to condition and students are nested within schools. Although both MRTs and CRTs are common in impact studies in education ( Spybrook & Raudenbush, 2009; Spybrook, Shi, & Kelcey, 2016), the focus of this article is the design of MRTs.

Initially, the focus of MRTs in education was to address "what works" questions or questions about main effects. More recently, researchers and policymakers have broadened the focus to include questions regarding "for whom, and under what circumstances" programs work, or questions about moderated treatment effects. The impetus for broadening the scope of questions stems in part from empirical research that suggests treatment effects frequently vary across site or individual characteristics ( Weiss et al., 2017). Understanding the context in which an intervention is likely to be effective is fundamental to understanding the extent to which results are applicable and scalable to a wide range of schools and students and also facilitates the development of more nuanced theories.

In this study, we consider the design of MRTs that seek to answer questions about moderated treatment effects. Recall that in an MRT, individuals (students) are randomly assigned to condition within sites (schools). Hence, students represent Level 1 and schools represent Level 2 with treatment varying across Level 1 units. Our analyses consider the intersections of three facets of multilevel moderation that are common in practice: (a) Level 1 and Level 2 moderator variables, (b) random and nonrandomly varying slopes (coefficients) of the treatment variable and the interaction term between the treatment and moderator variables, and (c) binary and continuous moderators. We consider moderators at the student level (e.g., gender) and at the school level (e.g., school size, Title 1 status). For both levels, we consider binary and continuous moderators.

In planning an MRT, a key design consideration is the sample size necessary to achieve adequate statistical power (probability of detecting the main treatment effect and moderated treatment effect). A strong literature base exists for conducting power analyses for the main effects of MRTs already exist (e.g., Borenstein & Hedges, 2012; Dong & Maynard, 2013; Konstantopoulos, 2008; Raudenbush et al., 2011) and for conducting power analyses for main effects and moderator effects in CRTs (e.g., Dong et al., 2018; Spybrook, Kelcey, & Dong, 2016). However, there is less work on power calculations for moderated treatment effects in MRTs. Raudenbush and Liu (2000) developed power formulas for the site-level (Level 2) binary moderator effect in MRTs, and Bloom and Spybrook (2017) developed formulas for the minimum detectable effect size difference (MDESD) for the site-level binary moderator in MRTs. However, the scope of such studies has largely been limited to binary site-level moderators in MRTs. Missing from this literature is a more comprehensive statistical framework for power analyses of moderated treatment effects in MRTs that incorporates the considerations noted above (e.g., continuous moderators, random slopes) and a careful analysis delineating the parameters that govern power and their proportional influence (e.g., how does the intraclass correlation [ICC] coefficient or treatment effect variation/heterogeneity of coefficients affect power).

The purpose of this study is to develop a more comprehensive statistical framework and set of tools for the effective and efficient design of MRTs probing moderated treatment effects. As noted above, the framework we develop considers the intersections of three facets of multilevel moderation that are common in practice: (a) Level 1 and Level 2 moderator variables, (b) random and nonrandomly varying slopes (coefficients) of the treatment variable and the interaction term between the treatment and moderator variables, and (c) binary and continuous moderators. Our investigation of these facets developed formulas that delineate statistical power, the MDESD, and their corresponding confidence intervals (CIs). We also created software to assist researchers conducting power analyses for various moderated treatment effects.

This article is organized as follows: First, we outline a working example to provide the context to our formulations, structure, and expressions. Second, we present the formulas for the standard error ( SE), statistical power, and the MDESD and its CIs for the moderator effect at Level 1 followed by Level 2. Within this scope, we first detail the case of continuous moderators with random slopes and then extend these cases to allow for binary moderators and nonrandomly varying slope models. We follow with Monte Carlo simulations to assess the validity of the formulas we derived. Third, we compare the statistical power and MDESD among the moderated treatment effect and main treatment effect both conceptually and practically followed by demonstrating the calculation of MDESD and power using several examples. We then summarize our findings and discuss the implications of powering for moderated treatment effects in the design of two-level MRTs. Finally, we conclude with considering directions for future work.

We develop an illustrative example to frame our study. Our example focuses on a computer-assisted tutoring program intended to improve students' reading achievement. For example, Chambers et al. (2008) used an MRT to test the effect of a computer-assisted tutoring program on reading achievement. The MRT included a total of 412 first graders randomly assigned to the computer-assisted tutoring or the traditional tutoring groups within each of 25 schools. The findings revealed no significant overall treatment effect. However, the study also suggested the potential for treatment effect heterogeneity. For instance, one common site- or school-level moderator variable that is commonly considered in moderation analyses is the average pretest. The follow-up question is how to design an MRT to systematically probe the moderated treatment effect of the computer-assisted tutoring program.

In this illustrative example and our larger study, we consider three design facets that are common in this literature. As outlined above, the first facet considers the level of the moderator (e.g., student vs. school level). For instance, the effect of the computer-assisted tutoring program may vary by the student characteristics (e.g., pretest and gender) or the site characteristics (e.g., average pretest). The levels of the moderators examine for whom (Level 1 moderators) and under what condition (Level 2 moderators) the computer-assisted tutoring program works.

The second facet concerns the quantitative nature of the moderator-that is, whether the moderator is binary (e.g., gender and program implementation [high vs. low]) or continuous (pretest and school size). When the moderator is a binary variable (e.g., gender), the moderator effect indicates the treatment effect difference between two categorical groups or the gender achievement gap in treatment effectiveness. When the moderator is a continuous variable (e.g., pretest), the moderator effect describes the disparate impact of the treatment on the outcome for different increments of the pretest.

The final facet examines whether the design calls for a random or nonrandomly varying term for the treatment and moderated treatment effects. More specifically, when the moderator is a Level 1 variable, the moderated treatment effect may randomly vary across sites (school) or be constant across schools. For instance, the treatment effect difference between males and females for the computer-assisted tutoring program may or may not be same across schools. In addition, at the school level, the treatment effect may still randomly vary across schools after accounting for the school-level moderator effect or may be constant across schools. For example, if the average pretest of a school explains some of the heterogeneity in treatment effects across schools but not all of it, there may be other factors contributing to the treatment effect heterogeneity. However, if the treatment effect is constant across schools after accounting for the differences among schools in terms of the average pretest, then it may be the only factor causing the treatment effect heterogeneity. The choice of random versus nonrandomly slope depends on the program theory and evidence from prior studies.

Below we describe how we develop the formulas of the statistical power and the MDESD for Level 1 and Level 2 moderators in two-level MRTs. Suppose there are n students in each school, where a proportion ( P) of the students within each school are randomly assigned to the treatment group to receive a computer-assisted tutoring intervention, and there are a total of J schools which serves as blocks or sites. The research questions include whether the effects of the tutoring intervention on student achievement vary by the students' pretest or gender, or by the schools' characteristics, and if the moderated treatment effects vary randomly across schools.

Random slope models allow us to test whether the treatment effect varies across moderator subgroups and whether the moderated treatment effects vary randomly across schools. To test for the Level 1 moderation, we use two-level random slope hierarchical linear modeling (HLM; Raudenbush & Bryk, 2002):

The combined model is:

is the achievement for student i in school j. The treatment variable, , is a binary variable indicating whether the student receives the tutoring intervention (e.g., = −0.5 for control, and 0.5 for treatment). is a Level 1 covariate. is a continuous Level 1 moderator, and . can be viewed as a grand-mean centered variable. The parameter, , estimates the average treatment effect. represents the random site-specific deviation from the average treatment effect. Of interest for the moderator analysis is the collection of the site-specific moderation effects () that can be summarized using the cross-site average moderated treatment effect () and the random site-specific deviation from that average (). We use to describe the variance of and its covariances of and with the random effects for the intercept and treatment, respectively. Note that we use fixed slopes for the covariate () and moderator () because we focus on the setting in which random intercepts, treatment effects, and moderated treatment effects sufficiently capture variation among schools. In addition, it is often hard to estimate more than three random effects with sample sizes typical to these types of educational experiments. However, additional random slopes for covariates and the moderator are possible.

By extending Snijders's (2001, 2005) work, the SE of the Level 1 moderator effect estimate () in Model 3 can be expressed as (see Appendix A in the online version of the journal for details):

To test for the Level 2 moderation, we use two-level random slope HLM ( Raudenbush & Bryk, 2002):

The combined model is:

By extending Snijders's (2001, 2005) work, the estimate of the SE of the Level 2 moderator effect estimate () can be expressed as (see online Appendix A for details):

and

and

Summary of Standardized Noncentrality Parameters, MDESD, and 100 × (1− α)% Confidence Intervals (CIs) for Two-Level MRTs

The hierarchical linear models with a nonrandomly varying slope assume that the treatment effect varies by the moderators but does not randomly vary across sites (Models MRT2-1N-1 and MRT2-1N-2 in Table 1 and below).

The models with a nonrandomly varying slope for a Level 1 moderator (MRT2-1N-1) are as follows:

The models with a non-randomly varying slope for a Level 2 moderator (MRT2-1N-2) are as follows:

To validate the SE and power formulas we derived, we conducted a Monte Carlo simulation to examine whether the formulas were consistent with the simulated results. The procedures for the Monte Carlo simulation are below:

1. We generated data using the hierarchical linear models in Equations 1 and 2, and 5 and 6 for random slope models with Level 1 and Level 2 moderators, respectively, Equations 27 and 28, and 29 and 30 for nonrandomly varying slope models with Level 1 and Level 2 moderators, respectively.

2. We used SAS PROC MIXED to analyze the data sets. We computed the SEs using the Kacker and Harville (1984) approximation, and the degrees of freedom are calculated using the Kenward and Roger (1997) method, which is recommended for small sample size ( Verbeke & Molenberghs, 2000, p. 57). We calculated the moderator effect, standardized effect variability of the Level 1 moderation across sites (), and proportions of variance () explained by Level 1 covariates using the same estimation models as the models for generating data, estimate the standardized treatment effect variability across sites () using the estimation models that only included the treatment variable, and estimate the unconditional ICC using the unconditional hierarchical linear models.

3. The moderator effect was standardized to the standardized mean difference for the binary moderators or the standardized coefficient for the continuous moderators; a p value of the moderator effect that is less than .05 was coded a rejection of the null hypothesis of no moderation.

4. We replicated Steps 1 through 3 2,000 times and calculated the means of the moderator effect size, , , R ², and unconditional ICC; The standard deviation of 2,000 moderator effect sizes served as the SE estimate based on the empirical distribution of the moderator effect; we also calculated the SE based on our formulas and constructed the 95% CI for each point estimate; we calculate the absolute difference and relative difference between the SE s based on our formulas and that from the empirical distribution; we calculate the coverage rate of the 95% CI as the percentage of the 95% CI based on our formulas covering the true moderator effect. The proportion of times the null was rejected across the 2,000 replications estimated the Type I error rate when the moderation effect was set to 0 and the empirical power when the moderation effect was not set as 0; we compared the power and Type I error rate calculated from our derived formulas with those estimated from simulation.

Our Monte Carlo simulation considered several scenarios by changing the sample size, the moderator effect size, random slopes and nonrandomly varying slopes, and binary and continuous Level 1 and Level 2 moderators.

Tables 2 through 5 present the results of SE and power (or Type I error rate) estimates from the Monte Carlo simulation and that were calculated based on the formulas using the same design parameters and the coverage rate of 95% CI. The results provided evidence of the close correspondence on SEs and power (or Type I error) between our formulas and the empirical distribution from the simulation. For example, in all scenarios, the absolute difference and relative difference between the SE based on the empirical distribution of the moderator effect estimates and SE calculated from our formulas range from −0.007 to 0.005 and from −7.23% to 3.51%, respectively. The coverage rate of the 95% CI ranges from 0.93 to 0.96. The differences between the power calculated from the formulas and that estimated from simulation ranges from −0.006 to 0.039.

In addition, our derived formulas are based on the balanced design, that is, equal site sizes , equal proportion of individuals assigned to the treatment group (), and equal proportions of individuals in the moderator subgroup () across sites. In practice, it is likely that the multisite moderation studies are imbalanced. For power analysis of the main effect in CRTs and MRTs, it is common to use the harmonic mean when the sample sizes across sites/clusters are imbalanced ( Bloom, 2006; Konstantopoulos, 2010). We conducted a small simulation for MRTs with imbalanced , , and Q 1j using the similar procedures described above with some modifications. Specially, for MRTs with imbalanced sample sizes, sample size () for site j ranges from 4 to 40, increases 4 for every four sites when J = 40 and for every eight sites when J = 80. For MRTs with imbalanced or Q 1j, or Q 1j ranges from 0.3 to 0.7. or Q 1j increases 0.1 for every eight sites with site number j when J = 40 and for every 16 sites when J = 80. We used our formulas to calculate power based on the arithmetic mean ( = 22), the harmonic mean ( = 14), and the geometric mean ( = 18), respectively. For imbalanced or Q 1j, we calculated the arithmetic mean, the harmonic mean, and the geometric mean of their variance ((1 − ) or Q 1j(1 − )), then solved for P or Q ₁. We also calculated power based on the balanced design. Table 6 presents the results of power from simulation and from the formulas for a binary moderator with random slopes with combinations of imbalanced , , and Q 1j.

The results for the individual effects of imbalanced , , and Q 1j are presented in Tables B1 through B3 in online Appendix B. Our simulation suggests that the power calculation based on the harmonic mean underestimates the actual power and the power calculation based on the arithmetic mean overestimates the actual power. The power calculation based on the geometric mean approximates the power from the simulation very well. In addition, the imbalanced design has smaller power than balanced design as expected.

Furthermore, we derived our formulas when the continuous moderators are assumed to be normally distributed. In practice, the continuous moderators may not be normally distributed ( Micceri, 1989). Although the conventional normality assumption for the linear models applies to the residuals, not the dependent variables or predictors, and linear models are robust to violations of the normality assumption when the sample size is large, we conducted a small Monte Carlo simulation to assess how the distributions of continuous moderators affect the estimated power. We used the SAS Macro RandFleishman ( Wicklin, 2013), which implemented Fleishman's (1978) cubic transformation method, to generate the variables with specified skewness and Pearson's kurtosis. We simulated moderators with combined skewness (ranging from 1.18 to 1.95) and Pearson's kurtosis (ranging from 2.21 to 7.44). The absolute difference between the power calculated from the formulas and that estimated from simulation ranged from 0.013 to 0.058. For many scenarios, the simulation results are very close to those from the power formulas. However, as the number of sites decreases the differences increase some, for example, the biggest difference (0.058) occurs for the smallest sample size of sites ( J = 20; see Table B4 in online Appendix B). Overall, the results suggest that our formulas are fairly robust to violations of normality assumption for moderators; however, power can be overestimated when the sample size is small, and the normality assumption is violated.

We also simulated moderators with a bimodal distribution. The moderator variables were generated from the mixture distribution with two mixture components: one normal distribution ( M = 1.4, variance = 0.3) with a mixture weight of 0.3, and another normal distribution ( M = −0.6, variance = 0.1) with a mixture weight of 0.7. The results suggest that our formulas estimated the power fairly close to the simulation (absolute difference ranging from 0.013 to 0.039, Table B5 in online Appendix B).

In this section, we compare the statistical power and MDESD among the moderation designs and main effect designs in two-level MRTs both conceptually (e.g., examining the formulas) and practically (e.g., using examples).

In this section, we compare the MDESD/MDES and power among four moderated treatment effects and the main effect in a two-level MRT using several examples. The MDESD and power for the moderated treatment effects are calculated using the software we developed, which is a Microsoft Excel-based software package implementing formulas in Table 1. The MDES and power for the main effects are calculated using PowerUp! ( Dong & Maynard, 2013). Suppose a team of researchers are designing a two-level MRT to test the efficacy of the computer-assisted tutoring intervention on mathematics achievement for the eighth graders. They are interested in student-level moderator effects and school-level moderator effects. They approach the moderator power analyses from two perspectives: (1) What is the MDESD given power of 0.80 and (2) what is the power for a meaningful moderation effect size.

Just like conducting a power analysis for the main effect, the researchers need to determine the meaningful effect size difference with practical significance they would like to detect and make reasonable assumptions of other design parameter values in their power analysis of moderator effects in MRTs. To determine the meaningful effect size differences, researchers may refer to the empirical benchmarks regarding normative expectations of annual gain, policy-relevant performance gaps, and moderation effect size results from similar studies ( Bloom et al., 2008; Dong et al., 2016; Hill et al., 2008). For example, Hill et al. (2008) reported students' math achievement gaps in effect size units from the National Assessment of Educational Progress in Grade 8 are −1.04 for Blacks versus Whites, −0.82 for Hispanics versus Whites, and −0.80 for the eligible versus ineligible for free/reduced-price lunch. The researchers may consider an effect size difference of 0.20 for the computer-assisted tutoring intervention to have a meaningful moderation effect because it is equivalent to one fifth a reduction of Black-White achievement gap and one fourth a reduction of Hispanic-White and eligible-ineligible for free/reduced-price lunch gaps. They may refer to moderation effect size results from similar studies; however, these results are very limited. For demonstration purposes, suppose they decide to use 0.20 as their desired effect size difference in their power analysis.

For other design parameter values, the researchers need to justify their choice based on the literature or pilot studies. Recently, several studies have reported the ICC and the proportion of variance explained by the covariates for academic achievement outcome measures (e.g., Bloom et al., 2007, and Hedges & Hedberg, 2007, 2013, on mathematics and reading; Westine et al., 2013, and Spybrook, Westine, & Taylor, 2016, on science achievement), outcome measures for teacher professional development ( Kelcey & Phelps, 2013), and social and behavioral outcomes ( Dong et al., 2016). The researchers assume a of 0.25, and the proportion of variance explained by the covariates at Level 1 of 0.5 ( = 0.5; Bloom et al., 2007; Hedges & Hedberg, 2007, 2013).

There are very few studies reporting the effect heterogeneity across sites values. We only identified Weiss et al. (2017) reporting the treatment effect heterogeneity values () across sites and did not identify any study reporting the heterogeneity values () for the Level 1 moderated treatment effect across sites. Weiss et al. (2017) studied 51 outcome measures in 16 MRTs and reported that ranged from 0 to 0.35, with 37% ranging from 0 to 0.05, 33% ranging from 0.05 to 0.15, and 29% ranging from 0.15 to 0.35. In this example, suppose the researchers decide to use the moderate effect size heterogeneity by assuming = 0.05 and 0.15, respectively. Because there are no empirical reference values available for , they assume the same values of as , that is, = = 0.05 and 0.15, respectively. Note that the researchers may choose (and other design parameters) based on pilot studies, and we provide the SAS code to estimate these design parameters in online Appendix D (the SAS code and example data set can be downloaded from the website: https://www.causalevaluation.org/).

They use a balanced design with equal assignment of students to the treatment and control groups ( P = 0.5) within a school (site) and 20 students per school. They are interested in the results for a binary moderator and a continuous moderator. For the binary case, they assume half of the sample is in one moderator subgroup ( Q = 0.5). Table 7 shows the results of MDESD and power for the total numbers ( J) of schools of 30 and 60 under the above assumptions. Tables C1 through C4 in online Appendix C provide examples of calculation of MDESD and power using our software.

Furthermore, we demonstrate the relationship between power and total sample size of sites by comparing the main treatment effect design with four moderation designs with binary moderators in and . The power was calculated independently for main effects and moderated effects based on the same assumptions as in Table 7: n = 20, = 0.25, P = 0.5, = 0.5, and Q ₁ = Q ₂ = 0.5. For the treatment effect heterogeneity, we set = = 0.05 for the random slope design in and = = 0.15 for the random slope design in In addition, for comparison purposes, we assume the effect size (the standardized mean difference) for the main treatment effect and the effect size difference for the moderator effect at 0.2 using a two-sided test with α = 0.05. This is equivalent to effect sizes for the two moderator subgroups of 0.3 and 0.1, respectively. Thus, the resulting power curves are for the moderation analyses with a binary Level 2 moderator random effect (gray solid line), a binary Level 1 moderator random effect (long dotted line), a binary (Level 1 or Level 2) moderator with nonrandomly varying effect (short dotted line), and the main treatment effect (black solid line).

The findings in Table 7, and , and conceptual comparisons are discussed below. First, as for all power analyses, the power increases with the sample sizes ( J and n). However, the importance of Level 1 and Level 2 sample sizes is different in different designs. Recall that the power and MDESD are same for Level 1 and Level 2 moderators with nonrandomly varying effects. This suggests that the sample sizes at Level 1 and Level 2 are equally important to the power and the MDESD for the nonrandomly varying moderator effect. In contrast, the sample size at Level 2 ( J) is more important than Level 1 ( n) for the moderator effect with the random slopes. Note that we set the site size ( n) as 20 and vary the total number of sites ( J) for demonstration in Table 7 and and . In practice, researchers may choose n and J based on their research goals, budget, and sample availability. For example, the average n ranges from 11 to 1,176 and J ranges from 9 to 318 in 16 MRTs reported in Weiss et al. (2017). When it is not feasible for researchers to increase J, they may aim to increase n to increase statistical power.

Second, the proportion of the sample allocation to the treatment and control group ( P) and to the moderator subgroup ( Q) are related to the power and MDESD. The power (MDESD) increases (decreases) when P and Q is close to 0.5.

Third, the power (MDESD) increases (decreases) when the ICC increases. This is because the sites explain more Level 2 variance, reduce Level 1 variance, and hence reduce the SE of the moderated treatment effect estimates when increases.

Fourth, the power increases with the proportion of variance explained by the covariates (). A covariate can improve power through reducing the SE of the moderator effect estimate. Hence, in a two-level MRT, a Level 1 covariate ( > 0) can always improve power; however, a Level 2 moderator that is included in the intercept model at Level 2 does not contribute to the power.

Fifth, a design for detecting main effects always has larger power than detecting moderation effects in a two-level MRT. This is different from CRTs, in which, the power for detecting the effects of a Level 1 moderator with nonrandomly varying slope can be larger than the power for the main treatment effect analysis ( Dong et al., 2018).

Sixth, the MDESD is larger or the power is smaller for a random moderator effect than a nonrandomly varying moderator effect. The differences for the power and MDESD between the two models (random slope and nonrandomly varying slope models) decreases when the number of clusters ( J) increases and the effect heterogeneity () decreases. Note that we set the (moderated) treatment effect heterogeneity values, = = 0.05 and 0.15, for the random slope designs based on the range of the moderate effect size heterogeneity reported by Weiss et al. (2017) for demonstration in Table 7 and and . The power would be bigger/smaller if or is smaller/bigger. In practice, researchers need to carefully justify the (moderated) treatment effect heterogeneity values. Finally, the MDESD as defined by the standardized mean difference for the binary moderator when Q = 0.5 is always twice the value of the MDESD defined by the standardized coefficient for the continuous moderator with a nonrandomly varying effect.

As researchers and policy makers are increasingly interested in the moderated treatment effects to answer the "what works for whom, and under what circumstances" questions in MRTs, a power analysis is a critical step. This study fills the gap in the literature by developing a more comprehensive statistical framework and software for power analyses to detect a wide variety of moderated treatment effects in MRTs. We provide some suggestions below.

First, we need to consider three facets of multilevel moderation that are common in practice: (a) Level 1 and Level 2 moderator variables, (b) random and nonrandomly varying slopes (coefficients) of the treatment variable and the interaction term between the treatment and moderator variables, and (c) binary and continuous moderators. We consider binary moderators (e.g., gender) when we are interested in detecting the treatment effect difference between boys and girls or whether the intervention can reduce boys-girls achievement gap; we consider continuous moderators (e.g., pretest) when we are interested in testing whether the association of pretest and posttest is different between the treatment and control groups or whether the treatment effect varies by the pretest. Sometimes we may dichotomize our continuous moderators to produce meaningful subgroups and facilitate the interpretation of moderated treatment effects. We consider Level 1 moderators (e.g., student characteristics) when we are interested in answering "for whom the program works," and Level 2 moderators (e.g., school characteristics) when we are interested in answering "under what condition the program works." Furthermore, we consider random (moderated) treatment effects when the theory or prior studies suggest that the (moderated) treatment effect may vary across sites and nonrandomly varying treatment effects otherwise. However, it would be beneficial to assume random effect if there is not clear theory or prior studies suggesting nonrandomly varying treatment effects.

Second, the power for all moderated treatment effects is smaller than the main effect in two-level MRTs. We need larger sample sizes to detect a moderated treatment effect with the same magnitude as the main effect. Regarding improving power, the sample size at the site level is more important than that at the individual level for random (moderated) treatment effects, and they are equally important for nonrandomly varying (moderated) treatment effects. Including Level 1 covariates that are correlated with the outcome, for example, pretest, can improve power. In addition, the power is bigger when the sample size is more balanced among the treatment-by-moderator groups and across sites, for example, the power is the maximum when P = 0.5 and Q ₁ or Q ₂ = 0.5 with equal site size. When the site size (), the proportion () of individuals assigned to treatment group, or the proportion of individuals ( Q 1j) in one moderator subgroup is imbalanced across sites ( j), the power based on the harmonic mean is very conservative whereas the power based on the geometric mean approximates the power from the simulations very well and hence is what we recommend for power calculations.

This study focused on two-level MRTs. There are many important directions for further work. First, extending the work to three-level MRTs is necessary. For example, in three-level MRTs, where the treatment variable could be at Level 1 or Level 2, the moderator could be at any of three levels, and the (moderated) treatment effect can be either random or nonrandomly varying. The three-level MRTs provide more opportunities to probe moderated treatment effects. Second, accurate empirical estimates of the design parameters are critical for a power analysis. Hence, more empirical studies of design parameters (e.g., ICC, treatment effect heterogeneity, and meaningful size regarding the moderator effects) are important as we move forward.

ORCID iD
Nianbo Dong https://orcid.org/0000-0003-0128-7106

References

Bloom, H. S. (2005). Randomizing groups to evaluate place-based programs. In Bloom, H. S. (Ed.), Learning more from social experiments: Evolving analytic approaches (pp. 115-172). Russell Sage Foundation.
Google Scholar

Bloom, H. S. (2006). The core analytics of randomized experiments for social research (MDRC Working Papers on Research Methodology). http://www.mdrc.org/publications/437/full.pdf
Google Scholar

Borenstein, M., Hedges, L. V. (2012). CRT-Power-Power analysis for cluster-randomized and multi-site studies [Computer software]. Biostat.
Google Scholar

Kacker, R. N., Harville, D. A. (1984). Approximations for standard errors of estimators of fixed and random effects in mixed linear models. Journal of the American Statistical Association, 79, 853-862.
Google Scholar

Kenward, M., Roger, J. (1997). Small sample inference for fixed effects from restricted maximum likelihood. Biometrics, 53, 983-997.
Google Scholar

Murray, D. (1998). Design and analysis of group-randomized trials. Oxford University Press.
Google Scholar

Raudenbush, S. W., Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (p. 485). Sage.
Google Scholar

Raudenbush, S. W., Liu, X. (2000). Statistical power and optimal design for multisite randomized trials. Psychological Methods, 5(2), 199-213.
Google Scholar

Raudenbush, S. W., Spybrook, J., Congdon, R., Liu, X., Martinez, A. (2011). Optimal design software for multi-level and longitudinal research (Version 2.01) [Computer software]. www.wtgrantfoundation.org.
Google Scholar

Snijders, T. (2001). Sampling. In Leyland, A. H., Goldstein, H. (Eds.), Multilevel modeling of health statistics (pp. 159-173). John Wiley.
Google Scholar

Snijders, T. (2005). Power and sample size in multilevel linear models. In Everitt, B. S., Howell, D. C. (Eds.), Encyclopedia of statistics in behavioral science (Vol. 3, pp. 1570-1573). Wiley.
Google Scholar

Spybrook, J., Westine, C. D., Taylor, J. A. (2016). Design parameters for impact research in science education. AERA Open, 2(1), http://doi.org/10.1080/2332858415625975
Google Scholar

U.S. Department of Education & National Science Foundation . (2013). Common guidelines for education research and development (NSF 13-126). Retrieved February 15, 2014, from http://ies.ed.gov/pdf/CommonGuidelines.pdf
Google Scholar

Verbeke, G., Molenberghs, G. (2000). Linear mixed models for longitudinal data. Springer.
Google Scholar

Wicklin, R. (2013). Simulating data with SAS. SAS Institute Inc.
Google Scholar

Authors

NIANBO DONG is an associate professor in the School of Education at the University of North Carolina at Chapel Hill, 116 Peabody Hall, Chapel Hill, NC 27599; email: dong.[email protected]com . His primary research interests include causal inference, statistical power analysis, multilevel modeling, and program and policy evaluation.

BENJAMIN KELCEY is an associate professor in the Quantitative Research Methodologies Program at the University of Cincinnati, Teachers/Dyer Hall, Cincinnati, OH 45221; email: benjamin.[email protected]edu . His research interests are causal inference and measurement methods within the context of multilevel and multidimensional settings such as classrooms and schools.

JESSACA SPYBROOK is a professor in the Evaluation, Measurement, and Research Program at Western Michigan University, 1903 W. Michigan Avenue, Kalamazoo, MI 49008; email: jessaca.[email protected]edu . Her research interests are the design of evaluations of educational interventions, multilevel models, and statistical power.

# data[].attributes.attachment_link.keywords[]
minimum detectable effect size difference

# data[].attributes.attachment_link.title
Design Considerations in Multisite Randomized Trials Probing Moderated Treatment Effects - Nianbo Dong, Benjamin Kelcey, Jessaca Spybrook, 2020

# data[].attributes.attachment_link.description
Recent efforts by a broad range of societies and funding agencies have emphasized rigorous study design as an important lever for improving the quality of evidence produced by impact evaluations (e.g., U.S. Department of Education & National Science Foundation, 2013).

# data[].attributes.headline
6 The Use of the Posterior Probability in Score Differencing

# data[].attributes.subject
6 The Use of the Posterior Probability in Score Differencing

# data[].attributes.body
interiretivism
The author explore and try to understand how the suggested approach may be effective in detecting fraudulent examinees. and call for more attention to the use of posterior probabilities,

# data[].attributes.attachment_link.readable_text
Score differencing is one of the six categories of statistical methods used to detect test fraud (Wollack & Schoenig, 2018) and involves the testing of the null hypothesis that the performance of an examinee is similar over two item sets versus the alternative hypothesis that the performance is better on one of the item sets. We suggest, to perform score differencing, the use of the posterior probability of better performance on one item set compared to another. In a simulation study, the suggested approach performs satisfactory compared to several existing approaches for score differencing. A real data example demonstrates how the suggested approach may be effective in detecting fraudulent examinees. The results in this article call for more attention to the use of posterior probabilities, and Bayesian approaches in general, in investigations of test fraud.

References

Berger, J. O., Sellke, T. (1987). Testing a point null hypothesis: The irreconcilability of p values and evidence. Journal of the American Statistical Association, 82, 112.
Google Scholar

Borchers, H. W. (2019). pracma: Practical numerical math functions (R package version 2.2.5). Available at https://CRAN.R-project.org/package=pracma.
Google Scholar

Carlin, B. P., Louis, T. A. (2008). Bayesian methods for data analysis. Chapman & Hall.
Google Scholar

Cox, D. R. (2006). Principles of statistical inference. Cambridge University Press.
Google Scholar

Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., Rubin, D. B. (2014). Bayesian data analysis (3rd ed.). Chapman & Hall.
Google Scholar

Hanson, B. A., Harris, D. J., Brennan, R. L. (1987). A comparison of several statistical methods for examining allegations of copying (ACT research report series no. 87-15). American College Testing.
Google Scholar

Holland, P. W. (1996). Assessing unusual agreement between the incorrect answers of two examinees using the K-index: Statistical theory and empirical support (ETS Research Report No. RR-94-4). Educational Testing Service.
Google Scholar

Johnson, M. S., Sinharay, S. (2016). Bayesian estimation. In van der Linden, W. (Ed.), Handbook of item response theory, Volume 2: Statistical tools (pp. 237-257). Chapman & Hall/CRC.
Google Scholar

Pratt, J. W. (1965). Bayesian interpretation of standard inference statements. Journal of the Royal Statistical Society: Series B, 27, 169-192.
Google Scholar

R Core, Team . (2019). R: A language and environment for statistical computing. Vienna, Austria. Available at https://www.R-project.org/.
Google Scholar

Sinharay, S., Jensen, J. L. (2019). Higher-order asymptotics and its application to testing the equality of the examinee ability over two sets of items. Psychometrika, 84, 484-510.
Google Scholar

Skorupski, W. P., Wainer, H. (2017). The case for Bayesian methods when investigating test fraud. In Cizek, G. J., Wollack, J. A. (Eds.), Handbook of detecting cheating on tests (pp. 214-231). Routledge.
Google Scholar

Stern, H. S. (2005). Model inference or model selection: Discussion of Klugkist, Laudy, and Hoijtink (2005). Psychological Methods, 10, 494-499.
Google Scholar

Wollack, J. A., Schoenig, R. W. (2018). Cheating. In Frey, B. B. (Ed.), The SAGE encyclopedia of educational research, measurement, and evaluation (pp. 260-265). Sage.
Google Scholar

Zopluoglu, C. (2017). Similarity, answer copying, and aberrance: Understanding the status quo. In Cizek, G. J., Wollack, J. A. (Eds.), Handbook of detecting cheating on tests (pp. 25-46). Routledge.
Google Scholar

# data[].attributes.attachment_link.title
The Use of the Posterior Probability in Score Differencing - Sandip Sinharay, Matthew S. Johnson, 2020

# data[].attributes.attachment_link.description
Score differencing is one of the six categories of statistical methods used to detect test fraud (Wollack & Schoenig, 2018) and involves the testing of the null hypothesis that the performance of an examinee is similar over two item sets versus the alternative hypothesis that the performance is better on one of the item sets.

# data[].attributes.headline
7. Testing the Within-State Distribution in Mixture Models for Responses and Response Times

# data[].attributes.subject
7. Testing the Within-State Distribution in Mixture Models for Responses and Response Times

# data[].attributes.body
Interpretivism
the author used mixture models  to enable detection of within-subject differences in responses and response times to psychometric test items.

# data[].attributes.attachment_link.readable_text
Abstract

Mixture models have been developed to enable detection of within-subject differences in responses and response times to psychometric test items. To enable mixture modeling of both responses and response times, a distributional assumption is needed for the within-state response time distribution. Since violations of the assumed response time distribution may bias the modeling results, choosing an appropriate within-state distribution is important. However, testing this distributional assumption is challenging as the latent within-state response time distribution is by definition different from the observed distribution. Therefore, existing tests on the observed distribution cannot be used. In this article, we propose statistical tests on the within-state response time distribution in a mixture modeling framework for responses and response times. We investigate the viability of the newly proposed tests in a simulation study, and we apply the test to a real data set.

1. Introduction

Recently, research interest has grown in modeling response times next to the item responses in order to investigate individual differences in ability and speed. Focusing on the item response times in addition to the item responses has facilitated various aspects of psychological testing including, for instance, item selection in computerized adaptive testing ( van der Linden et al., 1999; Veldkamp, 2016), test design ( van der Linden, 2007), and item calibration (T. Wang & Hanson, 2005). In addition, response times have been shown useful in detecting item preknowledge ( McLeod et al., 2003), aberrant response patterns ( Marianti et al., 2014; van der Linden & Guo, 2008; C. Wang, Xu, Shang, & Kuncel, 2018), and individual differences in the use of solution strategies. For instance, van der Maas and Jansen (2003) showed that response times can give detailed information on the type and duration of different solution strategies children use to solve a balance scale task. Suitable models to enable these inferences concerning individual differences in responses and response times include the model by Roskam (1987) and more recently the hierarchical model by van der Linden (2007, 2009), which was elaborated by Molenaar et al. (2015a).

Besides these applications of response times to individual differences research, response times have been used to facilitate the study of within-subject differences in solution strategies or psychological processes that underlie the responses to psychometric tests and questionnaires. For instance, response times have been used to identify fast guessing (e.g., Schnipke & Scrams, 1997) and within-subject differences in solution strategies (e.g., Molenaar et al., 2016). Other applications include the study of within-subject differences in motivation ( Wise & Kong, 2005) and faking on personality test items ( Holden & Kroner, 1992).

To facilitate the detection of within-subject differences in responses and response times, various approaches based on mixture modeling have been proposed. For instance, the earliest contribution by Schnipke and Scrams (1997) focused on a two-state within-subject mixture model for the response times only. Here, one state represented rapid-guessing behavior of examinees and the other state modeled the responses of examinees who actually tried to solve the item (i.e., a regular response process). The model by Schnipke and Scrams did not include a latent speed variable but can be seen as one of the first models within this framework. In their model, the mean and variance of the response times are estimated freely for each item in the regular process state, while in the rapid-guessing state, a common mean and variance parameter is assumed to underly the items. On the basis of this model, C. Wang and Xu (2015) and C. Wang, Xu and Shang (2018) proposed a mixture model in which separate measurement models are proposed for modeling the responses and response times in the slower response state, whereas for the faster guessing state, only a guessing parameter is estimated for the responses, and a mean and variance parameter is estimated for the response times. Molenaar et al. (2016) generalized this approach by proposing a mixture model that specifies a measurement model for the responses and a measurement model for the response times separately in each state.

Inspired by the aforementioned models, the hierarchical mixture modeling approach ( Molenaar et al., 2016; Schnipke & Scrams, 1997; C. Wang & Xu, 2015; C. Wang, Xu, & Shang, 2018; C. Wang, Xu, Shang, & Kuncel, 2018) that we focus on in this article is a mixture extension of the hierarchical model by van der Linden (2007, 2009) to allow for within-subject differences in ability and speed. In the van der Linden model, between-subject differences in ability level are captured by means of a continuous random latent ability variable , which underlies the item responses of respondent p=1, . . . ,N to item i=1, . . . ,J . Individual differences with respect to the speed with which the responses to the test items are given are modeled by a continuous random latent speed variable , which underlies the response times. In contrast to the van der Linden model, which assumes speed and ability to be constant within subjects, the hierarchical mixture modeling approach allows speed and ability to differ within subjects. Therefore, an item-specific latent class variable is assumed to underlie the response and the response time of person p to item i. Although the latent class variable can have K states in principle, it is commonly assumed to have two states, K=2 : In one state, the item properties of the faster responses are modeled and in the other state the item properties of the slower responses are modeled. Each respondent is allowed to switch between the slow and fast response state from item to item, such that within-subject differences are captured by the latent class variables.

To enable mixture modeling of both the responses and response times, a distributional assumption is needed for the within-state response time distribution. Correct specification of this distribution is important as it has been shown that violations of the assumed response time distribution may bias modeling results for mixture models in general ( Vermunt, 2011), for growth mixture models ( Bauer & Curran, 2003), and for the hierarchical mixture modeling framework for responses and response times as discussed above ( Molenaar et al., 2018). More specifically, Molenaar et al. (2018) showed that if the observed response time distribution differs from the assumed distribution, within-state parameter estimates and information criteria like Akaike's Information Criterion (AIC; Akaike, 1974), the Bayesian Information Criterion (BIC; Schwarz, 1978), and the consistent AIC (CAIC; Bozdogan, 1987) will be biased, and spurious states will be detected even if there are no states underlying the data. Thus, specifying an appropriate within-state response time distribution is important. In practice, however, it is commonly unknown which type of distribution would fit the within-state response times best. Often, statistically convenient distributions are chosen, for example, the log-normal, the exponential, or the chi-square distribution. These distributions are considered convenient as respectively the logarithmic, the reciprocal, and the square root transformation will result in normally distributed response times. Once a distribution is chosen, this assumed distribution should ideally be tested to ensure that the mixture modeling results are valid. However, testing this distributional assumption is challenging as the within-state distribution is by definition different from the observed distribution since the latter is aggregated over states. That is, if the within-state log-transformed response time distribution is normal, the observed log-response time distribution will be skewed (assuming that the two states differ in their expected log-response time and their log-response time variance). As a result, it is not clear whether skewness in the observed log-transformed distribution reflects a mixture of two states or a misspecification of the response time distribution ( Molenaar et al., 2018). Therefore, traditional statistical tests (e.g., the Shapiro-Wilk test [SW test], 1965) on the observed response time distribution cannot be used.

In this article, we propose statistical tests on the within-state response time distribution in the hierarchical mixture modeling framework for responses and response times. Specifically, we propose tests on normality of the transformed response time distribution using the SW test ( Shapiro & Wilk, 1965; see also Royston, 1982a, 1982b, 1992) and the Kolmogorov-Smirnov test (KS test; Kolmogorov, 1933; Smirnov, 1948). Although the SW test and the KS test are well-established methods to test a hypothesized distribution to hold for an observed variable, the innovative aspect of the present study is that we apply these tests to investigate a hypothesized distribution to hold for the latent within-state distribution of the hierarchical mixture modeling framework for responses and response times. We focus on the log-transformation for the response times, making our approach a test on log-normality. We prefer to focus on log-normality as this is the most commonly used assumption in mixture modeling of response times. However, the proposed methodology can readily be used to test for other distributions by using a different response time transformation. That is, one can consider for instance the square root transformation to test for a chi-square distribution and the reciprocal transformation to test for an exponential distribution. In addition, for the KS test, it is straightforward to accommodate any other distribution (e.g., the Weibull, ex-Gaussian, or Wald distribution) as long as its cumulative distribution function exists and can be evaluated.

The proposed normality tests can be used for various types of (response time) mixture models; however, in this study, we apply the tests to the Markov-dependent item states model and the independent item states model ( Molenaar et al., 2016). This article is organized as follows: First, we discuss the hierarchical mixture modeling approach with log-response times. Next, we present the normality tests for the within-state response time distribution. We then present a simulation study to investigate the performance of the different normality tests. In addition, we illustrate the use of the tests by means of a real data application, and we end with a general discussion.

2. The Hierarchical Mixture Modeling Approach

In the hierarchical mixture modeling approach ( Molenaar et al., 2016; Schnipke & Scrams, 1997; C. Wang & Xu, 2015; C. Wang, Xu, & Shang, 2018), an item-specific latent class variable is assumed to underlie the response and the response time of person p to item i. The levels of this latent class variable are referred to as item states. The item states for respondent p on the items of a test can be collected in vector =[,, . . . ,] . In addition, the item responses are collected in vector =[,, . . . ,] , and the log-transformed response times are collected in vector =[,, . . . ,] .

Although the latent class variable can have multiple states in principle, it is commonly assumed to have two states: a state capturing the measurement properties of the slower responses and a state capturing the measurement properties of the faster responses. The slower state, with a larger expected response time, is denoted by =0 , the faster state by =1 . As discussed above, the response times are assumed to follow a log-normal distribution in each state, such that the log-response times follow a normal distribution (e.g., C. Wang & Xu, 2015). In addition, the item responses and the log-response times are assumed to be independent conditional on the overall latent ability and the overall speed within the states .

In order to separate the effects of the item, the person and the latent class variable , a measurement model for the responses and a measurement model for the response times, is specified. Various models can be considered for modeling the responses, like the Rasch model ( Loeys et al., 2014), the graded response model (e.g., Molenaar et al., 2015b; Ranger & Ortner, 2011), the two-parameter logistic model (e.g., Molenaar et al., 2015a, 2015b; Ranger & Ortner, 2012), or the three-parameter model (e.g., van der Linden, 2007). Here, we specify a two-parameter logistic model to model the item responses, that is

P(|θ p,)=∏N ω(×θ p+) ×ω−[α k i×θ p+β k i] 1−,

1

where ω(.) denotes the logistic function, and α k i and β k i denote the discrimination and easiness parameters for item i and state k. Note that the item response parameters are allowed to differ across states; however, they are treated as fixed parameters within each state.

Although other models have been proposed as well, a normal one-factor model is commonly assumed for modeling the log-response times ( van der Linden, 2007). Here, we thus assume the log-response times to follow a conditional multivariate normal distribution. As follows, f(|τ p,θ p,) models a person's response times, given their ability, latent states, and overall speed by means of

f(|τ p,θ p,)=∏N 1 2 π σ ε i 2 ×exp−1 2(T p i−μ p i)2 σ ε i 2 ,

2

with

=E(|τ p,θ p,)=v i−δ−τ p,     δ>0,

3

where σ ε i 2 is the residual log-response time variance, vi is the time intensity parameter, and δ denotes the difference in expected speed between the slower and faster state. Here, the constraint δ>0 ensures that =1 corresponds to the faster state, which is the state with the smaller response times.

Next, by assuming a bivariate normal distribution for the latent ability variable and the latent speed variable , the general log marginal likelihood of item response vector and the log-response time vector , given the model parameter vector η, is given by

ℓ(,; η)=ln∫∫−∞ ∑C p 1 ∑C p 2 ...∑C p J P(|θ p,)×f(|τ p,θ p,)P()×g(θ p,τ p)d θ d τ,

4

where g(.) denotes the bivariate normal density function. P() denotes the state probabilities for a set of items, which, for each particular item, models the probability that the response of person p to item i belongs to a given state. As follows, P()=P(C p 1),P(C p 2), . . . ,P(C p J) . P(|θ p,) thus models the probability of obtaining a particular response pattern, given a person's ability and their latent states for the different items, and f(|τ p,θ p,) models a person's response times.

In the hierarchical mixture modeling framework, the item-specific latent class variables are commonly assumed to be independent from item to item. However, in practice, these latent class variables may be dependent, for instance, if a respondent guesses on one item, they may be more likely to guess on the next item. To account for such a possible dependency between the latent class variables in the hierarchical mixture framework, Molenaar et al. (2016) also considered a model with a time homogenous first-order Markov-structure on the item-specific latent class variables. As a result, in this model, P() , the state a person is in regarding a certain item, depends on the state of the previous item. That is, the state probability P() is decomposed as follows:

where P() is referred to as the initial state probability and P(|C p(i−1)) is the so-called transition probability. Since the number of states in our model equals two, there are two initial state probabilities. In addition, as the Markov dependency is assumed to be time homogenous, the transition probabilities are assumed equal for all subsequent items, resulting in four transition probabilities. In model estimation, only one initial state probability and two transition probabilities have to be estimated as the others follow from these estimates (see Figure 1 for a graphical representation of the model).

3. Normality Tests

In the model discussed above, the log-response times within each state are assumed to follow a normal distribution. To test this assumption, we use the SW test and the KS test. Specifically, we propose the following procedure: First, the hierarchical mixture model is fit to the responses and response times. Next, the resulting posterior state probabilities are obtained for each response which in turn are used to draw posterior state assignments to state 0 or 1 for each person's response to each item. Then the normality tests are conducted on either (1) the response times in state 0 and state 1 according to the posterior state assignment or (2) on the response times weighted by the posterior state probabilities.

The rationale for the above procedure is that if the within-state log-response time distribution is correctly specified, the resulting posterior state probabilities and posterior state assignments are correct. As a result, the SW and KS test statistics will follow their theoretical distribution under the null hypothesis of normality. However, if the within-state log-response time distribution is incorrectly specified, the resulting posterior state probabilities and posterior state assignments are wrong, and the SW and KS test statistics will not follow their null distributions. Below we discuss the SW and KS tests and apply them to the mixture modeling framework.

The SW test, also called the W test for normality, tests the null hypothesis that an observed variable comes from a normally distributed population. Since the original test as proposed by Shapiro and Wilk (1965) could not be used for sample sizes larger than 50, Royston (1982b, 1992) extended it to sample sizes up to 2,000. Then, suppose that x 1<x 2< ....<x N is an ordered sample of size N on which the normality test is carried out. The W statistic, as calculated for an item , is defined as

where a=a 1,a 2,...,a N are the normalized best linear unbiased coefficients, and p=1,...,N . Vector a is defined by

where m =m 1, . . . ,m N is a vector of expected values of standard normal order statistics, and V is the corresponding covariance matrix. Note that vector a is antisymmetric, that is, a N=−a 1 , and for odd N, a(N/2)+1=0 ( Royston, 1992). More details on the calculation of the Wi statistic can be found in the Appendix.

The KS test is a nonparametric goodness of fit test, which measures the distance between an empirical distribution function of a sample and a hypothesized cumulative distribution function (the one-sample KS test) or which compares the distribution functions of two samples (the two-sample KS test). Here, we will focus on the one-sample KS test for testing the within-state response time distribution of item i for normality, which is defined by

where F N(x) is the empirical distribution function, and F(x) the reference distribution. In this article, we assume a normal distribution for the log-response times; however, other distributions like the Wald distribution can be used as a reference distribution as well. Therefore, the KS test can not only be used to test the within-state response time distribution for normality but can be used to test for all kinds of distributions. As Monahan (2011) noted, the Kolmogorov-Smirnov statistic is distribution free for continuous random variables. As a result, the distribution function is evaluated at the observations, and then sorted, such that

D N i=max p p/N−F(X(p)),F(X(p))−(p−1)/N ,

9

where Xp denotes the response time of respondent p on item i, with ; and X(1)≤ . . . ≤X(p)≤ . . . ≤X(N) denote the ordered response times of a sample of respondents. The accompanying p value for the statistic needs to be bootstrapped, since the resulting sampling distribution is unknown, and therefore, the mean and standard deviation are unknown and thus must be estimated from the data when testing for normality ( Lilliefors, 1967).1 When testing for other types of distributions, estimating only the mean and standard deviation might not be sufficient, and additional parameters might need to be estimated (e.g., a skewness parameter).

Since the posterior state probabilities differ for each respondent on each item they are taken into account by including them as weights in the KS test, Equation 9 needs to be modified to account for the weighted response times ( Monahan, 2011). The empirical distribution function can be estimated by

where P(|,) are the posterior state probabilities, and I(X p≤x) is an indicator function, which equals 1 if X p≤x and 0 otherwise. Then, the Xp must be sorted together with the corresponding posterior state probabilities, the increments can be computed by

As follows, F N(x)=F k for X k−1<x≤X k , which can be calculated by estimating quantiles for the underlying normal distribution, weighted by the posterior state probabilities, that correspond to N ordered probabilities. Using Fk from Equation 11, the Kolmogorov-Smirnov statistic for item i modifies to

D N i=sup x|F N(x)−F(x)|=max k F k−F(X(k)),F(X(k))−F k−1 .

12

Like in the unweighted case, the p value of needs to be bootstrapped, since the mean and standard deviation of the log-response time distribution are estimated from the data ( Lilliefors, 1967).

4. Simulation Study

In the simulation study, we compared the performance of the unweighted SW test, the unweighted KS test, and the weighted KS test. Data are simulated according to nine different scenarios, mostly based on Molenaar et al. (2018) who found biased modeling results of the hierarchical mixture model in the case of nonnormality. The first three scenarios concern Markov mixture models that include Markov-dependent item states, the next three scenarios concern mixture models with independent item states, and the final three scenarios are generated according to a baseline model that does not include item states (i.e., a static model without mixtures). The scenarios differ in the distribution that is used for the log-response times, which is a normal, a truncated, or a skewed distribution. The responses are modeled using the two-parameter logistic model, with item parameters α k i and β k i , which denote the discrimination and easiness of an item i in state k, respectively. For each scenario, we conducted 50 replications with 20 items, the sample size was equal to 500 or 1,000.

The three Markov mixture model scenarios are the following:

Normal Markov mixture: In this scenario, we use the Markov-dependent item states model with normal log-response times to simulate the data. We use δ=.1 . That is, the expected log-response times differ by 0.1 between the fast and slow state. The item parameters for the two different states are set as follows: the discrimination parameters are set to α 0 i=1.5 for State 0 (slow state) and to α 1 i=1 for State 1 (fast state). We set the easiness parameters to increasing, equally spaced values: for the slower state, β 0 i is between −2 and 0, for the faster state β 1 i is between 0 and 2. The response time parameters for all items i are chosen as v i=2 , and the residual response time variances as σ ε i 2=0.13 . Furthermore, σ τ 2=0.13 and σ θτ 2=0.14 , so that the correlation between θ p and τ p equals 0.4. Finally, the initial state probability, P(C p 1=1) , is fixed to 0.5, and the transition probabilities, P(=1|C p(i−1)=0) and P(=0|C p(i−1)=1) , are fixed to .231, which corresponds to mildly instable states according to Bacci et al. (2014).

Truncated Markov mixture: In this scenario, the data are generated using the same parameter values as in the normal Markov mixture scenario above. However, now we use a right-truncated normal distribution for the log-response times, with truncation at ln(12) such that the response times are cut off at 12 seconds. This scenario mirrors data from time-pressured tests, where respondents only have a limited amount of time to answer the items.

Skewed mixture: Here, the data are generated using the same parameter values as in the normal Markov mixture scenario. However, the normally distributed log-response times are transformed using a Box-Cox transformation ( Box & Cox, 1964). In general, the transformation is used to transform skewed variables in such a way that they are closer to a normal distribution. Here, we use the transformation the other way around, so that we transform the normally distributed log-response times into skewed variables using the Box-Cox transformation. That is, we transform the normally distributed log-response times using ln()′=(λ ln()+1)λ with λ=.3 .

Figure 2 shows the resulting log-response time distribution of an arbitrary item from an example run of the three different Markov mixture model scenarios. Note that in the truncated scenario, the log-response times are negatively skewed, while in the skewed scenario, the log-response times are positively skewed.

In the three independent mixture scenarios, we used the same parameter values and setup as for the Markov mixture scenarios above; however, the item states are assumed to be independent, that is, the Markov structure is omitted (i.e., P(=1|C p(i−1)=0)=P(=1) and P(=0|C p(i−1)=1)=P(=0) ). Next, for the three baseline scenarios, the data do not include item states, and a baseline model (i.e., the traditional hierarchical model without mixtures by van der Linden, 2007) is used to generate the data. Like the mixture models, the scenarios differ in the distribution used for the log-response times, which again are either a normal, a truncated or a skewed distribution:

Normal baseline: In this scenario, the log-response times are normally distributed. The item parameters are as follows: The discrimination parameters for all items are set to α i=1 , the easiness parameters β i are set to increasing, equally spaced values between −2 and 2. The response time parameters vi, σ ε i 2 , σ τ 2 , and σ θτ 2 are set to the same values as in the normal mixture model scenario.

Truncated baseline: Here, the data are generated using the same parameter values as in the normal baseline scenario. Like in the truncated mixture model scenario, we use a right-truncated normal distribution for the log-response times, with truncation at ln(12) , such that the response times are cut off at 12 seconds.

Skewed baseline: Here, the data are generated using the same parameter values as in the normal baseline scenario. However, like the skewed mixture model scenario, the normally distributed log-response times are transformed using a reverse Box-Cox transformation ( Box & Cox, 1964), with the same value for λ.

5. Results

For the truncated scenarios, convergence problems occurred in 17 replications (4 in the truncated Markov mixture, 9 in the truncated independent mixture, and 4 in the truncated baseline scenario). In the results below, those replications are excluded. Tables 1, 2, and 3 contain the results for the normal, truncated, and skewed scenarios, respectively. Specifically, the tables depict the proportion of items for which the null hypothesis of normality is rejected at a .05 level of significance, averaged over all items within a state. That is, a proportion of .900 for a given state indicates that-averaged over the replications-the normality hypothesis is rejected for 90% of all items within that state.

First, for the normal scenarios in Table 1, these proportions indicate the Type I error rate of our approach. Ideally, these rates are close to the level of significance for the tests to be viable. As can be seen from the table, for the Markov mixture, the independent mixture, and the baseline scenarios, the results are similar. That is, the SW test, and the unweighted KS test have acceptable Type I error rates. In addition, the weighted KS test is associated with an inflated Type I error rate.

For the truncated and skewed scenarios, the mean proportion of normality rejections reflects the power to detect within-state departures from normality. Here, we used Cohen's (1988, p. 56) rule of thumb, considering a power coefficient of .80 or higher to be acceptable. As can be seen from Tables 2 and 3, the results indicate that generally the power is acceptable in one state and substantially smaller in the other state. In the truncated scenario, State 0 is associated with larger power, while in the skewed scenario, State 1 is associated with larger power as compared to State 0. This can be explained from Table 4 which contains the average initial state parameter estimates for State 1, together with the transition parameters in the different scenarios (i.e., the table reflects the proportions of persons in the different states). As can be seen from the mean initial state parameter estimates for State 1, in the truncated scenario, State 0 (slower state; larger log-response times) is the larger state, and in the skewed scenario, State 1 (faster state; smaller log-response times) is the larger state. This is due to the log-response times being positively skewed in the skewed scenario and negatively skewed in the truncated scenario (as mentioned above). As a result, due to these larger sample sizes in State 0 for the truncated scenarios and State 1 for the skewed scenarios, power differs between the two states. That is, when fitting a normal mixture to nonnormal data, the nonnormality is best detected in the largest state. Furthermore, even though class sizes are comparable, power tends to be larger in the larger state of the truncated scenarios when compared to the larger state of the skewed scenarios. This is due to the fact that the truncated distribution departs more from normality than the skewed distribution (see Figure 2). In general, the power of the KS tests (weighted and unweighted) is smaller as compared to the SW test. We return to this point in the discussion. Furthermore, the weighted KS test has slightly more power as compared to the unweighted KS test (but is also associated with an increased Type I error rate, see above).

Taken together the above, Type I error rate and the power of the proposed tests seem acceptable for the SW test and the unweighted KS test with more power for the SW test. The weighted KS test is associated with an inflated Type I error rate. There are no important differences between the Markov mixtures, independent mixtures, and baseline model. It turns out that, generally, violations of normality are only detected in one of the states. We note that of course the power of our approach depends on the severity of the normality violations (this is why the power seems somewhat larger in the truncated scenario as compared to the skewed scenario: the data in the truncated scenario is heavier skewed). In that sense, we consider our simulation study as a prove of principle (i.e., given the effect size we have chosen, we demonstrated that the approach is viable).

Our results indicate that nonnormality is detected if the data contain nonnormal mixtures (i.e., the Markov mixture and independent mixture scenarios) or if the data are nonnormal without mixtures (i.e., the baseline scenarios). In practice, where one does not know the data generating model, a significant normality test thus indicates that (1) the data follow a mixture model with a nonnormal within-state distribution or (2) the data are nonnormal but do not contain mixtures. For the present purposes, the distinction between (1) and (2) is not of importance as the implications are the same: In both cases, there is no mixture of normal distributions in the data, so the results of a normal (Markov-)mixture model should not be trusted. If our proposed tests are insignificant in both states, it can safely be concluded that (1) the data follow a mixture model with a normal within-state distribution or (2) the data are normal without mixtures (i.e., the baseline scenario's). As in both cases, the (within-state) data are normal, (1) and (2) can be distinguished by comparing the baseline model and the mixture models using common information criteria (e.g., BIC and CAIC) as demonstrated by Molenaar et al. (2018). Therefore, we propose the procedure summarized in the flow chart in Figure 3. That is, first, the fit of a normal (Markov-)mixture model is compared to that of a normal baseline model. If the baseline model fits better, it can be concluded that the transformed response times are normally distributed and that there is no mixture of normal distributions underlying the data. If the mixture model fits better, one can consult the statistics proposed in this article. If these statistics are insignificant in both states, it can be concluded that there is a true mixture of normal distributions underlying the response time data, and the results of the mixture model can be validly interpreted. However, if the proposed statistics are significant, it can be concluded that there is no mixture of normal distributions underlying the data, and the results of the normal (Markov-)mixture model cannot be trusted.

6. Application

The within-state normality tests are illustrated by means of a real-data application. The data consist of the responses and response times of 389 psychology freshman of the University of Amsterdam to 28 items of the knowledge subtest of the Dutch version of the Intelligence Structure Test ( Amthauer et al., 2001). The knowledge subtest measures essential types of knowledge, which people acquire in schools, higher education and other educational institutions, as well as daily life knowledge acquired from within their culture ( Hogrefe Ltd., 2016). The items in the subtest cover a broad range of topics, like economics, geography, mathematics, history, art and culture, natural sciences, and daily life facts ( de Vries, 2017). The items are dichotomously scored, with 0 indicating a false response and 1 indicating a correct response. Looking more closely at the response time distributions, Figures 4 and 5 show for a selection of items that they are not normally distributed. The observed response time distributions seem skewed, and the question is whether this nonnormality can be explained by a mixture of a fast and a slow state or whether there is an alternative explanation.

In the paper by Molenaar et al. (2016) mixture models with various types of Markov dependencies are fitted to the data and are shown to fit better than a baseline model without item states. However, as the modeling results of a fitting mixture model with a Markov dependency are only interpretable if the assumed normal distribution for the log-response times holds, we test this assumption using the proposed methodology.

Using a significance level of α=.05 , Table 5 shows that especially in State 1, the fast state, the continuous response times for most of the 28 items do not follow a normal distribution. Furthermore, all three tests indicate that for State 0, the slow state, the response times for the majority of the items are normally distributed. However, since normality does not hold for the majority of the items in State 1, the parameter estimates are biased and states detected in the data can be spurious. Therefore, we cannot use a mixture model with a Markov dependency to interpret the responses and response times, and we need to fit an alternative model in order to explain the data.

7. Discussion

If the within-state response time distribution in the hierarchical mixture modeling framework is misspecified, parameter estimates and model fit indices will be biased and spurious states can be detected in the data as a result. Therefore, in this article, we proposed statistical tests for normality of the within-state log-response time distribution.

In a simulation study, we found that violations of nonnormality can successfully be detected using our tests based on the SW and KS tests. Most importantly, our test has demonstrated an acceptable Type I error rate, which indicates that our test can be used to successfully identify situations where normality holds, and where model fit indices like BIC and CAIC can successfully be used to test between models that include and do not include normal mixtures. If normality is violated, it cannot be concluded whether the violations of normality are due to within-state nonnormality or due to observed nonnormality. However, in such cases, normal mixture modeling should not be adopted anyway, and our test is shown to be a good indicator for such situations.

We also found that the weighted and unweighted KS tests had a slightly increased Type I error rate. In addition, the SW test was associated with larger power as compared to the KS tests. This is line with for instance Razali and Wah (2011), Stephens (1974), and Yap and Sim (2011) who all noted that the KS test in general tends to have smaller power than the SW test. In addition, Shapiro et al. (1968) furthermore showed that in the case of misspecifying the parameters of the hypothesized null distribution, the power and Type I error can be influenced. Type I error rates at the 5% significance level can increase to 61% for a sample size of 50, and the effect becomes more pronounced when sample size increases. Monahan (2011) on the other hand noted that since the KS tests are less powerful, it should not be used in small samples but could be used in larger samples. Although the SW test thus seems preferable over the KS test in the present study, the KS test is more flexible as it can be used to test any assumed distribution, where the SW test can only be used on distributions that can be transformed to a normal one.

As noted above, if in practice, normality is rejected, it is advisable to not interpret the modeling results since they are unreliable. An alternative in that case may be to categorize the response times and use a model for categorical variables as is shown in the application. As Molenaar et al. (2018) showed, such an approach hardly produces parameter bias and false positives with respect to states underlying the data. Furthermore, the approach is comparable to the parametric within-subject mixture modeling approach regarding power. A second solution would be to use a nonparametric or semi-parameter alternative for modeling the responses and response times. However, such an approach is yet to be developed. The semi-parametric models by C. Wang, Fan, et al. (2013) and C. Wang, Chang, and Douglas (2013) can possibly provide a point of departure.

In this article, we have assumed a Markov structure for the dependency between the states. However, the present approach to test for normality is equally amenable to mixture models without Markov structure (e.g., the model by C. Wang & Xu, 2015). In the simulation study, we did not find any important differences between the scenario's in which the data included a Markov structure and scenario's in which the data did not included a Markov structure. However, if the Markov dependency becomes stronger (i.e., higher transition probabilities), power for a Markov model may be larger.

Another aspect of the present approach is that of the fit of the mixture model under consideration. That is, we tested for normality by assuming a certain mixture model (in this case, a Markov-dependent item states model). We showed that minor departures from normality can be detected if the model is otherwise correctly specified. If the initial mixture model is misspecified, false positives may arise. The severity of this inflation will depend on the size of the misfit. We therefore think that, first, in practice, were-hopefully-a researcher has a well (theoretically) motivated model that is not severely misspecified, the consequences will not be large. Second, the consequences of this inflation do not have serious consequences as also discussed with respect to the baseline model above. That is, if the tests proposed in the present article are significant, the conclusion should be that the results of the mixture model cannot be trusted.

The normality tests presented in this article are all tests for univariate normality. Testing normality thus needs to be conducted on an item-by-item basis, so in practice, a correction for multiple testing is appropriate (e.g., a Bonferroni or Bonferroni-Holm correction). A focus of future research may be the development of an overall test for multivariate normality, which considers all items at once. Then, the present univariate tests can be used as post hoc tests to investigate whether individual items are responsible for violations of the assumed distribution.

Even though m and V in vector a can be computed using various algorithms (see Royston, 1982a, 1982b; also see Davies & Stephens, 1978; Shea & Scallan, 1988), Royston (1992, also see Verrill & Johnson, 1988) offers a close approximation for vector a which is based on the Weisberg and Bingham (1975) statistic ∑ b p x p 2/∑ x p−x¯ 2 , where b p=(m˜ T m˜ )−1 2 m˜p , m˜p=Φ−1{(p−3/8)/(N+1/4)} , and Φ is the normal cumulative distribution function. Only for the first two components, and the last two since a is antisymmetric, b differs from a. The value for the last two (and first two) components from a can, for 4≤N≤1,000 , be approximated by

a˜N=b N+0.221157 y−0.147981 y 2−2.071190 y 3+4.434685 y 4−2.706056 y 5,

A1

a˜ =b +0.042981 y−0.293762 y 2−1.752461 y 3+5.682633 y 4−3.582663 y 5,

A2

where y=N−1 2 . The remaining ap are approximated by

for when and when . In Equation A3, the m˜p are normalized by using

ϕ=(m˜ T m˜ −2)/(1−2)  if  N≤5,

A4

=(m˜ T m˜ −2−2 m˜2)/(1−2−2 a˜2)  if  N>5.

A5

After calculation of the approximated a, the Wi statistic can be computed using Equation 6. Royston (1992) showed that small values of Wi indicated nonnormality, therefore, Wi needs to be normalized. For 4≤N≤11 , the transformed Wi (denoted by w), the mean μ and standard deviation σ are defined by

μ=0.5440−0.39978 N+0.025054 N 2−0.0006714 N 3,

σ=exp(1.3822−0.77857 N+0.062767 N 2−0.0020322 N 3).

For 12≤N≤2,000 , they equal

μ=−1.5861−0.31082 x−0.083751 x 2+0.0038915 x 3,

σ=exp(−0.4803−0.082676 x+0.0030302 x 2).

As follows, the p value of Wi can be found by using z=(w−μ)/σ , which corresponds to the upper tail of N(0,1) if and to the lower tail if .

ORCID iD
Renske E. Kuijpers https://orcid.org/0000-0002-2268-3995

Ingmar Visser https://orcid.org/0000-0003-3855-2778

Notes

References

Amthauer, R., Brocke, B., Liepmann, D., Beauducel, A. (2001). Intelligenz-struktur-test 2000 R [Intelligence-structure-test 2000 R]. Hogrefe.
Google Scholar

Bozdogan, H. (1987). Model selection and Akaike's information criterion (AIC): The general theory and its analytical extensions. Psychometrika, 52, 345-370. | https://doi.org/10.1007/BF02294361
Google Scholar Crossref

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum.
Google Scholar

de Vries, J. (2017). IST: Intelligentie Structuur Test [Intelligence Structure Test] (HTS Report no. 5105-7035). Retrieved from Hogrefe Uitgevers B.V. website: https://www.hogrefe.nl/shop/media/downloads/sample-reports/5700601_ISTbasisrapport_mr.pdf
Google Scholar

Hogrefe Ltd . (2016). IST: Intelligence Structure Test: English version of the Intelligenz-Struktur-Test 2000 R (I-S-T 2000 R) (HTS Report no.13216-189). Retrieved from Hogrefe Ltd website: https://www.hogrefe.co.uk/shop/media/downloads/sample-reports/5535902_mr.pdf
Google Scholar

McLeod, L., Lewis, C., Thissen, D. (2003). A Bayesian method for the detection of item preknowledge in computerized adaptive testing. Applied Psychological Measurement, 27, 121-137. | https://doi.org/10.1177/0146621602250534
Google Scholar SAGE Journals

Monahan, J. F. (2011). Numerical methods of statistics (2nd ed.). Cambridge University Press. |
Google Scholar Crossref

Roskam, E. E. (1987). Toward a psychometric theory of intelligence. In Roskam, E. E., Suck, R. (Eds.), Progress in mathematical psychology (pp. 151-171). North Holland.
Google Scholar

Royston, J. P. (1982b). An extension of Shapiro and Wilk's | W test for normality to large samples. Applied Statistics, 31, 115-124.
Google Scholar Crossref

Yap, B. W., Sim, C. H. (2011). Comparisons of various types of normality tests. Journal of Statistical Computation and Simulation, 81(12), 2141-2155. https://doi.org/10.1080/00949655.2010.520163
Google Scholar

Authors

# data[].attributes.attachment_link.title
Testing the Within-State Distribution in Mixture Models for Responses and Response Times - Renske E. Kuijpers, Ingmar Visser, Dylan Molenaar, 2020

# data[].attributes.attachment_link.description
Abstract Mixture models have been developed to enable detection of within-subject differences in responses and response times to psychometric test items. To enable mixture modeling of both responses and response times, a distributional assumption is needed for the within-state response time distribution.

# data[].attributes.headline
8. Commentary on “Validation Methods for Aggregate-Level Test Scale Linking: A Case Study Mapping School District Test Score Distributions to a Common Scale”

# data[].attributes.subject
8. Commentary on “Validation Methods for Aggregate-Level Test Scale Linking: A Case Study Mapping School District Test Score Distributions to a Common Scale”

# data[].attributes.body
Interpretivism
The author used case study as research approaches, and the reseacher is involved.

# data[].attributes.attachment_link.readable_text
In this commentary, I share my perspective on the goals of assessments in general, on linking assessments that were developed according to different specifications and for different purposes, and I propose several considerations for the authors and the readers. This brief commentary is structured around three perspectives (1) the context of this research, (2) the methodology proposed here, and (3) the consequences for applied research.

References

Braun, H., Qian, J. (2007). An enhanced method for mapping state standards onto the NAEP scale. In Dorans, N. J., Pommerich, M., Holland, P. W. (Eds.), Linking and aligning scores and scales. Statistics for social and behavioral sciences (pp. 313-338). Springer.
Google Scholar

Feuer, M. J., Holland, P. W., Green, B. F., Bertenthal, M. W., Hemphill, F. C. (1999). Uncommon measures: Equivalence and linkage among educational tests. National Academy Press.
Google Scholar

Kell, H. J., Lang, J. (2018). The great debate: General ability and specific abilities in the prediction of important outcomes. Journal of Intelligence, 6(3), 39. https://doi.org/10.3390/jintelligence6030039
Google Scholar

Koretz, D. (2007). Using aggregate-level linkages for estimation and validation: Comments on Thissen and Braun & Qian. In Dorans, N. J., Pommerich, M., Holland, P. W. (Eds.), Linking and aligning scores and scales (pp. 339-353). Springer.
Google Scholar

Lockwood, J. R., Castellano, K. E., Shear, B. R. (2018). Flexible Bayesian models for inferences from coarsened, group-level achievement data. Journal of Educational and Behavioral Statistics, 43(6), 663-692. |
Google Scholar SAGE Journals

Savi, A. O., Marsman, M., van der Maas, H. L., Maris, G. K. (2019). The wiring of intelligence. Perspectives on Psychological Science, 14(6), 1034-1061. |
Google Scholar SAGE Journals

Thissen, D. (2007). Linking assessments based on aggregate reporting: Background and issues. In Dorans, N. J., Pommerich, M., Holland, P. W. (Eds.), Linking and aligning scores and scales (pp. 287-312). Springer.
Google Scholar

Thissen, D. (2012). Validity issues involved in cross-grade statements about NAEP results (NAEP Validity Studies Panel). National Center for Education Statistics.
Google Scholar

van der Maas, H. L. J., Dolan, C. V., Grasman, R. P. P. P., Wicherts, J. M., Huizenga, H. M., Raijmakers, M. E. J. (2006). A dynamical model of general intelligence: The positive manifold of intelligence by mutualism. Psychological Review, 113, 842-861. http://doi.org/10.1037/0033-295x.113.4.842
Google Scholar

von Davier, A. A., Deonovic, B., Yudelson, M., Polyak, S., Woo, A. (2019). Computational psychometrics approach for holistic learning and assessment systems. Frontiers in Education. https://www.frontiersin.org/articles/10.3389/feduc.2019.00069/full
Google Scholar

# data[].attributes.attachment_link.title
Commentary on "Validation Methods for Aggregate-Level Test Scale Linking: A Case Study Mapping School District Test Score Distributions to a Common Scale" - Alina A. von Davier, 2020

# data[].attributes.attachment_link.description
In this commentary, I share my perspective on the goals of assessments in general, on linking assessments that were developed according to different specifications and for different purposes, and I propose several considerations for the authors and the readers.

# data[].attributes.headline
9. A Class of Cognitive Diagnosis Models for Polytomous Data

# data[].attributes.subject
9. A Class of Cognitive Diagnosis Models for Polytomous Data

# data[].attributes.body
positivism
the author used quantitative methods to proposes a class of cognitive diagnosis models (CDMs) for polytomously scored items with different link functions.

# data[].attributes.attachment_link.readable_text
In this commentary, I share my perspective on the goals of assessments in general, on linking assessments that were developed according to different specifications and for different purposes, and I propose several considerations for the authors and the readers. This brief commentary is structured around three perspectives (1) the context of this research, (2) the methodology proposed here, and (3) the consequences for applied research.

References

Braun, H., Qian, J. (2007). An enhanced method for mapping state standards onto the NAEP scale. In Dorans, N. J., Pommerich, M., Holland, P. W. (Eds.), Linking and aligning scores and scales. Statistics for social and behavioral sciences (pp. 313-338). Springer.
Google Scholar

Feuer, M. J., Holland, P. W., Green, B. F., Bertenthal, M. W., Hemphill, F. C. (1999). Uncommon measures: Equivalence and linkage among educational tests. National Academy Press.
Google Scholar

Kell, H. J., Lang, J. (2018). The great debate: General ability and specific abilities in the prediction of important outcomes. Journal of Intelligence, 6(3), 39. https://doi.org/10.3390/jintelligence6030039
Google Scholar

Koretz, D. (2007). Using aggregate-level linkages for estimation and validation: Comments on Thissen and Braun & Qian. In Dorans, N. J., Pommerich, M., Holland, P. W. (Eds.), Linking and aligning scores and scales (pp. 339-353). Springer.
Google Scholar

Lockwood, J. R., Castellano, K. E., Shear, B. R. (2018). Flexible Bayesian models for inferences from coarsened, group-level achievement data. Journal of Educational and Behavioral Statistics, 43(6), 663-692. |
Google Scholar SAGE Journals

Savi, A. O., Marsman, M., van der Maas, H. L., Maris, G. K. (2019). The wiring of intelligence. Perspectives on Psychological Science, 14(6), 1034-1061. |
Google Scholar SAGE Journals

Thissen, D. (2007). Linking assessments based on aggregate reporting: Background and issues. In Dorans, N. J., Pommerich, M., Holland, P. W. (Eds.), Linking and aligning scores and scales (pp. 287-312). Springer.
Google Scholar

Thissen, D. (2012). Validity issues involved in cross-grade statements about NAEP results (NAEP Validity Studies Panel). National Center for Education Statistics.
Google Scholar

van der Maas, H. L. J., Dolan, C. V., Grasman, R. P. P. P., Wicherts, J. M., Huizenga, H. M., Raijmakers, M. E. J. (2006). A dynamical model of general intelligence: The positive manifold of intelligence by mutualism. Psychological Review, 113, 842-861. http://doi.org/10.1037/0033-295x.113.4.842
Google Scholar

von Davier, A. A., Deonovic, B., Yudelson, M., Polyak, S., Woo, A. (2019). Computational psychometrics approach for holistic learning and assessment systems. Frontiers in Education. https://www.frontiersin.org/articles/10.3389/feduc.2019.00069/full
Google Scholar

# data[].attributes.attachment_link.title
Commentary on "Validation Methods for Aggregate-Level Test Scale Linking: A Case Study Mapping School District Test Score Distributions to a Common Scale" - Alina A. von Davier, 2020

# data[].attributes.attachment_link.description
In this commentary, I share my perspective on the goals of assessments in general, on linking assessments that were developed according to different specifications and for different purposes, and I propose several considerations for the authors and the readers.

# data[].attributes.headline
10. Testing Latent Variable Distribution Fit in IRT Using Posterior Residuals

# data[].attributes.subject
10. Testing Latent Variable Distribution Fit in IRT Using Posterior Residuals

# data[].attributes.body
positivism:
it enable detection of within-subject differences in responses and response times to psychometric test items

# data[].attributes.attachment_link.readable_text
Abstract

Mixture models have been developed to enable detection of within-subject differences in responses and response times to psychometric test items. To enable mixture modeling of both responses and response times, a distributional assumption is needed for the within-state response time distribution. Since violations of the assumed response time distribution may bias the modeling results, choosing an appropriate within-state distribution is important. However, testing this distributional assumption is challenging as the latent within-state response time distribution is by definition different from the observed distribution. Therefore, existing tests on the observed distribution cannot be used. In this article, we propose statistical tests on the within-state response time distribution in a mixture modeling framework for responses and response times. We investigate the viability of the newly proposed tests in a simulation study, and we apply the test to a real data set.

1. Introduction

Recently, research interest has grown in modeling response times next to the item responses in order to investigate individual differences in ability and speed. Focusing on the item response times in addition to the item responses has facilitated various aspects of psychological testing including, for instance, item selection in computerized adaptive testing ( van der Linden et al., 1999; Veldkamp, 2016), test design ( van der Linden, 2007), and item calibration (T. Wang & Hanson, 2005). In addition, response times have been shown useful in detecting item preknowledge ( McLeod et al., 2003), aberrant response patterns ( Marianti et al., 2014; van der Linden & Guo, 2008; C. Wang, Xu, Shang, & Kuncel, 2018), and individual differences in the use of solution strategies. For instance, van der Maas and Jansen (2003) showed that response times can give detailed information on the type and duration of different solution strategies children use to solve a balance scale task. Suitable models to enable these inferences concerning individual differences in responses and response times include the model by Roskam (1987) and more recently the hierarchical model by van der Linden (2007, 2009), which was elaborated by Molenaar et al. (2015a).

Besides these applications of response times to individual differences research, response times have been used to facilitate the study of within-subject differences in solution strategies or psychological processes that underlie the responses to psychometric tests and questionnaires. For instance, response times have been used to identify fast guessing (e.g., Schnipke & Scrams, 1997) and within-subject differences in solution strategies (e.g., Molenaar et al., 2016). Other applications include the study of within-subject differences in motivation ( Wise & Kong, 2005) and faking on personality test items ( Holden & Kroner, 1992).

To facilitate the detection of within-subject differences in responses and response times, various approaches based on mixture modeling have been proposed. For instance, the earliest contribution by Schnipke and Scrams (1997) focused on a two-state within-subject mixture model for the response times only. Here, one state represented rapid-guessing behavior of examinees and the other state modeled the responses of examinees who actually tried to solve the item (i.e., a regular response process). The model by Schnipke and Scrams did not include a latent speed variable but can be seen as one of the first models within this framework. In their model, the mean and variance of the response times are estimated freely for each item in the regular process state, while in the rapid-guessing state, a common mean and variance parameter is assumed to underly the items. On the basis of this model, C. Wang and Xu (2015) and C. Wang, Xu and Shang (2018) proposed a mixture model in which separate measurement models are proposed for modeling the responses and response times in the slower response state, whereas for the faster guessing state, only a guessing parameter is estimated for the responses, and a mean and variance parameter is estimated for the response times. Molenaar et al. (2016) generalized this approach by proposing a mixture model that specifies a measurement model for the responses and a measurement model for the response times separately in each state.

Inspired by the aforementioned models, the hierarchical mixture modeling approach ( Molenaar et al., 2016; Schnipke & Scrams, 1997; C. Wang & Xu, 2015; C. Wang, Xu, & Shang, 2018; C. Wang, Xu, Shang, & Kuncel, 2018) that we focus on in this article is a mixture extension of the hierarchical model by van der Linden (2007, 2009) to allow for within-subject differences in ability and speed. In the van der Linden model, between-subject differences in ability level are captured by means of a continuous random latent ability variable , which underlies the item responses of respondent p=1, . . . ,N to item i=1, . . . ,J . Individual differences with respect to the speed with which the responses to the test items are given are modeled by a continuous random latent speed variable , which underlies the response times. In contrast to the van der Linden model, which assumes speed and ability to be constant within subjects, the hierarchical mixture modeling approach allows speed and ability to differ within subjects. Therefore, an item-specific latent class variable is assumed to underlie the response and the response time of person p to item i. Although the latent class variable can have K states in principle, it is commonly assumed to have two states, K=2 : In one state, the item properties of the faster responses are modeled and in the other state the item properties of the slower responses are modeled. Each respondent is allowed to switch between the slow and fast response state from item to item, such that within-subject differences are captured by the latent class variables.

To enable mixture modeling of both the responses and response times, a distributional assumption is needed for the within-state response time distribution. Correct specification of this distribution is important as it has been shown that violations of the assumed response time distribution may bias modeling results for mixture models in general ( Vermunt, 2011), for growth mixture models ( Bauer & Curran, 2003), and for the hierarchical mixture modeling framework for responses and response times as discussed above ( Molenaar et al., 2018). More specifically, Molenaar et al. (2018) showed that if the observed response time distribution differs from the assumed distribution, within-state parameter estimates and information criteria like Akaike's Information Criterion (AIC; Akaike, 1974), the Bayesian Information Criterion (BIC; Schwarz, 1978), and the consistent AIC (CAIC; Bozdogan, 1987) will be biased, and spurious states will be detected even if there are no states underlying the data. Thus, specifying an appropriate within-state response time distribution is important. In practice, however, it is commonly unknown which type of distribution would fit the within-state response times best. Often, statistically convenient distributions are chosen, for example, the log-normal, the exponential, or the chi-square distribution. These distributions are considered convenient as respectively the logarithmic, the reciprocal, and the square root transformation will result in normally distributed response times. Once a distribution is chosen, this assumed distribution should ideally be tested to ensure that the mixture modeling results are valid. However, testing this distributional assumption is challenging as the within-state distribution is by definition different from the observed distribution since the latter is aggregated over states. That is, if the within-state log-transformed response time distribution is normal, the observed log-response time distribution will be skewed (assuming that the two states differ in their expected log-response time and their log-response time variance). As a result, it is not clear whether skewness in the observed log-transformed distribution reflects a mixture of two states or a misspecification of the response time distribution ( Molenaar et al., 2018). Therefore, traditional statistical tests (e.g., the Shapiro-Wilk test [SW test], 1965) on the observed response time distribution cannot be used.

In this article, we propose statistical tests on the within-state response time distribution in the hierarchical mixture modeling framework for responses and response times. Specifically, we propose tests on normality of the transformed response time distribution using the SW test ( Shapiro & Wilk, 1965; see also Royston, 1982a, 1982b, 1992) and the Kolmogorov-Smirnov test (KS test; Kolmogorov, 1933; Smirnov, 1948). Although the SW test and the KS test are well-established methods to test a hypothesized distribution to hold for an observed variable, the innovative aspect of the present study is that we apply these tests to investigate a hypothesized distribution to hold for the latent within-state distribution of the hierarchical mixture modeling framework for responses and response times. We focus on the log-transformation for the response times, making our approach a test on log-normality. We prefer to focus on log-normality as this is the most commonly used assumption in mixture modeling of response times. However, the proposed methodology can readily be used to test for other distributions by using a different response time transformation. That is, one can consider for instance the square root transformation to test for a chi-square distribution and the reciprocal transformation to test for an exponential distribution. In addition, for the KS test, it is straightforward to accommodate any other distribution (e.g., the Weibull, ex-Gaussian, or Wald distribution) as long as its cumulative distribution function exists and can be evaluated.

The proposed normality tests can be used for various types of (response time) mixture models; however, in this study, we apply the tests to the Markov-dependent item states model and the independent item states model ( Molenaar et al., 2016). This article is organized as follows: First, we discuss the hierarchical mixture modeling approach with log-response times. Next, we present the normality tests for the within-state response time distribution. We then present a simulation study to investigate the performance of the different normality tests. In addition, we illustrate the use of the tests by means of a real data application, and we end with a general discussion.

2. The Hierarchical Mixture Modeling Approach

In the hierarchical mixture modeling approach ( Molenaar et al., 2016; Schnipke & Scrams, 1997; C. Wang & Xu, 2015; C. Wang, Xu, & Shang, 2018), an item-specific latent class variable is assumed to underlie the response and the response time of person p to item i. The levels of this latent class variable are referred to as item states. The item states for respondent p on the items of a test can be collected in vector =[,, . . . ,] . In addition, the item responses are collected in vector =[,, . . . ,] , and the log-transformed response times are collected in vector =[,, . . . ,] .

Although the latent class variable can have multiple states in principle, it is commonly assumed to have two states: a state capturing the measurement properties of the slower responses and a state capturing the measurement properties of the faster responses. The slower state, with a larger expected response time, is denoted by =0 , the faster state by =1 . As discussed above, the response times are assumed to follow a log-normal distribution in each state, such that the log-response times follow a normal distribution (e.g., C. Wang & Xu, 2015). In addition, the item responses and the log-response times are assumed to be independent conditional on the overall latent ability and the overall speed within the states .

In order to separate the effects of the item, the person and the latent class variable , a measurement model for the responses and a measurement model for the response times, is specified. Various models can be considered for modeling the responses, like the Rasch model ( Loeys et al., 2014), the graded response model (e.g., Molenaar et al., 2015b; Ranger & Ortner, 2011), the two-parameter logistic model (e.g., Molenaar et al., 2015a, 2015b; Ranger & Ortner, 2012), or the three-parameter model (e.g., van der Linden, 2007). Here, we specify a two-parameter logistic model to model the item responses, that is

P(|θ p,)=∏N ω(×θ p+) ×ω−[α k i×θ p+β k i] 1−,

1

where ω(.) denotes the logistic function, and α k i and β k i denote the discrimination and easiness parameters for item i and state k. Note that the item response parameters are allowed to differ across states; however, they are treated as fixed parameters within each state.

Although other models have been proposed as well, a normal one-factor model is commonly assumed for modeling the log-response times ( van der Linden, 2007). Here, we thus assume the log-response times to follow a conditional multivariate normal distribution. As follows, f(|τ p,θ p,) models a person's response times, given their ability, latent states, and overall speed by means of

f(|τ p,θ p,)=∏N 1 2 π σ ε i 2 ×exp−1 2(T p i−μ p i)2 σ ε i 2 ,

2

with

=E(|τ p,θ p,)=v i−δ−τ p,     δ>0,

3

where σ ε i 2 is the residual log-response time variance, vi is the time intensity parameter, and δ denotes the difference in expected speed between the slower and faster state. Here, the constraint δ>0 ensures that =1 corresponds to the faster state, which is the state with the smaller response times.

Next, by assuming a bivariate normal distribution for the latent ability variable and the latent speed variable , the general log marginal likelihood of item response vector and the log-response time vector , given the model parameter vector η, is given by

ℓ(,; η)=ln∫∫−∞ ∑C p 1 ∑C p 2 ...∑C p J P(|θ p,)×f(|τ p,θ p,)P()×g(θ p,τ p)d θ d τ,

4

where g(.) denotes the bivariate normal density function. P() denotes the state probabilities for a set of items, which, for each particular item, models the probability that the response of person p to item i belongs to a given state. As follows, P()=P(C p 1),P(C p 2), . . . ,P(C p J) . P(|θ p,) thus models the probability of obtaining a particular response pattern, given a person's ability and their latent states for the different items, and f(|τ p,θ p,) models a person's response times.

In the hierarchical mixture modeling framework, the item-specific latent class variables are commonly assumed to be independent from item to item. However, in practice, these latent class variables may be dependent, for instance, if a respondent guesses on one item, they may be more likely to guess on the next item. To account for such a possible dependency between the latent class variables in the hierarchical mixture framework, Molenaar et al. (2016) also considered a model with a time homogenous first-order Markov-structure on the item-specific latent class variables. As a result, in this model, P() , the state a person is in regarding a certain item, depends on the state of the previous item. That is, the state probability P() is decomposed as follows:

where P() is referred to as the initial state probability and P(|C p(i−1)) is the so-called transition probability. Since the number of states in our model equals two, there are two initial state probabilities. In addition, as the Markov dependency is assumed to be time homogenous, the transition probabilities are assumed equal for all subsequent items, resulting in four transition probabilities. In model estimation, only one initial state probability and two transition probabilities have to be estimated as the others follow from these estimates (see Figure 1 for a graphical representation of the model).

3. Normality Tests

In the model discussed above, the log-response times within each state are assumed to follow a normal distribution. To test this assumption, we use the SW test and the KS test. Specifically, we propose the following procedure: First, the hierarchical mixture model is fit to the responses and response times. Next, the resulting posterior state probabilities are obtained for each response which in turn are used to draw posterior state assignments to state 0 or 1 for each person's response to each item. Then the normality tests are conducted on either (1) the response times in state 0 and state 1 according to the posterior state assignment or (2) on the response times weighted by the posterior state probabilities.

The rationale for the above procedure is that if the within-state log-response time distribution is correctly specified, the resulting posterior state probabilities and posterior state assignments are correct. As a result, the SW and KS test statistics will follow their theoretical distribution under the null hypothesis of normality. However, if the within-state log-response time distribution is incorrectly specified, the resulting posterior state probabilities and posterior state assignments are wrong, and the SW and KS test statistics will not follow their null distributions. Below we discuss the SW and KS tests and apply them to the mixture modeling framework.

The SW test, also called the W test for normality, tests the null hypothesis that an observed variable comes from a normally distributed population. Since the original test as proposed by Shapiro and Wilk (1965) could not be used for sample sizes larger than 50, Royston (1982b, 1992) extended it to sample sizes up to 2,000. Then, suppose that x 1<x 2< ....<x N is an ordered sample of size N on which the normality test is carried out. The W statistic, as calculated for an item , is defined as

where a=a 1,a 2,...,a N are the normalized best linear unbiased coefficients, and p=1,...,N . Vector a is defined by

where m =m 1, . . . ,m N is a vector of expected values of standard normal order statistics, and V is the corresponding covariance matrix. Note that vector a is antisymmetric, that is, a N=−a 1 , and for odd N, a(N/2)+1=0 ( Royston, 1992). More details on the calculation of the Wi statistic can be found in the Appendix.

The KS test is a nonparametric goodness of fit test, which measures the distance between an empirical distribution function of a sample and a hypothesized cumulative distribution function (the one-sample KS test) or which compares the distribution functions of two samples (the two-sample KS test). Here, we will focus on the one-sample KS test for testing the within-state response time distribution of item i for normality, which is defined by

where F N(x) is the empirical distribution function, and F(x) the reference distribution. In this article, we assume a normal distribution for the log-response times; however, other distributions like the Wald distribution can be used as a reference distribution as well. Therefore, the KS test can not only be used to test the within-state response time distribution for normality but can be used to test for all kinds of distributions. As Monahan (2011) noted, the Kolmogorov-Smirnov statistic is distribution free for continuous random variables. As a result, the distribution function is evaluated at the observations, and then sorted, such that

D N i=max p p/N−F(X(p)),F(X(p))−(p−1)/N ,

9

where Xp denotes the response time of respondent p on item i, with ; and X(1)≤ . . . ≤X(p)≤ . . . ≤X(N) denote the ordered response times of a sample of respondents. The accompanying p value for the statistic needs to be bootstrapped, since the resulting sampling distribution is unknown, and therefore, the mean and standard deviation are unknown and thus must be estimated from the data when testing for normality ( Lilliefors, 1967).1 When testing for other types of distributions, estimating only the mean and standard deviation might not be sufficient, and additional parameters might need to be estimated (e.g., a skewness parameter).

Since the posterior state probabilities differ for each respondent on each item they are taken into account by including them as weights in the KS test, Equation 9 needs to be modified to account for the weighted response times ( Monahan, 2011). The empirical distribution function can be estimated by

where P(|,) are the posterior state probabilities, and I(X p≤x) is an indicator function, which equals 1 if X p≤x and 0 otherwise. Then, the Xp must be sorted together with the corresponding posterior state probabilities, the increments can be computed by

As follows, F N(x)=F k for X k−1<x≤X k , which can be calculated by estimating quantiles for the underlying normal distribution, weighted by the posterior state probabilities, that correspond to N ordered probabilities. Using Fk from Equation 11, the Kolmogorov-Smirnov statistic for item i modifies to

D N i=sup x|F N(x)−F(x)|=max k F k−F(X(k)),F(X(k))−F k−1 .

12

Like in the unweighted case, the p value of needs to be bootstrapped, since the mean and standard deviation of the log-response time distribution are estimated from the data ( Lilliefors, 1967).

4. Simulation Study

In the simulation study, we compared the performance of the unweighted SW test, the unweighted KS test, and the weighted KS test. Data are simulated according to nine different scenarios, mostly based on Molenaar et al. (2018) who found biased modeling results of the hierarchical mixture model in the case of nonnormality. The first three scenarios concern Markov mixture models that include Markov-dependent item states, the next three scenarios concern mixture models with independent item states, and the final three scenarios are generated according to a baseline model that does not include item states (i.e., a static model without mixtures). The scenarios differ in the distribution that is used for the log-response times, which is a normal, a truncated, or a skewed distribution. The responses are modeled using the two-parameter logistic model, with item parameters α k i and β k i , which denote the discrimination and easiness of an item i in state k, respectively. For each scenario, we conducted 50 replications with 20 items, the sample size was equal to 500 or 1,000.

The three Markov mixture model scenarios are the following:

Normal Markov mixture: In this scenario, we use the Markov-dependent item states model with normal log-response times to simulate the data. We use δ=.1 . That is, the expected log-response times differ by 0.1 between the fast and slow state. The item parameters for the two different states are set as follows: the discrimination parameters are set to α 0 i=1.5 for State 0 (slow state) and to α 1 i=1 for State 1 (fast state). We set the easiness parameters to increasing, equally spaced values: for the slower state, β 0 i is between −2 and 0, for the faster state β 1 i is between 0 and 2. The response time parameters for all items i are chosen as v i=2 , and the residual response time variances as σ ε i 2=0.13 . Furthermore, σ τ 2=0.13 and σ θτ 2=0.14 , so that the correlation between θ p and τ p equals 0.4. Finally, the initial state probability, P(C p 1=1) , is fixed to 0.5, and the transition probabilities, P(=1|C p(i−1)=0) and P(=0|C p(i−1)=1) , are fixed to .231, which corresponds to mildly instable states according to Bacci et al. (2014).

Truncated Markov mixture: In this scenario, the data are generated using the same parameter values as in the normal Markov mixture scenario above. However, now we use a right-truncated normal distribution for the log-response times, with truncation at ln(12) such that the response times are cut off at 12 seconds. This scenario mirrors data from time-pressured tests, where respondents only have a limited amount of time to answer the items.

Skewed mixture: Here, the data are generated using the same parameter values as in the normal Markov mixture scenario. However, the normally distributed log-response times are transformed using a Box-Cox transformation ( Box & Cox, 1964). In general, the transformation is used to transform skewed variables in such a way that they are closer to a normal distribution. Here, we use the transformation the other way around, so that we transform the normally distributed log-response times into skewed variables using the Box-Cox transformation. That is, we transform the normally distributed log-response times using ln()′=(λ ln()+1)λ with λ=.3 .

Figure 2 shows the resulting log-response time distribution of an arbitrary item from an example run of the three different Markov mixture model scenarios. Note that in the truncated scenario, the log-response times are negatively skewed, while in the skewed scenario, the log-response times are positively skewed.

In the three independent mixture scenarios, we used the same parameter values and setup as for the Markov mixture scenarios above; however, the item states are assumed to be independent, that is, the Markov structure is omitted (i.e., P(=1|C p(i−1)=0)=P(=1) and P(=0|C p(i−1)=1)=P(=0) ). Next, for the three baseline scenarios, the data do not include item states, and a baseline model (i.e., the traditional hierarchical model without mixtures by van der Linden, 2007) is used to generate the data. Like the mixture models, the scenarios differ in the distribution used for the log-response times, which again are either a normal, a truncated or a skewed distribution:

Normal baseline: In this scenario, the log-response times are normally distributed. The item parameters are as follows: The discrimination parameters for all items are set to α i=1 , the easiness parameters β i are set to increasing, equally spaced values between −2 and 2. The response time parameters vi, σ ε i 2 , σ τ 2 , and σ θτ 2 are set to the same values as in the normal mixture model scenario.

Truncated baseline: Here, the data are generated using the same parameter values as in the normal baseline scenario. Like in the truncated mixture model scenario, we use a right-truncated normal distribution for the log-response times, with truncation at ln(12) , such that the response times are cut off at 12 seconds.

Skewed baseline: Here, the data are generated using the same parameter values as in the normal baseline scenario. However, like the skewed mixture model scenario, the normally distributed log-response times are transformed using a reverse Box-Cox transformation ( Box & Cox, 1964), with the same value for λ.

5. Results

For the truncated scenarios, convergence problems occurred in 17 replications (4 in the truncated Markov mixture, 9 in the truncated independent mixture, and 4 in the truncated baseline scenario). In the results below, those replications are excluded. Tables 1, 2, and 3 contain the results for the normal, truncated, and skewed scenarios, respectively. Specifically, the tables depict the proportion of items for which the null hypothesis of normality is rejected at a .05 level of significance, averaged over all items within a state. That is, a proportion of .900 for a given state indicates that-averaged over the replications-the normality hypothesis is rejected for 90% of all items within that state.

First, for the normal scenarios in Table 1, these proportions indicate the Type I error rate of our approach. Ideally, these rates are close to the level of significance for the tests to be viable. As can be seen from the table, for the Markov mixture, the independent mixture, and the baseline scenarios, the results are similar. That is, the SW test, and the unweighted KS test have acceptable Type I error rates. In addition, the weighted KS test is associated with an inflated Type I error rate.

For the truncated and skewed scenarios, the mean proportion of normality rejections reflects the power to detect within-state departures from normality. Here, we used Cohen's (1988, p. 56) rule of thumb, considering a power coefficient of .80 or higher to be acceptable. As can be seen from Tables 2 and 3, the results indicate that generally the power is acceptable in one state and substantially smaller in the other state. In the truncated scenario, State 0 is associated with larger power, while in the skewed scenario, State 1 is associated with larger power as compared to State 0. This can be explained from Table 4 which contains the average initial state parameter estimates for State 1, together with the transition parameters in the different scenarios (i.e., the table reflects the proportions of persons in the different states). As can be seen from the mean initial state parameter estimates for State 1, in the truncated scenario, State 0 (slower state; larger log-response times) is the larger state, and in the skewed scenario, State 1 (faster state; smaller log-response times) is the larger state. This is due to the log-response times being positively skewed in the skewed scenario and negatively skewed in the truncated scenario (as mentioned above). As a result, due to these larger sample sizes in State 0 for the truncated scenarios and State 1 for the skewed scenarios, power differs between the two states. That is, when fitting a normal mixture to nonnormal data, the nonnormality is best detected in the largest state. Furthermore, even though class sizes are comparable, power tends to be larger in the larger state of the truncated scenarios when compared to the larger state of the skewed scenarios. This is due to the fact that the truncated distribution departs more from normality than the skewed distribution (see Figure 2). In general, the power of the KS tests (weighted and unweighted) is smaller as compared to the SW test. We return to this point in the discussion. Furthermore, the weighted KS test has slightly more power as compared to the unweighted KS test (but is also associated with an increased Type I error rate, see above).

Taken together the above, Type I error rate and the power of the proposed tests seem acceptable for the SW test and the unweighted KS test with more power for the SW test. The weighted KS test is associated with an inflated Type I error rate. There are no important differences between the Markov mixtures, independent mixtures, and baseline model. It turns out that, generally, violations of normality are only detected in one of the states. We note that of course the power of our approach depends on the severity of the normality violations (this is why the power seems somewhat larger in the truncated scenario as compared to the skewed scenario: the data in the truncated scenario is heavier skewed). In that sense, we consider our simulation study as a prove of principle (i.e., given the effect size we have chosen, we demonstrated that the approach is viable).

Our results indicate that nonnormality is detected if the data contain nonnormal mixtures (i.e., the Markov mixture and independent mixture scenarios) or if the data are nonnormal without mixtures (i.e., the baseline scenarios). In practice, where one does not know the data generating model, a significant normality test thus indicates that (1) the data follow a mixture model with a nonnormal within-state distribution or (2) the data are nonnormal but do not contain mixtures. For the present purposes, the distinction between (1) and (2) is not of importance as the implications are the same: In both cases, there is no mixture of normal distributions in the data, so the results of a normal (Markov-)mixture model should not be trusted. If our proposed tests are insignificant in both states, it can safely be concluded that (1) the data follow a mixture model with a normal within-state distribution or (2) the data are normal without mixtures (i.e., the baseline scenario's). As in both cases, the (within-state) data are normal, (1) and (2) can be distinguished by comparing the baseline model and the mixture models using common information criteria (e.g., BIC and CAIC) as demonstrated by Molenaar et al. (2018). Therefore, we propose the procedure summarized in the flow chart in Figure 3. That is, first, the fit of a normal (Markov-)mixture model is compared to that of a normal baseline model. If the baseline model fits better, it can be concluded that the transformed response times are normally distributed and that there is no mixture of normal distributions underlying the data. If the mixture model fits better, one can consult the statistics proposed in this article. If these statistics are insignificant in both states, it can be concluded that there is a true mixture of normal distributions underlying the response time data, and the results of the mixture model can be validly interpreted. However, if the proposed statistics are significant, it can be concluded that there is no mixture of normal distributions underlying the data, and the results of the normal (Markov-)mixture model cannot be trusted.

6. Application

The within-state normality tests are illustrated by means of a real-data application. The data consist of the responses and response times of 389 psychology freshman of the University of Amsterdam to 28 items of the knowledge subtest of the Dutch version of the Intelligence Structure Test ( Amthauer et al., 2001). The knowledge subtest measures essential types of knowledge, which people acquire in schools, higher education and other educational institutions, as well as daily life knowledge acquired from within their culture ( Hogrefe Ltd., 2016). The items in the subtest cover a broad range of topics, like economics, geography, mathematics, history, art and culture, natural sciences, and daily life facts ( de Vries, 2017). The items are dichotomously scored, with 0 indicating a false response and 1 indicating a correct response. Looking more closely at the response time distributions, Figures 4 and 5 show for a selection of items that they are not normally distributed. The observed response time distributions seem skewed, and the question is whether this nonnormality can be explained by a mixture of a fast and a slow state or whether there is an alternative explanation.

In the paper by Molenaar et al. (2016) mixture models with various types of Markov dependencies are fitted to the data and are shown to fit better than a baseline model without item states. However, as the modeling results of a fitting mixture model with a Markov dependency are only interpretable if the assumed normal distribution for the log-response times holds, we test this assumption using the proposed methodology.

Using a significance level of α=.05 , Table 5 shows that especially in State 1, the fast state, the continuous response times for most of the 28 items do not follow a normal distribution. Furthermore, all three tests indicate that for State 0, the slow state, the response times for the majority of the items are normally distributed. However, since normality does not hold for the majority of the items in State 1, the parameter estimates are biased and states detected in the data can be spurious. Therefore, we cannot use a mixture model with a Markov dependency to interpret the responses and response times, and we need to fit an alternative model in order to explain the data.

7. Discussion

If the within-state response time distribution in the hierarchical mixture modeling framework is misspecified, parameter estimates and model fit indices will be biased and spurious states can be detected in the data as a result. Therefore, in this article, we proposed statistical tests for normality of the within-state log-response time distribution.

In a simulation study, we found that violations of nonnormality can successfully be detected using our tests based on the SW and KS tests. Most importantly, our test has demonstrated an acceptable Type I error rate, which indicates that our test can be used to successfully identify situations where normality holds, and where model fit indices like BIC and CAIC can successfully be used to test between models that include and do not include normal mixtures. If normality is violated, it cannot be concluded whether the violations of normality are due to within-state nonnormality or due to observed nonnormality. However, in such cases, normal mixture modeling should not be adopted anyway, and our test is shown to be a good indicator for such situations.

We also found that the weighted and unweighted KS tests had a slightly increased Type I error rate. In addition, the SW test was associated with larger power as compared to the KS tests. This is line with for instance Razali and Wah (2011), Stephens (1974), and Yap and Sim (2011) who all noted that the KS test in general tends to have smaller power than the SW test. In addition, Shapiro et al. (1968) furthermore showed that in the case of misspecifying the parameters of the hypothesized null distribution, the power and Type I error can be influenced. Type I error rates at the 5% significance level can increase to 61% for a sample size of 50, and the effect becomes more pronounced when sample size increases. Monahan (2011) on the other hand noted that since the KS tests are less powerful, it should not be used in small samples but could be used in larger samples. Although the SW test thus seems preferable over the KS test in the present study, the KS test is more flexible as it can be used to test any assumed distribution, where the SW test can only be used on distributions that can be transformed to a normal one.

As noted above, if in practice, normality is rejected, it is advisable to not interpret the modeling results since they are unreliable. An alternative in that case may be to categorize the response times and use a model for categorical variables as is shown in the application. As Molenaar et al. (2018) showed, such an approach hardly produces parameter bias and false positives with respect to states underlying the data. Furthermore, the approach is comparable to the parametric within-subject mixture modeling approach regarding power. A second solution would be to use a nonparametric or semi-parameter alternative for modeling the responses and response times. However, such an approach is yet to be developed. The semi-parametric models by C. Wang, Fan, et al. (2013) and C. Wang, Chang, and Douglas (2013) can possibly provide a point of departure.

In this article, we have assumed a Markov structure for the dependency between the states. However, the present approach to test for normality is equally amenable to mixture models without Markov structure (e.g., the model by C. Wang & Xu, 2015). In the simulation study, we did not find any important differences between the scenario's in which the data included a Markov structure and scenario's in which the data did not included a Markov structure. However, if the Markov dependency becomes stronger (i.e., higher transition probabilities), power for a Markov model may be larger.

Another aspect of the present approach is that of the fit of the mixture model under consideration. That is, we tested for normality by assuming a certain mixture model (in this case, a Markov-dependent item states model). We showed that minor departures from normality can be detected if the model is otherwise correctly specified. If the initial mixture model is misspecified, false positives may arise. The severity of this inflation will depend on the size of the misfit. We therefore think that, first, in practice, were-hopefully-a researcher has a well (theoretically) motivated model that is not severely misspecified, the consequences will not be large. Second, the consequences of this inflation do not have serious consequences as also discussed with respect to the baseline model above. That is, if the tests proposed in the present article are significant, the conclusion should be that the results of the mixture model cannot be trusted.

The normality tests presented in this article are all tests for univariate normality. Testing normality thus needs to be conducted on an item-by-item basis, so in practice, a correction for multiple testing is appropriate (e.g., a Bonferroni or Bonferroni-Holm correction). A focus of future research may be the development of an overall test for multivariate normality, which considers all items at once. Then, the present univariate tests can be used as post hoc tests to investigate whether individual items are responsible for violations of the assumed distribution.

Even though m and V in vector a can be computed using various algorithms (see Royston, 1982a, 1982b; also see Davies & Stephens, 1978; Shea & Scallan, 1988), Royston (1992, also see Verrill & Johnson, 1988) offers a close approximation for vector a which is based on the Weisberg and Bingham (1975) statistic ∑ b p x p 2/∑ x p−x¯ 2 , where b p=(m˜ T m˜ )−1 2 m˜p , m˜p=Φ−1{(p−3/8)/(N+1/4)} , and Φ is the normal cumulative distribution function. Only for the first two components, and the last two since a is antisymmetric, b differs from a. The value for the last two (and first two) components from a can, for 4≤N≤1,000 , be approximated by

a˜N=b N+0.221157 y−0.147981 y 2−2.071190 y 3+4.434685 y 4−2.706056 y 5,

A1

a˜ =b +0.042981 y−0.293762 y 2−1.752461 y 3+5.682633 y 4−3.582663 y 5,

A2

where y=N−1 2 . The remaining ap are approximated by

for when and when . In Equation A3, the m˜p are normalized by using

ϕ=(m˜ T m˜ −2)/(1−2)  if  N≤5,

A4

=(m˜ T m˜ −2−2 m˜2)/(1−2−2 a˜2)  if  N>5.

A5

After calculation of the approximated a, the Wi statistic can be computed using Equation 6. Royston (1992) showed that small values of Wi indicated nonnormality, therefore, Wi needs to be normalized. For 4≤N≤11 , the transformed Wi (denoted by w), the mean μ and standard deviation σ are defined by

μ=0.5440−0.39978 N+0.025054 N 2−0.0006714 N 3,

σ=exp(1.3822−0.77857 N+0.062767 N 2−0.0020322 N 3).

For 12≤N≤2,000 , they equal

μ=−1.5861−0.31082 x−0.083751 x 2+0.0038915 x 3,

σ=exp(−0.4803−0.082676 x+0.0030302 x 2).

As follows, the p value of Wi can be found by using z=(w−μ)/σ , which corresponds to the upper tail of N(0,1) if and to the lower tail if .

ORCID iD
Renske E. Kuijpers https://orcid.org/0000-0002-2268-3995

Ingmar Visser https://orcid.org/0000-0003-3855-2778

Notes

References

Amthauer, R., Brocke, B., Liepmann, D., Beauducel, A. (2001). Intelligenz-struktur-test 2000 R [Intelligence-structure-test 2000 R]. Hogrefe.
Google Scholar

Bozdogan, H. (1987). Model selection and Akaike's information criterion (AIC): The general theory and its analytical extensions. Psychometrika, 52, 345-370. | https://doi.org/10.1007/BF02294361
Google Scholar Crossref

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum.
Google Scholar

de Vries, J. (2017). IST: Intelligentie Structuur Test [Intelligence Structure Test] (HTS Report no. 5105-7035). Retrieved from Hogrefe Uitgevers B.V. website: https://www.hogrefe.nl/shop/media/downloads/sample-reports/5700601_ISTbasisrapport_mr.pdf
Google Scholar

Hogrefe Ltd . (2016). IST: Intelligence Structure Test: English version of the Intelligenz-Struktur-Test 2000 R (I-S-T 2000 R) (HTS Report no.13216-189). Retrieved from Hogrefe Ltd website: https://www.hogrefe.co.uk/shop/media/downloads/sample-reports/5535902_mr.pdf
Google Scholar

McLeod, L., Lewis, C., Thissen, D. (2003). A Bayesian method for the detection of item preknowledge in computerized adaptive testing. Applied Psychological Measurement, 27, 121-137. | https://doi.org/10.1177/0146621602250534
Google Scholar SAGE Journals

Monahan, J. F. (2011). Numerical methods of statistics (2nd ed.). Cambridge University Press. |
Google Scholar Crossref

Roskam, E. E. (1987). Toward a psychometric theory of intelligence. In Roskam, E. E., Suck, R. (Eds.), Progress in mathematical psychology (pp. 151-171). North Holland.
Google Scholar

Royston, J. P. (1982b). An extension of Shapiro and Wilk's | W test for normality to large samples. Applied Statistics, 31, 115-124.
Google Scholar Crossref

Yap, B. W., Sim, C. H. (2011). Comparisons of various types of normality tests. Journal of Statistical Computation and Simulation, 81(12), 2141-2155. https://doi.org/10.1080/00949655.2010.520163
Google Scholar

Authors

# data[].attributes.attachment_link.title
Testing the Within-State Distribution in Mixture Models for Responses and Response Times - Renske E. Kuijpers, Ingmar Visser, Dylan Molenaar, 2020

# data[].attributes.attachment_link.description
Abstract Mixture models have been developed to enable detection of within-subject differences in responses and response times to psychometric test items. To enable mixture modeling of both responses and response times, a distributional assumption is needed for the within-state response time distribution.

# data[].attributes.headline
6.Between lethal and local adaptation: Lesson study as an organizational routine

# data[].attributes.subject
6.Between lethal and local adaptation: Lesson study as an organizational routine

# data[].attributes.body
Paradigm：positivism
Reason： it investigates whether and how teachers continued to perform the professional development practice of lesson study in their own schools after participating in a cross-school Lesson Study Professional Learning Network for four years.

# data[].attributes.attachment_link.readable_text
Highlights

*

We examine whether and how teachers embed lesson study in their schools.

*

Teachers who maintained the script of lesson study often embedded the practice.

*

Teachers who modified the script discontinued or lethally adjusted lesson study.

Abstract

Professional development practices remain at the forefront of ways to support teacher learning but are difficult to sustain. We investigate whether and how teachers continued to perform the professional development practice of lesson study in their own schools after participating in a cross-school Lesson Study Professional Learning Network for four years. We found different perceptions of the general idea of lesson study. If the general idea was modified, teachers rarely continued to perform lesson study; if they did, they transformed lesson study in such a way that core elements were removed. When teachers maintained the general idea of lesson study they were more likely to continue to perform lesson study in their own school and to consider the practice useful.

Recommended articles Citing articles (0)

# data[].attributes.attachment_link.title
Between lethal and local adaptation: Lesson study as an organizational routine

# data[].attributes.attachment_link.description
We examine whether and how teachers embed lesson study in their schools. * Teachers who maintained the script of lesson study often embedded the practice. * Teachers who modified the script discontinued or lethally adjusted lesson study. Professional development practices remain at the forefront of ways to support teacher learning but are difficult to sustain.

# data[].attributes.headline
7.Examining Leisure Activity Engagement of Students with Intellectual Disability

# data[].attributes.subject
7.Examining Leisure Activity Engagement of Students with Intellectual Disability

# data[].attributes.body
Paradigm： positivism
Reason：It has conducted research, and has related models and data, with volunteer participation.

# data[].attributes.headline
8.A systematic review of integrative practices in physical education (2009–2018)

# data[].attributes.subject
8.A systematic review of integrative practices in physical education (2009–2018)

# data[].attributes.body
Available online:  24 October 2020
Paradigm：positivism
Reason: It adopts the method of systematic analysis to analyze the empirical research as a whole.

# data[].attributes.attachment_link.readable_text
Highlights

*

Forty-five integrative physical education studies were established for 2009-2018.

*

Technology was integrated most frequently, and art was integrated zero times.

*

Knowledge assessments and physical activity tracking were largely lacking.

*

An increase in publications found for Math and English language arts integration.

*

An increase in research from around the world of integrative physical education.

Abstract

The purpose of this systematic literature review was to provide a holistic analysis of the empirical research surrounding the integration of other content areas (e.g., math, science) into physical education class settings for the years 2009-2018. The systematic analysis took a three-phased approach: (a) extensive search, (b) researcher negotiations of coding classifications, and (c) analyzing results. Forty-five studies were established, showing technology integrated most frequently throughout the literature. There was a steady increase in publishing trends for 2010-2017, except in 2018. Additional findings reveal that most studies in this review did not utilize knowledge assessments or physical activity tracker measures. These measures are essential when integrating outside content into physical education to provide evidence of outcomes for both fields.

Recommended articles Citing articles (0)

# data[].attributes.attachment_link.title
A systematic review of integrative practices in physical education (2009-2018)

# data[].attributes.attachment_link.description
Forty-five integrative physical education studies were established for 2009-2018. * Technology was integrated most frequently, and art was integrated zero times. * Knowledge assessments and physical activity tracking were largely lacking. * An increase in publications found for Math and English language arts integration.

# data[].attributes.headline
9.Exploring the communities of learning policy in New Zealand using social network analysis: A case study of leadership, expertise, and networks

# data[].attributes.subject
9.Exploring the communities of learning policy in New Zealand using social network analysis: A case study of leadership, expertise, and networks

# data[].attributes.body
Paradigm：interpretivism
Reason: This article seeks to consult a model and put it into action to investigate and research, but it is based on explanation.

# data[].attributes.attachment_link.readable_text
Highlights

*

There is a pattern of strong and dense within-school ties.

*

There was less across-school collaboration and advice seeking than the policy intended.

*

More network members than is desirable remain on the periphery of the network.

*

Despite a strong core of network actors, those in formal CoL roles are not necessarily represented.

*

The potential for accessing and capitalizing on social resources is not yet being realized.

Abstract

Educators are increasingly being called upon to collaborate in ways that leverage expertise across schools as they work together to improve teaching and learning. In this study we use social network analysis to investigate the patterns of collaboration and advice-seeking within and across five schools that comprise one Community of Learning (CoL) in New Zealand. Our analyses indicate that while within school networks are connected, the network overall has low levels of between-school collaboration and advice-seeking activity. We discuss these findings with attention to misalignment between policy and practice, changing CoL leadership roles, the strength of within-school ties and the role of social networks in educational improvement.

Keywords

Expertise education policy

Recommended articles Citing articles (0)

# data[].attributes.attachment_link.title
Exploring the communities of learning policy in New Zealand using social network analysis: A case study of leadership, expertise, and networks

# data[].attributes.attachment_link.description
There is a pattern of strong and dense within-school ties. * There was less across-school collaboration and advice seeking than the policy intended. * More network members than is desirable remain on the periphery of the network. * Despite a strong core of network actors, those in formal CoL roles are not necessarily represented.

# data[].attributes.headline
10.Governing rationalities in children’s human rights education

# data[].attributes.subject
10.Governing rationalities in children’s human rights education

# data[].attributes.body
Paradigm：positivism
Reason：This research synthesis examines earlier studies within the field of children’s human rights in education, with the specific aim of clarifying which rationalities of teaching and learning children’s human rights are activated as significant for teachers’ thinking and work.

# data[].attributes.attachment_link.readable_text
Highlights

*

Human rights education research was analysed for activated rationalities.

*

5 governing rationalities in human rights education for children were identified.

*

The rationalities support and contradict each other in complex ways.

*

Governing rationalities connect to differing conceptions of rights for children.

Abstract

This research synthesis examines earlier studies within the field of children's human rights in education, with the specific aim of clarifying which rationalities of teaching and learning children's human rights are activated as significant for teachers' thinking and work. Theoretically framed by a Foucauldian governmentality approach, the qualitative analysis identifies five different rationalities in the teaching and learning of children's human rights in previous research. The paper discusses how the identified rationalities support or contradict each other in a complex way and their connection to differing conceptions of rights for children. Further, the possible consequences for teachers, teacher educators and policymakers are considered.

Recommended articles Citing articles (0)

# data[].attributes.attachment_link.title
Governing rationalities in children's human rights education

# data[].attributes.attachment_link.description
Human rights education research was analysed for activated rationalities. * 5 governing rationalities in human rights education for children were identified. * The rationalities support and contradict each other in complex ways. * Governing rationalities connect to differing conceptions of rights for children.