# data[].attributes.body
Assessment practitioners, educational developers, educators, and others have collaborated to improve student learning. Read their stories.

# data[].attributes.attachment_link.readable_text
Assessment practitioners, educational developers, educators, and others have collaborated to improve student learning. Read their stories.

Story Summaries & Keywords

Faculty in the Computer Information Systems undergraduate program at James Madison University worked with assessment and faculty development colleagues to develop a rubric and implement curricular changes to foster the development of key skills; interventions resulted in increased student performance and a budding culture of learning improvement within the program.

Keywords: program, undergraduate, business, discipline-specific learning outcome, curriculum change, assignment design, rubric

The faculty teaching a consumer sciences course at the University of Alabama collaborated to increase application/practice and teamwork activities during class after assessment results indicated students' shortcomings in critical thinking skills. Rubric evaluation of reflection papers showed students' skills increased after the pedagogical changes. Additionally, students shared that their perception of practice went from uncomfortable and unnecessary to essential to build skills.

Keywords: course, undergraduate, social science, critical thinking learning outcome, pedagogy change, rubric

As part of an initiative to improve student success in its STEM programs, York College of Pennsylvania piloted adaptive learning courseware in a General Chemistry course. The courseware yielded improved student scores, with positive implications for use in the College's other STEM programs.

Keywords: course, undergraduate, science, discipline-specific learning outcome, pedagogy change, test

The faculty member responsible for an American History course (a general education and transfer course) at Waubonsee Community College improved her students' information literacy skills by determining student areas of weakness and implementing scaffolded assignments as the solution. She used a modified AAC&U Information Literacy VALUE rubric to evaluate student performance on an essay question given as part the final exam.

Keywords: course, general education, humanities, information literacy learning outcome, assignment design, rubric

The Pacific Islands Studies program at the University of Hawai'i at Mānoa engaged faculty and creative-thinking graduate assistants in improving student writing skills and assessment measures. In addition to improving student achievement, the learning improvement process created opportunities for faculty reflection and discussion of teaching and assessment practice and led to a shared sense of purpose and accomplishment in meeting students' needs.

Keywords: program, undergraduate, interdisciplinary, written communication learning outcome, curriculum changes, pedagogy changes, assignment design, rubric

The Learning Improvement Community's goal is to provide a space for successful learning improvement stories. We recognize that the teaching, learning, and assessment practices represented take place in a "local" institutional context. The information contained in this website is for information purposes only. We do not control or guarantee the accuracy, relevance, timeliness, or completeness of information and practices presented in these stories.

# data[].attributes.attachment_link.title
Read Stories | Learning Improvement Community

# data[].attributes.attachment_link.description
Assessment practitioners, educational developers, educators, and others have collaborated to improve student learning. Read their stories. Story Summaries & Keywords Faculty in the Computer Information Systems undergraduate program at James Madison University worked with assessment and faculty development colleagues to develop a rubric and implement curricular changes to foster the development of key skills; interventions resulted in increased student performance and a budding culture of learning improvement within the program.

# data[].attributes.attachment_link.readable_text
Each year the Assessment Institute posts the handouts of those presenters who chose to submit their handouts electronically. Please note that not all presenters choose to submit their handouts.
Presenter Handouts:

# data[].attributes.attachment_link.description
Each year the Assessment Institute posts the handouts of those presenters who chose to submit their handouts electronically. Please note that not all presenters choose to submit their handouts. Presenter Handouts:

# data[].attributes.headline
Applying the RARE Model to General Education Assessment

# data[].attributes.subject
Applying the RARE Model to General Education Assessment

# data[].attributes.headline
Assessment with Benefits: Faculty Engagement and Community Building through GE Assessment

# data[].attributes.subject
Assessment with Benefits: Faculty Engagement and Community Building through GE Assessment

# data[].attributes.headline
Assessing General Education: Identifying Outcomes, Data Analysis, and Improvements

# data[].attributes.subject
Assessing General Education: Identifying Outcomes, Data Analysis, and Improvements

# data[].attributes.attachment_link.provider_name
Association of American Colleges & Universities

# data[].attributes.attachment_link.readable_text
Below is a list of the VALUE rubrics, organized by learning outcome. Click on an outcome to preview, download, and learn more about a particular rubric. For information on acceptable use of the VALUE rubrics, as well as how to reference and cite the rubrics, visit: How to Cite the VALUE Rubrics.

Follow the instructions to download all VALUE rubrics at no cost. All rubrics are offered via AAC&U's Shopping Cart:

If you experience any difficulty downloading the rubrics, please contact
Intellectual and Practical Skills Personal and Social Responsibility Integrative and Applied Learning
Below is a link to the Japanese Translation of the VALUE Rubrics, organized by learning outcome:

# data[].attributes.attachment_link.description
Below is a list of the VALUE rubrics, organized by learning outcome. Click on an outcome to preview, download, and learn more about a particular rubric. For information on acceptable use of the VALUE rubrics, as well as how to reference and cite the rubrics, visit: How to Cite the VALUE Rubrics.

# data[].attributes.body
The analysis contains an excellent overview of Assessment and uncovers five themes, two of which relate to using "indirect and direct measures of student development or academic success as a proxy for program performance" (6.4 Theme 4) and the prevalence of highlighting "positive evaluative claims about program performance" instead of areas for change or the impact of changes (6.5 Theme 5).

# data[].attributes.attachment_link.readable_text
Teirra K. Holloman , Walter C. Lee , Jeremi S. London , Chaneé D. Hawkins Ash , Bevlee A. Watford ,

Bevlee A. Watford

Department of Engineering Education, Virginia Tech, Blacksburg, Virginia, USA
Search for more papers by this author

Chaneé D. Hawkins Ash

Foresight Strategy Solutions, San Francisco, California, USA
Search for more papers by this author

Jeremi S. London

Department of Engineering Education, Virginia Tech, Blacksburg, Virginia, USA
Search for more papers by this author

Walter C. Lee

Department of Engineering Education, Virginia Tech, Blacksburg, Virginia, USA
Search for more papers by this author

Corresponding Author

Teirra K. Holloman

Department of Engineering Education, Virginia Tech, Blacksburg, Virginia, USA
Correspondence
Teirra K. Holloman, Department of Engineering Education, Virginia Tech, 345 Goodwin Hall, 635 Prices Fork Road, Blacksburg, VA 24061, USA.

Email: teirrah@vt.edu
Search for more papers by this author

First published:

Funding information: National Science Foundation, Grant/Award Numbers: 1647327, 1926935

Abstract

Background

In the field of engineering education, assessment and evaluation have been given insufficient attention as they relate to broadening participation. We posit that this lack of attention negatively impacts our ability to develop and implement sustainable solutions at scale.

Purpose

The purpose of this article is to explore the presence of an assessment cycle in the literature related to broadening participation in engineering. The assessment cycle refers to the process by which assessment/evaluation is planned, enacted, and used to improve educational processes.

Scope/Method

The scope of this study was influenced by it being situated in a larger study focused on broadening participation of Black Americans in engineering and computer science. We completed a literature map that illuminated a subset of literature broadly related to assessment/evaluation, and subsequently conducted a systematic literature review of 33 publications reporting on assessment/evaluation efforts. Our analysis of these efforts was grounded in a six-stage assessment cycle.

Results

The results of this study highlight common publishing practices related to the assessment cycle in the context of broadening participation. We find that assessment/evaluation is generally published at the program level, focused on student development or academic success as a proxy for program effectiveness, and concentrated on positive claims.

Conclusion

There is room to significantly improve how assessment/evaluation information is published. By highlighting productive and unproductive publishing practices related to assessment/evaluation, this research has important implications for the use and publishing of assessment/evaluation, particularly as it relates to broadening participation in engineering.

1 INTRODUCTION

As the field of engineering aspires to diversify, organizations continue to implement efforts aimed at bolstering the participation of groups that are historically and systematically marginalized (Lichtenstein et al., 2014). This challenge is commonly referred to as "broadening participation in engineering" (BPE). Here, we argue that our ability to develop and implement sustainable solutions in the context of BPE is intertwined with our ability to assess and evaluate existing interventions, programs, and institutions.

Assessment and evaluation are integral to BPE solutions because practitioners continually compare efforts and exchange best practices, relying on insights from others to determine which practices add the most value and should continue being implemented (George et al., 2019). Moreover, because BPE efforts are often overlooked and undervalued, there is also immense pressure on practitioners to secure external funding and retain internal budgets (i.e., soft funding); such work is often devalued due to its association with what is seen as feminized and immaterial labor (Chen et al., 2019). This circumstance raises the stakes, as it relates to accountability and demonstrating return-on-investment (Rincón & George-Jackson, 2016). As noted by Patton et al. ( 2019), "when making tough fiscal decisions, key diversity efforts are often cut or minimized" (p. 178). This reality results in BPE practitioners being forced to balance the pressure to minimize costs while maintaining and improving the outcomes about which institutions are concerned (Baber, 2015). In short, in the context of BPE, the pressure to assess and evaluate comes from the need to (1) improve one's own practices and exchange insights with other practitioners, and (2) justify the need for and use of resources.

The need for improved BPE assessment and evaluation is also echoed by national calls in this area. In fact, "inadequate assessments, metrics, and data tracking" were identified as a key impediment in "Surmounting the Barriers: Ethnic Diversity in Engineering Education" (National Academy of Engineering, 2014, p. 1). Performing self-assessments/evaluations of interventions and programs has also been historically recommended to assist in the recruitment, retention, and employment of people from underrepresented groups in engineering, both at the pre-college and undergraduate levels (Holloman et al., 2018). However, little work has been done to advance the scholarship of assessment and evaluation, particularly in the context of BPE.

First, we describe the limited focus on assessment and evaluation in engineering education research. While publication venues central to the field of engineering education (e.g., Journal of Engineering Education, Journal of Women & Minorities in Science and Engineering, Advances in Engineering Education) address topics related to assessment and evaluation (Cooper et al., 2019; Finelli & Froyd, 2019; e.g., Olds et al., 2005), the conversation primarily focuses on collecting and analyzing data (i.e., issues of measurement). The limitations of this focus become evident when considered through the lens of the assessment cycle. The assessment cycle refers to the process by which assessment (and evaluation) is planned, enacted, and used to improve education processes (Kuh et al., 2015), the latter of which is rarely discussed in engineering education scholarship on this topic. The field of engineering education must pay more attention to the process in its entirety; otherwise, it will miss important aspects that may result in more effective and informed change.

Next, we describe the lack of focus on assessment and evaluation in the context of engineering education. Prior conversations about assessment and evaluation in engineering education have largely focused on outcomes associated with the American Board of Engineering and Technology (ABET; D. B. Knight & Novoselich, 2017; Koehn, 2001; e.g., Moskal et al., 2002; Riley, 2016; Stagner & Jorendt, 2015). This narrow focus has limited our understanding of the role of assessment and evaluation in engineering education. According to J. T. Brown ( 2017), the pressure to assess or evaluate can come from the need to demonstrate efficiency and return-on-investment (ROI), compliance with regulations and standards, and learning or value added. Because of the central role ABET plays in engineering education (Felder & Brent, 2003), the need to demonstrate learning and compliance with regulations/standards is mostly well documented, leaving the needs of BPE practitioners (i.e., to demonstrate efficiency, ROI, and value) less understood.

1.1 Purpose

1.2 Scope and positionality

Before proceeding any further, it is important to situate this study in the positionality and interests of the authors. Each of the five authors identify as Black/African American, and the inspiration for this literature review comes from a larger study focused on the state of broadening participation of Black Americans in engineering and CS. As part of this larger study, we completed a literature map (J. S. London et al., 2020) that illuminated a subset of literature broadly related to the assessment and evaluation of BPE-related efforts, where Black Americans were the focus or among the participants in the article. This review is scoped to this subset of literature. Due to the exploratory nature of this study, we did not further disaggregate the papers by any attributes of the assessment/evaluation efforts themselves.

We thought this continued focus was justifiable because BPE practitioners focused on Black Americans often experience the aforementioned pressures to the greatest extent, largely due to Black Americans being the primary target of most attacks on affirmative action. This truth is a byproduct of the US higher education system being originally designed to educate only the White majority and equip White men for leadership roles (Geiger, 2005; Karabel, 2005; Thelin, 2011). In response to desegregation and the civil rights movement, policies such as affirmative action were put in place to aid access to higher education for students from marginalized communities. Unfortunately, many institutions faced and continue to face attacks on affirmative action policies and practices (Rodriguez, 2007; Shehab et al., 2012).

Combined, these two points led us to conclude that Black Americans provided an advantageous lens through which to examine the scholarship of assessment and evaluation in the context of BPE. As a result, many of the decisions we made as researchers were guided by this focus. We acknowledge that our results may have been different had we primarily focused on other populations (e.g., the Latinx community, women, and/or people with disabilities). However, because much of the literature focuses on the population beyond Black Americans, we believe our findings have implications for BPE broadly and discuss the subsequent literature accordingly.

2 DISCUSSING ASSESSMENT, EVALUATION, AND RESEARCH

Because BPE practitioners must regularly communicate with the broader education community, it is important to use language that is commonly accessible. However, finding a common language as it relates to monitoring progress and informing change (or accountability) is difficult because different communities have their own preferred terms. This difficulty is exacerbated in higher education, where there are multiple conflicting approaches to accountability. According to J. T. Brown ( 2017), there are at least seven fields (e.g., institutional effectiveness, educational measurement) that take a unique approach to this process and many more affiliated associates-such as the Association for the Assessment of Learning in Higher Education (AAHLE), the Association for Higher Education Effectiveness (AHEE), and the American Evaluation Association (AEA). Across these associations, numerous terms are regularly used regarding data collection that informs our decision making, such as assessment, evaluation, institutional research, and institutional effectiveness.

In considering the scholarship of assessment and evaluation, it is important to discuss the complicated relationship among assessment, evaluation, and research. Research is primarily concerned with producing knowledge that is either generalizable or transferable (Tracy, 2010; Winter, 2000). It is the process of inquiry and acquisition of knowledge about a topic or phenomenon. The process is guided by various research methods and methodologies which are ideally informed by clearly identified research questions. The execution of these methods produces data that is analyzed, interpreted, and used to confirm, deny, or create new knowledge about some theory, topic, or phenomenon. Though this rational model does not always describe the reality of the research process, it does provide a general overview (J. Martin, 1982).

In contrast to research, a ssessment and evaluation are primarily concerned with gathering useful information to be used for determining the effectiveness of localized efforts (Pellegrino et al., 2014). Though distinct, assessment and evaluation overlap significantly and are often used interchangeably. We acknowledge that the terms assessment and evaluation emerge from different traditions and offer the following definitions to establish a common language for this paper. According to Palomba and Banta ( 1999), assessment is "the systematic collection, review, and use of information about educational programs undertaken for the purpose of improving student learning and development" (p. 4). In the educational context, it is a mechanism for monitoring progress and informing change, widely perceived as a valuable and essential part of education (Suskie, 2008). According to Allen et al. ( 2002), evaluation is "the use of assessment findings (evidence/data) to judge program effectiveness; used as a basis for making decisions about program changes or improvement" (p. 1). It aids stakeholders and evaluators in determining if an effort (e.g., initiative, program, course, class) is worthy of adoption, continuation, or expansion (Fitzpatrick et al., 2011).

Given our choice to focus on the collection, review, and use of information about BPE interventions, we hereafter anchor on the process of assessment. This process is iterative and comes in many forms (e.g., quantitative, qualitative), fulfills many purposes (e.g., accountability, improvement), includes many stages (e.g., planning, collecting data, etc.), and is conducted at many levels of analysis (e.g., individual, program, institutions; Leskes, 2002; R. Miller & Leskes, 2005; Town & Pierce, 2013). To discuss the breadth of this process, we incorporate insights related to research and evaluation as well. When discussing the collection of assessment data, we leverage the language commonly associated with research, and when discussing the use of assessment data, we leverage the language commonly associated with evaluation. Further disentanglement of these terms and processes is beyond the scope of this paper.

3 AN OVERVIEW OF ASSESSMENT IN EDUCATION

Before narrowing our focus on BPE, it is important to first understand how assessment is used in education broadly. We will discuss (1) attributes by which assessment can be classified, (2) quality considerations associated with assessment, and (3) barriers to implementation.

3.1 Assessment attributes

For BPE practitioners to accurately assess their efforts, well-thought-out assessment plans are needed. A sound assessment plan has four attributes that should be considered and aligned. The first two attributes relate to the focus of the assessment plan (e.g., what and why?), and include the (1) level and (2) purpose. The next two attributes relate to the implementation of the assessment plan (e.g., how?), and include the (3) function and (4) form. Each attribute is discussed further below.

3.1.1 The focus of assessment plans

There are several approaches to discussing the focus of an assessment plan, either focusing on the different purposes (i.e., why an assessment plan is needed) or the different levels (i.e., aspects of the experience or environment). Regarding the levels of assessment, according to R. Miller and Leskes ( 2005), there are five different categories: (1) assessing individual student learning within courses, (2) assessing individual student learning across courses, (3) assessing courses, (4) assessing programs, and (5) assessing the institution. Regarding the purpose of assessment, common categories include accountability, improvement (Leskes, 2002), grading, learning, and coaching (Hansen, 2012). It is important to understand the different classifications because they have implications for how assessment questions are formulated. For example, at the individual level, one might ask "How well has the student achieved the learning outcomes set for the course?" whereas at the program level one might ask, "Does the program advance institution-wide goals as planned?" To assist readers with exploring the various levels and purposes of assessment across the literature, we provide an overview in Table 1.

Classification

Category

Definition

Level

Individuals

Investigating how well individual participants are learning and meeting predetermined goals within and across courses (R. Miller & Leskes, 2005)

Purpose

Assist learning

Provide specific information about students' strengths and difficulties with learning (National Research Council, 2001)

3.1.2 The implementation of assessment plans

Once the level and purpose have been identified, practitioners can determine the function (i.e., how the results will be used) and form (e.g., appropriate methods). To assist readers with exploring the functions and forms to consider, we provide an overview in Tables 2 and 3.

Category

Definition

Timing of data collection

Formative

Gathering information about student learning during the progression of a course or program, usually repeatedly

During

Summative

Gathering information at the conclusion of a course, program, or undergraduate career

End

Readiness

Exploring the satisfaction of prerequisite entry skills to remedy entry deficiencies and/or categorize learning groups

Before

Placement

Determine necessary course or unit objectives

Before

Diagnostic

Identify common breakdowns in mental models

During

Category

Definition

Embedded

Gathering data about student learning in a way that is ongoing and, thus, built into the teaching-learning process

Direct

Gathering data that demonstrates the learning objectives/outcomes of interest

Indirect

Gathering data that explores participant perception about learning and the surrounding environment

Qualitative

Collecting and analyzing textual data (surveys, interviews, focus groups, conversational analysis, observation, ethnographies) with emphasis on the context within which the study occurs

Quantitative

Collecting and analyzing data often from surveys, with the intent to generalize and make inferences

Regarding function, the most common approach to differentiating is to distinguish between formative assessment and summative assessment: formative assessment is focused on making improvements for current students or participants, whereas summative assessment is focused on following cohorts (Leskes, 2002). However, other categories include readiness, placement, and diagnostic. These distinctions are important because they enable educators to identify the focus of measurement, nature of the sample, item difficulty, time of administration, and how the results will be used. For example, according to M. D. Miller et al. ( 2008), assessments that occur prior to instruction or intervention can either assist with remedying entry deficiencies, assigning students to groups, informing instructional planning, or determining advanced placement ( readiness and placement); an assessment that occurs during instruction can either improve and direct learning through ongoing feedback, or remedy errors related to persistent learning difficulties ( formative and diagnostic); and assessment that occurs at the end of instruction can be used to assign grades, certify accomplishment, or evaluate teaching ( summative).

The forms of assessment tend to come in five main categories: embedded, direct, indirect, qualitative, and quantitative. Several factors implicate which form of assessment may be most appropriate to use. For example, the assessment form tends to indicate the type of data that will be collected in light of the assessment question. It is important for the assessment form to be aligned with the overarching assessment purpose, questions, and/or objectives. Resources, time constraints, accessibility of data, and analysis tools must also be considered. Lastly, because assessment results are often used to convince stakeholders, it is also important to consider the epistemological frameworks of those who will be receiving the results.

3.2 Assessment quality

As previously mentioned, there are some similarities between research and assessment, namely when it comes to quality. We define "quality" as the extent a process exemplifies standards and criteria noted in germane literature. For example, Moskal et al. ( 2002) note that the only way for engineering educators-and in our case BPE practitioners-to adequately monitor progress is to be aware of the important issues that surround assessment measures. Much like with engineering education research, two major issues related to assessment quality are validity and reliability. Validity refers to the degree to which a measurement measures what it is intended to measure or the degree to which interpretations of results are used correctly. Reliability refers to the degree of consistency of assessment scores (Moskal et al., 2002). These terms are often used when researchers employ quantitative methods. When researchers use qualitative methods, they are likely to use terms like transferability, trustworthiness, and dependability as a criterion by which to judge the work (Golafshani, 2003). Transferability is another way of looking at external validity, trustworthiness at internal validity, and dependability at reliability (Leydens et al., 2004). Ultimately, regardless of the approach, it is imperative that criteria are established, systematically used, and openly discussed in reports and publications.

Considering the pragmatic nature of assessment and pressure for BPE practitioners to demonstrate their value to a variety of external stakeholders, it can be difficult to determine what is considered "good assessment" given the many levels, purposes, forms, and functions. Fortunately, criteria regarding assessment best practices have been well documented over the years. For example, Suskie ( 2008) provides a concise list of five best practices, which summarizes over a dozen of these various sources. These characteristics state that good assessments (1) are used, (2) are cost effective, (3) yield reasonably accurate and truthful results, (4) are valued, and (5) focus on and flow from clear and important goals. Some suggestions to achieve these characteristics are to consider the purpose of assessment, have collaboration occurring between the student and the teacher, and use the feedback from the assessment when making future decisions. While these suggestions are informative, practitioners must interpret them as needed to fit their varying and ever-changing contexts.

In short, there is no "one-size-fits-all" approach to assessment, and it should not be perceived as such. "To say that assessment is [a] good assessment is akin to saying that a medical test is a good test; each can provide useful information only under certain circumstances" (National Research Council, 2001, p. 222). In other words, assessment can be considered good only when all parts are in alignment, it is implemented with a goal or need in mind, and has been validated and deemed reliable.

3.3 The barrier to assessment

Nonetheless, numerous factors make achieving "good assessment" difficult. According to Hoey and Nault ( 2008), these barriers primarily relate to aspects of trust (e.g., trust in the motives, questions, methods, data, etc.), which they viewed as fundamental to assessment design and implementation. Trust will either facilitate or pose obstacles to change implementation. Accordingly, it is important to foster trust via three main avenues: (1) culture, (2) structure, and (3) leadership. Each of these avenues must operate in a way that promotes assessment to establish stakeholders' buy-in. For example, an organization's culture would ideally promote open communication, value evidence-based decision making, offer incentives for improvement, and provide resources for those charged with an assessment. To promote the value of assessment, leaders must also make it a normalized practice within the organization. This process should include using those results in ongoing discussions and/or in creating new initiatives; and allocating funds, training, and/or personnel to aid in conducting assessment and the implementation of results. Unfortunately, for many BPE practitioners, trust is not there. With the overwhelming and constant pressure to show value, it is difficult to report any shortcomings or failures. Additionally, being often among the first to lose internal funding makes it challenging for BPE practitioners to trust that their work is valued.

The scholarship of assessment requires an understanding of how assessment can be operationalized, classified, evaluated, and promoted. Next, it is important to understand how each of these components fit together.

4 CONCEPTUAL FRAMEWORK: AN ASSESSMENT CYCLE

To systematically examine the use of assessment, we situate our investigation in a six-stage assessment cycle (Kuh et al., 2015). This assessment cycle is a continual process that begins with developing a plan and identifying desired outcomes, followed by collecting and analyzing data. Next, the assessment results are shared so that necessary change can be identified, implemented, and assessed. An overview of each stage will be discussed in the following sections.

4.1 Step 1: Plan and identify outcomes

4.2 Step 2: Collect data

Once data sources have been identified, the collection of data can begin. According to Worthen, Borg, and White (as cited in Fitzpatrick et al., 2011, p. 348), there are five categories by which data collection can be obtained: (1) directly from individuals identified as sources of information, (2) by an independent observer, (3) by a technological device, (4) with unobtrusive measures, and (5) from existing organizational information or formal repositories or databases. Information collection arrangements must also be made, identifying who will collect data, under what conditions, and when will collection take place. Important deadlines, timelines, and resources should also be considered. Given the focus on collecting data, this step in the cycle often overlaps with education research.

4.3 Step 3: Analyze data

For each assessment question, evaluators should ensure that they have a plan for analyzing the corresponding data or information. Fitzpatrick et al. ( 2011) note that it is important to specify how the information will be analyzed. They suggest doing so through two steps. First, one should determine what techniques will be used during analysis (e.g., descriptive statistics, emergent coding). Next, one should then determine by what means these analyses will take place (e.g., Excel, NVivo). When interpreting these results, Fitzpatrick et al. recommend involving stakeholders and clients because varying perspectives can result in varying interpretations. This step also overlaps with education research.

4.4 Step 4: Share results

When sharing results, it is important to be aware of when this information is needed, who needs it, and in what format. Fitzpatrick et al. ( 2011) offer a matrix to be considered for each assessment question to help organize this information. It includes understanding the audience, the necessary content, reporting format, date of the report, and the context it will be presented; this level of detail and organization also emphasizes the notion that questions that are to be investigated should be intentional, important, and useful to stakeholders. The timing, audience, and format of the results when they are shared is another notable difference between assessment and research.

4.5 Step 5: Identify and implement changes

After results have been disseminated, the hardest part of the cycle begins, requiring stakeholders to act on the results. Kuh et al. ( 2015) attribute this difficulty to an overemphasis on data collection and research design and suggests that emphasis should shift to implementation and assessing impact. Ensuring the utility at the beginning of an investigation will help in mitigating future oversight. Additionally, Kuh and colleagues suggest linking assessment to both internal and external processes such as accreditation and promotion. This step in the cycle also makes it difficult to separate assessment from evaluation because both processes require that the information previously collected be used to advance some desired end.

4.6 Step 6: Assess the impact of changes

This final stage of the cycle officially closes the assessment loop by ascertaining impact, and by extension, propelling the process into another iteration. Implementing changes does no good if stakeholders are unaware of the positive or negative consequences that resulted.

5 RESEARCH DESIGN

To explore the use of assessment/evaluation in the context of BPE, we conducted a systematic literature review (SLR). SLRs have been used to summarize, critically evaluate, and reconcile conflicting evidence to inform policy and practice (Borrego et al., 2014). Despite the well-established use of literature reviews in engineering education, SLRs are not used as often (Borrego et al., 2014). The value in an SLR can be found in the varying degrees of its purpose from describing the current state of knowledge to evaluating a theory, to identify gaps in the literature (Petticrew & Roberts, 2006).

In order to conduct this SLR, two primary sources were used to guide our research design. Petticrew and Roberts ( 2006) explain the details of the process as it relates to various disciplines conducting SLRs, and Borrego et al. ( 2014) operationalize the SLR method for the engineering education community. The five major steps of this SLR include the following:

1. Formulate guiding research questions and corresponding inclusion criteria.

2. Find and catalog sources.

3. Critique and appraise the quality of selected literature.

4. Address bias, validity, and reliability concerns.

5. Synthesize insights.

These steps will serve as a framework for the structure of the remainder of this section. The following sections will break down the process of collecting and identifying articles as a part of a larger study and identifying those appropriate for this study.

5.1 Formulating guiding research questions and inclusion criteria

Our exploration of the literature was guided by the following research question: How are assessment and evaluation practices related to broadening participation communicated in the literature? The initial inclusion criteria used during the collection of literature are as follows: (1) publication date range, (2) database selection, and (3) search string selection. This paper is part of a larger project, and more about these processes can be found in a previously published article about the overall process of conducting an SLR (Phillips et al., 2017). The date range (e.g., 1970-2017) was informed by historical events associated with efforts to broaden the participation of Black Americans in engineering and CS, specifically the establishment of several national-level efforts focused on BPE in the 1970s. For example, the National Action Council for Minorities in Engineering, Inc. (NACME) was established in 1974 and the NSBE was established in 1975. With the help of a librarian, databases and search strings were identified, as illustrated in Table 4. They helped to limit publication bias by keeping the database options as wide-ranging as possible. Additionally, their expertise was used to identify search strings, alongside input from researchers in our professional network and keywords associated with "Diversity" from the Engineering Education Research Taxonomy (Finelli et al., 2015).

Database name

Search string

Note

Education Source and PsycINFO (EBSCOhost interface)

((bias OR discrimination OR multicultural* OR inclusive* OR racism OR prejudice) OR

(motivation OR attainment OR achievement OR aspiration OR persist* OR retention)) AND

((AB african w2 american OR SU african w2 american OR TI african w2 american) OR (AB black OR SU black OR TI black) OR (AB people N2 color* OR SU people N2 color* OR TI people N2 color*))

AND

((AB STEM OR SU STEM OR TI STEM) OR (AB engineer* OR SU engineer* OR TI engineer) OR (AB "computer science" OR SU "computer science" OR TI "computer science"))

Search all fields for words used to include or exclude people

Search abstract, title, subject headings for terms used for African American

Search abstract, title, subject headings for STEM, engineering, and computer science

Compendex and INSPEC (Ei Village interface)

< ((motivation OR attainment OR achievement OR aspiration OR persist* OR retention) WN All fields) > OR < ((bias or discrimination or multicultural* or inclusive* or racism or prejudice) WN All fields) >

AND

< ((((african ONEAR/2 american) WN KY) OR ((black) WN KY)) OR ((people NEAR/2 color) WN KY)) >

AND

< ((STEM OR engineer* OR "computer science") WN KY) >

Quick search, autostemming off, search all fields

Search subject/title/abstract, autostemming off

Search subject/title/abstract, autostemming off

Abbreviation: STEM, science, technology, engineering, and mathematics.

5.2 Finding and cataloging sources

After removing 100 duplicate publications, 1080 articles were evaluated against three additional hierarchical inclusion criteria: (1) Is the article written in English and about education or the science, technology, engineering, and mathematics (STEM) workforce in the United States? (2) Is the article focused on engineering or CS in any context, or STEM disciplines in a K-12 context? (3) Is the article focused on issues or the experiences of Black Americans, or on some aspect of the wide variety of topics associated with broadening participation? Criterion 3 allowed for the inclusion of publications that might not focus solely on Black Americans. This inclusion was done to account for the possibility of learning from publications that emphasized Black Americans without solely focusing on them. We intended to be mindful of the realities of targeting a specific group in the current legal landscape and the frequency with which multiple marginalized groups are included in a single intervention. In total, 470 out of 1180 (40%) articles met the criteria and spanned across the education-to-workforce pathway, as illustrated in Figure 1. Additional details about the information in Table 4 is included in a systematic mapping review that has been published elsewhere (J. S. London et al., 2020). This paper focuses on a subset of 51 articles related to assessment.

5.2.1 Limitations of finding sources

Given that all initiatives, programs, and institutions are not mandated to publish assessment/evaluation results, this review can only speak of the work made available at the time of the literature search. Additionally, we must acknowledge that assessment was not a search term used in data collection, as this study is the result of a larger project that dictated data collection. Lastly, due to our initial interest and the exploratory nature of this study, we did not disaggregate literature based on assessment level or intervention type and we may have overlooked articles focused solely on the Latinx community, women, and/or people with disabilities. Each of these aspects bring limitations to this work and therefore cannot speak to the total landscape of BPE assessment.

5.3 Critique and appraise quality of literature

Following the documentation of the literature map results, the initial 51 assessment documents underwent a quality check. Four questions were used to appraise all documents being utilized in this SLR. The questions were as follows:

1. Is the problem/purpose/aim of the study clearly stated?

2. Is the sampling strategy apparent and appropriate? Does the sample represent the target population?

3. Is information about data collection procedures apparent and appropriate?

4. Is information about the approach to analyzing data apparent and appropriate for addressing the study's purpose?

Following this final step of appraisal, only 34 documents remained. Unfortunately, one document was not accessible given available resources and thus was discarded from our list of articles to review. This resulted in a final total of 33 assessment documents.

5.4 Address bias, validity, and reliability concerns

Although efforts were made to minimize bias, limitations remain. According to Cook et al. ( 1997) the limitations of an SLR stem from (1) the quality and quantity of the selected studies, and (2) the quality of the systematic review procedures. One fault in this study lies in the fact that the literature in this review was collected in January of 2017; therefore publications after that time have been excluded from this review. Lastly, this study is a result of seeking BPE assessment literature across the education pathway and therefore we must acknowledge the varying histories that exist between K-12 and postsecondary education and their use of assessment.

5.5 Synthesize insights

The synthesis step included extracting data and coming up with themes. After the final quality check, data were extracted from each of the 33 documents. For each document, we identified Where (what is the context of the program?), Who (which populations are being used to generate information?), What (what assessment questions or objectives are communicated?), How (what metrics and outcomes are being assessed?), When (what is the temporal, duration, proximity to the experience, the longitudinal nature of the assessment?), Findings (what sort of evaluative claims are people making?), and Implications (how will the results be used?), followed by a critique. A summary table was composed to provide a comprehensive view and summary of each study. These documents were compared across each section of every document and the pros, cons, similarities, and suggestions were identified for improvement based on our conceptual framework of the assessment cycle.

6 RESULTS AND DISCUSSION

Following the data extraction from the final 33 documents, we utilized Kuh and colleagues' assessment cycle to inform our analysis. Five major themes emerged as it relates to the initial stages of the assessment cycle, covering topics related to levels of assessment, purpose, and design. More specifically, Theme 1 focused on where the published assessment tends to occur. Theme 2 describes the sparse use of assessment at some levels. Theme 3 recounts the research designs used in BPE assessment plans. Theme 4 depicts the types of data being collected as a proxy for program performance. Lastly, the kinds of claims that are made based on the results of these assessment plans are discussed in Theme 5. Each theme is discussed below.

6.1 Theme 1: The assessment articles in our study most frequently focused on programs at the K-12 level and predominantly White institutions of higher education, and incorporated students with various marginalized identities

The papers included in this SLR focused on assessments most frequently used in the context of programs at the K-12 level and predominantly White institutions (PWIs) of higher education. Of the 33 articles identified, 9 were at the K-12 level, 20 were at the undergraduate level, 1 was at the graduate level, and 3 included participants across levels. It is important to note that none of the articles focused on community college and workforce segments.

6.2 Theme 2: The assessment articles in our study were most frequently program-level assessment and seldom focused on institution-level assessment or the broader context surrounding underrepresented students

Levels of assessment and their purpose play a major role in guiding the types of questions and topics explored in the assessment. Our analysis found that most articles investigated program effectiveness based on either a direct or indirect measure of student development or performance. Occasionally, some investigations would stray away from the status quo by moving beyond the program level. For example, Litzler et al. ( 2010) investigated climates in engineering at various universities in the United States using a 132-item survey instrument, while McCullough et al. ( 2014) created a rubric for students to complete weekly that informed their mentoring sessions. An included dissertation (Jia, 2015) assessed how stricter high school math prep impacted the performance of underrepresented students (i.e., Black Americans, Latinos/as/xs, and women) as compared to White men. While these topics were not focused on Black Americans, results shed light on findings directly related to Black Americans' persistence in engineering. Lastly, White et al. ( 2008) presented challenges and issues that arose during the creation of a program evaluation plan, but they did not present the evaluation plan itself.

Moreover, the seemingly narrow focus of assessment also limits the degree to which we understand the broader context within which BPE stakeholders work. If we understand that BPE is a multifaceted complex task, we need to consider our overall understanding of the problem space. When thinking about the context in which Black Americans are situated, there is much to be understood about how the engineering ecosystem does and does not adequately support them. One means of gathering more information is through assessment at various levels within various contexts. Course-level and institution-level assessment from the perspective of Black American students is necessary. We need to understand the parts of the environment that were not designed for Black American students to succeed just as much as we need to understand the interventions that were created with them in mind. Our understanding of one helps to inform the changes implemented to improve the other.

6.3 Theme 3: The assessment articles in our study utilized both quasi-experimental designs and nonexperimental designs

Assessment designs varied across articles and could be categorized into one of five types. Three quasi-experimental designs were frequently employed: (1) pretest-posttest design, (2) posttest-only design, and (3) posttest only design with repeated measures. Additionally, nonexperimental designs emphasized both formative and summative functions of assessment. Table 5 provides an overview of each design including references to a few example articles.

Design

Description

Example articles

Posttest only

A single observation of any kind performed after a treatment

Donnelly ( 2007); Gardner-McCune et al. ( 2013); J. Good et al. ( 2002); Lee and Cross ( 2013)

Pretest-posttest

Two observations of any kind performed before and after a treatment

Hrabowski and Maton ( 1995); Clark et al. ( 2015); D. Knight et al. ( 2013)

Posttest only with repeated measures

Performing multiple observations on the same sample over time (i.e., cohort performance over time)

Murphy et al. ( 2010); Jiang et al. ( 2005)

Nonexperimental-formative

Multiple observations of participants during the progression of the intervention

E. Brown ( 2007); J. M. Good et al. ( 2000); McCullough et al. ( 2014); Ogilvie ( 2007); Wilson et al. ( 2015)

Nonexperimental-summative

Multiple observations of an intervention occurring at the end of each intervention cycle

Lam et al. ( 1997); Kendricks and Arment ( 2011); Lasser and Snelsire ( 1996)

6.3.1 Posttest only

This design describes assessment plans that involved collecting data at the end of the intervention-sometimes including more than one data type. For example, Donnelly ( 2007) administered multiple assessment tools after the intervention. Post intervention, data were collected (via surveys, interviews, and focus groups) from the participants to assess the effectiveness of the program. Additionally, faculty who participated in the program completed surveys, providing insights on what they perceive to be important elements of the program.

6.3.2 Pretest-posttest

This design describes assessment plans that involved collecting data before and after an intervention, typically the same types of data, to demonstrate growth and development. For example, D. Knight et al. ( 2013) administered pre- and post-surveys to measure identity development in program participants. One survey was administered before and after two summer bridge programs. Additionally, another survey was conducted before and after two first-semester courses. Data were used to compare pre- to post-results, across demographic groups, and across interventions.

6.3.3 Posttest only design with repeated measures-Cohort overtime

This design describes assessment plans that involved collecting data from a particular cohort of students at the end of an intervention and longitudinally thereafter. At the undergraduate level, for example, some studies would follow students from their participation in a bridge program to their completion of the first/second year of an engineering program or through graduation. Murphy et al. ( 2010) observed the graduation rates of program participants and determined the likelihood that someone would graduate if they participated as opposed to those that did not; Jiang et al. ( 2005) tracked five North Carolina Louis Stokes Alliance for Minority Participation cohorts over a 5-year period.

6.3.4 Nonexperimental-Formative

This design describes assessment plans that involved observing participants and collecting data throughout the intervention. For example, J. M. Good et al. ( 2000) analyzed student journal entries that were collected throughout the entire academic year. It was noted that the program coordinator engaged with students and their journal entries weekly to "encourage discourse regarding program development and improvement with each mentor" (J. M. Good et al., 2000, p. 377). Similarly, E. Brown ( 2007) used measures like student accolades and changes in academic major during the intervention as factors of program success.

6.3.5 Nonexperimental-Summative

We previously described assessment plans that involved collecting data about a cohort of students at the end of an intervention cycle, repeatedly over time (i.e., posttest only design with repeated measures). The nonexperimental (summative) design is somewhat similar in that data is collected at the end of an intervention. However, the difference for this design is that data are not collected from the same group of participants. Lam et al. ( 1997) illustrate this. They aimed to better understand a multifaceted program, Increasing Diversity in Engineering Academics (IDEAS), that provides support for students academically, socially, emotionally, financially, and professionally year-round throughout their undergraduate careers. Therefore, an analysis of the pros, cons, and problems associated with the IDEAS program was conducted at the end of the intervention cycle. Similarly, Lasser and Snelsire ( 1996) evaluated the benefits of proactive mentoring over several years, using various measures to represent snapshots of the program over time.

6.4 Theme 4: The assessment articles in our study most frequently investigated program outcomes using indirect and direct measures of student development or academic success as a proxy for program performance

While a variety of assessment questions were expressed in the articles, most articles sought to determine the impact of an intervention program on participants. This entailed investigating how successful a program was based on the participants' likelihood of pursuing or persisting in STEM (Anderson-Rowland & Ruben, 2008; Dunn & Veltman, 1989; Wilson et al., 2015), affecting participants' attitudes and interest toward STEM (Clark et al., 2015), or participants' self-efficacy in STEM-related areas (Di Salvo et al., 2013).

6.5 Theme 5: The assessment articles in our study most frequently highlighted positive evaluative claims about program performance rather than assessing areas for change or assessing the impact of those changes

Given the pressures for BPE practitioners to prove their value, it is no surprise that not all authors highlighted places for improvement or any shortcomings that were revealed through their assessments. Similarly, knowing that resources are often scarce, the lack of discussions regarding the impact of changes that were made was not surprising. Just as it costs to host the initial intervention, it is also costly and time consuming to revisit and make changes to it.

7 IMPLICATIONS AND FUTURE WORK

Our research has important implications for the use of assessment and evaluation, and the assessment cycle, in educational practices related to BPE. Our paper sought to examine assessment/evaluation practices as they relate to BPE efforts. Although we have discussed a variety of topics, attributes, and considerations, we now offer implications grounded in the assessment cycle to assist practitioners with (1) planning and identifying outcomes; (2) collecting and analyzing data; (3) reporting and sharing results; and (4) identifying, implementing, and assessing change. Though our study was motivated by our research teams' focus on Black Americans, our findings are broadly applicable to BPE stakeholders focused on a variety of underrepresented or otherwise marginalized groups. Future exploration should consider disaggregating data based on levels (i.e., individual, program, institution) given that the associated assessment or evaluation may look differently as purposes differ and explore the extent to which BPE assessment/evaluation is implemented differently depending on the target population.

7.1 Implications for planning and identifying outcomes

In doing so, it is important to focus not only on the success stories-as most assessment articles we reviewed sought to demonstrate how well their programs were preparing students-but also on deficiencies and areas for improvement. If we are to make progress in this area, we must be willing to critically review existing efforts and constantly share ways to improve. Doing so requires establishing a culture of trust, where practitioners do not fear that their programs' sustainability will be jeopardized by sharing results about things that did not go well.

Practitioners should also specify the level of assessment in their assessment reports. Doing so is important because this decision greatly informs the types of questions and objectives being set in an assessment plan (R. Miller & Leskes, 2005). As our results suggest, assessment or evaluation in the context of BPE is seldom used (or published) at the institution level; this gap in the literature presents an opportunity for future work and scholarship.

Lastly, there is a need for more publications that focus on efforts beyond the context of PWIs and the K-12 level. Program context matters when it comes to complex issues, and the lack of assessment publications focused on efforts in other settings, such as HBCUs and industry, has implications for our ability to address this challenge, particularly as it relates to Black Americans. If a large percentage of the population of interest attends a particular type of university, for example, it is important to capture what interventions do and do not work in those contexts. Additionally, there is a need for more work that focuses on analyzing the short- and long-term impact of interventions in various environments.

7.2 Implications for data collection and analysis

There is also a need for more consistent reporting of results using quality metrics. The use of reliable and valid instruments increases the credibility of results tremendously, and reporting on the validity and reliability of the instruments used is just as important as selecting them. While some studies included validity and reliability measures, they were rare. Without such information, it is difficult for the reader to evaluate the credibility of the results and/or determine which results are transferable. If the internal validity is not confirmed, the external validity is compromised. When considering the indicators used to evaluate the quality of qualitative work (e.g., transparency, credibility, ethical considerations, and meaningful coherence) in educational research (Tracy, 2010), it should be noted that these indicators are applicable here as it relates to reporting on both qualitative and quantitative methods.

Lastly, there is also a need for more longitudinal designs that connect to the societal need for a more diverse engineering workforce. The absence of such designs creates a gap in understanding the long-term effectiveness of programs because many BPE interventions implicitly aim to contribute to the production of a more diverse engineering workforce. Longitudinal designs can also aid in tracking changes across all levels of assessment, specifically at the institutional level. Given that some schools of thought surrounding change in higher education view change as slow, continuous, and long-term (Kezar, 2001), longitudinal designs aid in our assessment and evaluation at the institutional level.

7.3 Implications for reporting and sharing results

Future assessment reports should be more transparent. Here, we learn from suggestions intended for educational research: "Transparent research is marked by [the] disclosure of the study's challenges and unexpected twists and turns, and [the] revelation of the ways research foci transformed over time" (Tracy, 2010, p. 842). We argue that the same could apply to assessment because one of the major barriers to change in assessment is a lack of trust; increased transparency could help strengthen trust between BPE practitioners and their stakeholders.

With transparency in mind, it may be harder to forget specific details about various aspects of the assessment process. For example, providing more clarity on the program structure in conjunction with discussions about the results is imperative. The easier it is for readers to understand the context and the findings, the more trustworthy a report will read. The use of properly labeled tables and figures reporting on participant demographics and program components (e.g., logic models) is also needed, as this was not included in many of the documents. Additionally, reports should document the participant selection criteria because this information allows for stakeholders to make better supported claims about student performance. Lastly, we reiterate that negative findings were rarely published. We encourage practitioners to share both their success and their shortcomings and lessons learned. According to the literature we reviewed, it would appear that almost everything is working effectively, which we know anecdotally is not the case. This reality leads us to also call upon administrators and other stakeholders to support and believe the reports and statements by BPE practitioners who discuss and advocate for the needs of their students. BPE practitioners need this support from stakeholders to ensure that their work is valued and uplifted so that they can be transparent without the fear of repercussions (i.e., loss of job, demotion) for reporting on failures, shortcomings, and aspects that require improvement, change, or reform.

Lastly, when reporting and sharing results, practitioners should also be mindful of narratives they are creating in the process (i.e., deficit vs. asset-based). The framing of these results can impact how minoritized communities are represented within engineering and thus impact the types of efforts created to address BPE-related issues. Practitioners should be particularly mindful of this concern when planning to use student achievement as data metrics.

7.4 Implications for identifying, implementing, and assessing change

Future assessment/evaluation reports should document insights related to all parts of the assessment cycle. We acknowledge that attending to the end of the assessment cycle is the hardest to complete (Kuh et al., 2015). As a result, it is not surprising that the implications of the assessment results and future work were seldom mentioned. It is also not surprising that more thorough discussions surrounding the end of the assessment cycle in BPE were seldom found in our literature search. Unfortunately, this significantly limits our ability to learn from the work done by others. While this omission does not mean that assessment/evaluation results did not inform changes, the absence of this information may leave a reader wondering about the utility of the findings-for those hosting the intervention and stakeholders desiring to learn from them.

Finally, it is worth acknowledging the challenges associated with documenting, tracing, and making sense of the impact of an intervention, regardless of the context. J. London and Cox ( 2015) note that these challenges center around issues of attribution, assessment/evaluation, and interpretation. The inherent challenges associated with demonstrating impact in any context simply add even more pressure when engaging in impact-related exercises in the BPE context. Until the issues associated with impact, in general, are resolved, BPE practitioners can only do so much regarding reporting when key stakeholders (especially donors or sponsors) desire absolute certainty of impact as the minimum starting place. Instead of focusing on executing high-quality interventions with fidelity, they are forced to focus more attention on gathering evidence to demonstrate value.

8 CONCLUSION

The purpose of this study was to explore the presence of the assessment cycle in the literature related to BPE. In doing so, we hoped to illuminate the extent to which the entire assessment cycle is explored in the engineering education literature. We also intended to provide an overview of the nuances associated with planning, enacting, and using assessment/evaluation in educational practices. By grounding our SLR of assessment articles in a six-stage assessment cycle, we were able to not only highlight aspects of the cycle that were present but also emphasize that which was missing. For example, we note the omission of negative results as well as information about the change that resulted from the process. This bias toward positive reporting in the articles we reviewed is of concern because we have to be comfortable saying "This did not work" if we are going to move forward. We argue that the constant and overwhelming pressures to justify the need for and use of resources or demonstrate efficiency and ROI make doing valuable and effective work harder for BPE practitioners.

The uneven amount of pressure applied to BPE efforts must also be revisited if we (as a field) agree that broadening participation of historically underrepresented ethnic/racial groups is important. Moreover, there is room to strengthen the trust between BPE practitioners and other stakeholders. BPE practitioners need to receive unwavering support from stakeholders that have the power to reduce (ideally eliminate) the fear of repercussions (i.e., loss of job, demotion) to be transparent about failures and aspects that are problematic. Ultimately, there is room to improve how assessment information is reported, particularly as it relates to BPE.

ACKNOWLEDGMENTS

This material is based on work supported by the National Science Foundation under Award Numbers 1647327 and 1926935. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. We thank Amy Van Epps for her guidance in conducting this work.

Biographies

Teirra K. Holloman is a Doctoral Candidate in Engineering Education at Virginia Tech, 635 Prices Fork Road, Blacksburg, VA 24061; teirrah@vt.edu.

Walter C. Lee is an Associate Professor of Engineering Education and the Director for Research in the Center for the Enhancement of Engineering Diversity (CEED) at Virginia Tech, 635 Prices Fork Road, Blacksburg, VA 24061; walterl@vt.edu.

Jeremi S. London is an Assistant Professor of Engineering Education at Virginia Tech, 635 Prices Fork Road, Blacksburg, VA 24061; jslondon@vt.edu.

Chaneé D. Hawkins Ash is the Co-founder and Principal of Foresight Strategy Solutions, 2033 Illinois Street, Vallejo, CA 94590; chanee.ash@gmail.com.

Bevlee A. Watford is a Professor of Engineering Education, Associate Dean for Equity and Engagement, and Executive Director of the Center for the Enhancement of Engineering Diversity in the College of Engineering at Virginia Tech, 490 Old Turner Street, Blacksburg, VA 24061; deuce@vt.edu.

Abstract

Background

In the field of engineering education, assessment and evaluation have been given insufficient attention as they relate to broadening participation. We posit that this lack of attention negatively impacts our ability to develop and implement sustainable solutions at scale.

Purpose

The purpose of this article is to explore the presence of an assessment cycle in the literature related to broadening participation in engineering. The assessment cycle refers to the process by which assessment/evaluation is planned, enacted, and used to improve educational processes.

Scope/Method

The scope of this study was influenced by it being situated in a larger study focused on broadening participation of Black Americans in engineering and computer science. We completed a literature map that illuminated a subset of literature broadly related to assessment/evaluation, and subsequently conducted a systematic literature review of 33 publications reporting on assessment/evaluation efforts. Our analysis of these efforts was grounded in a six-stage assessment cycle.

Results

The results of this study highlight common publishing practices related to the assessment cycle in the context of broadening participation. We find that assessment/evaluation is generally published at the program level, focused on student development or academic success as a proxy for program effectiveness, and concentrated on positive claims.

Conclusion

There is room to significantly improve how assessment/evaluation information is published. By highlighting productive and unproductive publishing practices related to assessment/evaluation, this research has important implications for the use and publishing of assessment/evaluation, particularly as it relates to broadening participation in engineering.

1 INTRODUCTION

As the field of engineering aspires to diversify, organizations continue to implement efforts aimed at bolstering the participation of groups that are historically and systematically marginalized (Lichtenstein et al., 2014). This challenge is commonly referred to as "broadening participation in engineering" (BPE). Here, we argue that our ability to develop and implement sustainable solutions in the context of BPE is intertwined with our ability to assess and evaluate existing interventions, programs, and institutions.

Assessment and evaluation are integral to BPE solutions because practitioners continually compare efforts and exchange best practices, relying on insights from others to determine which practices add the most value and should continue being implemented (George et al., 2019). Moreover, because BPE efforts are often overlooked and undervalued, there is also immense pressure on practitioners to secure external funding and retain internal budgets (i.e., soft funding); such work is often devalued due to its association with what is seen as feminized and immaterial labor (Chen et al., 2019). This circumstance raises the stakes, as it relates to accountability and demonstrating return-on-investment (Rincón & George-Jackson, 2016). As noted by Patton et al. ( 2019), "when making tough fiscal decisions, key diversity efforts are often cut or minimized" (p. 178). This reality results in BPE practitioners being forced to balance the pressure to minimize costs while maintaining and improving the outcomes about which institutions are concerned (Baber, 2015). In short, in the context of BPE, the pressure to assess and evaluate comes from the need to (1) improve one's own practices and exchange insights with other practitioners, and (2) justify the need for and use of resources.

The need for improved BPE assessment and evaluation is also echoed by national calls in this area. In fact, "inadequate assessments, metrics, and data tracking" were identified as a key impediment in "Surmounting the Barriers: Ethnic Diversity in Engineering Education" (National Academy of Engineering, 2014, p. 1). Performing self-assessments/evaluations of interventions and programs has also been historically recommended to assist in the recruitment, retention, and employment of people from underrepresented groups in engineering, both at the pre-college and undergraduate levels (Holloman et al., 2018). However, little work has been done to advance the scholarship of assessment and evaluation, particularly in the context of BPE.

First, we describe the limited focus on assessment and evaluation in engineering education research. While publication venues central to the field of engineering education (e.g., Journal of Engineering Education, Journal of Women & Minorities in Science and Engineering, Advances in Engineering Education) address topics related to assessment and evaluation (Cooper et al., 2019; Finelli & Froyd, 2019; e.g., Olds et al., 2005), the conversation primarily focuses on collecting and analyzing data (i.e., issues of measurement). The limitations of this focus become evident when considered through the lens of the assessment cycle. The assessment cycle refers to the process by which assessment (and evaluation) is planned, enacted, and used to improve education processes (Kuh et al., 2015), the latter of which is rarely discussed in engineering education scholarship on this topic. The field of engineering education must pay more attention to the process in its entirety; otherwise, it will miss important aspects that may result in more effective and informed change.

Next, we describe the lack of focus on assessment and evaluation in the context of engineering education. Prior conversations about assessment and evaluation in engineering education have largely focused on outcomes associated with the American Board of Engineering and Technology (ABET; D. B. Knight & Novoselich, 2017; Koehn, 2001; e.g., Moskal et al., 2002; Riley, 2016; Stagner & Jorendt, 2015). This narrow focus has limited our understanding of the role of assessment and evaluation in engineering education. According to J. T. Brown ( 2017), the pressure to assess or evaluate can come from the need to demonstrate efficiency and return-on-investment (ROI), compliance with regulations and standards, and learning or value added. Because of the central role ABET plays in engineering education (Felder & Brent, 2003), the need to demonstrate learning and compliance with regulations/standards is mostly well documented, leaving the needs of BPE practitioners (i.e., to demonstrate efficiency, ROI, and value) less understood.

1.1 Purpose

1.2 Scope and positionality

Before proceeding any further, it is important to situate this study in the positionality and interests of the authors. Each of the five authors identify as Black/African American, and the inspiration for this literature review comes from a larger study focused on the state of broadening participation of Black Americans in engineering and CS. As part of this larger study, we completed a literature map (J. S. London et al., 2020) that illuminated a subset of literature broadly related to the assessment and evaluation of BPE-related efforts, where Black Americans were the focus or among the participants in the article. This review is scoped to this subset of literature. Due to the exploratory nature of this study, we did not further disaggregate the papers by any attributes of the assessment/evaluation efforts themselves.

We thought this continued focus was justifiable because BPE practitioners focused on Black Americans often experience the aforementioned pressures to the greatest extent, largely due to Black Americans being the primary target of most attacks on affirmative action. This truth is a byproduct of the US higher education system being originally designed to educate only the White majority and equip White men for leadership roles (Geiger, 2005; Karabel, 2005; Thelin, 2011). In response to desegregation and the civil rights movement, policies such as affirmative action were put in place to aid access to higher education for students from marginalized communities. Unfortunately, many institutions faced and continue to face attacks on affirmative action policies and practices (Rodriguez, 2007; Shehab et al., 2012).

Combined, these two points led us to conclude that Black Americans provided an advantageous lens through which to examine the scholarship of assessment and evaluation in the context of BPE. As a result, many of the decisions we made as researchers were guided by this focus. We acknowledge that our results may have been different had we primarily focused on other populations (e.g., the Latinx community, women, and/or people with disabilities). However, because much of the literature focuses on the population beyond Black Americans, we believe our findings have implications for BPE broadly and discuss the subsequent literature accordingly.

2 DISCUSSING ASSESSMENT, EVALUATION, AND RESEARCH

Because BPE practitioners must regularly communicate with the broader education community, it is important to use language that is commonly accessible. However, finding a common language as it relates to monitoring progress and informing change (or accountability) is difficult because different communities have their own preferred terms. This difficulty is exacerbated in higher education, where there are multiple conflicting approaches to accountability. According to J. T. Brown ( 2017), there are at least seven fields (e.g., institutional effectiveness, educational measurement) that take a unique approach to this process and many more affiliated associates-such as the Association for the Assessment of Learning in Higher Education (AAHLE), the Association for Higher Education Effectiveness (AHEE), and the American Evaluation Association (AEA). Across these associations, numerous terms are regularly used regarding data collection that informs our decision making, such as assessment, evaluation, institutional research, and institutional effectiveness.

In considering the scholarship of assessment and evaluation, it is important to discuss the complicated relationship among assessment, evaluation, and research. Research is primarily concerned with producing knowledge that is either generalizable or transferable (Tracy, 2010; Winter, 2000). It is the process of inquiry and acquisition of knowledge about a topic or phenomenon. The process is guided by various research methods and methodologies which are ideally informed by clearly identified research questions. The execution of these methods produces data that is analyzed, interpreted, and used to confirm, deny, or create new knowledge about some theory, topic, or phenomenon. Though this rational model does not always describe the reality of the research process, it does provide a general overview (J. Martin, 1982).

In contrast to research, a ssessment and evaluation are primarily concerned with gathering useful information to be used for determining the effectiveness of localized efforts (Pellegrino et al., 2014). Though distinct, assessment and evaluation overlap significantly and are often used interchangeably. We acknowledge that the terms assessment and evaluation emerge from different traditions and offer the following definitions to establish a common language for this paper. According to Palomba and Banta ( 1999), assessment is "the systematic collection, review, and use of information about educational programs undertaken for the purpose of improving student learning and development" (p. 4). In the educational context, it is a mechanism for monitoring progress and informing change, widely perceived as a valuable and essential part of education (Suskie, 2008). According to Allen et al. ( 2002), evaluation is "the use of assessment findings (evidence/data) to judge program effectiveness; used as a basis for making decisions about program changes or improvement" (p. 1). It aids stakeholders and evaluators in determining if an effort (e.g., initiative, program, course, class) is worthy of adoption, continuation, or expansion (Fitzpatrick et al., 2011).

Given our choice to focus on the collection, review, and use of information about BPE interventions, we hereafter anchor on the process of assessment. This process is iterative and comes in many forms (e.g., quantitative, qualitative), fulfills many purposes (e.g., accountability, improvement), includes many stages (e.g., planning, collecting data, etc.), and is conducted at many levels of analysis (e.g., individual, program, institutions; Leskes, 2002; R. Miller & Leskes, 2005; Town & Pierce, 2013). To discuss the breadth of this process, we incorporate insights related to research and evaluation as well. When discussing the collection of assessment data, we leverage the language commonly associated with research, and when discussing the use of assessment data, we leverage the language commonly associated with evaluation. Further disentanglement of these terms and processes is beyond the scope of this paper.

3 AN OVERVIEW OF ASSESSMENT IN EDUCATION

Before narrowing our focus on BPE, it is important to first understand how assessment is used in education broadly. We will discuss (1) attributes by which assessment can be classified, (2) quality considerations associated with assessment, and (3) barriers to implementation.

3.1 Assessment attributes

For BPE practitioners to accurately assess their efforts, well-thought-out assessment plans are needed. A sound assessment plan has four attributes that should be considered and aligned. The first two attributes relate to the focus of the assessment plan (e.g., what and why?), and include the (1) level and (2) purpose. The next two attributes relate to the implementation of the assessment plan (e.g., how?), and include the (3) function and (4) form. Each attribute is discussed further below.

3.1.1 The focus of assessment plans

There are several approaches to discussing the focus of an assessment plan, either focusing on the different purposes (i.e., why an assessment plan is needed) or the different levels (i.e., aspects of the experience or environment). Regarding the levels of assessment, according to R. Miller and Leskes ( 2005), there are five different categories: (1) assessing individual student learning within courses, (2) assessing individual student learning across courses, (3) assessing courses, (4) assessing programs, and (5) assessing the institution. Regarding the purpose of assessment, common categories include accountability, improvement (Leskes, 2002), grading, learning, and coaching (Hansen, 2012). It is important to understand the different classifications because they have implications for how assessment questions are formulated. For example, at the individual level, one might ask "How well has the student achieved the learning outcomes set for the course?" whereas at the program level one might ask, "Does the program advance institution-wide goals as planned?" To assist readers with exploring the various levels and purposes of assessment across the literature, we provide an overview in Table 1.

Classification

Category

Definition

Level

Individuals

Investigating how well individual participants are learning and meeting predetermined goals within and across courses (R. Miller & Leskes, 2005)

Purpose

Assist learning

Provide specific information about students' strengths and difficulties with learning (National Research Council, 2001)

3.1.2 The implementation of assessment plans

Once the level and purpose have been identified, practitioners can determine the function (i.e., how the results will be used) and form (e.g., appropriate methods). To assist readers with exploring the functions and forms to consider, we provide an overview in Tables 2 and 3.

Category

Definition

Timing of data collection

Formative

Gathering information about student learning during the progression of a course or program, usually repeatedly

During

Summative

Gathering information at the conclusion of a course, program, or undergraduate career

End

Readiness

Exploring the satisfaction of prerequisite entry skills to remedy entry deficiencies and/or categorize learning groups

Before

Placement

Determine necessary course or unit objectives

Before

Diagnostic

Identify common breakdowns in mental models

During

Category

Definition

Embedded

Gathering data about student learning in a way that is ongoing and, thus, built into the teaching-learning process

Direct

Gathering data that demonstrates the learning objectives/outcomes of interest

Indirect

Gathering data that explores participant perception about learning and the surrounding environment

Qualitative

Collecting and analyzing textual data (surveys, interviews, focus groups, conversational analysis, observation, ethnographies) with emphasis on the context within which the study occurs

Quantitative

Collecting and analyzing data often from surveys, with the intent to generalize and make inferences

Regarding function, the most common approach to differentiating is to distinguish between formative assessment and summative assessment: formative assessment is focused on making improvements for current students or participants, whereas summative assessment is focused on following cohorts (Leskes, 2002). However, other categories include readiness, placement, and diagnostic. These distinctions are important because they enable educators to identify the focus of measurement, nature of the sample, item difficulty, time of administration, and how the results will be used. For example, according to M. D. Miller et al. ( 2008), assessments that occur prior to instruction or intervention can either assist with remedying entry deficiencies, assigning students to groups, informing instructional planning, or determining advanced placement ( readiness and placement); an assessment that occurs during instruction can either improve and direct learning through ongoing feedback, or remedy errors related to persistent learning difficulties ( formative and diagnostic); and assessment that occurs at the end of instruction can be used to assign grades, certify accomplishment, or evaluate teaching ( summative).

The forms of assessment tend to come in five main categories: embedded, direct, indirect, qualitative, and quantitative. Several factors implicate which form of assessment may be most appropriate to use. For example, the assessment form tends to indicate the type of data that will be collected in light of the assessment question. It is important for the assessment form to be aligned with the overarching assessment purpose, questions, and/or objectives. Resources, time constraints, accessibility of data, and analysis tools must also be considered. Lastly, because assessment results are often used to convince stakeholders, it is also important to consider the epistemological frameworks of those who will be receiving the results.

3.2 Assessment quality

As previously mentioned, there are some similarities between research and assessment, namely when it comes to quality. We define "quality" as the extent a process exemplifies standards and criteria noted in germane literature. For example, Moskal et al. ( 2002) note that the only way for engineering educators-and in our case BPE practitioners-to adequately monitor progress is to be aware of the important issues that surround assessment measures. Much like with engineering education research, two major issues related to assessment quality are validity and reliability. Validity refers to the degree to which a measurement measures what it is intended to measure or the degree to which interpretations of results are used correctly. Reliability refers to the degree of consistency of assessment scores (Moskal et al., 2002). These terms are often used when researchers employ quantitative methods. When researchers use qualitative methods, they are likely to use terms like transferability, trustworthiness, and dependability as a criterion by which to judge the work (Golafshani, 2003). Transferability is another way of looking at external validity, trustworthiness at internal validity, and dependability at reliability (Leydens et al., 2004). Ultimately, regardless of the approach, it is imperative that criteria are established, systematically used, and openly discussed in reports and publications.

Considering the pragmatic nature of assessment and pressure for BPE practitioners to demonstrate their value to a variety of external stakeholders, it can be difficult to determine what is considered "good assessment" given the many levels, purposes, forms, and functions. Fortunately, criteria regarding assessment best practices have been well documented over the years. For example, Suskie ( 2008) provides a concise list of five best practices, which summarizes over a dozen of these various sources. These characteristics state that good assessments (1) are used, (2) are cost effective, (3) yield reasonably accurate and truthful results, (4) are valued, and (5) focus on and flow from clear and important goals. Some suggestions to achieve these characteristics are to consider the purpose of assessment, have collaboration occurring between the student and the teacher, and use the feedback from the assessment when making future decisions. While these suggestions are informative, practitioners must interpret them as needed to fit their varying and ever-changing contexts.

In short, there is no "one-size-fits-all" approach to assessment, and it should not be perceived as such. "To say that assessment is [a] good assessment is akin to saying that a medical test is a good test; each can provide useful information only under certain circumstances" (National Research Council, 2001, p. 222). In other words, assessment can be considered good only when all parts are in alignment, it is implemented with a goal or need in mind, and has been validated and deemed reliable.

3.3 The barrier to assessment

Nonetheless, numerous factors make achieving "good assessment" difficult. According to Hoey and Nault ( 2008), these barriers primarily relate to aspects of trust (e.g., trust in the motives, questions, methods, data, etc.), which they viewed as fundamental to assessment design and implementation. Trust will either facilitate or pose obstacles to change implementation. Accordingly, it is important to foster trust via three main avenues: (1) culture, (2) structure, and (3) leadership. Each of these avenues must operate in a way that promotes assessment to establish stakeholders' buy-in. For example, an organization's culture would ideally promote open communication, value evidence-based decision making, offer incentives for improvement, and provide resources for those charged with an assessment. To promote the value of assessment, leaders must also make it a normalized practice within the organization. This process should include using those results in ongoing discussions and/or in creating new initiatives; and allocating funds, training, and/or personnel to aid in conducting assessment and the implementation of results. Unfortunately, for many BPE practitioners, trust is not there. With the overwhelming and constant pressure to show value, it is difficult to report any shortcomings or failures. Additionally, being often among the first to lose internal funding makes it challenging for BPE practitioners to trust that their work is valued.

The scholarship of assessment requires an understanding of how assessment can be operationalized, classified, evaluated, and promoted. Next, it is important to understand how each of these components fit together.

4 CONCEPTUAL FRAMEWORK: AN ASSESSMENT CYCLE

To systematically examine the use of assessment, we situate our investigation in a six-stage assessment cycle (Kuh et al., 2015). This assessment cycle is a continual process that begins with developing a plan and identifying desired outcomes, followed by collecting and analyzing data. Next, the assessment results are shared so that necessary change can be identified, implemented, and assessed. An overview of each stage will be discussed in the following sections.

4.1 Step 1: Plan and identify outcomes

4.2 Step 2: Collect data

Once data sources have been identified, the collection of data can begin. According to Worthen, Borg, and White (as cited in Fitzpatrick et al., 2011, p. 348), there are five categories by which data collection can be obtained: (1) directly from individuals identified as sources of information, (2) by an independent observer, (3) by a technological device, (4) with unobtrusive measures, and (5) from existing organizational information or formal repositories or databases. Information collection arrangements must also be made, identifying who will collect data, under what conditions, and when will collection take place. Important deadlines, timelines, and resources should also be considered. Given the focus on collecting data, this step in the cycle often overlaps with education research.

4.3 Step 3: Analyze data

For each assessment question, evaluators should ensure that they have a plan for analyzing the corresponding data or information. Fitzpatrick et al. ( 2011) note that it is important to specify how the information will be analyzed. They suggest doing so through two steps. First, one should determine what techniques will be used during analysis (e.g., descriptive statistics, emergent coding). Next, one should then determine by what means these analyses will take place (e.g., Excel, NVivo). When interpreting these results, Fitzpatrick et al. recommend involving stakeholders and clients because varying perspectives can result in varying interpretations. This step also overlaps with education research.

4.4 Step 4: Share results

When sharing results, it is important to be aware of when this information is needed, who needs it, and in what format. Fitzpatrick et al. ( 2011) offer a matrix to be considered for each assessment question to help organize this information. It includes understanding the audience, the necessary content, reporting format, date of the report, and the context it will be presented; this level of detail and organization also emphasizes the notion that questions that are to be investigated should be intentional, important, and useful to stakeholders. The timing, audience, and format of the results when they are shared is another notable difference between assessment and research.

4.5 Step 5: Identify and implement changes

After results have been disseminated, the hardest part of the cycle begins, requiring stakeholders to act on the results. Kuh et al. ( 2015) attribute this difficulty to an overemphasis on data collection and research design and suggests that emphasis should shift to implementation and assessing impact. Ensuring the utility at the beginning of an investigation will help in mitigating future oversight. Additionally, Kuh and colleagues suggest linking assessment to both internal and external processes such as accreditation and promotion. This step in the cycle also makes it difficult to separate assessment from evaluation because both processes require that the information previously collected be used to advance some desired end.

4.6 Step 6: Assess the impact of changes

This final stage of the cycle officially closes the assessment loop by ascertaining impact, and by extension, propelling the process into another iteration. Implementing changes does no good if stakeholders are unaware of the positive or negative consequences that resulted.

5 RESEARCH DESIGN

To explore the use of assessment/evaluation in the context of BPE, we conducted a systematic literature review (SLR). SLRs have been used to summarize, critically evaluate, and reconcile conflicting evidence to inform policy and practice (Borrego et al., 2014). Despite the well-established use of literature reviews in engineering education, SLRs are not used as often (Borrego et al., 2014). The value in an SLR can be found in the varying degrees of its purpose from describing the current state of knowledge to evaluating a theory, to identify gaps in the literature (Petticrew & Roberts, 2006).

In order to conduct this SLR, two primary sources were used to guide our research design. Petticrew and Roberts ( 2006) explain the details of the process as it relates to various disciplines conducting SLRs, and Borrego et al. ( 2014) operationalize the SLR method for the engineering education community. The five major steps of this SLR include the following:

1. Formulate guiding research questions and corresponding inclusion criteria.

2. Find and catalog sources.

3. Critique and appraise the quality of selected literature.

4. Address bias, validity, and reliability concerns.

5. Synthesize insights.

These steps will serve as a framework for the structure of the remainder of this section. The following sections will break down the process of collecting and identifying articles as a part of a larger study and identifying those appropriate for this study.

5.1 Formulating guiding research questions and inclusion criteria

Our exploration of the literature was guided by the following research question: How are assessment and evaluation practices related to broadening participation communicated in the literature? The initial inclusion criteria used during the collection of literature are as follows: (1) publication date range, (2) database selection, and (3) search string selection. This paper is part of a larger project, and more about these processes can be found in a previously published article about the overall process of conducting an SLR (Phillips et al., 2017). The date range (e.g., 1970-2017) was informed by historical events associated with efforts to broaden the participation of Black Americans in engineering and CS, specifically the establishment of several national-level efforts focused on BPE in the 1970s. For example, the National Action Council for Minorities in Engineering, Inc. (NACME) was established in 1974 and the NSBE was established in 1975. With the help of a librarian, databases and search strings were identified, as illustrated in Table 4. They helped to limit publication bias by keeping the database options as wide-ranging as possible. Additionally, their expertise was used to identify search strings, alongside input from researchers in our professional network and keywords associated with "Diversity" from the Engineering Education Research Taxonomy (Finelli et al., 2015).

Database name

Search string

Note

Education Source and PsycINFO (EBSCOhost interface)

((bias OR discrimination OR multicultural* OR inclusive* OR racism OR prejudice) OR

(motivation OR attainment OR achievement OR aspiration OR persist* OR retention)) AND

((AB african w2 american OR SU african w2 american OR TI african w2 american) OR (AB black OR SU black OR TI black) OR (AB people N2 color* OR SU people N2 color* OR TI people N2 color*))

AND

((AB STEM OR SU STEM OR TI STEM) OR (AB engineer* OR SU engineer* OR TI engineer) OR (AB "computer science" OR SU "computer science" OR TI "computer science"))

Search all fields for words used to include or exclude people

Search abstract, title, subject headings for terms used for African American

Search abstract, title, subject headings for STEM, engineering, and computer science

Compendex and INSPEC (Ei Village interface)

< ((motivation OR attainment OR achievement OR aspiration OR persist* OR retention) WN All fields) > OR < ((bias or discrimination or multicultural* or inclusive* or racism or prejudice) WN All fields) >

AND

< ((((african ONEAR/2 american) WN KY) OR ((black) WN KY)) OR ((people NEAR/2 color) WN KY)) >

AND

< ((STEM OR engineer* OR "computer science") WN KY) >

Quick search, autostemming off, search all fields

Search subject/title/abstract, autostemming off

Search subject/title/abstract, autostemming off

Abbreviation: STEM, science, technology, engineering, and mathematics.

5.2 Finding and cataloging sources

After removing 100 duplicate publications, 1080 articles were evaluated against three additional hierarchical inclusion criteria: (1) Is the article written in English and about education or the science, technology, engineering, and mathematics (STEM) workforce in the United States? (2) Is the article focused on engineering or CS in any context, or STEM disciplines in a K-12 context? (3) Is the article focused on issues or the experiences of Black Americans, or on some aspect of the wide variety of topics associated with broadening participation? Criterion 3 allowed for the inclusion of publications that might not focus solely on Black Americans. This inclusion was done to account for the possibility of learning from publications that emphasized Black Americans without solely focusing on them. We intended to be mindful of the realities of targeting a specific group in the current legal landscape and the frequency with which multiple marginalized groups are included in a single intervention. In total, 470 out of 1180 (40%) articles met the criteria and spanned across the education-to-workforce pathway, as illustrated in Figure 1. Additional details about the information in Table 4 is included in a systematic mapping review that has been published elsewhere (J. S. London et al., 2020). This paper focuses on a subset of 51 articles related to assessment.

5.2.1 Limitations of finding sources

Given that all initiatives, programs, and institutions are not mandated to publish assessment/evaluation results, this review can only speak of the work made available at the time of the literature search. Additionally, we must acknowledge that assessment was not a search term used in data collection, as this study is the result of a larger project that dictated data collection. Lastly, due to our initial interest and the exploratory nature of this study, we did not disaggregate literature based on assessment level or intervention type and we may have overlooked articles focused solely on the Latinx community, women, and/or people with disabilities. Each of these aspects bring limitations to this work and therefore cannot speak to the total landscape of BPE assessment.

5.3 Critique and appraise quality of literature

Following the documentation of the literature map results, the initial 51 assessment documents underwent a quality check. Four questions were used to appraise all documents being utilized in this SLR. The questions were as follows:

1. Is the problem/purpose/aim of the study clearly stated?

2. Is the sampling strategy apparent and appropriate? Does the sample represent the target population?

3. Is information about data collection procedures apparent and appropriate?

4. Is information about the approach to analyzing data apparent and appropriate for addressing the study's purpose?

Following this final step of appraisal, only 34 documents remained. Unfortunately, one document was not accessible given available resources and thus was discarded from our list of articles to review. This resulted in a final total of 33 assessment documents.

5.4 Address bias, validity, and reliability concerns

Although efforts were made to minimize bias, limitations remain. According to Cook et al. ( 1997) the limitations of an SLR stem from (1) the quality and quantity of the selected studies, and (2) the quality of the systematic review procedures. One fault in this study lies in the fact that the literature in this review was collected in January of 2017; therefore publications after that time have been excluded from this review. Lastly, this study is a result of seeking BPE assessment literature across the education pathway and therefore we must acknowledge the varying histories that exist between K-12 and postsecondary education and their use of assessment.

5.5 Synthesize insights

The synthesis step included extracting data and coming up with themes. After the final quality check, data were extracted from each of the 33 documents. For each document, we identified Where (what is the context of the program?), Who (which populations are being used to generate information?), What (what assessment questions or objectives are communicated?), How (what metrics and outcomes are being assessed?), When (what is the temporal, duration, proximity to the experience, the longitudinal nature of the assessment?), Findings (what sort of evaluative claims are people making?), and Implications (how will the results be used?), followed by a critique. A summary table was composed to provide a comprehensive view and summary of each study. These documents were compared across each section of every document and the pros, cons, similarities, and suggestions were identified for improvement based on our conceptual framework of the assessment cycle.

6 RESULTS AND DISCUSSION

Following the data extraction from the final 33 documents, we utilized Kuh and colleagues' assessment cycle to inform our analysis. Five major themes emerged as it relates to the initial stages of the assessment cycle, covering topics related to levels of assessment, purpose, and design. More specifically, Theme 1 focused on where the published assessment tends to occur. Theme 2 describes the sparse use of assessment at some levels. Theme 3 recounts the research designs used in BPE assessment plans. Theme 4 depicts the types of data being collected as a proxy for program performance. Lastly, the kinds of claims that are made based on the results of these assessment plans are discussed in Theme 5. Each theme is discussed below.

6.1 Theme 1: The assessment articles in our study most frequently focused on programs at the K-12 level and predominantly White institutions of higher education, and incorporated students with various marginalized identities

The papers included in this SLR focused on assessments most frequently used in the context of programs at the K-12 level and predominantly White institutions (PWIs) of higher education. Of the 33 articles identified, 9 were at the K-12 level, 20 were at the undergraduate level, 1 was at the graduate level, and 3 included participants across levels. It is important to note that none of the articles focused on community college and workforce segments.

6.2 Theme 2: The assessment articles in our study were most frequently program-level assessment and seldom focused on institution-level assessment or the broader context surrounding underrepresented students

Levels of assessment and their purpose play a major role in guiding the types of questions and topics explored in the assessment. Our analysis found that most articles investigated program effectiveness based on either a direct or indirect measure of student development or performance. Occasionally, some investigations would stray away from the status quo by moving beyond the program level. For example, Litzler et al. ( 2010) investigated climates in engineering at various universities in the United States using a 132-item survey instrument, while McCullough et al. ( 2014) created a rubric for students to complete weekly that informed their mentoring sessions. An included dissertation (Jia, 2015) assessed how stricter high school math prep impacted the performance of underrepresented students (i.e., Black Americans, Latinos/as/xs, and women) as compared to White men. While these topics were not focused on Black Americans, results shed light on findings directly related to Black Americans' persistence in engineering. Lastly, White et al. ( 2008) presented challenges and issues that arose during the creation of a program evaluation plan, but they did not present the evaluation plan itself.

Moreover, the seemingly narrow focus of assessment also limits the degree to which we understand the broader context within which BPE stakeholders work. If we understand that BPE is a multifaceted complex task, we need to consider our overall understanding of the problem space. When thinking about the context in which Black Americans are situated, there is much to be understood about how the engineering ecosystem does and does not adequately support them. One means of gathering more information is through assessment at various levels within various contexts. Course-level and institution-level assessment from the perspective of Black American students is necessary. We need to understand the parts of the environment that were not designed for Black American students to succeed just as much as we need to understand the interventions that were created with them in mind. Our understanding of one helps to inform the changes implemented to improve the other.

6.3 Theme 3: The assessment articles in our study utilized both quasi-experimental designs and nonexperimental designs

Assessment designs varied across articles and could be categorized into one of five types. Three quasi-experimental designs were frequently employed: (1) pretest-posttest design, (2) posttest-only design, and (3) posttest only design with repeated measures. Additionally, nonexperimental designs emphasized both formative and summative functions of assessment. Table 5 provides an overview of each design including references to a few example articles.

Design

Description

Example articles

Posttest only

A single observation of any kind performed after a treatment

Donnelly ( 2007); Gardner-McCune et al. ( 2013); J. Good et al. ( 2002); Lee and Cross ( 2013)

Pretest-posttest

Two observations of any kind performed before and after a treatment

Hrabowski and Maton ( 1995); Clark et al. ( 2015); D. Knight et al. ( 2013)

Posttest only with repeated measures

Performing multiple observations on the same sample over time (i.e., cohort performance over time)

Murphy et al. ( 2010); Jiang et al. ( 2005)

Nonexperimental-formative

Multiple observations of participants during the progression of the intervention

E. Brown ( 2007); J. M. Good et al. ( 2000); McCullough et al. ( 2014); Ogilvie ( 2007); Wilson et al. ( 2015)

Nonexperimental-summative

Multiple observations of an intervention occurring at the end of each intervention cycle

Lam et al. ( 1997); Kendricks and Arment ( 2011); Lasser and Snelsire ( 1996)

6.3.1 Posttest only

This design describes assessment plans that involved collecting data at the end of the intervention-sometimes including more than one data type. For example, Donnelly ( 2007) administered multiple assessment tools after the intervention. Post intervention, data were collected (via surveys, interviews, and focus groups) from the participants to assess the effectiveness of the program. Additionally, faculty who participated in the program completed surveys, providing insights on what they perceive to be important elements of the program.

6.3.2 Pretest-posttest

This design describes assessment plans that involved collecting data before and after an intervention, typically the same types of data, to demonstrate growth and development. For example, D. Knight et al. ( 2013) administered pre- and post-surveys to measure identity development in program participants. One survey was administered before and after two summer bridge programs. Additionally, another survey was conducted before and after two first-semester courses. Data were used to compare pre- to post-results, across demographic groups, and across interventions.

6.3.3 Posttest only design with repeated measures-Cohort overtime

This design describes assessment plans that involved collecting data from a particular cohort of students at the end of an intervention and longitudinally thereafter. At the undergraduate level, for example, some studies would follow students from their participation in a bridge program to their completion of the first/second year of an engineering program or through graduation. Murphy et al. ( 2010) observed the graduation rates of program participants and determined the likelihood that someone would graduate if they participated as opposed to those that did not; Jiang et al. ( 2005) tracked five North Carolina Louis Stokes Alliance for Minority Participation cohorts over a 5-year period.

6.3.4 Nonexperimental-Formative

This design describes assessment plans that involved observing participants and collecting data throughout the intervention. For example, J. M. Good et al. ( 2000) analyzed student journal entries that were collected throughout the entire academic year. It was noted that the program coordinator engaged with students and their journal entries weekly to "encourage discourse regarding program development and improvement with each mentor" (J. M. Good et al., 2000, p. 377). Similarly, E. Brown ( 2007) used measures like student accolades and changes in academic major during the intervention as factors of program success.

6.3.5 Nonexperimental-Summative

We previously described assessment plans that involved collecting data about a cohort of students at the end of an intervention cycle, repeatedly over time (i.e., posttest only design with repeated measures). The nonexperimental (summative) design is somewhat similar in that data is collected at the end of an intervention. However, the difference for this design is that data are not collected from the same group of participants. Lam et al. ( 1997) illustrate this. They aimed to better understand a multifaceted program, Increasing Diversity in Engineering Academics (IDEAS), that provides support for students academically, socially, emotionally, financially, and professionally year-round throughout their undergraduate careers. Therefore, an analysis of the pros, cons, and problems associated with the IDEAS program was conducted at the end of the intervention cycle. Similarly, Lasser and Snelsire ( 1996) evaluated the benefits of proactive mentoring over several years, using various measures to represent snapshots of the program over time.

6.4 Theme 4: The assessment articles in our study most frequently investigated program outcomes using indirect and direct measures of student development or academic success as a proxy for program performance

While a variety of assessment questions were expressed in the articles, most articles sought to determine the impact of an intervention program on participants. This entailed investigating how successful a program was based on the participants' likelihood of pursuing or persisting in STEM (Anderson-Rowland & Ruben, 2008; Dunn & Veltman, 1989; Wilson et al., 2015), affecting participants' attitudes and interest toward STEM (Clark et al., 2015), or participants' self-efficacy in STEM-related areas (Di Salvo et al., 2013).

6.5 Theme 5: The assessment articles in our study most frequently highlighted positive evaluative claims about program performance rather than assessing areas for change or assessing the impact of those changes

Given the pressures for BPE practitioners to prove their value, it is no surprise that not all authors highlighted places for improvement or any shortcomings that were revealed through their assessments. Similarly, knowing that resources are often scarce, the lack of discussions regarding the impact of changes that were made was not surprising. Just as it costs to host the initial intervention, it is also costly and time consuming to revisit and make changes to it.

7 IMPLICATIONS AND FUTURE WORK

Our research has important implications for the use of assessment and evaluation, and the assessment cycle, in educational practices related to BPE. Our paper sought to examine assessment/evaluation practices as they relate to BPE efforts. Although we have discussed a variety of topics, attributes, and considerations, we now offer implications grounded in the assessment cycle to assist practitioners with (1) planning and identifying outcomes; (2) collecting and analyzing data; (3) reporting and sharing results; and (4) identifying, implementing, and assessing change. Though our study was motivated by our research teams' focus on Black Americans, our findings are broadly applicable to BPE stakeholders focused on a variety of underrepresented or otherwise marginalized groups. Future exploration should consider disaggregating data based on levels (i.e., individual, program, institution) given that the associated assessment or evaluation may look differently as purposes differ and explore the extent to which BPE assessment/evaluation is implemented differently depending on the target population.

7.1 Implications for planning and identifying outcomes

In doing so, it is important to focus not only on the success stories-as most assessment articles we reviewed sought to demonstrate how well their programs were preparing students-but also on deficiencies and areas for improvement. If we are to make progress in this area, we must be willing to critically review existing efforts and constantly share ways to improve. Doing so requires establishing a culture of trust, where practitioners do not fear that their programs' sustainability will be jeopardized by sharing results about things that did not go well.

Practitioners should also specify the level of assessment in their assessment reports. Doing so is important because this decision greatly informs the types of questions and objectives being set in an assessment plan (R. Miller & Leskes, 2005). As our results suggest, assessment or evaluation in the context of BPE is seldom used (or published) at the institution level; this gap in the literature presents an opportunity for future work and scholarship.

Lastly, there is a need for more publications that focus on efforts beyond the context of PWIs and the K-12 level. Program context matters when it comes to complex issues, and the lack of assessment publications focused on efforts in other settings, such as HBCUs and industry, has implications for our ability to address this challenge, particularly as it relates to Black Americans. If a large percentage of the population of interest attends a particular type of university, for example, it is important to capture what interventions do and do not work in those contexts. Additionally, there is a need for more work that focuses on analyzing the short- and long-term impact of interventions in various environments.

7.2 Implications for data collection and analysis

There is also a need for more consistent reporting of results using quality metrics. The use of reliable and valid instruments increases the credibility of results tremendously, and reporting on the validity and reliability of the instruments used is just as important as selecting them. While some studies included validity and reliability measures, they were rare. Without such information, it is difficult for the reader to evaluate the credibility of the results and/or determine which results are transferable. If the internal validity is not confirmed, the external validity is compromised. When considering the indicators used to evaluate the quality of qualitative work (e.g., transparency, credibility, ethical considerations, and meaningful coherence) in educational research (Tracy, 2010), it should be noted that these indicators are applicable here as it relates to reporting on both qualitative and quantitative methods.

Lastly, there is also a need for more longitudinal designs that connect to the societal need for a more diverse engineering workforce. The absence of such designs creates a gap in understanding the long-term effectiveness of programs because many BPE interventions implicitly aim to contribute to the production of a more diverse engineering workforce. Longitudinal designs can also aid in tracking changes across all levels of assessment, specifically at the institutional level. Given that some schools of thought surrounding change in higher education view change as slow, continuous, and long-term (Kezar, 2001), longitudinal designs aid in our assessment and evaluation at the institutional level.

7.3 Implications for reporting and sharing results

Future assessment reports should be more transparent. Here, we learn from suggestions intended for educational research: "Transparent research is marked by [the] disclosure of the study's challenges and unexpected twists and turns, and [the] revelation of the ways research foci transformed over time" (Tracy, 2010, p. 842). We argue that the same could apply to assessment because one of the major barriers to change in assessment is a lack of trust; increased transparency could help strengthen trust between BPE practitioners and their stakeholders.

With transparency in mind, it may be harder to forget specific details about various aspects of the assessment process. For example, providing more clarity on the program structure in conjunction with discussions about the results is imperative. The easier it is for readers to understand the context and the findings, the more trustworthy a report will read. The use of properly labeled tables and figures reporting on participant demographics and program components (e.g., logic models) is also needed, as this was not included in many of the documents. Additionally, reports should document the participant selection criteria because this information allows for stakeholders to make better supported claims about student performance. Lastly, we reiterate that negative findings were rarely published. We encourage practitioners to share both their success and their shortcomings and lessons learned. According to the literature we reviewed, it would appear that almost everything is working effectively, which we know anecdotally is not the case. This reality leads us to also call upon administrators and other stakeholders to support and believe the reports and statements by BPE practitioners who discuss and advocate for the needs of their students. BPE practitioners need this support from stakeholders to ensure that their work is valued and uplifted so that they can be transparent without the fear of repercussions (i.e., loss of job, demotion) for reporting on failures, shortcomings, and aspects that require improvement, change, or reform.

Lastly, when reporting and sharing results, practitioners should also be mindful of narratives they are creating in the process (i.e., deficit vs. asset-based). The framing of these results can impact how minoritized communities are represented within engineering and thus impact the types of efforts created to address BPE-related issues. Practitioners should be particularly mindful of this concern when planning to use student achievement as data metrics.

7.4 Implications for identifying, implementing, and assessing change

Future assessment/evaluation reports should document insights related to all parts of the assessment cycle. We acknowledge that attending to the end of the assessment cycle is the hardest to complete (Kuh et al., 2015). As a result, it is not surprising that the implications of the assessment results and future work were seldom mentioned. It is also not surprising that more thorough discussions surrounding the end of the assessment cycle in BPE were seldom found in our literature search. Unfortunately, this significantly limits our ability to learn from the work done by others. While this omission does not mean that assessment/evaluation results did not inform changes, the absence of this information may leave a reader wondering about the utility of the findings-for those hosting the intervention and stakeholders desiring to learn from them.

Finally, it is worth acknowledging the challenges associated with documenting, tracing, and making sense of the impact of an intervention, regardless of the context. J. London and Cox ( 2015) note that these challenges center around issues of attribution, assessment/evaluation, and interpretation. The inherent challenges associated with demonstrating impact in any context simply add even more pressure when engaging in impact-related exercises in the BPE context. Until the issues associated with impact, in general, are resolved, BPE practitioners can only do so much regarding reporting when key stakeholders (especially donors or sponsors) desire absolute certainty of impact as the minimum starting place. Instead of focusing on executing high-quality interventions with fidelity, they are forced to focus more attention on gathering evidence to demonstrate value.

8 CONCLUSION

The purpose of this study was to explore the presence of the assessment cycle in the literature related to BPE. In doing so, we hoped to illuminate the extent to which the entire assessment cycle is explored in the engineering education literature. We also intended to provide an overview of the nuances associated with planning, enacting, and using assessment/evaluation in educational practices. By grounding our SLR of assessment articles in a six-stage assessment cycle, we were able to not only highlight aspects of the cycle that were present but also emphasize that which was missing. For example, we note the omission of negative results as well as information about the change that resulted from the process. This bias toward positive reporting in the articles we reviewed is of concern because we have to be comfortable saying "This did not work" if we are going to move forward. We argue that the constant and overwhelming pressures to justify the need for and use of resources or demonstrate efficiency and ROI make doing valuable and effective work harder for BPE practitioners.

The uneven amount of pressure applied to BPE efforts must also be revisited if we (as a field) agree that broadening participation of historically underrepresented ethnic/racial groups is important. Moreover, there is room to strengthen the trust between BPE practitioners and other stakeholders. BPE practitioners need to receive unwavering support from stakeholders that have the power to reduce (ideally eliminate) the fear of repercussions (i.e., loss of job, demotion) to be transparent about failures and aspects that are problematic. Ultimately, there is room to improve how assessment information is reported, particularly as it relates to BPE.

ACKNOWLEDGMENTS

This material is based on work supported by the National Science Foundation under Award Numbers 1647327 and 1926935. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. We thank Amy Van Epps for her guidance in conducting this work.

Biographies

Teirra K. Holloman is a Doctoral Candidate in Engineering Education at Virginia Tech, 635 Prices Fork Road, Blacksburg, VA 24061; teirrah@vt.edu.

Walter C. Lee is an Associate Professor of Engineering Education and the Director for Research in the Center for the Enhancement of Engineering Diversity (CEED) at Virginia Tech, 635 Prices Fork Road, Blacksburg, VA 24061; walterl@vt.edu.

Jeremi S. London is an Assistant Professor of Engineering Education at Virginia Tech, 635 Prices Fork Road, Blacksburg, VA 24061; jslondon@vt.edu.

Chaneé D. Hawkins Ash is the Co-founder and Principal of Foresight Strategy Solutions, 2033 Illinois Street, Vallejo, CA 94590; chanee.ash@gmail.com.

Bevlee A. Watford is a Professor of Engineering Education, Associate Dean for Equity and Engagement, and Executive Director of the Center for the Enhancement of Engineering Diversity in the College of Engineering at Virginia Tech, 490 Old Turner Street, Blacksburg, VA 24061; deuce@vt.edu.

# data[].attributes.attachment_link.title
The assessment cycle: Insights from a systematic literature review on broadening participation in engineering and computer science

# data[].attributes.attachment_link.description
Teirra K. Holloman , Walter C. Lee , Jeremi S. London , Chaneé D. Hawkins Ash , Bevlee A. Watford , Bevlee A. Watford Department of Engineering Education, Virginia Tech, Blacksburg, Virginia, USA Search for more papers by this author Chaneé D.

# data[].attributes.attachment_link.title
Research Guides: GIS and Data Resources: Data Analysis & Visualization

# data[].attributes.attachment_link.description
Research Guides: GIS and Data Resources: Data Analysis & Visualization

# data[].attributes.attachment_link.readable_text
The Journal of Student Affairs Inquiry (JSAI) is an online, peer-reviewed publication hosted by Student Affairs Assessment Leaders (SAAL).

Inquiry is an intentional process of deep learning, reflection, planning, and action. It involves individual and interconnected components of assessment, evaluation, and research that support problem-solving and decision-making. JSAI supports inquiry across all functional areas within student affairs. If you have questions about your idea for a submission, reach out to the editorial leadership team ( jsai.editors@gmail.com).

Purpose

To this end, JSAI provides a space for practitioner-scholars to engage with the inquiry process to

understand and improve student learning, development, success, and experiences;

promote and enhance equity-minded inquiry, policy, and practice in student affairs; and

advance the practice and utility of inquiry in student affairs across programs, institutions, and the field.

Commitment

The editorial leadership team is committed to diversity, equity, accessibility, and inclusion. In addition to intentionally and strategically diversifying the editorial board to represent the field, we have expanded the type of manuscripts we publish to invite more voices to engage in the inquiry process. We are working to provide alternative formats (e.g., DAISY, MP3, Text, PDF) to enhance the accessibility and usability of JSAI.

These efforts align with the SAAL ethos, CAS Standards, and the Professional Competencies in Student Affairs.

Publisher: Student Affairs Assessment Leaders

# data[].attributes.attachment_link.description
The Journal of Student Affairs Inquiry (JSAI) is an online, peer-reviewed publication hosted by Student Affairs Assessment Leaders (SAAL). Inquiry is an intentional process of deep learning, reflection, planning, and action. It involves individual and interconnected components of assessment, evaluation, and research that support problem-solving and decision-making.

# data[].attributes.headline
Mapping and Assessing Student Learning in Student Affairs

# data[].attributes.subject
Mapping and Assessing Student Learning in Student Affairs

# data[].attributes.attachment_link.provider_name
Assessment and Curriculum Support Center

# data[].attributes.attachment_link.title
Rubric Bank - Assessment and Curriculum Support Center

# data[].attributes.attachment_link.description
We would like to share your rubric too. Please email it to us.

# data[].attributes.headline
Why aren't grades good enough to assess student learning.

# data[].attributes.subject
Why aren't grades good enough to assess student learning.

# data[].attributes.attachment_link.readable_text
Posted on November 17, 2013 at 6:55 AM

Why aren't grades sufficient evidence of student learning?

1. Grades alone do not usually provide meaningful information on exactly what students have and have not learned. So it's hard to use grades alone to decide how to improve teaching and learning.

2. Grading and assessment criteria sometimes differ. Some components of grades reflect classroom management strategies (attendance, timely submission of assignments) rather than achievement of key learning outcomes.

3. Grading standards are sometimes vague or inconsistent. They may weight relatively unimportant (but easier to assess) outcomes more heavily than some major (but harder to assess) outcomes.

4. Grades do not reflect all learning experiences. They provide information on student performance in individual courses and assignments but not student progress in achieving program-wide or institution-wide outcomes.

That said, the grading process can provide excellent evidence of achievement of key learning outcomes, and using information from the grading process in this way can make assessment faster, easier, and more meaningful. NILOA (the National Institute for Learning Outcomes Assessment) has recently published a paper on how Prince George's Community College in Maryland is doing exactly this: http://learningoutcomesassessment.org/OccasionalPapernineteen.html.

You'll see from the NILOA paper that using the grading process to collect assessment evidence works only when faculty are willing to collaborate and agree on at least base grading criteria. I often suggest a two-part rubric: the top half provides the common criteria everyone agrees to, and the bottom half is class-specific criteria that individual faculty want to factor into grades.

# data[].attributes.attachment_link.description
Posted on November 17, 2013 at 6:55 AM Why aren't grades sufficient evidence of student learning? 1. Grades alone do not usually provide meaningful information on exactly what students have and have not learned. So it's hard to use grades alone to decide how to improve teaching and learning. 2.

# data[].attributes.attachment_link.title
Part 4  Inclusive and Equitable Assessment

# data[].attributes.attachment_link.readable_text
From Abstract to Concrete: Creating a Peer Review Assignment

If your course analysis and planning revealed the need for an assignment that includes peer review, now it is time to design that assignment. This resource will guide you in creating the peer review components of an assignment. It includes guidance for describing the assignment, clarifying expectations related to peer review, and how to provide feedback. Examples from a course that used peer review are included.

Any assignment design should include:

A clear description of the assignment

Learning outcomes

Evaluation breakdown and criteria

An outline of what is expected of students

A rubric or questions and criteria to consider when reviewing a peer's work

Procedural or technical information needed to complete the assignment

Note

The example content on this page under the Show/Hide links is from a first year Folklore course.

Information That Goes in the Course Syllabus

Below are items to consider including in your course syllabus. This information may be dispersed among pages such as the "Course Evaluation" page and the "Course Format and Expectations" page.

Assignment description

Evaluation information

General information about the assignment - specific information may go in the module

Assignment type: essay, presentation, etc.

Associated learning outcomes

Evaluation breakdown - include marks for:

participation in peer review

peer ratings

instructor grading

A grading rubric

About peer review

How to use peerScholar

A description or details for each phase of the peer review process

Tips on how to critique a peer's work

Tips on how to provide effective feedback to a peer

When using a new technology in a course, it is important for students to have good instructions on how to use the tool or app.

Show/Hide excerpt from a Course Evaluation page

Short Essays

There are three (3) short essays that will go through the peer review process-each essay will be worth 20%.

Part of your grade will be based on your work within the peer review process. Learning to offer great peer feedback is a skill that requires practice, and can be very rewarding for all involved. Learning to receive peer feedback and use it effectively can help you improve your work in this course and future courses. To get the most out of this peer review process, you need to make each draft a complete, full-length essay with a controlling idea, organization, and conclusion.

For each short essay you will:

1. Submit a draft, by the due date, for your peers to review (1%)

2. Review the work of two of your peers and provide feedback that they can use to improve their essay (2%)

3. Reflect on your peers' feedback of your work and revise your draft using feedback from your peers and your own reflections (1%)

4. Revise your draft using feedback from your peers and your own reflections and submit a final version of your essay for your instructor (16%)

Show/Hide sample information about how to provide feedback

How to Provide Feedback

Think about how to share written feedback in a constructive way. Read the papers of each of your group members and provide feedback that they can use to improve their paper. An effective peer review includes the following:

general comments about the paper;

specific descriptions of what you liked / didn't like or what was effective / ineffective; and

specific advice about what can be improved.

The following statements stems will help you respond to your peers:

"I'd like to hear more about..."

"This is what I find interesting..."

And, as always, think about the questions:

"What surprised me?"

"What intrigued me?"

"What disturbed me?"

Additional Tips

Don't withhold constructive feedback about areas of improvement.

Pay attention to the language used - positive and constructive versus judgemental.

Focus on the strengths and weakness of the individual's work, not the individual themselves or their personality.

Be specific.

If possible, begin and end with positive comments, include areas for improvement in the middle.

Be realistic - are the suggested changes doable within the content of the assignment?

Present your own thoughts on your peer's work versus stating your suggested changes as facts.

When providing feedback via comments, be sure that it has the right level of detail and that it is clear and states your objective opinion. If feedback is excessively brief or vague, excessively detailed, or subjective rather than objective, it becomes about personal taste and preference and may be confusing, off-putting or difficult to use.

Show/Hide sample instructions for how to use peerScholar

Peer Review Using peerScholar

When your assignment draft is complete you will use peerScholar to complete the peer review process.

Peer review has three phases in peerScholar:

1. Create - submit your work for your peers to review

2. Assess - review your peer's assignment and provide feedback to your to them

3. Reflect

1. review feedback that your peers provided to you

2. carefully consider how you would like to incorporate the feedback

3. revise your essay and resubmit it for your instructor

Create

Submit your assignment via peerScholar by the due date provided in the course schedule. Your assignment will be available to your peers for review when the Create phase is closed and the Assess phase begins.

Submission Steps:

1. On the page "Short Essay: Peer Review Phase 1" click the link "peerScholar".

2. Once peerScholar opens, select the "Create" link.

3. Copy and paste your essay into the compose window, or attach a file.

4. Save your work.

5. Select "Preview" to see what your saved essay will look like when you submit it.

Assess

Now it is time to review essays from two of your peers and provide feedback to them. Your peers will do the same for you! Feedback will be available when the Assess phase is closed and the Reflect phase begins.

Steps to provide feedback to your peers:

1. On the page "Short Essay: Peer Review Phase 2" click the link "peerScholar".

2. Once peerScholar opens, select the "Assess" link.

3. Carefully read the work of each or your peers and provide them with feedback using the inline comments tool and by answering the questions provided.

Reflect

Review the feedback you received from your peers on your work and think critically about it to decide how you want to incorporate their feedback. Then, revise your essay and resubmit for your instructor.

Steps to reflect on your peers' feedback and to resubmit your essay:

1. On the page "Short Essay: Peer Review Phase 1" click the link "peerScholar".

2. Once peerScholar opens, select the "Reflect" link.

3. Carefully read the feedback that each peer provided. Decide on how you want to incorporate the feedback.

4. Toggle to the Revision screen to see you a copy of your original composition. Revise your essay and resubmit it to peerScholar for your instructor to grade. You may also upload a revised document.

Rubrics

Where possible, include a rubric in a peer review assignment. The rubric will benefit instructors when they configure the assignment and create assessments to use during the peer review, it will benefit students when they complete the assignment, and it will benefit all of those involved in evaluating and providing feedback.

Educational Benefits of Using a Rubric

Students can score their peer's work using the rubric

A rubric can act as a guide in a student's exploration of their peer's work

Using a rubric encourages a discovery mind set

Students can identify examples of what good work looks like and what poor work looks like

A good reason for using a rubric when including peer review mark in a student's overall grade for the assignment:
If you average five or more peer-assessments that took place based on a rubric, the average score tends to be a very good estimate of the student's skill

Making the Rubric Available to Students

Make sure a copy of the rubric is available in Brightspace and peerScholar:

1. Attach a rubric as part of the assignment details in the course syllabus

2. In the Create phase of peerScholar, attach a rubric and refer to the attached rubric in the instructions

Information That Goes in a Module

You can add specific information about an assignment, phase, or deliverable directly in your module or weekly content. Information can include:

Descriptions and details about a specific topic or deliverable

Required readings and resources

Value for the deliverable

Link to launch the application, if applicable

Show/Hide a sample section of weekly overview page

Note: The title of the short essay has been changed and links are inactive.

Week 3: Overview

Short Essay: "Title TBD" (20%)

If you haven't started already, it's time to start your Short Essay: "Title TBD".

Instructions

Read the story "Title TBD", and watch the two videos of the stories being told. After carefully reading the tale, and watching the videos, please consider the following questions:

1. What "life lessons" are emphasized in the tale?

2. How might these tales reflect the values and beliefs of members of Newfoundland outport communities in the first half of the 20th century?

3. What "real world" lessons are highlighted through this tale?

4. Have you heard/seen/read alternate versions of this tale?

5. What does each version of the story emphasize? (Compare and contrast)

When your draft version of your essay is complete, submit it via peerScholar for your peers to review.

See the course evaluation page for general information about Short Essays and about peer review. See the course schedule for dues dates.

# data[].attributes.attachment_link.description
If your course analysis and planning revealed the need for an assignment that includes peer review, now it is time to design that assignment. This resource will guide you in creating the peer review components of an assignment. It includes guidance for describing the assignment, clarifying expectations related to peer review, and how to provide feedback.

# data[].attributes.headline
Dr. Shamini Dias: Equity in Assessing Learning.

# data[].attributes.subject
Dr. Shamini Dias: Equity in Assessing Learning.

# data[].attributes.body
Equity in Assessing Learning. Disrupting Legacy Practices for Antiracist, Inclusive Practices: In this presentation, we locate assessment at the heart of education; it is where we make teaching and learning visible, where we gauge how well we are journeying in the teaching-learning process. We will also explore how and why assessment becomes space of anxiety, how legacy practices invoke power and privilege to impose barriers to success for many learners. In doing this, we will examine principles and strategies for equity-minded assessments that integrate growth mindset, self-determination, and holistic learning that engages students’ lived experiences.

# data[].attributes.attachment_link.title
Friday SLO Talk Dr. Shamini Dias: Equity in Assessing Learning.

# data[].attributes.headline
Ensuring Meaningful, Equitable, and Manageable Learning

# data[].attributes.subject
Ensuring Meaningful, Equitable, and Manageable Learning

# data[].attributes.attachment_link.title
Friday SLO Talk: Ensuring Meaningful, Equitable, and Manageable Learning

# data[].attributes.attachment_link.entities[].name
Center for Innovation in Legal Education

# data[].attributes.attachment_link.entities[].name
High School English Language Arts Assessment

# data[].attributes.attachment_link.description
Rubrics are a powerful and flexible tool for assessing students' learning in our classrooms and programs. This workshop was presented at DePaul University by a regional expert in the design and use of rubrics, Dr. Ranfen Li from the University of Illinois at Chicago.

# data[].attributes.attachment_link.title
Tips for Writing a Strong Rubric - Rubric Examples Included!

# data[].attributes.attachment_link.description
Our Tips for Writing a Strong Rubric include ideas for creating both Holistic and Analytical Rubrics. This guide provides examples of rubrics and specific action items!

# data[].attributes.attachment_link.readable_text
How to Create a Rubric: Introduction

Perhaps you have never even thought about the care it takes to create a rubric. Perhaps you have never even heard of a rubric and its usage in education, in which case, you should take a peek at this article: "What is a rubric?" Basically, this tool that teachers and professors use to help them communicate expectations, provide focused feedback, and grade products, can be invaluable when the correct answer is not as cut and dried as Choice A on a multiple choice test. But creating a great rubric is more than just slapping some expectations on a paper, assigning some percentage points, and calling it a day. A good rubric needs to be designed with care and precision in order to truly help teachers distribute and receive the expected work.

Steps to Create a Rubric

The following six steps will help you when you decide to use a rubric for assessing an essay, a project, group work, or any other task that does not have a clear right or wrong answer.

Before you can create a rubric, you need to decide the type of rubric you'd like to use, and that will largely be determined by your goals for the assessment.

Ask yourself the following questions:

1. How detailed do I want my feedback to be?

2. How will I break down my expectations for this project?

3. Are all of the tasks equally important?

4. How do I want to assess performance?

5. What standards must the students hit in order to achieve acceptable or exceptional performance?

6. Do I want to give one final grade on the project or a cluster of smaller grades based on several criteria?

7. Am I grading based on the work or on participation? Am I grading on both?

Once you've figured out how detailed you'd like the rubric to be and the goals you are trying to reach, you can choose a type of rubric.

Step 2: Choose a Rubric Type

Although there are many variations of rubrics, it can be helpful to at least have a standard set to help you decide where to start. Here are two that are widely used in teaching as defined by DePaul University's Graduate Educational department:

1. Analytic Rubric: This is the standard grid rubric that many teachers routinely use to assess students' work. This is the optimal rubric for providing clear, detailed feedback. With an analytic rubric, criteria for the students' work is listed in the left column and performance levels are listed across the top. The squares inside the grid will typically contain the specs for each level. A rubric for an essay, for example, might contain criteria like "Organization, Support, and Focus," and may contain performance levels like "(4) Exceptional, (3) Satisfactory, (2) Developing, and (1) Unsatisfactory."​ The performance levels are typically given percentage points or letter grades and a final grade is typically calculated at the end. The scoring rubrics for the ACT and SAT are designed this way, although when students take them, they will receive a holistic score.

2. Holistic Rubric: This is the type of rubric that is much easier to create, but much more difficult to use accurately. Typically, a teacher provides a series of letter grades or a range of numbers (1-4 or 1-6, for example) and then assigns expectations for each of those scores. When grading, the teacher matches the student work in its entirety to a single description on the scale. This is useful for grading multiple essays, but it does not leave room for detailed feedback on student work.

Step 3: Determine Your Criteria

This is where the learning objectives for your unit or course come into play. Here, you'll need to brainstorm a list of knowledge and skills you would like to assess for the project. Group them according to similarities and get rid of anything that is not absolutely critical. A rubric with too much criteria is difficult to use! Try to stick with 4-7 specific subjects for which you'll be able to create unambiguous, measurable expectations in the performance levels. You'll want to be able to spot the criteria quickly while grading and be able to explain them quickly when instructing your students. In an analytic rubric, the criteria are typically listed along the left column.

Step 4: Create Your Performance Levels

Once you have determined the broad levels you would like students to demonstrate mastery of, you will need to figure out what type of scores you will assign based on each level of mastery. Most ratings scales include between three and five levels. Some teachers use a combination of numbers and descriptive labels like "(4) Exceptional, (3) Satisfactory, etc." while other teachers simply assign numbers, percentages, letter grades or any combination of the three for each level. You can arrange them from highest to lowest or lowest to highest as long as your levels are organized and easy to understand.

Step 5: Write Descriptors for Each Level of Your Rubric

This is probably your most difficult step in creating a rubric.Here, you will need to write short statements of your expectations underneath each performance level for every single criteria. The descriptions should be specific and measurable. The language should be parallel to help with student comprehension and the degree to which the standards are met should be explained.

Again, to use an analytic essay rubric as an example, if your criteria was "Organization" and you used the (4) Exceptional, (3) Satisfactory, (2) Developing, and (1) Unsatisfactory scale, you would need to write the specific content a student would need to produce to meet each level. It could look something like this:

Organization
Organization is coherent, unified, and effective in support of the paper's purpose and
consistently demonstrates
effective and appropriate
transitions
between ideas and paragraphs.

Organization is coherent and unified in support of the paper's purpose and usually demonstrates effective and appropriate transitions between ideas and paragraphs.

Organization is coherent in
support of the essay's purpose, but is ineffective at times and may demonstrate abrupt or weak transitions between ideas or paragraphs.

Organization is confused and fragmented. It does not support the essay's purpose and demonstrates a
lack of structure or coherence that negatively
affects readability.

A holistic rubric would not break down the essay's grading criteria with such precision. The top two tiers of a holistic essay rubric would look more like this:

6 = Essay demonstrates excellent composition skills including a clear and thought-provoking thesis, appropriate and effective organization, lively and convincing supporting materials, effective diction and sentence skills, and perfect or near perfect mechanics including spelling and punctuation. The writing perfectly accomplishes the objectives of the assignment.

5 = Essay contains strong composition skills including a clear and thought-provoking thesis, but development, diction, and sentence style may suffer minor flaws. The essay shows careful and acceptable use of mechanics. The writing effectively accomplishes the goals of the assignment.

Step 6: Revise Your Rubric

After creating the descriptive language for all of the levels (making sure it is parallel, specific and measurable), you need to go back through and limit your rubric to a single page. Too many parameters will be difficult to assess at once, and may be an ineffective way to assess students' mastery of a specific standard. Consider the effectiveness of the rubric, asking for student understanding and co-teacher feedback before moving forward. Do not be afraid to revise as necessary. It may even be helpful to grade a sample project in order to gauge the effectiveness of your rubric. You can always adjust the rubric if need be before handing it out, but once it's distributed, it will be difficult to retract.

Teacher Resources:

# data[].attributes.attachment_link.description
Wondering how to create a rubric for your classroom? Check out the six steps you need to design an effective rubric to assess your students' next essay or project.

# data[].attributes.headline
Using Self and Peer Assessment in Writing Courses

# data[].attributes.subject
Using Self and Peer Assessment in Writing Courses

# data[].attributes.attachment_link.readable_text
We detected a high number of errors from your connection. To continue, please confirm that youâ€™re a human (and not a spambot).

# data[].attributes.attachment_link.title
AALHE 2022 Assessment as Dialogue - Using Self and Peer Assessment in Writing Courses - Hinojosa

# data[].attributes.attachment_link.readable_text
We detected a high number of errors from your connection. To continue, please confirm that youâ€™re a human (and not a spambot).

# data[].attributes.attachment_link.title
AALHE 2022 Assessment as Dialogue - Using Self and Peer Assessment in Writing Courses - Hinojosa