Categorizing and Describing Validity Evidence by Jonathan Bostic

Categorizing and Describing Validity Evidence by Jonathan Bostic https://padlet.com/jonathan_bostic/rz646vpw9sy4 Made with validity en-us 2020-02-12 18:42:27 UTC 2020-02-20 22:20:56 UTC hello@padlet.com Item Design Blueprint & Construct Map https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448326998 Conceptualizing how the items and their hypothesized hierarchy aligns with the theory it seeks to measure.

]]> 2020-02-20 21:15:02 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448326998 - alignment to standards or criterion https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327349 - literature and expert supported evidence of construct over- and under-representation

- accessibility and equivalence of content meaning for different groups (fairness evidence)

]]> 2020-02-20 21:15:45 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327349 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327434 2020-02-20 21:15:59 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327434 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327490 2020-02-20 21:16:07 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327490 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327498 Cognitive Interviews with sample of people from population of intended use]]> 2020-02-20 21:16:08 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327498 Exploratory factor analysis to see how the items are grouping together as a construct. https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327508 2020-02-20 21:16:10 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327508 External field expertise vetting https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327515 2020-02-20 21:16:11 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327515 Framework https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327536 Explicit connection of content to existing relevant framework(s)]]> 2020-02-20 21:16:13 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327536 claims related to the construct of measurement/what the assessment is measuring https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327581 expert review
alignment studies]]> 2020-02-20 21:16:20 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327581 cognitive interviews https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327584 2020-02-20 21:16:20 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327584 Cognitive Interviews https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327605 Examining responses & interpretations of the items to determine "adequacy with which test content represents the content domain"]]> 2020-02-20 21:16:23 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327605 Correlations agwilhelm https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327638 2020-02-20 21:16:28 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327638 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327689 Learning trajectory ]]> 2020-02-20 21:16:36 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327689 Opinions from the experts to ensure that the all main ideas related to a component have been addressed in the set of items. https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327777 2020-02-20 21:16:46 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327777 Regression results agwilhelm https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327795 2020-02-20 21:16:48 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327795 Cognitive interviews https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327831 participants are engaging in the thinking that was intended by instrument developers]]> 2020-02-20 21:16:50 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327831 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327887 Revisions based on early explorations]]> 2020-02-20 21:16:55 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327887 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327953 CFA to see how the internal structure maps to your hypothesized structure - including looking at measurement invariance to be sure the structure holds over different groups]]> 2020-02-20 21:16:57 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327953 Not all assessments are tests https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327988 you also need to think about "content" that describes learning behavior, affect, teacher observations, and other appropriate assessments.

]]> 2020-02-20 21:16:59 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327988 Claim: Scores from the instrument represent the construct being measured. https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327992 Type of Evidence: Expert review of the instrument
Claim: Scoring with the instrument is applied by accurately and consistently (that is, the content of the instrument is sufficiently described so that users can accurately apply the instrument).
Type of Evidence: Compare the scores of a user to the scores of a master coder/user. ]]> 2020-02-20 21:17:00 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327992 erikdjacobson https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327994 2020-02-20 21:17:00 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448327994 Cognitive Interviews (and other response data like written work). https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328069 Examining the content of participant responses with whether such explanations align with what the item was intended to assess.]]> 2020-02-20 21:17:05 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328069 Evaluate the assessment content against the theoretical and operational definition of the measured construct (e.g., if this is a student content area assessment, it could a list of standards covered by items, if it's a multi-dimensional construct such as anxiety, it could be a list of measured types or subdomains ) https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328089 2020-02-20 21:17:09 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328089 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328145 Connection to theoretical construct]]> 2020-02-20 21:17:15 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328145 Expert review https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328162 Describe process by which experts (whose expertise is given/summarized) compare test content to framework/standards/ intended outcomes. Explicit mention of revisions undertaken due to expert review. Experts' representation and expertise is established.]]> 2020-02-20 21:17:19 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328162 Expert analysis https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328190 Experts assess whether content of test matches with construct intended to measure]]> 2020-02-20 21:17:23 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328190 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328225 Alignment study
Blueprint
Theoretical framework
Literature review for the content

]]> 2020-02-20 21:17:28 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328225 correlational evidence https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328227 2020-02-20 21:17:29 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328227 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328232 Expert analysis]]> 2020-02-20 21:17:30 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328232 Conducting one on one OR cognitive interviews with the participants for whom the interpretations will be made afterwards. https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328239 2020-02-20 21:17:30 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328239 Blueprints / specifications https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328329 Based on theoretical frameworks, prior research, curriculum materials, field work, etc. ]]> 2020-02-20 21:17:40 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328329 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328360 Links to published frameworks/association guidelines

]]> 2020-02-20 21:17:45 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328360 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328433 Relationships of item difficulty to theory such as LTs (wright maps, etc)]]> 2020-02-20 21:17:51 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328433 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328463 Different levels of cognitive processes]]> 2020-02-20 21:17:55 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328463 Item Response Theory https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328477 -Determine/confirm number of dimensions.
-Difficulty of items
-DIF analysis

]]> 2020-02-20 21:17:57 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328477 Test Taking Observation https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328494 Observing how test takers respond to the items to gather evidence of no cross-contamination from non-tested factors]]> 2020-02-20 21:17:59 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328494 claims related to the cognitive processes/item interpretation of respondents/raters https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328538 cognitive interviews]]> 2020-02-20 21:18:01 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328538 Factor analysis https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328546 Exploratory and confirmatory]]> 2020-02-20 21:18:02 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328546 - Predictive of expected future behavior/achievement https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328581
- Testing of the nomological network

- Invariance of external relationships across groups (fairness evidence)

- Distinct from similar constructs (divergent)

- Correspondence with expert opinion on the test users (e.g., scores correlate with teacher's knowledge of the student)]]> 2020-02-20 21:18:07 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328581 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328633 Internal-consistency if applicable (cronbach's alpha)
Test-retest reliability (correlation between tests if it's a construct that shouldn't vary over time)
Parallel forms reliability (correlation between scores on two forms)]]> 2020-02-20 21:18:10 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328633 consistency in terms of stability, equivalence; could include analyses from classical test theory that looks at internal consistecy https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328635 2020-02-20 21:18:10 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328635 Factor analysis agwilhelm https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328706 2020-02-20 21:18:15 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328706 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328763 Cognitive interviews]]> 2020-02-20 21:18:23 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328763 cognitive interviews; https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328798 open-ended survey items following Likert items to assess reasons for item responses

]]> 2020-02-20 21:18:29 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328798 Cronbach Alpha https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328812 2020-02-20 21:18:31 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328812 interview or observational data as individuals reason about tasks https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328817 2020-02-20 21:18:31 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328817 Confirmatory Factor Analysis https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328858 The items comprising the instrument reflect the theory on which it was designed (i.e., the items covary as intended)]]> 2020-02-20 21:18:34 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328858 claims related to how users interpret score reports https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328906 observation
interviews
surveys]]> 2020-02-20 21:18:37 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328906 Breadth of Coverage https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328908 Address what the test is intended to cover and acknowledge what it doesn't cover. Appropriateness of content to intended inferences.]]> 2020-02-20 21:18:38 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328908 correlations with other instruments match results from theory; factor analysis with items from an instrument with overlapping construct supports separate latent variables for two instruments https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328915 2020-02-20 21:18:39 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328915 Defining construct https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328938 Defining the construct to be measured based on theoretical framework]]> 2020-02-20 21:18:42 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328938 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328955 Completer details/instructions on item(s)]]> 2020-02-20 21:18:45 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328955 Correlation with the other constructs, for both which are related in a positive and negative way. https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328972 2020-02-20 21:18:47 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328972 Claim: The topics on the assessment are important/relevant to the domain. Evidence: Expert consensus (e.g., Delphi process, surveys, focus groups) https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328974 2020-02-20 21:18:47 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328974 Think alouds https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328977 2020-02-20 21:18:47 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448328977 claims related to the empirical clustering/relationships across items/dimensionality https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329031 EFA/CFA
Cronbach's alpha (standards also call this reliability but I believe it fits here)]]> 2020-02-20 21:18:51 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329031 erikdjacobson https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329036 Factor analysis]]> 2020-02-20 21:18:51 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329036 Evidence: Expert Review Panel https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329041 2020-02-20 21:18:52 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329041 Evaluate the overlap and or alignment between the cognitive processes involved in completing the assessment task and the intended measured construct. https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329064 2020-02-20 21:18:55 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329064 correlation matrix stefanielivers https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329107 2020-02-20 21:19:01 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329107 Study users https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329128 Study behaviors of users when responding to test items to see if fits construct]]> 2020-02-20 21:19:03 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329128 Cognitive interviews https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329139 Examining the extent to which respondent's verbalized cognitive processes correspond to the intended cognitive processes.
(e.g., via summary reports or quantitative results from coding cognitive interviews)]]> 2020-02-20 21:19:05 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329139 Psychometric Analysis https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329152 Using certain psychometric analyses (such as Rasch), evidence can point towards whether items perform in the way that aligns with theory, etc. (for example: using Rasch, do items on the Wright Map perform in a way corresponding to the Construct Map).]]> 2020-02-20 21:19:07 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329152 process data from online assessments if relevant https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329177 2020-02-20 21:19:10 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329177 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329184 Log data that shows how students solve problems/ complete activities]]> 2020-02-20 21:19:11 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329184 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329190 How many times has the study been repeated?]]> 2020-02-20 21:19:12 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329190 Item types https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329206 Justify the types of items involved in the assessment and how those formats can capture the intended outcome variables]]> 2020-02-20 21:19:14 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329206 claims related to the extent to which assessment scores can be used to predict future performance https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329237 HLM/regression
classification accuracy]]> 2020-02-20 21:19:19 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329237 description of scoring training and criteria https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329264 scorers need to be trained in how to score observation rubrics. This is critical to both the reliability and validity of the scores.

]]> 2020-02-20 21:19:23 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329264 IRT plots vs. learning trajectory measures https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329271 2020-02-20 21:19:24 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329271 differential impact, interviews or notes from respondents about how use of the instrument impacted them in unexpected ways https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329299 2020-02-20 21:19:26 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329299 Claim: Assessment content adequately covers the domain of interest https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329321 2020-02-20 21:19:30 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329321 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329348 Predictability]]> 2020-02-20 21:19:34 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329348 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329404 Speededness study
CTT for certain error producing response patterns]]> 2020-02-20 21:19:40 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329404 Measurement invariance https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329405 2020-02-20 21:19:40 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329405 Correlation https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329416 Assessing correlations among variables]]> 2020-02-20 21:19:42 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329416 examine degree of correlation of resulting scores to scores on related measures (not necessarily looking for higher correlations, more about whether it gives the types of relationships you'd expect based on the degree of relationship) https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329435 2020-02-20 21:19:46 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329435 test of discrimination https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329455 2020-02-20 21:19:50 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329455 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329479 2020-02-20 21:19:53 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329479 claims related to the extent to which assessment scores follow hypothesized relationships with other measures (convergent/divergent) https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329483 SEM
correlations
HLM/regression]]> 2020-02-20 21:19:53 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329483 Cognitive interviews agwilhelm https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329502 2020-02-20 21:19:56 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329502 Item Response Theory https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329523 DIF analysis]]> 2020-02-20 21:19:58 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329523 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329534 Concept map & theory developed from previous research]]> 2020-02-20 21:19:59 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329534 Sorting tasks https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329568 Useful when categorizing is involved. (Examining whether respondents' understanding of categories matches the researcher's understanding). Respondents are asked to sort items/behaviors/aspects/etc. into predetermined or not predetermined categories. ]]> 2020-02-20 21:20:05 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329568 Literature review agwilhelm https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329583 2020-02-20 21:20:07 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329583 Rasch rating scale effectiveness (Linacre's criteria) https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329586 factor analysis
assessment of group differences in scores

]]> 2020-02-20 21:20:07 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329586 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329596 Relationship of scores to other variables of interest]]> 2020-02-20 21:20:10 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329596 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329605 Correlations with existing measurements

]]> 2020-02-20 21:20:11 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329605 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329620 Expert review (looking for items matching constructs or items measuring what you purport that they measure)]]> 2020-02-20 21:20:13 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329620 Respondents constructing responses in ways that are meaningful to the context. https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329633 2020-02-20 21:20:15 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329633 Content of questions aligning with the domain the test is intended to measure. https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329634 2020-02-20 21:20:15 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329634 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329639 Factor structure in alignment with theoretical foundation(s)]]> 2020-02-20 21:20:17 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329639 measures construct it intended to measure stefanielivers https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329648 2020-02-20 21:20:17 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329648 - Think aloud data covering issues of motivation, understanding of tasks/content, social desirability, more. https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329688
- Vetting of writing/notes/etc... of students that indicates their process for responding to items

- Comparison of response processes across groups (fairness evidence)]]> 2020-02-20 21:20:23 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329688 Evidence: Cognitive Interviews https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329690 2020-02-20 21:20:23 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329690 Elaboration documents for learning trajectories describing the levels, cases, and related misconceptions https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329697 2020-02-20 21:20:24 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329697 Structural and temporal relationships matter! https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329705 If other variables are implicated by an assessment, then the causal links (mediators and moderators) and the temporal order of variables in the process being modeled must be accounted for.]]> 2020-02-20 21:20:26 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329705 reviews from experts https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329708 2020-02-20 21:20:27 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329708 User interviews agwilhelm https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329710 2020-02-20 21:20:28 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329710 Setting assessment was administered https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329721 In what ways was the assessment administered; what conditions?]]> 2020-02-20 21:20:29 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329721 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329792 Confounding variables]]> 2020-02-20 21:20:40 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329792 Conversations with experts agwilhelm https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329821 2020-02-20 21:20:44 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329821 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329829 How reliable are the test items in connection with curriculum that spans across different times and possibly for different purposes?]]> 2020-02-20 21:20:45 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329829 Correlation between Assessments https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329831 Certain constructs are hypothesized to be related (theoretically). Tests measuring such constructs should have correlations in line with theory.]]> 2020-02-20 21:20:46 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329831 Diagnostic interviews https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329870 It is often appropriate to conduct diagnostic interviews to ascertain the extent to which test-takers are interpreting items as intended.]]> 2020-02-20 21:20:51 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329870 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329893 Reliability coefficient]]> 2020-02-20 21:20:53 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329893 Expert Review https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329974 Experts on this construct confirm that the items appropriately map the construct domain for the intended use of the instrument]]> 2020-02-20 21:20:57 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448329974 elimination of items that proved problematic stefanielivers https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330016 2020-02-20 21:21:00 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330016 Claim: Responses align with expectations/theory/intended score inferences https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330071 2020-02-20 21:21:05 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330071 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330095 Correlation with other measures (for which there already exists validity evidence) or low/no correlation with measures that should be different conceptually]]> 2020-02-20 21:21:08 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330095 Reviews and syntheses of research on student learning in the LT topic https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330098 2020-02-20 21:21:08 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330098 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330117 -Expert review of items
-Think aloud]]> 2020-02-20 21:21:10 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330117 Study scorers https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330168 If assessment relies on scoring by a person, empirically study how people do that scoring and if they are attending to the right things]]> 2020-02-20 21:21:14 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330168 To ensure that the sub-groups within my population are perceiving the items in a similar way, so that I am not making any deductions which are biased for a specific subset of my group. Again depending upon if that is for each item or the assessment as a whole. https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330173 2020-02-20 21:21:15 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330173 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330205 Multiple users need to interpret the content from expert and novice user perspective.]]> 2020-02-20 21:21:21 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330205 included a bias review team/ analysis in terms of equity and access stefanielivers https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330225 2020-02-20 21:21:24 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330225 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330246 Evaluations about alignment of assessment to construct of interest (Webb method, GAAT method)]]> 2020-02-20 21:21:25 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330246 Study the literature to assess what specific items and ideas have been stated in the past to ensure that the items represent the construct in a fair way. https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330281 2020-02-20 21:21:28 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330281 DIF Analysis https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330345 Examine whether items perform in ways that are biased for certain groups or classifications.]]> 2020-02-20 21:21:33 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330345 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330389 Obviously alpha across raters but to level of expertise of user as well.]]> 2020-02-20 21:21:36 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330389 Differential item functioning https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330456 Address the extent to which the test is associated with differential item functioning for different subgroups. Address via theory/lit whether such diff is a concern.]]> 2020-02-20 21:21:40 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330456 Checking the representativeness of the samples by school type, course type etc. https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330537 2020-02-20 21:21:46 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330537 Claim: Item responses reflect test-takers' thinking/understanding. Evidence: cognitive interviews https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330576 2020-02-20 21:21:50 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330576 Simpler is better https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330587 Lord Occam, for a dead white guy, you're still pretty good! Complex internal structure of an assessment limits the applicability of the assessment to other situations where the relationships among internal contexts is different--e.g., teaching the same content with different contexts, in different semesters--you would expect the internal structure of a test to be different]]> 2020-02-20 21:21:51 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330587 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330606 Construct irrelevance]]> 2020-02-20 21:21:52 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330606 if the envisioned use is that the provided information leads to change, tracking on whether change happens and evidence that maps it to the provision of information from the assessment https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330664 2020-02-20 21:21:59 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330664 Demographics https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330674 If relevant, socioeconomic variables such as race/ethnicity, gender identity, previous experience, etc.]]> 2020-02-20 21:22:00 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330674 Group membership or demographic categories https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330681 2020-02-20 21:22:01 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330681 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330721 2020-02-20 21:22:05 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330721 Evidence: Examination of dimensionality (i.e., factor analysis, rasch) https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330735 2020-02-20 21:22:08 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330735 assessment for confounding variables stefanielivers https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330760 2020-02-20 21:22:11 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330760 claims about consistency of scores/measurement error https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330763 generalizability study/decision study
inter-rater agreement]]> 2020-02-20 21:22:11 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330763 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330875 -External/standardized measures
-Attitude measures (e.g. math self-efficacy)
-Analysis of student work]]> 2020-02-20 21:22:25 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330875 Reviewing the distribution of responses to look for evidence of lack of clarity in items https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330973 2020-02-20 21:22:37 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330973 EFA/CFA https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330976 Sometimes it is appropriate to establish the items work together as scale factors.]]> 2020-02-20 21:22:37 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330976 Assess differential item functioning https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330986 2020-02-20 21:22:38 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448330986 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331000 Interpretations/use acceptability not always related to score level but change in scores too.
]]> 2020-02-20 21:22:40 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331000 Conducting the multiple regressions to see the relationship between the constructs, if these relations are guided in the previous literature. https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331015 2020-02-20 21:22:42 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331015 process for alignment to identified content stefanielivers https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331023 2020-02-20 21:22:43 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331023 Cronbach's Alpha https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331029 A measure of internal consistency (i.e., that the items are all measuring the same thing; NOTE: does not speak to whether the items measure the RIGHT thing, just the SAME thing)]]> 2020-02-20 21:22:43 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331029 claims related to how scores are used to affect later outcomes (e.g., teacher/student performance, program evaluation) https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331045 interviews
observation
experimental/quasi-experimental studies]]> 2020-02-20 21:22:47 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331045 - Dimensionality https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331056 - Item properties
- Score distribution properties
- Reliability
- Classification consistency
- Application of psychometric theory, such as IRT, CDM, G-theory
- Measurement invariance analysis (fairness evidence)

]]> 2020-02-20 21:22:49 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331056 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331064 CFA
Definitely not EFA
Dimensionality analysis
Construct invariance analysis across time
Construct invariance analysis across sub groups
Differential item functioning analysis
Standard setting
Cut score analysis (which could also be a part of consequences analysis)]]> 2020-02-20 21:22:50 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331064 Claim: Scoring with the instrument is applied consistently/accurately OR the items on the instrument are interpreted as intended. https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331116 Type of Evidence: Cognitive interviews with the person using the instrument (e.g., the person who is responding to items); Qualitative comparison of how the user responds to items in comparison to how master coder/user responds to items

Claim: The resulting scores represent variation in the construct to be measured.
Type of Evidence: Examination of descriptive statistics (using full range of scores, expecting variation) ]]> 2020-02-20 21:22:57 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331116 Evidence: Comparison of item difficulty with theoretical expectations https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331132 2020-02-20 21:22:59 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331132 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331138 For measures with consequences (i.e., used in decision-making), how accurate is the measure in discerning who goes in which group? ]]> 2020-02-20 21:23:00 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331138 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331172 EFA/CFA]]> 2020-02-20 21:23:05 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331172 Prediction https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331256 To predict an outcome (e.g., via regression) ]]> 2020-02-20 21:23:13 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331256 This is present https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331279 Report includes connection to other variables, such as block of items from existing measure, or scores on other assessment. This could include test-taker demographics and looking by subgroups. If the test is supposed to predict a thing (such as teaching quality) then there should be a measure of that predicted outcome that can be connected to the given assessment]]> 2020-02-20 21:23:17 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331279 Do all questions designed to target a certain standard align with one another, or is there noise? https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331307 2020-02-20 21:23:20 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331307 Depending upon what type of data do we have: if we have information for two times, if only for one time data points (split half method), inter-rater reliability in case two different people get the chance to score the data. All depends upon how the scores need to be used for intended interpretations. https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331319 2020-02-20 21:23:23 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331319 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331323 Anticipated response process should articulated so that the assumptions about the use of the measurement are clearly specified for the users. ]]> 2020-02-20 21:23:23 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331323 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331410 Impact analysis on expected, observed, unexpected consequences

]]> 2020-02-20 21:23:38 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331410 Strength to other measures either measuring the same or related constructs or strength of predictive relationships to other variables of interest, evaluating evidence on sensitivity over time https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331414 2020-02-20 21:23:38 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331414 Lit Review? https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331421 How have people (ab)used a test developed for specific/explicitly stated purposes?

]]> 2020-02-20 21:23:40 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331421 Test-retest reliability measures https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331427 2020-02-20 21:23:41 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331427 Feedback from this category to intended use. https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331510 Though it might be speculative (comprehensive examination of external biases is not possible for most assessments of human behavior), the most egregious violations should be anticipated and assessed if possible, or caveats issued.]]> 2020-02-20 21:23:46 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331510 Convergent/discriminant https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331533 To show that the instrument measures the same construct as another existing instrument (convergent).
To show that the instrument measures construct distinct from others, especially conceptually similar ones (discriminant).
(e.g., via correlational analysis). ]]> 2020-02-20 21:23:49 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331533 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331538 Think aloud protocols from interviews with students]]> 2020-02-20 21:23:50 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331538 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331559 Factor analyses (construct validity)]]> 2020-02-20 21:23:55 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331559 Confirmation of coherence with context agwilhelm https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331622 2020-02-20 21:24:03 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331622 Focus group of potential test uses https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331628 A focus group of potential test users could provide data about how users could use the test in (possibly) unintended ways and to uncover possible unwanted consequences of test (mis)use]]> 2020-02-20 21:24:03 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331628 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331642 How well trained are teachers to interpret the implications of those responses? How are the students then followed up subsequently? Which kind of pedagogical interventions will result in the more desirable responses if they were to be any different at all?]]> 2020-02-20 21:24:06 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331642 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331684 Some categorizing could be grouping types of questions together that have a similar mathematical concept focus within the test. If the test has multiple choice questions, "how were the answer choices derived" (created by researcher or derived from student responses of an open-ended questions).]]> 2020-02-20 21:24:12 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331684 measurement invariance https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331750 2020-02-20 21:24:20 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331750 Internal consistency https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331785 2020-02-20 21:24:26 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331785 - Multiple estimates from different methods https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331791 - SEM
- Connection of reliability/SEM to intended use
- Separate estimates of reliability for different groups (fairness evidence)]]> 2020-02-20 21:24:27 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331791 check to end of year scores or growth on end of year scores https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331884 2020-02-20 21:24:42 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331884 Field notes from administration agwilhelm https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331900 2020-02-20 21:24:46 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331900 claims related to variability in responses https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331912 descriptive statistics (ICCs, SDs, etc.) ]]> 2020-02-20 21:24:47 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331912 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331964 Predictions of related variables (i.e. failed test means learning still can improve to pass mark to address potential bias)]]> 2020-02-20 21:24:55 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331964 Evidence: examination of scoring patterns between and within raters https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331986 2020-02-20 21:24:58 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448331986 Correlation to other tests https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332023 Study if scores are correlated with other scores that measure the same thing]]> 2020-02-20 21:25:04 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332023 Parallel forms https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332059 2020-02-20 21:25:07 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332059 Teacher knowledge related to student learning (pre-post on content-aligned test) https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332070 2020-02-20 21:25:08 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332070 - Thoughtful consideration and evaluation of unintended consequences https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332073 - Cost-benefit analysis (money cost or otherwise)
- Exploration into different consequences for different groups of test takers and users (fairness evidence)]]> 2020-02-20 21:25:09 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332073 Address potential bias https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332092 Explicit argument that includes anticipated consequences, including for differential item functioning and/or subgroups of the population. Report should at least attempt to speculate about unintended consequences, and warn against some anticipated mis-uses of an assessment (e.g., this observation protocol is for formative feedback not high-stakes teacher evaluation)]]> 2020-02-20 21:25:13 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332092 ρ https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332136 Measure of the same or similar constructs should be related, and measures of different constructs should not be related]]> 2020-02-20 21:25:20 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332136 Factor analysis results https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332156 Cluster analysis results
Network models]]> 2020-02-20 21:25:22 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332156 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332160 Reliability coefficient
Item-total correlation
Inter rater reliability
Test retest reliability

]]> 2020-02-20 21:25:23 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332160 Process of creating questions https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332167 Using a past assessment or what framework was being used if new questions created]]> 2020-02-20 21:25:24 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332167 Examining individual responses/data agwilhelm https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332189 2020-02-20 21:25:28 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332189 consistency across scorers stefanielivers https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332208 2020-02-20 21:25:30 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332208 "content" is not just the structure of the domain https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332219 it also includes the approach to instruction and the ways in which the content is expected to be demonstrated]]> 2020-02-20 21:25:32 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332219 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332235 Relation to other measures/instruments]]> 2020-02-20 21:25:35 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332235 Examining score distributions agwilhelm https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332281 2020-02-20 21:25:42 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332281 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332458 Connection of scores to similar measure]]> 2020-02-20 21:26:13 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332458 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332463 If time related, explanation or proof on rationale for time requirements.
]]> 2020-02-20 21:26:14 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332463 Cross-tabulation of results by sub-population agwilhelm https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332486 2020-02-20 21:26:17 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332486 Teacher knowledge related to observed classroom instruction https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332495 2020-02-20 21:26:18 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332495 pilot group of scorers/ test group stefanielivers https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332509 2020-02-20 21:26:20 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332509 Teacher knowledge scores increase after intervention with known efficacy. https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332560 2020-02-20 21:26:27 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332560 Evaluating evidence by sub populations to understand differential performance; evaluate decision accuracy if available against stakes https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332587 2020-02-20 21:26:30 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332587 reason about the relationship between the item and the level and what variables (numeric values, type of task, context, multiple steps, availability of calculator effects difficulty https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332644 2020-02-20 21:26:40 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332644 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332684 Does the reliability of our measurements imply that we have appropriately put the tests, assessments, or such in where they need to be? What are the evidences of these evidences?]]> 2020-02-20 21:26:47 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332684 DIF https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332686 If different sub-populations of test-takers consistently score differently on certain items it may indicate that those items are subject to unintended bias]]> 2020-02-20 21:26:47 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332686 Convergent evidence agwilhelm https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332730 2020-02-20 21:26:56 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332730 Claim: item hierarchy aligns with theoretical expectations https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332731 2020-02-20 21:26:56 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332731 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332768 Well conceived alignment to construct of interest ]]> 2020-02-20 21:27:02 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332768 Teacher knowledge scores related to selectivity of undergraduate institution. https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332775 2020-02-20 21:27:03 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332775 evaluation of whether the task makes available the opportunity for the test taker to engage in the full range of envisioned performance levels, ie, can someone who is somehow believed to be "proficient" respond in ways that show this the case given the parameters of the test. Note: it seems to me that this is an investigation related to content that might draw on response patterns or support conclusions about internal structure. I am not sure where to place it and these categories don't feel quite of a type. https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332787 2020-02-20 21:27:04 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332787 Describe sample https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332792 Thorough description of sample so that other assessment users can accurately judge how close their sample is to the original (aligned with intended use).]]> 2020-02-20 21:27:05 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332792 Divergent evidence agwilhelm https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332794 2020-02-20 21:27:05 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332794 identified content/ setting for use stefanielivers https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332862 2020-02-20 21:27:16 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332862 Gathering feedback from experts in the field on items https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332878 What was the process of gathering the feedback, in what ways, how were the experts determined/selected that were providing feedback?]]> 2020-02-20 21:27:19 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448332878 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333016 Unintended consequences for testing/ bias should be accumulated during the validation process for further refinement of the metric and its validity assurance protocol for its intended uses.]]> 2020-02-20 21:27:39 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333016 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333086 Teacher and/or expert reviews]]> 2020-02-20 21:27:49 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333086 differences in instructional approaches and/or opportunity to learn affect results but might be attributed to students https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333139 2020-02-20 21:27:56 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333139 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333307 Dimensionality of the assessment (factor analysis, etc)]]> 2020-02-20 21:28:19 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333307 Assessment administered over time? https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333415 What were the conditions, how were participants maintained over time, time period, effects on learning/attitudes?]]> 2020-02-20 21:28:37 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333415 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333438 Categorizing the types of student responses by how the response came about. (through examples, a deductive or inductive argument)]]> 2020-02-20 21:28:40 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333438 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333442 Interitem correlation]]> 2020-02-20 21:28:40 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333442 Dimensionality https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333459 Examining the number of dimensions. Can be exploratory or confirmatory, depending on the stage of instrument development). Might be tricky with tests where dimensionality analysis may produce "difficulty" dimensions rather than content. ]]> 2020-02-20 21:28:44 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333459 Evidence: Comparison of responses to responses on similar existing assessments https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333507 2020-02-20 21:28:51 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333507 Evaluate the amount of measurement error https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333605 2020-02-20 21:29:03 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333605 Claim: Scores from the instrument represent the construct being measured. https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333614 Type of evidence: CFA to empirically test the theoretical structure]]> 2020-02-20 21:29:04 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333614 Training for observation protocol use robust enough to accurately measure the intended construct. https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333677 2020-02-20 21:29:12 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333677 Student placement in programs based on unintended constructs being tested. https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333678 2020-02-20 21:29:12 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333678 alignment of items to standards stefanielivers https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333852 2020-02-20 21:29:40 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333852 Evidence: Comparison of responses based on contextual factors or person characteristics https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333903 2020-02-20 21:29:48 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333903 Item interviews https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333944 2020-02-20 21:29:53 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333944 Fieldnotes from administration agwilhelm https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333956 2020-02-20 21:29:55 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448333956 Textbook analysis https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448334021 2020-02-20 21:30:06 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448334021 Analysis of standards documents https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448334064 2020-02-20 21:30:12 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448334064 https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448334173 What kinds of conditions, whether or not they are necessary, might actually place the validity of these arguments inside the test or across different assessments?]]> 2020-02-20 21:30:28 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448334173 Inter rater reliability https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448334258 ]]> 2020-02-20 21:30:45 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448334258 Correlations agwilhelm https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448334442 2020-02-20 21:31:12 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448334442 Re agwilhelm https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448334535 2020-02-20 21:31:26 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448334535 Disaggregated response patterns https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448336938 The response process and its interpretation may be different for different students. This can be for demographic/cultural groups as well as different ways of thinking/history of education, and other ways of distinguishing probable different approaches to instruction]]> 2020-02-20 21:37:25 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448336938 Claim: Scores on the instrument demonstrate theoretically supported correlations to other constructs. https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448337170 Type of Evidence: Examine relationships between scores and scores on instruments that measure other constructs]]> 2020-02-20 21:38:04 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448337170 Claim: Scores reliably measure the construct of interest. https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448339604 Type of Evidence: In context of knowledge measure, this might be test-retest. In context of measuring instructional practice, conduct decision study to determine how many lessons are necessary to get reliable estimate of a teacher's practice.
Claim: Unexplained error is minimized.
Type of Evidence: Generalizability studies to determine sources of variance. ]]> 2020-02-20 21:44:16 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448339604 Sampling items so that they are balanced across content map and representative of the construct https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448340596 2020-02-20 21:46:50 UTC https://padlet.com/jonathan_bostic/rz646vpw9sy4/wish/448340596