# data[].attributes.headline
Sampling items so that they are balanced across content map and representative of the construct

# data[].attributes.subject
Sampling items so that they are balanced across content map and representative of the construct

# data[].attributes.headline
Claim: Scores reliably measure the construct of interest.  

# data[].attributes.subject
Claim: Scores reliably measure the construct of interest.  

# data[].attributes.body
Type of Evidence:  In context of knowledge measure, this might be test-retest.  In context of measuring instructional practice, conduct decision study to determine how many lessons are necessary to get reliable estimate of a teacher's practice.
Claim:  Unexplained error is minimized.
Type of Evidence: Generalizability studies to determine sources of variance.

# data[].attributes.headline
Claim:  Scores on the instrument demonstrate theoretically supported correlations to other constructs.  

# data[].attributes.subject
Claim:  Scores on the instrument demonstrate theoretically supported correlations to other constructs.  

# data[].attributes.body
Type of Evidence: Examine relationships between scores and scores on instruments that measure other constructs

# data[].attributes.body
The response process and its interpretation may be different for different students. This can be for demographic/cultural groups as well as different ways of thinking/history of education, and other ways of distinguishing probable different approaches to instruction

# data[].attributes.body
What kinds of conditions, whether or not they are necessary, might actually place the validity of these arguments inside the test or across different assessments?

# data[].attributes.headline
Evidence: Comparison of responses based on contextual factors or person characteristics

# data[].attributes.subject
Evidence: Comparison of responses based on contextual factors or person characteristics

# data[].attributes.headline
Student placement in programs based on unintended constructs being tested.

# data[].attributes.subject
Student placement in programs based on unintended constructs being tested.

# data[].attributes.headline
Training for observation protocol use robust enough to accurately measure the intended construct. 

# data[].attributes.subject
Training for observation protocol use robust enough to accurately measure the intended construct. 

# data[].attributes.headline
Claim: Scores from the instrument represent the construct being measured. 

# data[].attributes.subject
Claim: Scores from the instrument represent the construct being measured. 

# data[].attributes.body
Type of evidence: CFA to empirically test the theoretical structure

# data[].attributes.headline
Evaluate the amount of measurement error

# data[].attributes.subject
Evaluate the amount of measurement error

# data[].attributes.headline
Evidence: Comparison of responses to responses on similar existing assessments

# data[].attributes.subject
Evidence: Comparison of responses to responses on similar existing assessments

# data[].attributes.body
Examining the number of dimensions. Can be exploratory or confirmatory, depending on the stage of instrument development). Might be tricky with tests where dimensionality analysis may produce "difficulty" dimensions rather than content.

# data[].attributes.body
Interitem correlation

# data[].attributes.body
Categorizing the types of student responses by how the response came about. (through examples, a deductive or inductive argument)

# data[].attributes.body
What were the conditions, how were participants maintained over time, time period, effects on learning/attitudes?

# data[].attributes.body
Dimensionality of the assessment (factor analysis, etc)

# data[].attributes.headline
differences in instructional approaches and/or opportunity to learn affect results but might be attributed to students

# data[].attributes.subject
differences in instructional approaches and/or opportunity to learn affect results but might be attributed to students

# data[].attributes.body
Teacher and/or expert reviews

# data[].attributes.body
Unintended consequences for testing/ bias should be accumulated during the validation process for further refinement of the metric and its validity assurance protocol for its intended uses.

# data[].attributes.headline
Gathering feedback from experts in the field on items

# data[].attributes.subject
Gathering feedback from experts in the field on items

# data[].attributes.body
What was the process of gathering the feedback, in what ways, how were the experts determined/selected that were providing feedback?

# data[].attributes.body
Thorough description of sample so that other assessment users can accurately judge how close their sample is to the original (aligned with intended use).

# data[].attributes.headline
evaluation of whether the task makes available the opportunity for the test taker to engage in the full range of envisioned performance levels, ie, can someone who is somehow believed to be "proficient" respond in ways that show this the case given the parameters of the test. Note: it seems to me that this is an investigation related to content that might draw on response patterns or support conclusions about internal structure. I am not sure where to place it and these categories don't feel quite of a type. 

# data[].attributes.subject
evaluation of whether the task makes available the opportunity for the test taker to engage in the full range of envisioned performance levels, ie, can someone who is somehow believed to be "proficient" respond in ways that show this the case given the parameters of the test. Note: it seems to me that this is an investigation related to content that might draw on response patterns or support conclusions about internal structure. I am not sure where to place it and these categories don't feel quite of a type. 

# data[].attributes.headline
Teacher knowledge scores related to selectivity of undergraduate institution.

# data[].attributes.subject
Teacher knowledge scores related to selectivity of undergraduate institution.

# data[].attributes.body
Well conceived alignment to construct of interest

# data[].attributes.headline
Claim: item hierarchy aligns with theoretical expectations

# data[].attributes.subject
Claim: item hierarchy aligns with theoretical expectations

# data[].attributes.body
If different sub-populations of test-takers consistently score differently on certain items it may indicate that those items are subject to unintended bias

# data[].attributes.body
Does the reliability of our measurements imply that we have appropriately put the tests, assessments, or such in where they need to be? What are the evidences of these evidences?

# data[].attributes.headline
reason about the relationship between the item and the level and what variables (numeric values, type of task, context, multiple steps, availability of calculator effects difficulty

# data[].attributes.subject
reason about the relationship between the item and the level and what variables (numeric values, type of task, context, multiple steps, availability of calculator effects difficulty

# data[].attributes.headline
Evaluating evidence by sub populations to understand differential performance; evaluate decision accuracy if available against stakes

# data[].attributes.subject
Evaluating evidence by sub populations to understand differential performance; evaluate decision accuracy if available against stakes

# data[].attributes.headline
Teacher knowledge scores increase after intervention with known efficacy.

# data[].attributes.subject
Teacher knowledge scores increase after intervention with known efficacy.

# data[].attributes.headline
Teacher knowledge related to observed classroom instruction

# data[].attributes.subject
Teacher knowledge related to observed classroom instruction

# data[].attributes.headline
Cross-tabulation of results by sub-population

# data[].attributes.subject
Cross-tabulation of results by sub-population

# data[].attributes.body
If time related, explanation or proof on rationale for time requirements.

# data[].attributes.body
Connection of scores to similar measure

# data[].attributes.body
Relation to other measures/instruments

# data[].attributes.headline
"content" is not just the structure of the domain

# data[].attributes.subject
"content" is not just the structure of the domain

# data[].attributes.body
it also includes the approach to instruction and the ways in which the content is expected to be demonstrated

# data[].attributes.body
Using a past assessment or what framework was being used if new questions created

# data[].attributes.body
Reliability coefficient
Item-total correlation
Inter rater reliability
Test retest reliability

# data[].attributes.body
Cluster analysis results
Network models

# data[].attributes.body
Measure of the same or similar constructs should be related, and measures of different constructs should not be related

# data[].attributes.body
Explicit argument that includes anticipated consequences, including for differential item functioning and/or subgroups of the population. Report should at least attempt to speculate about unintended consequences, and warn against some anticipated mis-uses of an assessment (e.g., this observation protocol is for formative feedback not high-stakes teacher evaluation)

# data[].attributes.headline
- Thoughtful consideration and evaluation of unintended consequences

# data[].attributes.subject
- Thoughtful consideration and evaluation of unintended consequences

# data[].attributes.body
- Cost-benefit analysis (money cost or otherwise)
- Exploration into different consequences for different groups of test takers and users (fairness evidence)

# data[].attributes.headline
Teacher knowledge related to student learning (pre-post on content-aligned test)

# data[].attributes.subject
Teacher knowledge related to student learning (pre-post on content-aligned test)

# data[].attributes.body
Study if scores are correlated with other scores that measure the same thing

# data[].attributes.headline
Evidence: examination of scoring patterns between and within raters

# data[].attributes.subject
Evidence: examination of scoring patterns between and within raters

# data[].attributes.body
Predictions of related variables (i.e. failed test means learning still can improve to pass mark to address potential bias)

# data[].attributes.headline
claims related to variability in responses 

# data[].attributes.subject
claims related to variability in responses 

# data[].attributes.body
descriptive statistics (ICCs, SDs, etc.)

# data[].attributes.headline
check to end of year scores or growth on end of year scores

# data[].attributes.subject
check to end of year scores or growth on end of year scores

# data[].attributes.headline
- Multiple estimates from different methods

# data[].attributes.subject
- Multiple estimates from different methods

# data[].attributes.body
- SEM
- Connection of reliability/SEM to intended use
- Separate estimates of reliability for different groups (fairness evidence)

# data[].attributes.body
Some categorizing could be grouping types of questions together that have a similar mathematical concept focus within the test. If the test has multiple choice questions, "how were the answer choices derived" (created by researcher or derived from student responses of an open-ended questions).

# data[].attributes.body
How well trained are teachers to interpret the implications of those responses? How are the students then followed up subsequently? Which kind of pedagogical interventions will result in the more desirable responses if they were to be any different at all?

# data[].attributes.body
A focus group of potential test users could provide data about how users could use the test in (possibly) unintended ways and to uncover possible unwanted consequences of test (mis)use

# data[].attributes.body
Factor analyses (construct validity)

# data[].attributes.body
Think aloud protocols from interviews with students

# data[].attributes.body
To show that the instrument measures the same construct as another existing instrument (convergent).
To show that the instrument measures construct distinct from others, especially conceptually similar ones (discriminant).
(e.g., via correlational analysis).

# data[].attributes.headline
Feedback from this category to intended use.

# data[].attributes.subject
Feedback from this category to intended use.

# data[].attributes.body
Though it might be speculative (comprehensive examination of external biases is not possible for most assessments of human behavior), the most egregious violations should be anticipated and assessed if possible, or caveats issued.

# data[].attributes.body
How have people (ab)used a test developed for specific/explicitly stated purposes?

# data[].attributes.headline
Strength to other measures either measuring the same or related constructs or strength of predictive relationships to other variables of interest, evaluating evidence on sensitivity over time

# data[].attributes.subject
Strength to other measures either measuring the same or related constructs or strength of predictive relationships to other variables of interest, evaluating evidence on sensitivity over time

# data[].attributes.body
Impact analysis on expected, observed, unexpected consequences

# data[].attributes.body
Anticipated response process should articulated so that the assumptions about the use of the measurement are clearly specified for the users.

# data[].attributes.headline
Depending upon what type of data do we have: if we have information for two times, if only for one time data points (split half method), inter-rater reliability in case two different people get the chance to score the data. All depends upon how the scores need to be used for intended interpretations. 

# data[].attributes.subject
Depending upon what type of data do we have: if we have information for two times, if only for one time data points (split half method), inter-rater reliability in case two different people get the chance to score the data. All depends upon how the scores need to be used for intended interpretations. 

# data[].attributes.headline
Do all questions designed to target a certain standard align with one another, or is there noise?

# data[].attributes.subject
Do all questions designed to target a certain standard align with one another, or is there noise?

# data[].attributes.body
Report includes connection to other variables, such as block of items from existing measure, or scores on other assessment. This could include test-taker demographics and looking by subgroups. If the test is supposed to predict a thing (such as teaching quality) then there should be a measure of that predicted outcome that can be connected to the given assessment

# data[].attributes.body
To predict an outcome (e.g., via regression)

# data[].attributes.body
EFA/CFA

# data[].attributes.body
For measures with consequences (i.e., used in decision-making), how accurate is the measure in discerning who goes in which group?

# data[].attributes.headline
Evidence: Comparison of item difficulty with theoretical expectations

# data[].attributes.subject
Evidence: Comparison of item difficulty with theoretical expectations

# data[].attributes.headline
Claim: Scoring with the instrument is applied consistently/accurately OR the items on the instrument are interpreted as intended.

# data[].attributes.subject
Claim: Scoring with the instrument is applied consistently/accurately OR the items on the instrument are interpreted as intended.

# data[].attributes.body
Type of Evidence:  Cognitive interviews with the person using the instrument (e.g., the person who is responding to items); Qualitative comparison of how the user responds to items in comparison to how master coder/user responds to items

Claim: The resulting scores represent variation in the construct to be measured.
Type of Evidence:  Examination of descriptive statistics (using full range of scores, expecting variation)

# data[].attributes.body
CFA
Definitely not EFA
Dimensionality analysis
Construct invariance analysis across time
Construct invariance analysis across sub groups
Differential item functioning analysis
Standard setting
Cut score analysis (which could also be a part of consequences analysis)

# data[].attributes.body
- Item properties
- Score distribution properties
- Reliability
- Classification consistency
- Application of psychometric theory, such as IRT, CDM, G-theory
- Measurement invariance analysis (fairness evidence)

# data[].attributes.headline
claims related to how scores are used to affect later outcomes (e.g., teacher/student performance, program evaluation)

# data[].attributes.subject
claims related to how scores are used to affect later outcomes (e.g., teacher/student performance, program evaluation)

# data[].attributes.body
interviews
observation
experimental/quasi-experimental studies

# data[].attributes.body
A measure of internal consistency (i.e., that the items are all measuring the same thing; NOTE: does not speak to whether the items measure the RIGHT thing, just the SAME thing)

# data[].attributes.headline
process for alignment to identified content

# data[].attributes.subject
process for alignment to identified content

# data[].attributes.headline
Conducting the multiple regressions to see the relationship between the constructs, if these relations are guided in the previous literature. 

# data[].attributes.subject
Conducting the multiple regressions to see the relationship between the constructs, if these relations are guided in the previous literature. 

# data[].attributes.body
Interpretations/use acceptability not always related to score level but change in scores too.

# data[].attributes.body
Sometimes it is appropriate to establish the items work together as scale factors.

# data[].attributes.headline
Reviewing the distribution of responses to look for evidence of lack of clarity in items

# data[].attributes.subject
Reviewing the distribution of responses to look for evidence of lack of clarity in items

# data[].attributes.body
-External/standardized measures
-Attitude measures (e.g. math self-efficacy)
-Analysis of student work