# data[].attributes.headline
International Test Commission: Its History, Current Status, and Future Directions

# data[].attributes.subject
International Test Commission: Its History, Current Status, and Future Directions

# data[].attributes.body
https://www.intestcom.org/files/oakland_2001.pdf

# data[].attributes.headline
Achievement Criteria for ITC Guidelines on Test Adaptation

# data[].attributes.subject
Achievement Criteria for ITC Guidelines on Test Adaptation

# data[].attributes.body
https://www.cop.es/pdf/ITC-guidelines-for-test-adaptation-CRITERION-CHECKLIST.pdf

# data[].attributes.headline
Automatic boost articulation therapy in adults with dysarthria: Acceptability, usability and user interaction

# data[].attributes.subject
Automatic boost articulation therapy in adults with dysarthria: Acceptability, usability and user interaction

# data[].attributes.body
https://onlinelibrary.wiley.com/doi/10.1111/1460-6984.12647?af=R

# data[].attributes.headline
A Usability Study of Internet-Based Therapy for Naming Deficits in Aphasia

# data[].attributes.subject
A Usability Study of Internet-Based Therapy for Naming Deficits in Aphasia

# data[].attributes.body
https://pubmed.ncbi.nlm.nih.gov/27784033/

# data[].attributes.attachment_link.title
A Usability Study of Internet-Based Therapy for Naming Deficits in Aphasia - PubMed

# data[].attributes.attachment_link.description
It is important to consider usability and the clinician’s perspective in developing telepractice applications in speech-language pathology. Future directions include assessing the efficacy of remote treatment and collecting a larger sample of clinician data.

# data[].attributes.body
https://www.sciencedirect.com/book/9780128036563/usability-testing-for-survey-research

# data[].attributes.attachment_link.readable_text
Book * 2017

Authors:

Emily Geisen and Jennifer Romano Bergstrom

Book * 2017

Authors:

Emily Geisen and Jennifer Romano Bergstrom

Book description

Usability Testing for Survey Research provides researchers with a guide to the tools necessary to evaluate, test, and modify surveys in an iterative method during the survey pretes ... read full description

Browse content

Table of contents

Description

Usability Testing for Survey Research provides researchers with a guide to the tools necessary to evaluate, test, and modify surveys in an iterative method during the survey pretesting process. It includes examples that apply usability to any type of survey during any stage of development, along with tactics on how to tailor usability testing to meet budget and scheduling constraints.

The book's authors distill their experience to provide tips on how usability testing can be applied to paper surveys, mixed-mode surveys, interviewer-administered tools, and additional products.

Readers will gain an understanding of usability and usability testing and why it is needed for survey research, along with guidance on how to design and conduct usability tests, analyze and report findings, ideas for how to tailor usability testing to meet budget and schedule constraints, and new knowledge on how to apply usability testing to other survey-related products, such as project websites and interviewer administered tools.

Show more

Usability Testing for Survey Research provides researchers with a guide to the tools necessary to evaluate, test, and modify surveys in an iterative method during the survey pretesting process. It includes examples that apply usability to any type of survey during any stage of development, along with tactics on how to tailor usability testing to meet budget and scheduling constraints.

The book's authors distill their experience to provide tips on how usability testing can be applied to paper surveys, mixed-mode surveys, interviewer-administered tools, and additional products.

Readers will gain an understanding of usability and usability testing and why it is needed for survey research, along with guidance on how to design and conduct usability tests, analyze and report findings, ideas for how to tailor usability testing to meet budget and schedule constraints, and new knowledge on how to apply usability testing to other survey-related products, such as project websites and interviewer administered tools.

Key Features

Explains how to design and conduct usability tests and analyze and report the findings

Includes examples on how to conduct usability testing on any type of survey, from a simple three-question survey on a mobile device, to a complex, multi-page establishment survey

Presents real-world examples from leading usability and survey professionals, including a diverse collection of case studies and considerations for using and combining other methods

Discusses the facilities, materials, and software needed for usability testing, including in-lab testing, remote testing, and eye tracking

Show more

Explains how to design and conduct usability tests and analyze and report the findings

Includes examples on how to conduct usability testing on any type of survey, from a simple three-question survey on a mobile device, to a complex, multi-page establishment survey

Presents real-world examples from leading usability and survey professionals, including a diverse collection of case studies and considerations for using and combining other methods

Discusses the facilities, materials, and software needed for usability testing, including in-lab testing, remote testing, and eye tracking

Details

Copyright

Copyright © 2017 Elsevier Inc. All rights reserved.

You currently don't have access to this book, however you can purchase separate chapters directly from the table of contents or buy the full version.
Purchase the book

Authors

Emily Geisen

Jennifer Romano Bergstrom

# data[].attributes.attachment_link.description
Book * 2017 Authors: Emily Geisen and Jennifer Romano Bergstrom Book * 2017 Authors: Emily Geisen and Jennifer Romano Bergstrom Usability Testing for Survey Research provides researchers with a guide to the tools necessary to evaluate, test, and modify surveys in an iterative method during the survey pretes ...

# data[].attributes.body
https://www.iso.org/obp/ui/#iso:std:iso:ts:20282:-2:ed-2:v1:en

# data[].attributes.headline
Remote Articulation Test System for Supporting Special Education Classes for Language-disabled Children

# data[].attributes.subject
Remote Articulation Test System for Supporting Special Education Classes for Language-disabled Children

# data[].attributes.body
https://www.computer.org/csdl/proceedings-article/iiai-aai/2020/739700a128/1tGcpycQk2k

# data[].attributes.headline
A Quantitative Approach to Content Validity

# data[].attributes.subject
A Quantitative Approach to Content Validity

# data[].attributes.headline
ABC of Content Validation and CVI Calculation

# data[].attributes.subject
ABC of Content Validation and CVI Calculation

# data[].attributes.attachment_link.readable_text
4.2.5 Characteristics and Features

*

anonymity of experts, which assures free expression of opinions provided by the experts. This method helps to avoid social pressure from dominant or dogmatic individuals or even from the majority or minorities,

*

iteration-A t any point, experts can change their opinions or judgments without fear of being exposed to public criticism,

*

controlled feedback- Experts are informed about views of other experts who participate in the study,

*

some form of statistical aggregation permits a quantitative analysis. Thus, though qualitative, the Delphi method can provide quantitative results.

Read full chapter

URL: https://www.sciencedirect.com/science/article/pii/B9780128115138000042

Historical perspectives

As noted, the Delphi method was developed by the Rand Corporation in the 1950s. Most of its early uses were in the military and industrial fields. Since many of the early Delphi studies used the method to make forecasts of future occurrences, the name was first applied by some people at Rand as a joke, which was not entirely appreciated by its founders, Olaf Helmer or Norman Dalkey (Turoff and Hiltz 1997). The joke was that in Greek mythology, the famous Greek oracle, Pythia, 'performed' at Delphi where the temple sacred to the god Apollo was built. When Pythia was consulted, she would utter weird sounds as in a frenzy, much in the way that the witches in Shakespeare's Macbeth intoned: 'Double, double, toil and trouble ...'.At Delphi, temple priests interpreted these to the public, and thus they were able to exert considerable influence over the future actions of cities, as well as private individuals, who sought the oracle's advice.

The Delphi method as developed has been used both to predict the future and evaluate alternative courses of action (Reilly 1973, p. 188). However, the method has not been without its critics. For example, Busha and Harter (1980) saw the Delphi method as not having a sound theoretical base and of achieving results which are often less than spectacular.

Read full chapter

URL: https://www.sciencedirect.com/science/article/pii/B9781876938420500204

3.2 Public Partnership Methods

Several specialized methods are available for the development of participation processes. These methods propose a way of increasing public participation in order to solve a problem or conflict (MINVU, 2010). Bordagaray, Dell'Olio, Ibeas, Barreda, and Alonso (2015) developed a system for discovering influential variables to address the problems detected by users of public bicycles in order to improve the management of the service.

The most important methods for analyzing public participation are described below:

1.

The Delphi method is a formal, in-depth systematic qualitative methodology which was first studied by a team at the RAND Corporation in 1950, who made multiple practical applications of the method ( Dalkey & Helmer, 1963). The goal of the method is to reduce the diversity of opinions within a small group so they tend to converge towards one common opinion. The method applies a cycle made up of various stages of questions, answers, analysis which are then returned to the experts in the subject. The questionnaires aim to reduce the deviation of the group's opinions from that of the expert by defining the average of the answers obtained (Landeta, 1999). The first questionnaire calculates the dispersion of opinion; the second delivers the opinions of their group partners to each expert. Each expert will analyze the pros and cons of the opinions of the members of the group along with their own presented in the first cycle. The process repeats until the required number of questionnaires have been applied. At the end of the iterations, the replies that continue to be different from the group average can be analyzed as alternative scenarios to formulate alternative hypotheses about the future. This methodological approach is used to fulfill long term goals within a determined overall strategic plan.

2.

The "Charrette" method (MINVU, 2010) is understood to be a rigorous and inclusive collaborative process for elaborating strategies and plans for the practical application of design and planning projects. This method makes use of the opinions of all the agents involved considering them all to be equal, the idea being that everyone has a voice and is listened to and nobody dominates the decision making process. The methodology is based on meetings or workshops which are open to the public, these sessions are guided by a trained moderator who acts as mediator between contributors. The main benefit of this method is that it involves the community meaning that the chosen final approach to the desired project will probably be positive as the public have been involved in its formulation, planning, and design.

3.

The Livingstone (Mesías & Romero, 2004) method is a system of public participation which comes from the field of architecture. The method contemplates a series of dynamic participation techniques, such as, for example, role play, role swapping or questionnaires which look to collect information about what people want, examples could be: the more-less game, which aims to establish priorities for members of a group; the fiscal game, trying to define problems, although these problems appear not to have any solution, etc. If these games are performed correctly, a solid program of requirements can be established

4.

The Mixed Method of public participation is a new scientific method that can be applied to any field requiring research into demand and human behavior (Barreda, 2016). This method is characterized by not having rigid boundaries between qualitative and quantitative research as its mixed nature is able to apply both approaches to collect the opinions and perceptions of all the agents involved to arrive at an unbiased sample of the relevant population.

The mixed method follows three clearly differentiated phases (Fig. 3.3). The first phase uses qualitative techniques for the initial data collection process. A thorough state of the art review needs to be performed beforehand on the relevant bibliography in order to make an initial identification of the determinant variables on the behavior of the agents involved. The initial identification of the variables will support the design of scientific guidelines for the consultation processes which will be performed in two different ways:

*

In person: By conducting Mega Focus Groups MGF, FG, and in-depth interviews with the agents involved.

*

Online Participation Platform: The online participation platform uses a debating platform available to each of the agents which enables focus groups, debates and in-depth interviews, etc. to take place online.

The diverse group dynamics are then used to identify the variables to be considered in the design of the Stated Preferences (SP) and Revealed Preferences (PR) Pilot Surveys.

The following phases are of a quantitative nature: the second phase is based on the variables taken from the qualitative part of the process, and concentrates on the design and application of stated and/or revealed preferences pilot surveys. The Pilot Survey will be asked to a representative sample in each of the cases and after the required and opportune modifications have been made, the questionnaire is redesigned to create the definitive PD and/or PR survey. The third phase of this methodology involves asking the definitive survey to a sample of the population, the size of which depends on the goals of the study. These surveys will also be available through the online platform, thereby providing greater heterogeneity in the results and enabling a bigger sample size to allow transversal analysis of the information obtained. The data collected in the surveys is then applied to the development of Econometric Discrete Choice Models to quantify the parameters addressed in the surveys. The results of these are presented and discussed along with the conclusions drawn to form a basis on which to design future policies for the management and/or planning in the particular field of study.

Read full chapter

URL: https://www.sciencedirect.com/science/article/pii/B9780081020807000033

15.3 Introducing scenario planning

So how could we go about predicting what the future could hold for education institutions, libraries, and librarians-and how might that help us to understand the changes that Education 4.0 is likely to bring? We will look at scenario planning, the umbrella name for a family of techniques that have seen widespread adoption from large corporates such as Apple to policy strategy units such as the UK's Government Office for Science and innovation foundations like Nesta. Scenario planning was originally developed by Shell in the 1970s and subsequently adopted by groups as diverse as the fashion industry (Forum for the Future, 2019), the International Olympic Committee (IOC, 2020).

In their Compendium of Innovation Methods, Nesta (2019) characterize their scenario planning approach in terms of

●

Gathering intelligence about the future (e.g. seven Questions and the Delphi method).

●

Exploring the dynamics of change (e.g. Driver Mapping and the Axes of Uncertainty).

●

Describing what the future might be like (e.g. Visioning and SWOT Analysis).

●

Developing/testing policy and strategy (e.g. Backcasting and Roadmapping).

In their Futures Toolkit, the Government Office for Science (GO-Science, 2017) notes that they are working to deliver according to a five-stage policy lifecycle. This policy lifecycle could also be viewed through an organizational lens in terms of strategy and implementation:

●

Formulate policy using consultation, research, and fit with the wider strategic or policy aims of government to establish relevant goals and targets.

●

Implement policy by creating and funding appropriate initiatives and policy instruments.

●

Monitor events to track progress towards goals and targets and to adjust activity if necessary.

●

Evaluate impact against the project goals and targets and to test contribution to the overall strategy.

●

Modify the objectives if required to increase impact or strategic relevance.

Spaniol and Rowland (2017) describe the evolution of scenario planning from Cold War scenarios devised by defence strategists through to their adoption by corporate interests-notably Royal Dutch Shell under Pierre Wack during the oil crisis of the early 1970s. They also note that the theory underpinning scenario planning has largely been retrofitted to existing practice and as such can be viewed as exhibiting 'dismal theory and methodological chaos'. Nevertheless, they find that there is significant evidence that organizations of all types and sizes have found scenario planning to be a useful technique and indeed that they continue to use it.

Oliver and Parrett (2017) describe the application of a common scenario planning approach that builds on the pioneering work of the Shell team, in the context of media experts looking at possible strategies for Google's YouTube brand as it faces increasing competition in the streaming media market from both incumbents and challengers such as Netflix. They describe a scenario planning approach that we will draw upon for this chapter-'identify the key focal issue, driving forces, and critical uncertainties before designing four plausible futures and a series of strategic options'.

According to Oliver and Perrett, the participants in the YouTube strategy workshop 'generated 49 driving forces... in a lively debate that lasted 1.5 hours, during which time the participants contested the different views presented. Following this discussion, two critical uncertainties were identified: (1) an increase in the regulation of video content and (2) the extent to which video content can be monetized'. These uncertainties acted as drivers to prompt the participants to devise four scenarios to model:

1.

Low increase in the regulation of video content and the low extent to which video content can be monetized, aka 'Porn-riddled, cat-infested YouTube rebrands to YouCloud in last-ditch attempt to make £££'.

2.

Low increase in the regulation of video content and the high extent to which video content can be monetized, aka 'Trillionaires prefer Laissez-Faire'.

3.

High increase in the regulation of video content and the low extent to which video content can be monetized, aka 'No-Ella! Former online sensation gone down the (You)Tube'.

4.

High increase in the regulation of video content and the high extent to which video content can be monetized, aka 'YouTube takes first steps towards the monetization of freedom of speech as anonymous user pays £100k for live ISIS stream'.

Whilst these scenarios may seem overblown or hyperbolic, scenario planning practitioners would say that it is important to create a compelling narrative which will encourage participants to give the exercise their full attention. This approach is often characterized using a 2 × 2 matrix such as the one shown in Fig. 15.1, which shows how the scenarios relate to the uncertainty areas around regulation and potential to monetize.

In the YouTube case, alongside each of the scenarios, the participants considered early warning signals that would help YouTube to establish the likelihood of the scenario coming to pass. They also explored potential strategic responses that would help YouTube continue to thrive in each of the scenarios. We can do something similar when we consider the transition to Education 4.0 for libraries and the institutions that they exist to support, looking at possible scenarios and potential institutional responses.

Having identified critical uncertainties and modelled potential scenarios the organization undertaking scenario planning is often left with a variant of what Clayton Christensen characterizes as the Innovator's Dilemma: How should one go about setting up a new business unit or developing a new product for a business model, technology, or approach that is the antithesis of everything that the organization stands for or the way that it works at present?

As part of an interview with Christensen, Denning (2016) observes that

The classic pattern of disruptive innovation ... is one in which an unobtrusive competitor eats away at the low end of an incumbent's market with a lower quality product. The incumbent is happy to concede the low-value customers and concentrates on adding more features for its base of high-value customers. Next the disruptor steadily improves quality to move up-market, and then devours the whole market of the incumbent, who often doesn't perceive the threat until it's too late.

Somewhat ominously, Denning also notes that 'at one extreme, some incumbents were not innovating at all, such as universities'.

Read full chapter

URL: https://www.sciencedirect.com/science/article/pii/B978012822144000015X

11.4 Heuristics, hedging, and bias in subjective estimation

The pressing economic and policy issues stemming from the OPEC cartel were manifest in two resource appraisals of national scope: NURE for uranium [US DOE (1980)] and the oil resource appraisal resulting in USGS Circular 725 [Miller et al. (1975)]. Both of these appraisals employed (1) subjective geological analysis as a means to estimating the magnitude of undiscovered deposits, and (2) the description of quantitative estimates by subjective probabilities. Because of the urgency of the policy issues that were examined with regard to estimates of the quantity and quality of undiscovered deposits, methodologies were scrutinized more carefully than ever before.

Initially, subjective probabilities of an expert were accepted as useful, first because of his expertise and second because of the seemingly reasonable assumption that man functions well as an intuitive statistician, meaning that the expert's perception of likelihood of an event corresponds generally to its relative frequency in nature. Moreover, Delphi methods had become accepted means for treating differences of opinions of experts, having taken for granted that convergence of opinions of members of the group to a single value provided a desirable and useful resolution of differing expert opinion. These perceptions were seriously challenged by the work of Tversky and Kahneman (1972, 1974) and Sachman (1974).

Tversky and Kahneman (1972, 1974) identified some of the heuristics used by man to estimate an uncertain event: anchoring and adjustment, representativeness, and availability. These heuristics were found to lead to predictable biases. Psychometricians found that man generally behaves as though he knows far more than he actually does, tending to provide subjective probability distributions that are narrower than they should be by approximately 50% [Slovic (1972)]. This tendency results from the inability of man to image mentally the combinations that result in extreme values of the random variable; heuristics used to approximate actual calculations understate probabilities for events in the tails of the distribution when stochastic events are conjunctive.

Sachman's (1974) investigation of traditional Delphi as a means of inquiry found that repeated cycles of estimate and statistical feedback violate blatantly the condition that estimates by individuals be independent in order for statistics such as mean and variance and for relative frequencies of estimates to be statistically meaningful. Moreover, he documented that convergence can be solely a psychological phenomenon and usually is not the result of exchange of factual information and logical deliberation.

These findings led to (1) the total elimination of Delphi or the replacement of traditional Delphi by modified Delphi, and (2) the structuring and support of subjective probability estimation so as to improve the use by the expert of his science and available geodata and to minimize the effects of heuristic biases.

Read full chapter

URL: https://www.sciencedirect.com/science/article/pii/S1573443905800089

3.8 Qualitative Methods of Forecast of Transport Demand

Qualitative methods for the forecast of transport demand are based on perceptions, evaluations, and assessments made either by transport specialists or by the users of a transport service. They have a more or less subjective and fluctuating character, as they differ from one person to another and change from one moment to another. Transport science has devised methods to transform such subjective evaluations into objectively acceptable frames of decision-making and understanding the evolution of a specific transport demand problem. However, in order to reduce the degree of subjectivity of qualitative methods, simple statistical techniques have been suggested and are in use.

There are four principal qualitative methods for the assessment, evaluation, and forecast of transport demand:

*

Executive judgment: it is based on the experience and evaluation of transport specialists who have a deep knowledge of the transport problem under study. This method considers most of the variables of a transport problem and can be executed, particularly with the help of modern electronic communication (e-mail, video chat, forums, etc.), in rather short time periods and with a low cost.

*

Delphi method : it is based on the global assessment made by a group of transport experts and is used for the assessment of new transport problems (with unknown for the moment effects) or for existing transport problems, for which our data are either nonexisting or inaccurate or of poor quality. Delphi is conducted in successive rounds until a level of consensus is reached among experts. It can be supported by simple statistical techniques and usually it takes a rather long time.

*

Scenario writing: it is based on the construction of alternative scenarios, which describe one or more plausible future situations and the processes and pathways toward these situations; it should be noted that the departing point is usually the existing situation. Scenario writing can also be supported by simple statistical techniques.

*

Survey methods: these are based on the views and aspects of transport users who are interviewed over a number of transport issues. Key constituents of a survey are the questionnaire, which includes the points and issues that will be explored; and the sample of the survey, that is, the portion of the whole population which will be surveyed. Survey methods are supported by more sophisticated statistical techniques compared with other qualitative methods.

Qualitative methods for transport demand are extensively analyzed in Chapter 4 of the book.

Read full chapter

URL: https://www.sciencedirect.com/science/article/pii/B9780128115138000030

12.3.1 Design Phase

The design phase consisted of three stages: (1) an information phase, (2) an analytical phase, and (3) a feedback phase. In the information phase, stakeholders were first given all essential information concerning the performance indicators followed by a presentation on these indicators as a reminder. The analytical phase included a quantitative and qualitative assessment of the performance indicators. In the first stage, an internal stakeholder assessment with port experts from the EA&S Committee was held, based on the Delphi method, to ensure a consensus on the prioritization of the indicators based on stakeholder acceptance and implementation feasibility of the indicators. Alongside the internal stakeholder analysis, a quantitative analysis in the form of a survey yielding 338 responses of external stakeholders (other port stakeholders and experts) was held. The majority of the respondents (30%) are employed by port managing bodies. The rest of the stakeholders that were well represented in the sample were consultants and/or academics (19%), terminal operators (7%), ship owners (7%), governmental agents (6%), and individuals working in the offer of technical-nautical services (6%). On the contrary, responses coming from shippers (4%), rail transport (2%), trade unions (2%), local communities adjacent to a port (2%), barge transport (1%), and road transport (1%) were limited and accounted only for the 12% of the total of the sample. The vast majority of the responders (86%) are working or/and residing in a European Union member state, whereas the remaining 14% of the respondents are located on a different continent.

The most important assessment question took the following form: "Do you think that the indicator "X" is acceptable/feasible?". A 5-point Likert-style scale was used for responding to each question, anchored with strongly agreed (5) and strongly disagreed (1). The option of "No opinion" was also given. Finally, in the feedback phase, the results of the former stages were made public at conferences and technical committee meetings with dedicated discussions on port performance indicators.

The internal assessment shows, for the socioeconomic category, that stakeholder acceptance and implementation feasibility are somewhat linked to each other: the indicators with the highest acceptance average are also deemed feasible by the respondents, with the exception of gross value added (GVA). Direct employment, direct GVA, and investments receive the highest averages on acceptance. It is remarkable that a relative measure of direct GVA per full-time equivalent (FTE) scores lower than the absolute measures it is composed of. Here, comparability issues clearly play a role as some members from the expert committees (mainly port authority representatives) clearly expressed their concern that the diversity of ports should be taken into account when comparing these relative indicators. Also, potential problems at the data collection side, for employment figures, as well as GVA, could heavily distort the result of this relative indicator.

We also revealed differences between the internal and external stakeholder assessment. In the external assessment, the top two indicators are direct gross added value and direct employment, with investment having a lower acceptance among the broader representation of stakeholders. Confronting internal and external assessments, we obtain Table 12.2, where in addition to the results of the two assessments, we added a column "number of scores lower than 3" (based on the aggregated internal assessment dimensions and the external assessment score), which could lead to more synthesis. Here, we observe that again direct employment and investments could be ranked in the high-priority category. However, we note a discrepancy between the scores of investment and direct gross added value comparing the acceptance dimension. Both in the internal and external assessment, direct gross added value scores considerably better than investments in this dimension. In the external assessment, direct gross added value obtains the best score, while investment is only the fifth-ranked indicator.

Indicator

Internal Assessment

External Assessment

Priority (Low, Medium, High)

Stakeholder Acceptance (Mean)

Feasibility (Mean)

Acceptance (Mean)

Lower Than 3

FTE, full-time equivalent; GVA, gross value added.

In that context, we also need to mention the comments received by the qualitative analysis (informed statements from the Delphi method), in particular, regarding the nature and definition of the investment indicator (public vs. private investment), as well as data availability issues apparently not sufficiently captured during the internal stakeholder assessment phase; this probably also explains the lower score of the indicator investments in the external assessment. Therefore, for implementation phase purposes, it was decided not to include this indicator.

Read full chapter

URL: https://www.sciencedirect.com/science/article/pii/B9780128140543000128

Potential impact

The Delphi method is a unique methodology used to systematically generate consensus by using panels of thoughtfully selected experts. It enables researchers and educators to answer critical research questions that are not amenable to other types of research designs, such as experimental or quasi-experimental methods. By appropriately using the Delphi method, there is an opportunity to inform decision-making in pharmacy education in an organized and meaningful way. By providing multiple examples, we have provided a scaffolded overview of this method to empower more health professionals to use this method across a wide range of contexts and needs. Additional resources on the Delphi Method are provided in Box 1.

As we encounter situations that require consensus building or forecasting, we must expand our research toolkits with methods that enable systematic and reliable generation of new knowledge. The Delphi method is a purposeful way to inform decision-making, generate consensus, and answer research questions. Pharmacy educators can use the Delphi to meet a variety of purposes. While a growing number of studies in pharmacy education use Delphi processes, we must ensure our protocols uphold the quality and rigor needed to effectively answer complex questions that cannot be addressed by other research designs.

Read full article

URL: https://www.sciencedirect.com/science/article/pii/S1877129721002227

3.1.3 Combined method

Flexibility is one of the advantages of AHP. It is flexible to be integrated with different techniques like Delphi method and fuzzy mathematics etc. This allows users to benefit from the combined methods resulting in a better way for analysis and decision-making. Delphi method was first developed by RAND Corporation in the United States in 1950s. It is a method that use a series of questionnaire delivered using multiple iteration process to collect data from a panel of selected experts. The anonymity of Delphi panels, iteration that allows panelists to refine their views, controlled feedback and statistical aggregation of group response that allows for quantitative analysis and interpretation of data lead to a relatively representative and reliable result ( Arof, 2015). It takes the advantages of personal knowledge and experiences of experts, which has been used in ERA of industrial park (Yue et al., 2008). It also can be used with AHP. During a case study of 40 industrial parks in the coast area of Jiangsu Province, Yin et al. (2011) employed both Delphi method and AHP, and constructed a comprehensive risk ranking model to ensure a convincing result. The environmental risk levels of those CIPs were determined, and the main risk factors were identified.

AHP helps to manage complicated problem by converting it into manageable and comprehensible small problems, describing the relative importance of risks. However, the subjective pairwise comparison is prone to vagueness type uncertainty, consequently raise the need for exploring fuzzy-based techniques (Tesfamariam and Sadiq, 2006). This F-AHP approach formulates the hierarchic tree, and then create fuzzy pairwise comparison matrix followed by fuzzy analysis. The detailed fuzzy set theory will be introduced in Section 3.3. It has been widely used in environmental decision making. Liu (2015) used the AHP-multilevel fuzzy comprehensive evaluation to classify the risk levels of environmental risk source in Qinzhou petrochemical industrial park in Guangxi Province, China. The criteria and their weight were established based on AHP, then fuzzy set theory was employed to classify the risk sources into four levels, namely particularly significant, significant environmental, greater, and general environmental sources.

Read full article

URL: https://www.sciencedirect.com/science/article/pii/S0921344920301646

4.2 Effective use of leaflets

Leaflets have been used by the police, some Departments of Forensic Medicine, and the Japanese Cabinet Office at support settings. Examination of "the Guidelines for Acute Psychosocial Support for Crime Victims," prepared through consensus formation by supporters from many orientations using the Delphi method, has shown that the receipt of leaflets by victims during the acute-phase information-provision period is beneficial. Furthermore, explaining the flow of legal procedures to the victims is essential [ 136]. During the acute phase immediately after shock, psychological paralysis and numbness are some of the major symptoms faced by the victims (particularly the bereaved family). In such situations, a complete understanding of words and sentences is sometimes impossible because of psychological confusion; hence, providing informational leaflets that can be read later by the victims is important.

Leaflets containing information regarding where to obtain further information have also been created for places providing support to MVA orphans. Accessing information on their own is difficult for orphans, and support providers who notice that orphans want more information can provide them with relevant leaflets.

Read full article

URL: https://www.sciencedirect.com/science/article/pii/S0386111219300068

# data[].attributes.attachment_link.description
anonymity of experts, which assures free expression of opinions provided by the experts. This method helps to avoid social pressure from dominant or dogmatic individuals or even from the majority or minorities, * iteration-A t any point, experts can change their opinions or judgments without fear of being exposed to public criticism, * controlled feedback- Experts are informed about views of other experts who participate in the study, * some form of statistical aggregation permits a quantitative analysis.

# data[].attributes.headline
Gathering and Analyzing Content Validity Data

# data[].attributes.subject
Gathering and Analyzing Content Validity Data

# data[].attributes.attachment_link.title
Sci-Hub | Gathering and Analyzing Content Validity Data | 10.1207/s15326977ea0504_2

# data[].attributes.headline
ITC Guidelines on Computer-Based and Internet Delivered Testing 

# data[].attributes.subject
ITC Guidelines on Computer-Based and Internet Delivered Testing 

# data[].attributes.body
https://www.intestcom.org/files/guideline_computer_based_testing.pdf

# data[].attributes.headline
ITC Guidelines for Translating and Adapting Tests(Second Edition)

# data[].attributes.subject
ITC Guidelines for Translating and Adapting Tests(Second Edition)

# data[].attributes.body
https://www.intestcom.org/files/guideline_test_adaptation_2ed.pdf

# data[].attributes.body
https://docs.google.com/drawings/d/1v1t5_76s2xzxRQmcAfRdeT4KDQWKMAAmwbh6xwkNRSc/edit?usp=sharing

# data[].attributes.attachment_link.description
Choose from a wide variety of shapes to create diagrams and charts. Free from Google.

# data[].attributes.headline
Recommendations for elaboration, transcultural adaptation and validation process of tests in Speech, Hearing and Language Pathology

# data[].attributes.subject
Recommendations for elaboration, transcultural adaptation and validation process of tests in Speech, Hearing and Language Pathology

# data[].attributes.body
https://drive.google.com/drive/folders/1bQ4Y4cpsXej-Qkyk2S0xgtc3JL1I_aPK

# data[].attributes.attachment_link.description
Get access to files anywhere through secure cloud storage and file backup for your photos, videos, files and more with Google Drive.

# data[].attributes.headline
Resourcing speech-language pathologists to work with multilingual children

# data[].attributes.subject
Resourcing speech-language pathologists to work with multilingual children

# data[].attributes.body
https://www.tandfonline.com/doi/abs/10.3109/17549507.2013.876666?journalCode=iasl20

# data[].attributes.attachment_link.readable_text
Abstract
Speech-language pathologists play important roles in supporting people to be competent communicators in the languages of their communities. However, with over 7000 languages spoken throughout the world and the majority of the global population being multilingual, there is often a mismatch between the languages spoken by children and families and their speech-language pathologists. This paper provides insights into service provision for multilingual children within an English-dominant country by viewing Australia's multilingual population as a microcosm of ethnolinguistic minorities. Recent population studies of Australian pre-school children show that their most common languages other than English are: Arabic, Cantonese, Vietnamese, Italian, Mandarin, Spanish, and Greek. Although 20.2% of services by Speech Pathology Australia members are offered in languages other than English, there is a mismatch between the language of the services and the languages of children within similar geographical communities. Australian speech-language pathologists typically use informal or English-based assessments and intervention tools with multilingual children. Thus, there is a need for accessible culturally and linguistically appropriate resources for working with multilingual children. Recent international collaborations have resulted in practical strategies to support speech-language pathologists during assessment, intervention, and collaboration with families, communities, and other professionals. The International Expert Panel on Multilingual Children's Speech was assembled to prepare a position paper to address issues faced by speech-language pathologists when working with multilingual populations. The Multilingual Children's Speech website ( http://www.csu.edu.au/research/multilingual-speech) addresses one of the aims of the position paper by providing free resources and information for speech-language pathologists about more than 45 languages. These international collaborations have been framed around the World Health Organization's International Classification of Functioning, Disability and Health (ICF-CY) and have been established with the goal of supporting multilingual children to participate in society.

Acknowledgements

This paper is based on the Invited Elizabeth Usher Memorial address to the Speech Pathology Australia National Conference, Gold Coast, Australia, June 2013. Thanks are extended to Speech Pathology Australia for the honour of presenting the address and the invitation to publish it. The paper contains a summary of the work undertaken during my 4-year Australian Research Council Future Fellowship (FT0990588). I am grateful for the insights from and collaboration with my students and colleagues including Sarah Verdon, Kate Crowe, Karla N. Washington, Carol K. S. To, Brian A. Goldstein, Linda J. Harrison, Jane McCormack, Lindy McAllister, Elise Baker, Caroline Bowen, Rebekah Lockart, Bethany Toohill, Nicole Limbrick, Hannah Wilkin, Suzanne Hopf, and Sarah Masso.

Declaration of interest: The author alone is responsible for the content and writing of the paper. This paper went through peer review and revisions overseen by a guest editor.

Notes

1. Children who only communicated using Australian Sign Language (Auslan) were not included within this figure.

# data[].attributes.attachment_link.title
Resourcing speech-language pathologists to work with multilingual children

# data[].attributes.attachment_link.description
Abstract Speech-language pathologists play important roles in supporting people to be competent communicators in the languages of their communities. However, with over 7000 languages spoken throughout the world and the majority of the global population being multilingual, there is often a mismatch between the languages spoken by children and families and their speech-language pathologists.

# data[].attributes.headline
The Psychometrics of Standardized Tests of Articulation and Phonology

# data[].attributes.subject
The Psychometrics of Standardized Tests of Articulation and Phonology

# data[].attributes.body
https://pubs.asha.org/doi/10.1044/2018_PERS-SIG1-2018-0018

# data[].attributes.attachment_link.readable_text
Abstract

Purpose

The purpose of this tutorial is to provide speech-language pathologists with the knowledge and tools to (a) evaluate standardized tests of articulation and phonology and (b) utilize criterion-referenced approaches to assessment in the absence of psychometrically strong standardized tests.

Method

Relevant literature on psychometrics of standardized tests used to diagnose speech sound disorders in children is discussed. Norm-referenced and criterion-referenced approaches to assessment are reviewed, and a step-by-step guide to a criterion-referenced assessment is provided. Published criterion references are provided as a quick and easy resource guide for professionals.

Results

Few psychometrically strong standardized tests exist for the evaluation of speech sound disorders for monolingual and bilingual populations. The use of criterion-referenced testing is encouraged to avoid diagnostic pitfalls.

Discussion

Speech-language pathologists who increase their use of criterion-referenced measures and decrease their use of standardized tests will arrive at more accurate diagnoses of speech sound disorders.

Over 30 years ago, McCauley and Swisher (1984) published an article reviewing the psychometric merit of standardized tests of articulation. Their conclusion was that speech-language pathologists (SLPs) are in the "uncomfortable position" of having less than perfect standardized tests to assess the speech skills of children suspected of having a speech sound disorder (SSD; p. 41). Fast forward 30 years to Kirk and Vigeland (2014), who published a review of more recent standardized tests of articulation and phonology, their results were not unlike those of their predecessors; the tests included in their review did not meet many of the psychometric properties required for the adequate identification of typical and disordered skills in children acquiring their phonological system. The authors cautioned clinicians that, although single-word, standardized tests of articulation and phonology have value in the assessment battery of young children, they should not be the only component of the clinical decision-making process. Although Flipsen and Ogiela (2015) found modest gains in recent years in the quality of single-word standardized tests of articulation, most tests continue to lack the full range of psychometric criteria, and many must still be supplemented with additional measures to be diagnostically acceptable. Where did these tests fall short? How do we decide what tests are best? In the absence of psychometrically sound standardized tests, what are our options for speech sound assessment? This article will discuss (a) current practices regarding standardized tests, (b) limitations of standardized tests, and (c) how to avoid overreliance on standardized tests and adopt criterion-referenced approaches.

SLPs' Use of Standardized Tests: Current Practices

In order to understand what sort of impact the widespread use of standardized tests has on accuracy of diagnosis, we must first discuss the limitations of standardized tests. It is important to point out that Kirk and Vigeland (2014) placed the impetus on the developers of standardized tests to increase the inclusion of various psychometric properties at the time of test creation. Tests of higher quality would be a great service to the clinicians in our profession and would reduce the number of diagnostic errors made by clinicians. It is important that the message taken away from this article is that most standardized tests do not provide the sensitivity (identification of disorder when disorder is present) and specificity (identification of typical when a child is indeed typical) required to act as the sole component of an evaluation. More to the point, SLPs who are aware of the weaknesses of standardized tests will be more likely to design an evaluation battery that results in an accurate diagnosis.

How are we assessing SSD as a field? Skahan, Watson, and Lof (2007) published survey data on common assessment practices used by SLPs practicing in the United States. Just over 300 clinician responses were analyzed to observe how assessment protocols were selected for children with suspected SSDs. Clinicians who responded to this questionnaire had an average of 15 years' of experience in the field, ranging from 1 to 40 years, and worked in a variety of settings. More than 50% of respondents included the following procedures in their speech sound assessment protocol: (a) estimating intelligibility, (b) administration of a standardized single-word test, (c) a hearing screening, (d) stimulability testing, and (e) an oral motor examination (p. 249). Survey results indicated that the most widely used standardized test by SLPs was the Goldman-Fristoe Test of Articulation (GFTA, multiple editions; Goldman & Fristoe, 2000). It was reported by the majority of respondents that the GFTA is "always" used as part of their phonological assessment (p. 249). In addition, approximately 30% of respondents indicated using the Khan-Lewis Phonological Analysis ( Khan & Lewis, 2002) as a supplement to the GFTA (p. 249). Williams and McLeod (2012) examined the phonological assessment practices of Australian SLPs. When considering bilingual and multilingual children on their caseloads, over 40% of SLPs always used a standardized test in English to assess multilingual children, regardless of language background (p. 298). Skahan et al. found a similar result for SLPs practicing in the United States; English-only standardized tests were used by 35% of SLPs when assessing bilingual children (p. 251). In order to understand what sort of impact the widespread use of these tests has on accuracy of diagnosis, we must first discuss psychometric properties of tests and the impact they have on a test's utility.

What Do Standardized Tests Tell Us?

According to Peña, Spaulding, and Plante (2006), norm-referenced, standardized tests provide the opportunity to compare the speech of one particular child to a "standard" (p. 247). We can observe if a child is performing like his peers (i.e., in the average range) or how many levels above or below his peers (i.e., standard deviations). If a child is performing below average, we can observe how far below his peers he is performing. The number of standard deviations below average, derived from a child's performance on the test, often designates whether or not a child receives speech therapy services. These interpretations, however, are only accurate if the test is well constructed. The proverbial saying, "Garbage in, garbage out" applies here; if a test is weak, the results of that test will provide an inaccurate representation of a child's speech sound abilities. This is a major challenge in our field because (a) the majority of standardized tests of articulation and phonology are psychometrically weak ( Kirk & Vigeland, 2014) and (b) there is an overreliance on standardized tests by SLPs ( Skahan et al., 2007). The goal of this article is to provide information on navigating standardized assessment manuals and how to identify tests that are well constructed. If strong standardized tests are unavailable, we will explore alternative approaches to assessment.

The Psychometrics of Standardized Tests of Articulation and Phonology

How do we know if a test is well constructed? McCauley and Strand (2008) and Kirk and Vigeland (2014) are both excellent resources for reviewing the parameters of standardized tests of articulation and phonological and oral motor (speech and nonspeech) abilities in children. Here, we will focus on three major psychometric properties of standardized tests: (a) the normative sample, (b) reliability, and (c) validity. For a more complete explanation of psychometric parameters, Hutchinson (1996) provides a comprehensive list and a thorough explanation of psychometric properties. In addition, detail on the standards for educational and psychological testing is regularly published by the American Educational Research Association (2014). Furthermore, a five-part webinar on test psychometrics for child speech is available for online learning ( Pavleko, 2018). Because the focus of this tutorial is to focus on the core components a test must possess for accurate diagnosis of SSDs (in line with Kirk & Vigeland, 2014), we will focus on these three main criteria.

The Normative Sample

Plante and Vance (1994) indicated two major areas of importance when evaluating the normative sample of a test: the definition of the standardization population (including geographical location, socioeconomic status [SES], and the typical vs. disordered status of the children included) and that the normative sample includes at least 100 children in each subgroup. This information should be easily accessible in the test manual.

If the goal of standardized testing is to differentiate between typical and disordered speech skills, the normative sample must be made up of only typically developing children ( Peña et al., 2006). Including children with disorders in a normative sample shifts mean scores lower, causing underidentification of children with disorders. On the other hand, if you want to judge the severity of a particular disorder, a normative sample of individuals with that disorder is necessary. For example, if a clinician wants to compare a child with an already diagnosed phonological disorder to other children with phonological disorders in order to judge severity, the normative sample should consist of children with phonological disorders. This is not the case when the goal of testing is to diagnose the presence of a disorder; the normative sample must consist of only typically developing children to achieve that result. Many commonly used standardized tests of both speech and language include children with disorders in their normative samples, so this is certainly a psychometric characteristic that SLPs should be investigating first and foremost.

In addition to the level of phonological ability that children have in the normative sample, there is a requisite number of children that need to be included in that sample for comparison. The minimum amount of children in each subgroup necessary for comparison is 100 ( McCauley & Swisher, 1984). Fewer than 100 children in a subgroup allows too much variability in the sample, creating a less-than-reliable comparison between the child being tested and the children included in the sample. Therefore, in terms of the normative sample, the speech sound abilities of the children and the number of children in the normative sample are two main psychometric characteristics that should be clearly stated in the test manual.

Reliability

The second main area of focus in test psychometrics is reliability. According to McCauley and Swisher (1984), the reliability of a test is derived from how consistently a test measures a behavior. For example, if a child is evaluated on the same test item on more than one occasion (e.g., different times in the same day), the child should receive the same score on that item every time. Test-retest reliability, specifically, is the ability of a test to yield the same score when given more than once over short periods. Interexaminer reliability is the ability of more than one examiner to administer the same test to the same child and get the same result. Both test-retest reliability and interexaminer reliability should report a statistically significant correlation coefficient of .90 or higher to be psychometrically acceptable ( McCauley & Strand, 2008). These psychometric attributes instill confidence that a test is consistent, over time, in its ability to measure phonological skills in child speech. Again, this information should be easily accessible in the test manual. Clinicians should be cautious in selecting a test that does not report reliability criteria.

Validity

Our third area of psychometric focus is validity. Correlated with reliability, test validity refers to the ability of a test to measure what it says it measures. For example, if an SLP chooses the GFTA to assess articulation abilities in a preschooler, the SLP's question should be "Does the GFTA actually capture the articulation abilities of a child?" McCauley and Swisher (1984) break validity down into many specific categories, but here, we will focus on only those categories that are easy to measure. Psychometric properties that are easy to measure will be less prone to error or subjectivity.

If a test has strong content validity, that test provides the SLP with accurate information on the ability being tested (p. 35). Criteria supporting the content validity of a test should include (a) a justification of the methods used to choose content, (b) expert evaluation of the test's content, and (c) an item analysis ( McCauley & Strand, 2008). Content validity criteria are determined by the structure (e.g., number of members allowed in an onset cluster) and function (e.g., a sound's morphological status) of a language's phonology ( Eisenberg & Hitchcock, 2010). These criteria are applied to ensure that enough content, or opportunities, for each aspect of speech sound production is being tested ( Eisenberg & Hitchcock, 2010; Kirk & Vigeland, 2015). For example, all vowels must be targeted enough times to obtain an accurate representation of a child's abilities. Likewise, a test must include specific words that trigger all of the phonological error patterns common in a language. For a sound to be included in a child's phonetic inventory, at least two productions of that sound are required; therefore, a standardized test of articulation must provide at least two opportunities for production ( Dinnsen, Chin, Elbert, & Powell, 1990). Weston, Shriberg, and Miller (1989) suggested that 90 different words or 250 total words are needed for sufficient evaluation of a child's phonological abilities (although we know that most single-word tests of articulation contain far fewer than the suggested amount). In more recent work, item response theory has been applied to single-word tests of articulation to distinguish, more effectively and efficiently, the performance of children with and without SSDs ( Brackenbury, Zickar, Munson, & Storkel, 2017). More specifically, reducing single-word samples to only those words that children with SSDs consistently produce in error (whereas children with typical skills consistently produce them correctly) allows SLPs to "zero in" on performance in the disordered range, reducing the number of words needed to obtain an adequate sample. This approach is new, however, and most available tests do not implement this analysis in test construction. In terms of phonological patterns, Kirk and Vigeland (2015) suggested that a test should provide at least four opportunities for the 11 English phonological error pattern types and that there should be ample opportunity for errors across word position and in clusters.

On the other hand, face validity is a surface measure. Does the test appear to test what it says it does? Are the items appropriate, at face value, to test articulation and phonological skills? Face validity is mentioned here because it should not be confused with content validity, which is a much deeper measure. Face validity indicates if the test looks like it tests what it says it tests; content validity measures whether or not it actually accomplishes that aim. The content validity of a test is essential knowledge for an SLP choosing an appropriate standardized test, so SLPs should take caution in accepting reports of only face validity in the absence of content validity.

Standardized tests of articulation and phonology that are designed to differentiate typical from disordered speech must clearly report data on these two main psychometric properties. If the data are (a) not reported or accessible in the test manual or (b) do not meet the minimum criterion requirements for these parameters, the test should not be used as an assessment tool. The absence of standardized tests as part of an assessment protocol is difficult for many SLPs to adopt. For one, standardized tests are widely used, efficient, and familiar. What is important to recognize, however, is they might not be representing the true abilities of the children on our caseloads. This problem brings us to the topic of alternative methods of assessment: How do we fill the gap?

Criterion-Referenced Tests of Articulation and Phonology

As was covered thus far, evaluation of SSDs in children very often includes the use of standardized tests ( Skahan et al., 2007); however, criterion-referenced measures are a less utilized, but highly informative, way to evaluate child speech. What do we mean by norm-referenced versus criterion-referenced testing? Norm-referenced tests compare the child you are testing to a sample of children, given the same test, for comparison. Criterion-referenced testing uses published criteria from studies on child phonological development for comparison. The ideal normative sample in norm-referenced tests, when the aim is to make an accurate diagnosis of disorder, is a sample made up of typical children. In criterion-referenced studies, children with both typical and disordered speech sound abilities are included in the study sample to develop cutoff scores or boundaries between what is considered typical and disordered based on a child's age. Knowledge of cutoff scores allows an SLP to compare an individual child's performance to the cutoff score, or criterion, derived from a study (see Table 1).

Measure

Monolingual English

Bilingual English

Bilingual Spanish

Percent Occurrence of Phonological Error Patterns (Processes)

* Typically developing children, ages 2;10-5;2: Most common phonological patterns include gliding, weak syllable deletion, glottal replacement, cluster reduction, labial assimilation, vocalization, final consonant deletion; stopping, and fronting; use of phonological patterns is reduced by 50% between 3;0 and 4;0 ( Haelsig & Madison, 1986) * Children with disorders, ages 3;7-13;0: Number of processes used ranged from 3 to 10; Final consonant deletion, gliding, unstressed syllable deletion, stopping of fricatives, and vowelization were the most common patterns ( McReynolds & Elbert, 1981) * Typically developing monolingual English speakers living in a Spanish-English environment: At age 4;0, can demonstrate final consonant devoicing as high as 20%-30% ( Fabiano-Smith & Hoffman, 2018)

* At age 3;0 manner classes with low accuracy include stops and fricatives ( Fabiano-Smith & Goldstein, 2010b) * Typically developing children, at age 4;0, exhibit gliding, stopping, final consonant devoicing, and cluster reduction at rates 10% or lower with the exception of gliding (53%). Four-year-olds with disorders exhibit cluster reduction, gliding, final consonant devoicing, and stopping all at rates higher than 10% ( Fabiano-Smith & Hoffman, 2018). Consonant devoicing occurred at approximately 30% in both groups, but is linked to dialect rather than disorder.

* At age 3;0, manner classes with low accuracy include flap and trill ( Fabiano-Smith & Goldstein, 2010b) * Typically developing children, ages 3;0-4;0, Puerto Rican dialect: Phonological patterns with a frequency of occurrence greater than 10% of diagnostic significance; at age 3;0, cluster reduction occurs at 15% and is the most common; at age 4;0, no patterns occurred at a rate greater than 10%. * Other commonly occurring patterns included final consonant deletion, liquid simplification, weak syllable deletion, assimilation, fronting, and stopping ( Goldstein & Iglesias, 1996)

Percent Consonants Correct (PCC; Shriberg & Kwiatkowski, 1982)

* Age 4: > 90% = mild, 65%-85% = mild-moderate, 50%-65% = moderate-severe, and < 50% = severe ( Shriberg & Kwiatkowski, 1994)

* Age 3;0: PCC = 72% in bilingual children ( Fabiano-Smith & Goldstein, 2010b) * Age 5;0: PCC-R = 94% ( Goldstein et al., 2005); PCC = 88% ( Fabiano-Smith & Hoffman, 2018)

* Age 3;0: PCC = 66% in typically developing children ( Fabiano-Smith & Goldstein, 2010b) * Age 5;0: PCC-R = 91% ( Goldstein et al., 2005)

Phonetic Inventory Complexity

* Phonetic inventories complete by age 3;0 in typically developing children; in children with SSD, ages 3;0-6;0, the following characteristics were observed across children: Incomplete phonetic inventories, variation in the number and types of sounds, many sounds but no phonological distinctions, or many phonological distinctions and few sounds ( Dinnsen et al., 1990)

* Phonetic inventories complete by age 3;0 in typically developing bilingual children ( Fabiano-Smith & Barlow, 2010)

* Phonetic inventories complete, or nearly complete, by age 3;0 in typically developing bilingual children ( Fabiano-Smith & Barlow, 2010)

Percent Intelligibility

* Range of percent intelligibility for preschoolers: 68%-100% * Mean percent intelligibility: 85% * Cutoff: For a child 4;0 or older, intelligibility in connected speech that falls below 66% may be a potential indicator of speech sound disorder ( Gordon-Brannan & Hodson, 2000)

Unavailable

Unavailable

Scaffolding Scale of Stimulability (SSS) ( Glaspey & Stoel-Gammon, 2005)

* Using stimulability testing as dynamic assessment, higher scores on the SSS indicate higher likelihood of SSD

Unavailable

Unavailable

Like in the evaluation of standardized tests, the validity of these cutoff scores is not to be overlooked; there should be published evidence that one or more studies demonstrated correlated test scores on a given criterion-referenced measure and that it is supported by a strong underlying theoretical or developmental construct ( McCauley & Strand, 2008). For example, Shriberg and Kwiatkowski (1982) provide extensive detail on how severity cutoff scores were derived for the consonant accuracy measure, Percent Consonants Correct (PCC). Correlations were derived among the listening judgments of unskilled listeners and SLPs. Those judgments were then correlated with children's performance on PCC. These analyses resulted in cutoff scores that indicated mild, moderate, and severe severity categories for children with SSDs. Supporting published data such as these instills confidence that a criterion-referenced measure is truly measuring what it is intended to measure.

McCauley (1996) reported the three most commonly used criterion-referenced measures in the evaluation of SSD: percent occurrence of phonological patterns ( Preston, Hull, & Edwards, 2013), PCC (Percent Consonants Correct-Revised [PCC-R]; Shriberg & Kwiatkowski, 1994), and percent intelligibility ( Gordon, Brannan, & Hodson, 2000). Published criteria for these measures can be found in Table 1. To illustrate how criterion-referenced measures are used, take, for example, PCC-R ( Shriberg & Kwiatkowski, 1994). PCC-R is a measure of overall consonant accuracy in children. Shriberg and Kwiatkowski (1994) measured PCC-R in hundreds of preschoolers and provided cutoff scores that separate out typically developing children from children with SSDs. One of their findings was that 5-year-old monolingual English-speaking children should exhibit greater than 90% accuracy on consonant production. Presence of disorder and severity of disorder can be derived from how far below that criterion a child is performing. A clinician can ask himself if the 5-year-old he evaluated reached the 90% criterion or if he or she is exhibiting only 70% or 50% accuracy. More recently, Fabiano-Smith and Hoffman (2018) performed a similar study examining PCC-R in the English productions of bilingual Spanish- and English-speaking 5-year-olds. From a group of 44 children, with and without SSDs, it was observed that cutoff scores that differentiated typical from disordered speech production did not differ significantly between monolinguals and bilinguals. This is clinically important because, for bilingual children, standardized, norm-referenced tests are few. Therefore, using a criterion-referenced approach, rather than a norm-referenced approach, is essential in making an accurate diagnosis. It is important to point out, however, that deriving criterion-referenced data from standardized tests of articulation and phonology can lead to clinician error. Eisenberg and Hitchcock (2010) examined standardized tests that are often used to derive phonetic inventories for child speech. They found that standardized tests did not provide enough opportunities for children to produce all the sounds of their language, with enough frequency, to develop a representative phonetic inventory. They suggested the use of a single-word test plus additional words that supplement the missing sounds. Fabiano-Smith and Crouse-Matlock (2012) found that connected speech samples, specifically play samples in preschoolers, provided ample opportunities for phonetic inventory development in both monolingual English-speaking and bilingual Spanish- and English-speaking children.

It is essential that speech sound evaluations take into consideration a number of measures, at different levels of the sound system, to arrive at an accurate diagnosis (number of substitution errors, phonetic inventory complexity, overall consonant accuracy, etc.); however, because survey data indicated that over half of SLPs, both in the United States and in other English-speaking countries, are choosing standardized tests as part of their evaluation protocol, the utility of criterion-referenced measures is emphasized and encouraged. The relative strengths or weaknesses of a standardized test will depend on how it was developed, and each test should be thoroughly assessed for its psychometric merit before any are selected for use in assessment. Unfortunately, because the majority of standardized tests of articulation and phonology currently did not meet basic psychometric requirements ( Kirk & Vigeland, 2014), it is essential that they do not act as the cornerstone of speech sound assessment.

Evidence-Based Evaluation of SSDs

What is included in an evidence-based protocol for the assessment of SSDs? What are some alternative approaches to the use of standardized tests for the assessment of SSD? Here, we provide a step-by-step approach to assessment of speech production for either monolingual or bilingual children (adapted from work by Brian A. Goldstein and Leah Fabiano-Smith). The criterion references that can be used to interpret child data collected in this manner can be found on Table 1.

Step 1: Perform a Detailed Case History

According to a systematic review of the literature on pediatric screening for speech and language disorders in children 5 years old or younger ( Wallace et al., 2015), male sex, family history of speech and language disorders, and low SES were the most significant predictors of disorder. Importantly, African American children and children whose primary home language is not English are less likely to receive speech-language therapy services than their White peers, even when minority group status and SES are correlated ( Morgan et al., 2016). What these findings tell us is that we need to (a) ask if there is a family history of speech and language impairment in the child's biological family and (b) that we need to use culturally sensitive assessment practices to be sure we are identifying all children who present with speech-language impairment. In addition to standard background history that we gather from caretakers, we also want to ask what languages are spoken in the home, by whom, and how often (language input). We also want to ask what language(s) the child uses in the home and at school, how often, and with whom (language output). This information gives the SLP an idea of what level of language proficiency, in each language, a child exhibits. If a child uses English only 10% of the time, we cannot expect him to perform at a typical level on a standardized test in English, even if he has typical phonological skills. Ask parents how many hours a day the child is awake, ask what a typical day is like for that child (home and school, if applicable), and ask about weekends separately (weekends can often present a very different linguistic environment for children). Ask these questions each time you assess a child, as linguistic environments can change over time, impacting language proficiency levels.

Step 2: Routine Assessments

As with any evaluation, we want to include an oral peripheral examination and a hearing evaluation. Oral peripheral examinations test cranial nerve function and screen for motor speech impairments. Any child who is referred for a speech-language evaluation should receive a full audiologic evaluation, not simply a hearing screening.

Step 3: Obtain Speech Samples

A single-word sample should be collected in all languages of the child. We can use a standardized test in a nonstandardized way, but we need to acknowledge that it might not provide the number or type of opportunities needed to truly represent a child's skill set. We do not score the test-we simply use the word list. We supplement this list with a connected speech sample, in all languages, to (a) judge intelligibility subjectively or calculate a quantitative intelligibility measure and (b) supplement the word list for the development of a phonetic inventory. Phonetic transcription is required of the single-word speech sample only.

Step 4: Phonetic Inventory (Independent Analysis)

Determine the phonetic inventory of the child, in all languages, using the child's speech samples. At least two productions of each sound, whether produced in the correct context or not, is needed for a phone to be included in the inventory ( Dinnsen et al., 1990). Organize the inventory by place and manner of articulation for ease of evaluation. Be sure to supplement a single-word list with a connected speech sample to be sure the child has enough opportunities to produce each sound at least two times.

Step 5: Consonant Accuracy (Relational Analysis)

Again, here, we can use the single-word test in a nonstandardized way: Count the number of opportunities for consonant production in the sample (you will only need to do this once if you use the same test repeatedly). This can also be done for vowels in younger children and children who exhibit vowel errors. Then, count the total number of consonants the child produced correctly. Derive a percent accuracy with total number of opportunities for consonant production as a denominator and total consonants produced correctly as the numerator. Calculating overall consonant accuracy from a single-word sample is not the same approach used in the development of PCC ( Shriberg, Austin, Lewis, McSweeny, & Wilson, 1997), but for clinical ease and efficiency, this method has been used in research-based studies (e.g., Fabiano-Smith & Goldstein, 2010a, 2010b). Using the connected speech sample to supplement this analysis is also a useful approach.

Step 6: Error Analysis (Substitutions, Omissions, Distortions)

Using your phonetic transcriptions, examine the targets that a child is avoiding and the sounds she is selecting to use as substitutes for those targets. For children who speak more than one language, examine your samples for instances of cross-linguistic effects or sounds specific to one language being used as substitutes in the other language (e.g., the Spanish trill /r/ being used as a substitute in English). These productions should not be counted as errors.

Step 7: Phonological Error Pattern Analysis

The type and frequency of phonological error patterns should be calculated from your single-word sample, separately for each language. Shriberg and Kwiatkowski (1994) found that the average frequency of occurrence of phonological error patterns in monolingual English-speaking children with SSDs differs significantly from children with typically developing skills. Diagnostic criteria for phonological error patterns in monolingual English speakers are well established ( McReynolds & Elbert, 1981). Type and frequency parameters for bilingual Spanish- to English-speaking children are still in development ( Fabiano-Smith, Privette, & An, in preparation), but preliminary parameters derived from smaller data sets are readily available (e.g., Fabiano-Smith & Goldstein, 2010b; Fabiano-Smith & Hoffman, 2018; Goldstein, Fabiano, & Washington, 2005; Goldstein & Washington, 2001).

Step 8: Measures of Whole-Word Proximity (Phonological Mean Length of Utterance and Proportion of Whole-Word Proximity)

Measures of whole-word proximity are more nuanced analyses of word-level errors ( Ingram, 2002). By examining the relationship between a child's production and the adult target form, a word that would otherwise be scored as incorrect based on a single error receives credit to for how closely its error(s) approximate the adult target form. Phonological mean length of utterance (Pmlu) indicates how complex a child's production is in comparison to an adult target, and proportion of whole-word proximity indicates how closely a child's productions approximate the adult target. These measures can be used to identify a child's stage of acquisition, to assess proximity to target words, and to evaluate the complexity of words beyond the level of the segment ( Ingram, 2002). To calculate Pmlu, 1 point is given for each correct vowel and consonant and an extra point is given for every correct consonant. Phonological whole-word proximity is calculated by dividing the Pmlu of the child by the Pmlu of the target word. Babatsouli, Ingram, and Sotiropoulous (2014) found that the combination of these two measures might help to discriminate typical from disordered speech skills in the English and the Spanish of bilingual children. More specifically, examining only one of the two measures did not discriminate between typical children and children with disorders, but when these two measures were placed in a regression model for comparison, evidence of disorder was observed (see also Bunta, Fabiano-Smith, Goldstein, & Ingram, 2009). These analyses must be performed by hand, but the time invested might provide more in-depth insight into a child's phonological abilities.

Step 9: Stimulability

Stimulability testing can serve as a diagnostic indicator at the phonemic, manner class, and syllabic levels. Using the Scaffolding Scale of Stimulability (SSS; Glaspey & Stoel-Gammon, 2005), a dynamic assessment can be used to indicate whether or not a child presents with an SSD. Instead of the traditional "simulable/not stimulable" dichotomy, the SSS provides hierarchical scaffolding support for children as they demonstrate their knowledge, or partial knowledge, of the target sound or structure. This 21-point scale assesses phonemes, clusters, syllables, and word shapes across seven different phonetic environments, providing scaffolding options at each step. Children who accumulate more points on the SSS are more likely to need support in production (i.e., more likely to present with a disorder) versus those children who earn less points, indicating less support required for accurate production (i.e., less likely to present with a disorder).

Step 10: Intelligibility

Depending on the severity of a child's SSD, deriving a subjective estimate of intelligibility might not be sufficient for characterizing the nature of their disorder. There are multiple approaches to quantifying intelligibility of speech, and depending on the needs of a particular child during assessment, implementation of more than one approach might be useful. For an overview of multiple approaches, see Kent, Miolo, and Bloedel (1994).

Incorporating criterion-referenced measures into an evaluation protocol will provide the SLP with depth, as well as breath, of the phonological skills a child possesses. Whereas a standardized test provides one, static score (that might not be representative), criterion-referenced approaches provide detailed information, in some cases over time (e.g., the SSS), that allow the SLP to observe a child's true skill set and inform evidence-based treatment.

Case Studies

We can apply this assessment protocol to the three case studies woven throughout this issue. To start, both monolingual and bilingual children by the age of 3;0 (years;months), both with typical development and with SSDs, have nearly complete phonetic inventories ( Dinnsen et al., 1990; Fabiano-Smith & Barlow, 2010). If, at this age or older, a child presents with a sparse phonetic inventory, then it would be a cause for clinical concern. In the three case studies described, all children have nearly complete phonetic inventories. The first child is lacking late-acquired sound phonemically but not necessarily phonetically. He might be able to produce the late sounds as substitutes for other sounds but cannot do so accurately in the target context. The child in Case Study 2 may or may not have the approximant /ɹ/ in his inventory, but regardless, it is the only sound lacking. This places him at nearly the most complex level (see Fabiano-Smith & Barlow, 2010, for typological levels of complexity in both English and Spanish). The third child exhibits the error of lateralizing fricatives, so he might be lacking a stridency distinction between fricatives, but the rest of his skill set places him at the next to highest level of phonetic inventory complexity.

The measure on which we might see more of a distinction among children is on the relational measure, PCC. Low accuracy on late-acquired sounds (as a category) can pull down overall consonant accuracy (e.g., Fabiano-Smith & Goldstein, 2010a, 2010b); therefore, PCC will likely reveal the impairment for the child in Case Study 1 and not for the other two children who have a few sound-specific errors.

A substitution error analysis will yield qualitative information for the first child, as we can observe the substitutes he is using in place of the late-acquired targets he is avoiding. For Child 2, this analysis will not yield much more information, as only one sound is in error. For Child 3, however, a substitution error analysis will aid in determining if a specific set of fricatives are subject to lateralization or if all fricatives are at risk.

When we move to phonological error pattern analysis, impairment should be revealed for all three children, as the first child will likely exhibit a high level of gliding and stopping of fricatives, the child in Case Study 2 will also exhibit a high level of gliding, and the child in Case Study 3 will exhibit a high level of lateralization errors on fricatives. Measures of whole-word proximity will likely be more useful for Child 1 than for Children 2 and 3, as there are more sounds in error; therefore, obtaining information on how close to the target sound a child is may tell us if he or she is moving toward more accurate production (i.e., errors sharing some features with their targets). For Children 2 and 3, the targets that trigger their errors are already clear.

Stimulability would be a helpful measure for all three children in terms of making predictions about time frame for future mastery of the sound(s) in question. Finally, intelligibility ratings will likely be unnecessary for Child 2 because only one predictable sound is in error. For Children 1 and 3, however, a measure of intelligibility will aid in addressing how their particular errors are affecting their overall communication abilities.

Conclusions

Including a number of different measures, at different levels of the sound system, is necessary for accurate diagnosis of SSD. Psychometrically sound standardized tests can act as a piece of an assessment battery but should never act as the sole indicator of typical versus disordered development. Furthermore, a standardized test that is not psychometrically sound should never be scored and submitted as a diagnostic indicator. Criterion-referenced procedures and the utilization of standardized tests in nonstandardized ways will provide information that is reflective of a child's speech skills, supporting the SLP in accurate diagnosis of SSD.

Acknowledgments

This work was funded by National Institute on Child Health and Human Development Grant 1-R21-HD081382-01A1 (2015-2018) and National Institute on Deafness and Other Communication Disorders Grant 1-R01-DC016624-01A1 (2018-2023) received by the author. Open access fees were paid for by an award to Holly Storkel from the Friends of the Life Span Institute at the University of Kansas.

References

American Educational Research Association. (2014). Standards for educational and psychological testing (2014 ed.). Washington, DC: Author.

Babatsouli, E. , Ingram, D. , & Sotiropoulos, D. (2014). Phonological word proximity in child speech development. Chaotic Modeling and Simulation, 4(3), 295-313.

Fabiano-Smith, L. , & Crouse-Matlock, S. (2012). Phonetic inventory elicitation in bilingual Spanish-English speaking children: What toys are best? Poster presented at the annual convention of the American Speech-Language-Hearing Association, Atlanta, GA .

Fabiano-Smith, L. , Privette, C. , & An, L. (in preparation). Phonological abilities in bilingual Spanish-English speaking children: The language combination effect. Manuscript in preparation.

Goldman, R. , & Fristoe, M. (2000). Goldman-Fristoe Test of Articulation-Second Edition (GFTA-2). San Antonio, TX:Pearson Clinical.

Khan, L. , & Lewis, N. (2002). Khan-Lewis Phonological Analysis-Second Edition. San Antonio, TX: Pearson Clinical.

Pavleko, S. L. (2018). Evidence-based practices for selecting and using standardized tests: Considerations for speech testing. Retrieved from https://www.speechpathology.com/slp-ceus/course/evidence-based-practices-in-selecting-8104

Author Notes

Financial: Leah Fabiano-Smith has no relevant financial interests to disclose. Nonfinancial: Leah Fabiano-Smith has no relevant nonfinancial interests to disclose.

# data[].attributes.attachment_link.title
Standardized Tests and the Diagnosis of Speech Sound Disorders

# data[].attributes.attachment_link.description
Purpose The purpose of this tutorial is to provide speech-language pathologists with the knowledge and tools to (a) evaluate standardized tests of articulation and phonology and (b) utilize criterion-referenced approaches to assessment in the absence of psychometrically strong standardized tests.

# data[].attributes.headline
ITC Guidelines for Test Adaptation and Translation 

# data[].attributes.subject
ITC Guidelines for Test Adaptation and Translation 

# data[].attributes.body
https://www.intestcom.org/files/guideline_test_adaptation_2ed.pdf

https://www.researchgate.net/publication/326547947_ITC_Guidelines_for_Translating_and_Adapting_Tests_Second_Edition

# data[].attributes.body
https://jmr.tums.ac.ir/index.php/jmr/article/view/78

# data[].attributes.attachment_link.readable_text
Cynthia C. Fundamentals of hand therapy: Clinical reasoning and treatment guidelines for common diagnosis of the upper extremity. 2nd edition. Amsterdam: Elsevier; 2007

Azad A, Lajevardi L. [Hand and upper extremity rehabilitation: A practical guide (Persian)]. Ardabil: Soleymanzadeh Publisher; 2012.

Thrombly C, Radomski M. Occupational therapy for physical dysfunction. Philadelphia: Lippincott Williams & Wilkins; 2008.

Schubert HE. Scaphoid fracture. Review of diagnostic tests and treatment. Canadian Family Physician. 2000; 46(9):1825-32. PMCID: PMC2145017

MacDermid JC, Roth J, Richards R. Pain and disability reported in the year following a distal radius fracture: A cohort study. BMC Musculoskeletal Disorders. 2003; 4(1):24. doi: 10.1186/1471-2474-4-24

Salter M, Davis AT. Hand injuries: A therapeutic approach. London: Churchill Livingston; 1987.

MacDermid JC, Turgeon T, Richards RS, Beadle M, Roth JH. Patient-rating wrist pain and disability: A reliable and valid measurement tool. Journal of Orthopaedic Trauma. 1998; 12(8):577-86. doi: 10.1097/00005131-199811000-00009

Navarro CM, Ponzer S, Tornkvist H, Ahrengart L, Bergstrom G. Measuring outcome after wrist injury: Translation and validation of the Swedish version of the patient-Rated Wrist Evaluation (PRWE-Swe). BMC Musculoskeletal Disorders. 2011; 12(1):171. doi: 10.1186/1471-2474-12-171

Calfee RP, Adams AA. Clinicalresearch and patient-rated outcome measures in hand surgery. The Journal of Hand Surgery. 2012; 37(4):851-55. doi: 10.1016/j.jhsa.2012.01.043

Shauver MJ, Chang WCh, Chung KC. Contribution of functional parameters to patient-rated outcomes after surgical treatment of distal radius fractures. The Journal of Hand Surgery. 2014; 39(3):436-442. doi: 10.1016/j.jhsa.2013.11.015

Changulani M, Okonkwo U, Keswani T, Kalairajah Y. Outcome evaluation measures for wrist and hand-which one to choose? International Orthopaedics. 2007; 32(1):1-6. doi: 10.1007/s00264-007-0368-z

MacDermid JC. Development of a scale for client rating of wrist pain and disability. Journal of Hand Therapy. 1996; 9(2):178-83. doi: 10.1016/s0894-1130(96)80076-7

MacDermid JC, Richards RS, Roth JH. Distal radius fracture: A prospective outcome study of 275 patients. Journal of Hand Therapy. 2001; 14(2):154-69. doi: 10.1016/s0894-1130(01)80046-6

Schønnemann JO, Hansen TB, Søballe K. Translation and validation of the Danish version of the Patient Rated Wrist Evaluation questionnaire. Journal of Plastic Surgery and Hand Surgery. 2013; 47(6):489-92. doi: 10.3109/2000656X.2013.787934

Rodrigues E, Fanseca M, MacDermid JC. Brazilian version of the Patient-Rated Wrist Evaluation (PRWE-BR): Cross-cultural adaptation, internal consistency, test-retest reliability and construct validity. Journal of Hand Therapy. 2015; 28(1):69-76. doi: 10.1016/j.jht.2014.09.008

Kim JK, Kong JS. Evaluation of the Korean version of the patient-Rated Wrist Evaluation. Journal of Hand Therapy. 2013; 26(3):238-44. doi: 10.1016/j.jht.2013.01.003

Wong-Man-Wah J, Kwan-Wing-Wang M, Li-Tsang-Wai-Ping C. Construct validity of the Chinese version of the Patient-Rated Wrist Evaluation questionnaire (PRWE-Hong Kong version).Journal of Hand Therapy. 2006; 19(1):18-26. doi: 10.1197/j.jht.2005.10.003

Omokawa SH, Imaeda T, Sawaizumi T, Momose T, Gotani H, Abe Y, et al. Responsiveness of the Japanese version of the patient-Rated Wrist Evaluation (PRWE-J) and physical impairment measurements in evaluating recovery after treatment of ulnocarpal abutment syndrome. Journal of Orthopaedic Science. 2012; 17(5):551-5. doi: 10.1007/s00776-012-0265-1

Ozturk O, Sari Z, Ozgul B, Tasyikan L. Validity and reliability of the Turkish "Patient-Rated Wrist Evaluation" questionnaire. Acta Orthopaedica et Traumatologica Turcica. 2015; 49(2):120-5. doi: 10.3944/AOTT.2015.14.0208.

Mehta SP, Mhatre B, MacDermid JC, Mehta A. Cross-cultural adaptation and psychometric testing of the Hindi version of the Patient-rated Wrist Evaluation. Journal of Hand Therapy. 2012; 25(1):65-78. doi: 10.1016/j.jht.2011.08.001

Hemelaers L, Angst F, Simmen BR, Wood-Dauphinee Sh. Reliability and validity of the German version of the Patient-Rated Wrist Evaluation as an outcome measure of wrist pain and disability in patients with acute distal radius fractures. Journal of Hand Therapy. 2008; 21(4):366-76. doi: 10.1197/j.jht.2008.03.002

Chryssochou E, Hilfiker R, Deriaz O, Luthi F, Konzelmann M. Construct validity of the French version of the PRWE (Patient-Rated Wrist Evaluation) with the French version of the DASH (Disabilities Arm Shoulder and Hand) is good to very good in a population of patients with wrist injuries in an inpatient rehabilitation unit. Annals of Physical and Rehabilitation Medicine. 2011; 54:e242. doi: 10.1016/j.rehab.2011.07.343

Sandelin H, Jalanko T, Huhtala H, Lassila H, Haapala J, Helkamaa T. Translation and validation of the Finnish version of the Patient-Rated Wrist Evaluation questionnaire (PRWE) in patients with acute distal radius fracture. Scandinavian Journal of Surgery. 2016; 105(3):204-10. doi: 10.1177/1457496915613649

Jacobsen PB, Davis K, Cella D. Assessing quality of life in research and clinical practice. Journal of Oncology. 2002; 16 (10):133-9. PMID: 12380963

Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine. 2000; 25(24):3186-91. doi: 10.1097/00007632-200012150-00014

Yalghmale F. Content validity and its estimation. Journal of Medical Education. 2003; 3(1):25-7.

Lawshe CH. A quantitative approach to content validity. Personnel Psychology. 1975; 28(4): 563-75. doi: 10.1111/j.1744-6570.1975.tb01393.x

Waltz CF, Bausell RB. Nursing research: Design, statistics, and computer analysis. Philadelphia: W.B. Saunders Company; 1981

Polit DF, Beck CT. The content validity index: Are you sure you know what's being reported? Critique and recommendations. Research in Nursing & Health. 2006; 29(5):489-97. doi: 10.1002/nur.20147

Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health. 2007; 30(4):459-67. doi: /10.1002/nur.20199

Hosseini Z, Ghorbani Z, Ebn-Ahmady A. [Face and content validity and reliability assessment of change cycle questionnaire in smokers (Persian)]. Journal of Mashhad Dental School. 2015; 39(2):147-54.

Foroughan M, Jafari Z, Shirinbayan P, Ghaem-Magham Farahani Z, Rahgozar M. [Validation of Mini-Mental State Examination (MMSE) in the elderly population of Tehran city. (Persian)]. Advances in Cognitive Science. 2009; 10(2):29-37.

# data[].attributes.attachment_link.title
Translation, Cultural Adaptation, Face and Content Validity of the Persian Version "Patient-Rated Wrist Evaluation" (PRWE-Persian) Questionnaire

# data[].attributes.attachment_link.description
Cynthia C. Fundamentals of hand therapy: Clinical reasoning and treatment guidelines for common diagnosis of the upper extremity. 2nd edition. Amsterdam: Elsevier; 2007 Azad A, Lajevardi L. [Hand and upper extremity rehabilitation: A practical guide (Persian)]. Ardabil: Soleymanzadeh Publisher; 2012. Thrombly C, Radomski M. Occupational therapy for physical dysfunction.

# data[].attributes.headline
Note: Look into: ITC guideliens for test adaptation 

# data[].attributes.subject
Note: Look into: ITC guideliens for test adaptation 

# data[].attributes.body
https://etd.ohiolink.edu/apexprod/rws_etd/send_file/send?accession=ohiou1082054025&disposition=inline

# data[].attributes.body
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5740392/

# data[].attributes.attachment_link.readable_text
Rima A Abdul-Khalek, ¹ Andrea J Darzi, ¹ Mohammad W Godah, 1, ² Lama Kilzar, ² Chantal Lakis, ³ Arnav Agarwal, 4, ⁵ Elias Abou-Jaoude, ⁶ Joerg J Meerpohl, ⁷ Wojtek Wiercioch, ⁸ Nancy Santesso, ⁸ Hneine Brax, ⁹ Holger Schünemann, ⁴ and Elie A Akl 1, 2, 4, ¹⁰

Abstract

Background

Adaptation refers to the systematic approach for considering the endorsement or modification of recommendations produced in one setting for application in another as an alternative to de novo development.

Objective

To describe and assess the methods used for adapting health-related guidelines published in peer-reviewed journals, and to assess the quality of the resulting adapted guidelines.

Methods

We searched Medline and Embase up to June 2015. We assessed the method of adaptation, and the quality of included guidelines.

Results

Seventy-two papers were eligible. Most adapted guidelines and their source guidelines were published by professional societies (71% and 68% respectively), and in high-income countries (83% and 85% respectively). Of the 57 adapted guidelines that reported any detail about adaptation method, 34 (60%) did not use a published adaptation method. The number (and percentage) of adapted guidelines fulfilling each of the ADAPTE steps ranged between 2 (4%) and 57 (100%). The quality of adapted guidelines was highest for the "scope and purpose" domain and lowest for the "editorial independence" domain (respective mean percentages of the maximum possible scores were 93% and 43%). The mean score for "rigor of development" was 57%.

Conclusion

Most adapted guidelines published in peer-reviewed journals do not report using a published adaptation method, and their adaptation quality was variable.

Guideline adaptation provides an alternative to de novo guideline development bz making the process more efficient and avoiding duplication of efforts. Given that guidelines produced in one setting may not be applicable to other settings, adaptation takes into account the cultural and organizational differences in the new setting to guarantee their applicability [ 1]. This requires a well-structured adaptation methodology that takes into consideration affordability and availability of resources and services that would allow contextualizing global guidelines to countries of varying levels of income [ 2].

A number of adaptation methodologies have been proposed, including the Systematic Guideline Review Method [ 3], Making GRADE the Irresistible Choice (MAGIC) [ 4], GRADE-ADOLOPMENT [ 5], and ADAPTE [ 1]. GRADE is short for Grading of Recommendations Assessment, Development and Evaluation [ 6]. ADAPTE is one of the earliest systematic frameworks to adapt guidelines to a local context [ 1]. The ADAPTE framework consists of 24 steps in three main phases: (1) set-up (preparation), (2) adaptation and (3) finalization [ 7]. The evaluation of guideline adaptation has been recommended to ensure the reproducibility and efficiency of the methods in producing high quality guidelines [ 8].

In a recently conducted study, we systematically evaluated the reporting of processes employed in the national adaptation of World Health Organization (WHO) guidelines for Human Immunodeficiency Virus (HIV) and Tuberculosis [ 9]. We found that 32 out of 170 (19%) guideline adaptations reported their processes. It remains unclear to what extent these findings apply to fields other than HIV and tuberculosis, or to guideline adaptions published in the peer reviewed literature. This is particularly relevant as the adaptation methods used could affect the quality, trustworthiness and applicability of the adapted recommendations.

Our main objective was to describe and assess the methods used for adapting health-related guidelines published in peer-reviewed journals, and to assess the quality of the resulting adapted guidelines.

METHODS

Definitions

We adopted the WHO definition of guidelines as "systematically developed evidence-based statements which assist providers, recipients and other stakeholders to make informed decisions about appropriate health interventions" [ 10]. We considered the following definition of guideline adaptation that is based on the ones proposed by Fervers et al. [ 1] and by the GRADE-ADOLOPMENT methodology [ 5]: systematic approach for considering the endorsement or modification of recommendations produced in one setting for application in another as an alternative to de novo development.

Eligibility

We included documents meeting all the following eligibility criteria:

Meet the above definition of guideline adaptation;

Adaptations of specific health-related guidelines (e.g., management of asthma, screening mammography); and

Published in any language;

We also acquired papers that the authors referred to when describing their adaptation methods, and considered them in the review process.

We excluded documents:

Describing an adaptation methodology process but not an actual adaptation;

Meeting abstracts;

Restricted to implementation only (e.g., for clinical decision support);

Reporting appraisals of clinical practice guidelines (CPGs) for the purpose of adaptation; or

Reporting only algorithms.

Search strategy

We used the OVID interface to search Medline and Embase databases. The period of the search was from January 2000 to June 2015. Online Supplementary Document(Online Supplementary Document) provides the detailed search strategies for each database. Also, we searched the reference lists of relevant papers (eg, reviews on adaptation methods) identified in our search. We did not use any language restrictions.

Selection process

All reviewers underwent calibration exercises. They screened the same set of papers and received feedback on their performance. Then, teams of two reviewers independently screened abstracts and full texts of identified citations for potentially eligible guideline adaptations. Next, the review teams screened the full text of citations judged as potentially eligible by at least one of two reviewers using standardized screening forms. The two members compared their results and resolved disagreement by discussion or with the help of a third reviewer as needed. When excluding an adapted guideline, we recorded the reason for exclusion.

Data abstraction process

All reviewers underwent calibration exercises for data abstraction. Teams of two reviewers worked in duplicate and independently to abstract relevant information from the included adapted guidelines (except 122 non-English papers which were abstracted by only one person: 63 were in either French or Spanish in which one of the reviewers (RAA) was fluent, and 59 were translated using Google Translate). They used standardized online data abstraction forms on REDCap TM [ 11]. They compared results and resolved disagreements by discussion, or with the help of a third reviewer.

We abstracted the following characteristics from each included adapted guideline:

Characteristics of the adapted guideline: name and year of publication, country, contributors to guideline adaptation (governmental body, WHO Headquarter, Regional or National offices, Not for Profit Organizations (NGOs), professional society, "expert panel"), guideline area, reporting of source guideline.

Characteristics of the source guideline: number of source guidelines, name and year of publication, country, contributors to source guideline development (governmental body, WHO Headquarter, Regional or National offices, NGOs, and professional society, "expert panel").

Other information: disclosure of conflicts of interest, and funding (reporting, source, and role).

We classified income levels of countries (for both adapted and source guidelines) as per the World Bank classification into high, upper-middle, lower-middle and low income countries.

We also assessed whether the authors explicitly reported using an adaptation method. We considered the following options: ADAPTE or one of its variants, other published adaptation method, or an unpublished adaptation method.

Next, we wanted to explore the specific steps followed in the adaptation process. As we did not identify any validated or standardized tool, we decided to rely on the steps described in ADAPTE (Table 1 ) [ 7]. We chose ADAPTE because it is a well-structured tool, and represents the most widely used method for guideline adaptation. Thus, we abstracted information about the steps covered in the adaptation methodology from papers reporting on at least one element of the adaptation phase of the ADAPTE process [ 7]. So, the purpose was not to assess compliance with ADAPTE. Rather, we used ADAPTE to identify standard steps in the adaptation approach.

Table 1

Phases

Steps

Set-up phase
1. Establish an organizing committee

6. Write protocol

Adaptation phase
7. Determine the health questions

18. Prepare a document that respects the needs of the end users and provides a detailed transparent explanation of the process

Finalization phase
19. External review by target users

24. Produce high quality final guideline

To appraise the quality of adapted guidelines, we used the Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument [ 12]. This tool was designed to assess the quality of guidelines by evaluating the rigor of guidelines development and transparency in reporting its processes. It is a 23-item tool, in which the items are distributed into six quality domains: scope and purpose, stakeholder involvement, rigor of development, clarity of presentation, applicability, and editorial independence ( Online Supplementary Document(Online Supplementary Document) provides a listing of the tool's 23 items and a description of its domains). The items in each domain are rated on a 7-point scale, where 1 indicates 'Strongly Disagree' and 7 indicates 'Strongly Agree'. We calculated the AGREE II "scaled domain score" for each domain, as suggested by the AGREE II group. For this, we added the scores of individual items of a particular domain, then we scaled the total score as a percentage of the maximum possible score in that domain [ 13]. The resulting scores of the six domains are independent and are not compared to a minimum score as recommended by the AGREE Consortium [ 12].

Data analysis

We conducted a descriptive analysis of all variables. We used frequencies and percentages for categorical variables. For continuous variables, we assessed the distribution for normality using the Kolmogorov-Smirnov test. For non-normally distributed variables, we used median and Inter-Quartile Range (IQR); otherwise we planned to use mean and standard deviation.

We used the Mann-Whitney test to assess the association between reporting (vs not reporting), the use of a published adaptation method and the quality of the adapted guideline, measured as the mean score for each AGREE II domain.

RESULTS

Study selection

Figure 1 shows the study flow. Out of a total of 12 021 captured citations, we identified a total of 72 eligible papers, each reporting on one guideline adaptation project. One of those (the Guideline of the German Society for Nutritional Medicine (DGEM)) included 13 different chapters in clinical nutrition but we considered them as one guideline. We excluded 139 papers based on full text screening for the following reasons: 56 were not related to adaptation, 33 were adaptations for implementation only, 13 reported adaptation of a non-health-related guideline, 11 were appraisals for guideline quality, and 10 were review articles, 4 reported algorithms only, 5 were meeting abstracts, 6 were guideline adaptations without reporting of the resulting recommendations, and 1 was a guideline commentary.

Findings

Characteristics of the adapted guidelines

Table 2 provides a description of the 72 included adapted guidelines. Forty three percent (n = 31) of these adapted guidelines were published between the years 2012-2014. High-income countries produced 83% of the adaptations (n = 60), whereas lower-middle income countries produced only 6% (n = 4). Professional societies developed 71% (n = 51) of the adaptations, while governments were involved in only 11% (n = 8) of adapted guidelines.

Table 2

No.

%

Private not for profit

4

6

The guideline development group reported conflicts of interests in 38 (53%) of the adapted guidelines. Forty seven percent of adaptations reported on the funding source. The most common funding sources were governmental in 15 (21%) adaptations, followed by private-for-profit organizations in 12 (17%) adaptations. Of the thirty-one adapted guidelines that reported being funded, only six reported on the role of the funder in the adaptation process. While three explicitly reported having no role, the other three reported being involved in: preparing the manuscript, employing an author, and covering costs of meetings.

Characteristics of the source guideline

Ninety percent of adapted guidelines reported on the source guideline (Table 3 ). The median number of source guidelines was 2.5 with an IQR of 4 (75 th - 25 th percentiles being 5 and 1). The IQR for the publication year of the source guidelines was 6 years (75 th - 25 th percentiles being 2010 and 2004). Eighty five percent of source guideline originated from high income countries. Most common source guideline developers were professional societies (68%), and WHO (13%).

Adaptation method (n = 57)

Out of 72 included papers, 57 reported at least one detail about the adaptation method. Of the 57 adapted guidelines that reported at least one detail of the adaptation method, sixty percent (n = 34) did not report using a published method for guideline adaptation. Forty percent (n = 23) reported using an adaptation method: either ADAPTE (n = 13); ADAPTE as modified by the authors (n = 3); or some other previously published method (n = 7), including: the Practice Guideline Evaluation and Adaptation Cycle (n = 4), the Systematic Guideline Review Method (SGR) (n = 1), RAND consensus method (n = 1), and Registered Nurses' Association of Ontario (RNAO) Toolkit for Implementation of Clinical Practice Guidelines (n = 1). As planned, we collected for these 57 papers information about the adaption method (ADAPTE steps) and the quality of the adapted guideline (AGREE II score).

ADAPTE steps

The total number of steps of ADAPTE reported to be followed by each of the 57 adapted guidelines ranged between 6 and 22 (out of a maximum value of 23). The median number was 14 and IQR was 4 (75 th - 25 th percentiles being 17 and 13).

Figure 2 show the number of adapted guidelines fulfilling each of the ADAPTE steps. The distribution (out of a maximum value of 57) ranged between 2 (4%) and 57 (100%). The median number was 38 and IQR was 27 (75 th - 25 th percentiles being 51 and 24). The percentages of guidelines fulfilling ADAPTE steps are presented in Online Supplementary Document(Online Supplementary Document).

At least 95% of the adapted guidelines met the first four steps of ADAPTE (ie, setup or preparatory phase). In the adaptation phase (steps 7 to 18), all adapted guidelines determined health questions, while 88% reported searching for guidelines or any other relevant documents. Only 16% of adapted guidelines assessed guideline consistency ie, assessment of the search and study selection, and links between evidence and recommendation. In the finalization phase: 91% acknowledged the source, but only 16% reported consulting with sources guideline developers.

Quality of the adapted guideline (AGREE II score):

Table 4 reports the AGREE II mean scaled domain scores for the 57 adapted guidelines. The mean scores were highest for the "scope and purpose" domain (93%) and "clarity of presentation" domain (86%). The mean scores were lowest for the two "applicability" and "editorial independence" domains (50% and 43% respectively). The mean score for "rigor of development" was 57%.

Table 4

Domain

Mean %

Standard deviation

Minimum%

Maximum%

95% confidence Interval

Domain 6: Editorial Independence

42.54

34.88

0

100.00

33.28 to 51.80

With regards to evaluating the association between reporting the use of a published adaptation method and the quality of the adapted guideline, we found a statistically significant association for the applicability domain but not for the scope and purpose, stakeholder involvement, rigor of development, clarity of presentation and editorial independence domains. For the applicability domain, the scores were 64% when reporting the use of a published adaptation method and 41% when not reporting such use ( P  = 0.005).

DISCUSSION

Summary of findings

Our aim was to assess the methods used for adaptation and to assess the quality of health-related guideline adaptations. We identified 72 adapted guidelines published in the past 15 years through an electronic database search. Of the identified guidelines, the majority of both adapted guidelines and their source guidelines were published by professional societies, and in high-income countries. About a fifth of the adapted guidelines did not report any detail about their adaptation methodology. Of those that did, most did not use a published adaptation method. The ADAPTE framework was the most frequently used method but was used only in a quarter of the adapted guidelines. It is important to note that ADAPTE was developed and published after the publication of many of guidelines included in this study. The use of the different steps of guideline adaptation, as well as the quality of the guidelines was variable. A key step before adapting a guideline/recommendation should be an evaluation of how well the source guideline assessed, interpreted and made recommendations, but this may not be occurring. The use of a published adaptation method was associated with a higher score on applicability. This association suggests that the use of adaptation methodologies strengthens the relevance of the adapted guideline.

It is interesting that guideline adaptations are mostly published in high-income countries. It is very likely that adaptations of guidelines in low-income countries are not being reported in peer-reviewed journals, or are being reported in peer-reviewed journals that are not indexed. Capturing those guidelines would require searching governments' websites as well as national or regional journals.

Strengths and limitations

The main strength of this study is the use of standard systematic review methodology such as duplicate methods for guideline selection and data abstraction. Also, we used AGREE II instrument, a validated tool for assessment of guideline quality. While AGREE II was not specifically designed for adapted guidelines, we believe it still applies. While we used the ADAPTE steps to assess the process of guideline adaptation, the tool was actually developed for guiding the adaptation process and not for assessing it. Also, this tool was developed and published after the publication of many of adapted guidelines included in this study. Unfortunately, no tool for assessing the adaptation process is currently available. The main limitations relate to restricting the search to electronic databases, and not including adapted guidelines published only in governmental databases or websites, or published locally as reports.

Comparison to similar studies

We are not aware of any study that systematically evaluated the quality of adapted guidelines. Miguel-Garcia et al. assessed qualitatively the quality of the Spanish adaptation of the European Guidelines on Cardiovascular Disease Prevention in Clinical Practice [ 14]. The authors highlighted the importance of considering clinical evidence both in developing the source guidelines and in adapting them. Alonso-Coello et al. systematically reviewed studies that used AGREE instrument to appraise guidelines in general (ie, not necessarily adapted guidelines). They found that, despite the fact that quality of guidelines improved over the last two decades, it remained moderate to low when measured with the AGREE instrument [ 15].

We are aware of two methodological surveys that systematically evaluated guideline adaptation process. Fervers et al. published a literature review in 2006 that identified 18 reports of models, practical examples and experiences of guideline adaptation [ 1]. They reported that none of these used a validated process for guidelines adaptation. Indeed, that paper was the basis for the ADAPTE framework [ 7]. Our team conducted the second survey that focused on adaptation of WHO guidelines for HIV and Tuberculosis, and also assessed the number of ADAPTE steps met by the adaptation processes on the national level [ 8]. The median number of ADAPTE steps in the study of WHO adapted guidelines was 11.5 [IQR = 3.5 (75 th - 25 th percentiles being 13.5 and 10)] compared to 14 [IQR = 4 (75 th - 25 th percentiles being 17 and 13)] in the current study. This lower number amongst WHO adapted guidelines was mainly related to lower values for the 'adaptation phase'. The difference could be related to the fact that guidelines assessed in the current study underwent peer review, which might have improved their reporting, or led to the selection of those with better processes.

Implications for practice

Guideline adaptation projects need to improve the reporting of their methods. This work suggests that when adapting guidelines, developers have either not evaluated the consistency of the source guideline (links between evidence and recommendation), or have not reported that they did. Increasing awareness of the different phases of adaptation, providing additional tools to facilitate evaluation, facilitating collaboration between developers, or making the evidence to recommendation process more transparent, may be warranted. Guideline adaptation developers also need to follow methodologies specifically designed for this purpose. For example, a large number of guidelines are currently using the GRADE methodology. Also certain adaption methodologies, eg, the "GRADE-ADOLOPMENT" framework encompasses adoption, adaptation and de novo guideline development. This process is building on newly-developed but also already published systematic reviews and health technology assessment reports. Another methodological advancement is the use of the GRADE EtD frameworks, which could facilitate the adaptation process [ 16].

Implications for future research

There is a need to develop a standardized tool for assessing the quality of conduct and of reporting of guideline adaptations, given that this methodology has features that are distinct from guideline development. Such a tool would be helpful for both researchers in the field of guideline adaptation, and groups working on adapting guidelines. Also, research focusing on non-peer reviewed guidelines is needed to better assess methods used for adaptation efforts in low and middle-income countries (LMIC).

Footnotes

Authorship declaration: Conception and design: EAA, RAA, Data acquisition: RAA, MWG, LK, CL, AA, EAJ, JJM, WW, NS, HB; Data analysis: RAA, EAA; Interpretation of results: RAA, HJS, EAA; Manuscript drafting: RAA, EAA; Critical revision of the manuscript and approval of the final version: RAA, MWG, LK, CL, AA, EAJ, JJM, WW, NS, HB, HJS, EAA.

Funding: None.

Competing interests: The authors completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available upon request from the corresponding author), and report intellectual conflict of interest being members of the GRADE working group (AD, JJM, WW, NS, HJS, and EAA) and authors of studies reporting adaptation approaches (AD, JJM, WW, NS, HJS, and EAA).

Additional Material

Online Supplementary Document:

REFERENCES

3. Muth C, Gensichen J, Beyer M, Hutchinson A, Gerlach FM. The systematic guideline review: method, rationale, and test on chronic heart failure. BMC Health Serv Res. 2009;9:74. doi: 10.1186/1472-6963-9-74. [[ PMC free article] [ PubMed] [ CrossRef] Google Scholar]

5. Schünemann HJ, Wiercioch W, Brozek J, Etxeandia-Ikobaltzeta I, Mustafa RA, Manja V, et al. GRADE Evidence to Decision Frameworks for adoption, adaptation and de novo development of trustworthy recommendations: GRADE-ADOLOPMENT. J Clin Epidemiol. 2017 . Epub ahead of print. [[ PubMed] Google Scholar]

7. The ADAPTE Collaboration. The ADAPTE Process: Resource Toolkit for Guideline Adaptation. 2009. http://www.g-i- n.net/. Version 2.0. Accessed: 19 May 2016.

8. Harstall C, Taenzer P, Zuck N, Angus DK, Moga C, Scott NA. Adapting low back pain guidelines within a multidisciplinary context: a process evaluation. J Eval Clin Pract. 2013;19:773-81. [[ PubMed] Google Scholar]

10. World Health Organization. WHO Handbook for Guideline Development. 2nd ed. Geneva: Word Health Organization; 2014. [ Google Scholar]

11. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap) - A metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42:377-81. doi: 10.1016/j.jbi.2008.08.010. [[ PMC free article] [ PubMed] [ CrossRef] Google Scholar]

12. AGREE Next Steps Consortium. The AGREE II Instrument [Electronic version] 2009 [Available from: http://www.agreetrust.org

13. Brouwers MC, Browman GP, Cluzeau F. feder G, Fervers B, Hanna S, Makarski J on behalf of the AGREE, Consortium NS. AGREE II: Advancing guideline development, reporting and evaluation in healthcare. CMAJ. 2010;182:E839-42. doi: 10.1503/cmaj.090449. [[ PMC free article] [ PubMed] [ CrossRef] Google Scholar]

14. Miguel-Garcia F, Merino-Senovilla A, Montero-Alonso MJ, Garcia-Ortiz A, Sanz-Cantalapiedra R, Maderuelo-Fernandez JA. Cardiovascular prevention according to CEIPC: a critical appraisal. Rev Esp Salud Publica. 2010;84:185-201. [[ PubMed] Google Scholar]

15. Alonso-Coello P, Irfan A, Sola I, Gich I, Delgado-Noguera M, Rigau D, et al. The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies. Qual Saf Health Care. 2010;19:e58. [[ PubMed] Google Scholar]

Articles from Journal of Global Health are provided here courtesy of International Society for Global Health

# data[].attributes.attachment_link.title
Methods used in adaptation of health-related guidelines: A systematic survey

# data[].attributes.attachment_link.description
Rima A Abdul-Khalek, 1 Andrea J Darzi, 1 Mohammad W Godah, 1, 2 Lama Kilzar, 2 Chantal Lakis, 3 Arnav Agarwal, 4, 5 Elias Abou-Jaoude, 6 Joerg J Meerpohl, 7 Wojtek Wiercioch, 8 Nancy Santesso, 8 Hneine Brax, 9 Holger Schünemann, 4 and Elie A Akl 1, 2, 4, 10 Adaptation refers to the systematic approach for considering the endorsement or modification of recommendations produced in one setting for application in another as an alternative to de novo development.

# data[].attributes.body
https://drive.google.com/drive/u/1/folders/1AVgJW7gFf3I8i25XzvhpqrxxtMqFpExk

# data[].attributes.attachment_link.description
Get access to files anywhere through secure cloud storage and file backup for your photos, videos, files and more with Google Drive.