# data[].attributes.headline
18. An article which discusses the limitations of a small scale study

# data[].attributes.subject
18. An article which discusses the limitations of a small scale study

# data[].attributes.body
Aida Idris, Georgeta Ion & Aidan Seery (2019) Peer learning in international higher education: the experience of international students in an Irish university, Irish Educational Studies, 38:1, 1-24, DOI:10.1080/03323315.2018.1489299

# data[].attributes.attachment_link.title
Peer learning in international higher education: the experience of international students in an Irish university

# data[].attributes.attachment_link.description
Some of the main concerns in international higher education are the feeling of isolation among international students and their inability to adapt to the host environment, which may result in sub-optimal academic performance.

# data[].attributes.headline
17.An article which uses purposive or opportunistic sampling

# data[].attributes.subject
17.An article which uses purposive or opportunistic sampling

# data[].attributes.body
Huang, Qi, 2016. Robust spectrum sensing techniques for cognitive radio networks.

# data[].attributes.attachment_link.readable_text
Abstract

Cognitive radio is a promising technology that improves the spectral utilisation by allowing unlicensed secondary users to access underutilised frequency bands in an opportunistic manner. This task can be carried out through spectrum sensing: the secondary user monitors the presence of primary users over the radio spectrum periodically to avoid harmful interference to the licensed service. Traditional energy based sensing methods assume the value of noise power as prior knowledge. They suffer from the noise uncertainty problem as even a mild noise level mismatch will lead to significant performance loss. Hence, developing an efficient robust detection method is important. In this thesis, a novel sensing technique using the F-test is proposed. By assuming a multiple antenna assisted receiver, this detector uses the F-statistic as the test statistic which offers absolute robustness against the noise variance uncertainty. In addition, since the channel state information (CSI) is required to be known, the impact of CSI uncertainty is also discussed. Results show the F-test based sensing method performs better than the energy detector and has a constant false alarm probability, independent of the accuracy of the CSI estimate. Another main topic of this thesis is to address the sensing problem for non-Gaussian noise. Most of the current sensing techniques consider Gaussian noise as implied by the central limit theorem (CLT) and it offers mathematical tractability. However, it sometimes fails to model the noise in practical wireless communication systems, which often shows a non-Gaussian heavy-tailed behaviour. In this thesis, several sensing algorithms are proposed for non-Gaussian noise. Firstly, a non-parametric eigenvalue based detector is developed by exploiting the eigenstructure of the sample covariance matrix. This detector is blind as no information about the noise, signal and channel is required. In addition, the conventional energy detector and the aforementioned F-test based detector are generalised to non-Gaussian noise, which require the noise power and CSI to be known, respectively. A major concern of these detection methods is to control the false alarm probability. Although the test statistics are easy to evaluate, the corresponding null distributions are difficult to obtain as they depend on the noise type which may be unknown and non-Gaussian. In this thesis, we apply the powerful bootstrap technique to overcome this difficulty. The key idea is to reuse the data through resampling instead of repeating the experiment a large number of times. By using the nonparametric bootstrap approach to estimate the null distribution of the test statistic, the assumptions on the data model are minimised and no large sample assumption is invoked. In addition, for the F-statistic based method, we also propose a degrees-of-freedom modification approach for null distribution approximation. This method assumes a known noise kurtosis and yields closed form solutions. Simulation results show that in non-Gaussian noise, all the three detectors maintain the desired false alarm probability by using the proposed algorithms. The F-statistic based detector performs the best, e.g., to obtain a 90% detection probability in Laplacian noise, it provides a 2.5 dB and 4 dB signal-to-noise ratio (SNR) gain compared with the eigenvalue based detector and the energy based detector, respectively.

# data[].attributes.attachment_link.title
Robust spectrum sensing techniques for cognitive radio networks

# data[].attributes.attachment_link.description
Abstract Cognitive radio is a promising technology that improves the spectral utilisation by allowing unlicensed secondary users to access underutilised frequency bands in an opportunistic manner. This task can be carried out through spectrum sensing: the secondary user monitors the presence of primary users over the radio spectrum periodically to avoid harmful interference to the licensed service.

# data[].attributes.headline
16.An article which discusses the generalisability of the findings

# data[].attributes.subject
16.An article which discusses the generalisability of the findings

# data[].attributes.body
Wang, Cheng et al., 2018. Generalisability of an online randomised controlled trial: an empirical analysis. Journal of Epidemiology and Community Health, 72(2), pp.173–178.

# data[].attributes.attachment_link.description
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336, VAT Registration Number GB 592 9507 00, and is acknowledged by the UK authorities as a " Recognised body " which has been granted degree awarding powers.

# data[].attributes.headline
15.An article which reports descriptive statistics (e.g. mean and standard deviation)

# data[].attributes.subject
15.An article which reports descriptive statistics (e.g. mean and standard deviation)

# data[].attributes.body
Mishra, Prabhaker et al., 2019. Descriptive statistics and normality tests for statistical data. Annals of cardiac anaesthesia, 22(1), pp.67–72.

# data[].attributes.attachment_link.readable_text
Document controls

Main content

Byline: Prabhaker. Mishra, Chandra. Pandey, Uttam. Singh, Anshul. Gupta, Chinmoy. Sahu, Amit. Keshri

Descriptive statistics are an important part of biomedical research which is used to describe the basic features of the data in the study. They provide simple summaries about the sample and the measures. Measures of the central tendency and dispersion are used to describe the quantitative data. For the continuous data, test of the normality is an important step for deciding the measures of central tendency and statistical methods for data analysis. When our data follow normal distribution, parametric tests otherwise nonparametric methods are used to compare the groups. There are different methods used to test the normality of data, including numerical and visual methods, and each method has its own advantages and disadvantages. In the present study, we have discussed the summary measures and methods used to test the normality of the data.

A data set is a collection of the data of individual cases or subjects. Usually, it is meaningless to present such data individually because that will not produce any important conclusions. In place of individual case presentation, we present summary statistics of our data set with or without analytical form which can be easily absorbable for the audience. Statistics which is a science of collection, analysis, presentation, and interpretation of the data, have two main branches, are descriptive statistics and inferential statistics.[1]

Summary measures or summary statistics or descriptive statistics are used to summarize a set of observations, in order to communicate the largest amount of information as simply as possible. Descriptive statistics are the kind of information presented in just a few words to describe the basic features of the data in a study such as the mean and standard deviation (SD).[2],[3] The another is inferential statistics, which draw conclusions from data that are subject to random variation (e.g., observational errors and sampling variation). In inferential statistics, most predictions are for the future and generalizations about a population by studying a smaller sample.[2],[4] To draw the inference from the study participants in terms of different groups, etc., statistical methods are used. These statistical methods have some assumptions including normality of the continuous data. There are different methods used to test the normality of data, including numerical and visual methods, and each method has its own advantages and disadvantages.[5] Descriptive statistics and inferential statistics both are employed in scientific analysis of data and are equally important in the statistics. In the present study, we have discussed the summary measures to describe the data and methods used to test the normality of the data. To understand the descriptive statistics and test of the normality of the data, an example [Table 1] with a data set of 15 patients whose mean arterial pressure (MAP) was measured are given below. Further examples related to the measures of central tendency, dispersion, and tests of normality are discussed based on the above data.{Table 1}

There are three major types of descriptive statistics: Measures of frequency (frequency, percent), measures of central tendency (mean, median and mode), and measures of dispersion or variation (variance, SD, standard error, quartile, interquartile range, percentile, range, and coefficient of variation [CV]) provide simple summaries about the sample and the measures. A measure of frequency is usually used for the categorical data while others are used for quantitative data.

Frequency statistics simply count the number of times that in each variable occurs, such as the number of males and females within the sample or population. Frequency analysis is an important area of statistics that deals with the number of occurrences (frequency) and percentage. For example, according to [Table 1], out of the 15 patients, frequency of the males and females were 8 (53.3%) and 7 (46.7%), respectively.

Measures of Central Tendency

Data are commonly describe the observations in a measure of central tendency, which is also called measures of central location, is used to find out the representative value of a data set. The mean, median, and mode are three types of measures of central tendency. Measures of central tendency give us one value (mean or median) for the distribution and this value represents the entire distribution. To make comparisons between two or more groups, representative values of these distributions are compared. It helps in further statistical analysis because many techniques of statistical analysis such as measures of dispersion, skewness, correlation, t -test, and ANOVA test are calculated using value of measures of central tendency. That is why measures of central tendency are also called as measures of the first order. A representative value (measures of central tendency) is considered good when it was calculated using all observations and not affected by extreme values because these values are used to calculate for further measures.

Computation of Measures of Central Tendency

Mean is the mathematical average value of a set of data. Mean can be calculated using summation of the observations divided by number of observations. It is the most popular measure and very easy to calculate. It is a unique value for one group, that is, there is only one answer, which is useful when comparing between the groups. In the computation of mean, all the observations are used.[2],[5] One disadvantage with mean is that it is affected by extreme values (outliers). For example, according to [Table 2], mean MAP of the patients was 97.47 indicated that average MAP of the patients was 97.47 mmHg.{Table 2}

The median is defined as the middle most observation if data are arranged either in increasing or decreasing order of magnitude. Thus, it is one of the observations, which occupies the central place in the distribution (data). This is also called positional average. Extreme values (outliers) do not affect the median. It is unique, that is, there is only one median of one data set which is useful when comparing between the groups. There is one disadvantage of median over mean that it is not as popular as mean.[6] For example, according to [Table 2], median MAP of the patients was 95 mmHg indicated that 50% observations of the data are either less than or equal to the 95 mmHg and rest of the 50% observations are either equal or greater than 95 mmHg.

Mode is a value that occurs most frequently in a set of observation, that is, the observation, which has maximum frequency is called mode. In a data set, it is possible to have multiple modes or no mode exists. Due to the possibility of the multiple modes for one data set, it is not used to compare between the groups. For example, according to [Table 2], maximum repeated value is 116 mmHg (2 times) rest are repeated one time only, mode of the data is 116 mmHg.

Measures of dispersion is another measure used to show how spread out (variation) in a data set also called measures of variation. It is quantitatively degree of variation or dispersion of values in a population or in a sample. More specifically, it is showing lack of representation of measures of central tendency usually for mean/median. These are indices that give us an idea about homogeneity or heterogeneity of the data.[2],[6]

Variance, SD, standard error, quartile, interquartile range, percentile, range, and CV.

Computation of Measures of Dispersion

Standard deviation and variance

The SD is a measure of how spread out values is from its mean value. Its symbol is ? (the Greek letter sigma) or s. It is called SD because we have taken a standard value (mean) to measures the dispersion. Where x [sub]i is individual value, x? is mean value. If sample size is <30, we use ' n -1' in denominator, for sample size &#8805;30, use ' n ' in denominator. The variance (s2) is defined as the average of the squared difference from the mean. It is equal to the square of the SD (s).

For example, in the above, SD is 11.01 mmHg When n <30 which showed that approximate average deviation between mean value and individual values is 11.01. Similarly, variance is 121.22 [i.e., (11.01)2], which showed that average square deviation between mean value and individual values is 121.22 [Table 2].

Standard error is the approximate difference between sample mean and population mean. When we draw the many samples from same population with same sample size through random sampling technique, then SD among the sample means is called standard error. If sample SD and sample size are given, we can calculate standard error for this sample, by using the formula.

Standard error = sample SD/&#8730;sample size.

For example, according to [Table 2], standard error is 2.84 mmHg, which showed that average mean difference between sample means and population mean is 2.84 mmHg [Table 2].

Quartiles and interquartile range

The quartiles are the three points that divide the data set into four equal groups, each group comprising a quarter of the data, for a set of data values which are arranged in either ascending or descending order. Q1, Q2, and Q3 are represent the first, second, and third quartile's value.[7]

For ith Quartile = [i * (n + 1)/4]th observation, where i = 1, 2, 3.

For example, in the above, first quartile (Q1) = (n + 1)/4= (15 + 1)/4 = 4th observation from initial = 88 mmHg (i.e., first 25% number of observations of the data are either &#8804;88 and rest 75% observations are either &#8805;88), Q2 (also called median) = [2* (n + 1)/4] = 8th observation from initial = 95 mmHg, that is, first 50% number of observations of the data are either less or equal to the 95 and rest 50% observations are either &#8805;95, and similarly Q3 = [3* (n + 1)/4] = 12th observation from initial = 107 mmHg, i.e., indicated that first 75% number of observations of the data are either &#8804;107 and rest 25% observations are either &#8805;107. The interquartile range (IQR) is a measure of variability, also called the midspread or middle 50%, which is a measure of statistical dispersion, being equal to the difference between 75th (Q3 or third quartile) and 25th (Q1 or first quartile) percentiles. For example, in the above example, three quartiles, that is, Q1, Q2, and Q3 are 88, 95, and 107, respectively. As the first and third quartile in the data is 88 and 107. Hence, IQR of the data is 19 mmHg (also can write like: 88-107) [Table 2].

The percentiles are the 99 points that divide the data set into 100 equal groups, each group comprising a 1% of the data, for a set of data values which are arranged in either ascending or descending order. About 25% percentile is the first quartile, 50% percentile is the second quartile also called median value, while 75% percentile is the third quartile of the data.

For ith percentile = [i * (n + 1)/100]th observation, where i = 1, 2, 3.,99.

Example: In the above, 10th percentile = [10* (n + 1)/100] =1.6th observation from initial which is fall between the first and second observation from the initial = 1st observation + 0.6* (difference between the second and first observation) = 83.20 mmHg, which indicated that 10% of the data are either &#8804;83.20 and rest 90% observations are either &#8805;83.20.

Interpretation of SD without considering the magnitude of mean of the sample or population may be misleading. To overcome this problem, CV gives an idea. CV gives the result in terms of ratio of SD with respect to its mean value, which expressed in %. CV = 100 x (SD/mean). For example, in the above, coefficient of the variation is 11.3% which indicated that SD is 11.3% of its mean value [i.e., 100* (11.01/97.47)] [Table 2].

Difference between largest and smallest observation is called range. If A and B are smallest and largest observations in a data set, then the range (R) is equal to the difference of largest and smallest observation,th at is, R = A&#8722;B.

For example, in the above, minimum and maximum observation in the data is 82 mmHg and 116 mmHg. Hence, the range of the data is 34 mmHg (also can write like: 82-116) [Table 2].

Descriptive statistics can be calculated in the statistical software 'SPSS' (analyze &#8594; descriptive statistics &#8594; frequencies or descriptives.

Normality of data and testing

The standard normal distribution is the most important continuous probability distribution has a bell-shaped density curve described by its mean and SD and extreme values in the data set have no significant impact on the mean value. If a continuous data is follow normal distribution then 68.2%, 95.4%, and 99.7% observations are lie between mean [+ or -] 1 SD, mean [+ or -] 2 SD, and mean [+ or -] 3 SD, respectively.[2],[4]

Why to test the normality of data

Various statistical methods used for data analysis make assumptions about normality, including correlation, regression, t -tests, and analysis of variance. Central limit theorem states that when sample size has 100 or more observations, violation of the normality is not a major issue.[5],[8] Although for meaningful conclusions, assumption of the normality should be followed irrespective of the sample size. If a continuous data follow normal distribution, then we present this data in mean value. Further, this mean value is used to compare between/among the groups to calculate the significance level ( P value). If our data are not normally distributed, resultant mean is not a representative value of our data. A wrong selection of the representative value of a data set and further calculated significance level using this representative value might give wrong interpretation.[9] That is why, first we test the normality of the data, then we decide whether mean is applicable as representative value of the data or not. If applicable, then means are compared using parametric test otherwise medians are used to compare the groups, using nonparametric methods.

Methods used for test of normality of data

An assessment of the normality of data is a prerequisite for many statistical tests because normal data is an underlying assumption in parametric testing. There are two main methods of assessing normality: Graphical and numerical (including statistical tests).[3],[4] Statistical tests have the advantage of making an objective judgment of normality but have the disadvantage of sometimes not being sensitive enough at low sample sizes or overly sensitive to large sample sizes. Graphical interpretation has the advantage of allowing good judgment to assess normality in situations when numerical tests might be over or undersensitive. Although normality assessment using graphical methods need a great deal of the experience to avoid the wrong interpretations. If we do not have a good experience, it is the best to rely on the numerical methods.[10] There are various methods available to test the normality of the continuous data, out of them, most popular methods are Shapiro-Wilk test, Kolmogorov-Smirnov test, skewness, kurtosis, histogram, box plot, P-P Plot, Q-Q Plot, and mean with SD. The two well-known tests of normality, namely, the Kolmogorov-Smirnov test and the Shapiro-Wilk test are most widely used methods to test the normality of the data. Normality tests can be conducted in the statistical software 'SPSS' (analyze &#8594; descriptive statistics &#8594; explore &#8594; plots &#8594; normality plots with tests).

The Shapiro-Wilk test is more appropriate method for small sample sizes (<50 samples) although it can also be handling on larger sample size while Kolmogorov-Smirnov test is used for n &#8805;50. For both of the above tests, null hypothesis states that data are taken from normal distributed population. When P > 0.05, null hypothesis accepted and data are called as normally distributed. Skewness is a measure of symmetry, or more precisely, the lack of symmetry of the normal distribution. Kurtosis is a measure of the peakedness of a distribution. The original kurtosis value is sometimes called kurtosis (proper). Most of the statistical packages such as SPSS provide 'excess' kurtosis (also called kurtosis [excess]) obtained by subtracting 3 from the kurtosis (proper). A distribution, or data set, is symmetric if it looks the same to the left and right of the center point. If mean, median, and mode of a distribution coincide, then it is called a symmetric distribution, that is, skewness = 0, kurtosis (excess) = 0. A distribution is called approximate normal if skewness or kurtosis (excess) of the data are between &#8722; 1 and + 1. Although this is a less reliable method in the small-to-moderate sample size (i.e., n <300) because it can not adjust the standard error (as the sample size increases, the standard error decreases). To overcome this problem, a z -test is applied for normality test using skewness and kurtosis. A Z score could be obtained by dividing the skewness values or excess kurtosis value by their standard errors. For small sample size ( n <50), z value [+ or -] 1.96 are sufficient to establish normality of the data.[8] However, medium-sized samples (50&#8804; n <300), at absolute z -value [+ or -] 3.29, conclude the distribution of the sample is normal.[11] For sample size >300, normality of the data is depend on the histograms and the absolute values of skewness and kurtosis. Either an absolute skewness value &#8804;2 or an absolute kurtosis (excess) &#8804;4 may be used as reference values for determining considerable normality.[11] A histogram is an estimate of the probability distribution of a continuous variable. If the graph is approximately bell-shaped and symmetric about the mean, we can assume normally distributed data[12],[13] [Figure 1]. In statistics, a Q-Q plot is a scatterplot created by plotting two sets of quantiles (observed and expected) against one another. For normally distributed data, observed data are approximate to the expected data, that is, they are statistically equal [Figure 2]. A P-P plot (probability-probability plot or percent-percent plot) is a graphical technique for assessing how closely two data sets (observed and expected) agree. It forms an approximate straight line when data are normally distributed. Departures from this straight line indicate departures from normality [Figure 3]. Box plot is another way to assess the normality of the data. It shows the median as a horizontal line inside the box and the IQR (range between the first and third quartile) as the length of the box. The whiskers (line extending from the top and bottom of the box) represent the minimum and maximum values when they are within 1.5 times the IQR from either end of the box (i.e., Q1 &#8722; 1.5* IQR and Q3 + 1.5* IQR). Scores >1.5 times and 3 times the IQR are out of the box plot and are considered as outliers and extreme outliers, respectively. A box plot that is symmetric with the median line at approximately the center of the box and with symmetric whiskers indicate that the data may have come from a normal distribution. In case many outliers are present in our data set, either outliers are need to remove or data should treat as nonnormally distributed[8],[13],[14] [Figure 4]. Another method of normality of the data is relative value of the SD with respect to mean. If SD is less than half mean (i.e., CV <50%), data are considered normal.[15] This is the quick method to test the normality. However this method should only be used when our sample size is at least 50.{Figure 1}{Figure 2}{Figure 3}{Figure 4}

For example in [Table 1], data of MAP of the 15 patients are given. Normality of the above data was assessed. Result showed that data were normally distributed as skewness (0.398) and kurtosis (&#8722;0.825) individually were within [+ or -]1. Critical ratio ( Z value) of the skewness (0.686) and kurtosis (&#8722;0.737) were within [+ or -]1.96, also evident to normally distributed. Similarly, Shapiro-Wilk test ( P = 0.454) and Kolmogorov-Smirnov test ( P = 0.200) were statistically insignificant, that is, data were considered normally distributed. As sample size is <50, we have to take Shapiro-Wilk test result and Kolmogorov-Smirnov test result must be avoided, although both methods indicated that data were normally distributed. As SD of the MAP was less than half mean value (11.01 <48.73), data were considered normally distributed, although due to sample size <50, we should avoid this method because it should use when our sample size is at least 50 [Table 2] and [Table 3].{Table 3}

Descriptive statistics are a statistical method to summarizing data in a valid and meaningful way. A good and appropriate measure is important not only for data but also for statistical methods used for hypothesis testing. For continuous data, testing of normality is very important because based on the normality status, measures of central tendency, dispersion, and selection of parametric/nonparametric test are decided. Although there are various methods for normality testing but for small sample size ( n <50), Shapiro-Wilk test should be used as it has more power to detect the nonnormality and this is the most popular and widely used method. When our sample size ( n ) is at least 50, any other methods (Kolmogorov-Smirnov test, skewness, kurtosis, z value of the skewness and kurtosis, histogram, box plot, P-P Plot, Q-Q Plot, and SD with respect to mean) can be used to test of the normality of continuous data.

The authors would like to express their deep and sincere gratitude to Dr. Prabhat Tiwari, Professor, Department of Anaesthesiology, Sanjay Gandhi Postgraduate Institute of Medical Sciences, Lucknow, for his critical comments and useful suggestions that was very much useful to improve the quality of this manuscript.

Financial support and sponsorship

There are no conflicts of interest.

1. Lund Research Ltd. Descriptive and Inferential Statistics. Available from: http://www.statistics.laerd.com. [Last accessed on 2018 Aug 02].

2. Sundaram KR, Dwivedi SN, Sreenivas V. Medical Statistics: Principles and Methods. 2nd ed. New Delhi: Wolters Kluwer India; 2014.

3. Bland M. An Introduction to Medical Statistics. 4th ed. Oxford: Oxford University Press; 2015.

4. Campbell MJ, Machin D, Walters SJ. Medical Statistics: A text book for the health sciences, 4th ed. Chichester: John Wiley & Sons, Ltd.; 2007.

5. Altman DG, Bland JM. Statistics notes: The normal distribution. BMJ 1995;310:298.

6. Altman DG. Practical Statistics for Medical Research Chapman and Hall/CRC Texts in Statistical Science. London: CRC Press; 1999.

7. Indrayan A, Sarmukaddam SB. Medical Bio-Statistics. New York: Marcel Dekker Inc.; 2000.

8. Ghasemi A, Zahediasl S. Normality tests for statistical analysis: A guide for non-statisticians. Int J Endocrinol Metab 2012;10:486-9.

9. Indrayan A, Satyanarayana L. Essentials of biostatistics. Indian Pediatr 1999;36:1127-34.

10. Lund Research Ltd. Testing for Normality using SPSS Statistics. Available from: http://www.statistics.laerd.com. [Last accessed 2018 Aug 02].

11. Kim HY. Statistical notes for clinical researchers: Assessing normal distribution (2) using skewness and kurtosis. Restor Dent Endod 2013;38:52-4.

12. Armitage P, Berry G. Statistical Methods in Medical Research. 2nd ed. London: Blackwell Scientific Publications; 1987.

13. Barton B, Peat J. Medical Statistics: A Guide to SPSS, Data Analysis and Clinical Appraisal. 2nd ed. Sydney: Wiley Blackwell, BMJ Books; 2014.

14. Baghban AA, Younespour S, Jambarsang S, Yousefi M, Zayeri F, Jalilian FA. How to test normality distribution for a variable: A real example and a simulation study. J Paramed Sci 2013;4:73-7.

15. Jeyaseelan L. Short Training Course Materials on Fundamentals of Biostatistics, Principles of Epidemiology and SPSS. CMC Vellore: Biostatistics Resource and Training Center (BRTC); 2007.

Source Citation

# data[].attributes.attachment_link.title
Descriptive statistics and normality tests for statistical data

# data[].attributes.attachment_link.description
Byline: Prabhaker. Mishra, Chandra. Pandey, Uttam. Singh, Anshul. Gupta, Chinmoy. Sahu, Amit. Keshri Descriptive statistics are an important part of biomedical research which is used to describe the basic features of the data in the study. They provide simple summaries about the sample and the measures.

# data[].attributes.headline
14.An article which uses thematic analysis

# data[].attributes.subject
14.An article which uses thematic analysis

# data[].attributes.body
Jones, Marian V, Coviello, Nicole & Tang, Yee Kwan, 2011. International Entrepreneurship research (1989–2009): A domain ontology and thematic analysis. Journal of business venturing, 26(6), pp.632–659.

# data[].attributes.attachment_link.description
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336, VAT Registration Number GB 592 9507 00, and is acknowledged by the UK authorities as a " Recognised body " which has been granted degree awarding powers.

# data[].attributes.headline
13.An article which makes a statement about the position of the researcher

# data[].attributes.subject
13.An article which makes a statement about the position of the researcher

# data[].attributes.body
Pring, R., 2000. The 'False Dualism' of Educational Research. Journal of philosophy of education, 34(2), pp.247–260.

# data[].attributes.attachment_link.description
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336, VAT Registration Number GB 592 9507 00, and is acknowledged by the UK authorities as a " Recognised body " which has been granted degree awarding powers.

# data[].attributes.body
MacLennan, S., Imamura, M., Lapitan, M.C., Omar, M.I., Lam, T.B., Hilvano-Cabungcal, A.M., Royle, P., Stewart, F., MacLennan, G., MacLennan, S.J. and Canfield, S.E., 2012. Systematic review of oncological outcomes following surgical management of localised renal cancer. European urology, 61(5), pp.972-993.

# data[].attributes.attachment_link.readable_text
Abstract

Context

Renal cell carcinoma (RCC) accounts for 2-3% of adult malignancies. There remain uncertainties over the oncological outcomes for the surgical management of localised RCC.

Objective

Systematically review relevant literature comparing oncological outcomes of surgical management of localised RCC (T1-2N0M0).

Evidence acquisition

Relevant databases including Medline, Embase, and the Cochrane Library were searched up to October 2010, and an updated scoping search was performed up to January 2012. Randomised controlled trials (RCTs) or quasi-RCTs, prospective observational studies with controls, retrospective matched-pair studies, and comparative studies from well-defined registries/databases were included. The main outcomes were overall survival, cancer-specific survival, recurrence, and metastases. The Cochrane risk of bias tool was used to assess RCTs, and an extended version was used to assess nonrandomised studies (NRSs). The quality of evidence was assessed using Grading of Recommendations Assessment, Development, and Evaluation (GRADE).

Evidence synthesis

A total of 4580 abstracts and 389 full-text articles were assessed. Thirty-four studies met the inclusion criteria (6 RCTs and 28 NRSs). Meta-analyses were planned but were deemed inappropriate due to data heterogeneity. There were high risks of bias and low-quality evidence across the evidence base. Open radical nephrectomy and open partial nephrectomy showed similar cancer-specific and overall survival, but when both open and laparoscopic approaches are considered together, the evidence showed improved survival for partial nephrectomy for tumours ≤4 cm. The overall evidence suggests either equivalent or better survival with partial nephrectomy. Laparoscopic radical nephrectomy offered equivalent survival to open radical nephrectomy, and all laparoscopic approaches achieved equivalent survival. Open and laparoscopic partial nephrectomy achieved equivalent survival. The issue of ipsilateral adrenalectomy or complete lymph node dissection with radical nephrectomy or partial nephrectomy remains unresolved.

Conclusions

The evidence base suggests localised RCCs are best managed by nephron-sparing surgery where technically feasible. However, the current evidence base has significant limitations due to studies of low methodological quality marked by high risks of bias.

View full text

# data[].attributes.attachment_link.title
Systematic Review of Oncological Outcomes Following Surgical Management of Localised Renal Cancer

# data[].attributes.attachment_link.description
Renal cell carcinoma (RCC) accounts for 2-3% of adult malignancies. There remain uncertainties over the oncological outcomes for the surgical management of localised RCC. Systematically review relevant literature comparing oncological outcomes of surgical management of localised RCC (T1-2N0M0).

# data[].attributes.headline
5. An article reporting narrative research

# data[].attributes.subject
5. An article reporting narrative research

# data[].attributes.body
Spector-Mersel, G., 2010. Narrative research: Time for a paradigm. Narrative inquiry, 20(1), pp.204-224.

# data[].attributes.attachment_link.readable_text
1. Home

2. e-Journals

3. Narrative Inquiry

4. Volume 20, Issue 1

5. Article

ISSN 1387-6740

E-ISSN: 1569-9935

Abstract

As a result of the popularization of the narrative idea and the considerable diversity existing among narrative studies, a rather "all included" conception has arisen, in which the framework of narrative inquiry has been significantly blurred. For narrative inquiry to persist as a unique mode of investigation into human nature, a complementary dialogue is required that aims at outlining its core, alongside the emphasis given in the literature on diversity as its hallmark. As a possible reference point for this debate, recognizing the narrative paradigm that has crystallized since the "narrative turn" is suggested. The narrative paradigm is discussed in light of six major dimensions - ontology, epistemology, methodology, inquiry aim, inquirer posture and participant/narrator posture - indicating that it coincides with other interpretive paradigms in certain aspects yet proffers a unique philosophical infrastructure that gives rise to particular methodological principles and methods. Considering the narrative paradigm as the essence of narrative inquiry asserts that the latter is not confined to a methodology, as often implied. Rather it constitutes a full-fledged research Weltanschauung that intimately connects the "hows" of investigation to the "whats", namely premises about the nature of reality and our relationships with it.

# data[].attributes.attachment_link.description
As a result of the popularization of the narrative idea and the considerable diversity existing among narrative studies, a rather "all included" conception has arisen, in which the framework of narrative inquiry has been significantly blurred.

# data[].attributes.headline
4. An article reporting an experiment or a quasi-experiment

# data[].attributes.subject
4. An article reporting an experiment or a quasi-experiment

# data[].attributes.body
Cohen, S.G. and Ledford Jr, G.E., 1994. The effectiveness of self-managing teams: A quasi-experiment. Human relations, 47(1), pp.13-43.

# data[].attributes.attachment_link.readable_text
This study used a quasi-experimental design to assess the effectiveness of self-managing teams in a telecommunications company. These teams performed customer service, technical support, administrative support, and managerial functions in a variety of locations. The balance of evidence indicates that self-managing teams were more effective than comparable traditionally-managed groups that performed the same type of work. The study illustrates the value of a collaborative research project in which researchers and clients jointly define the research questions, study design, and methods.

CORDERY, J. L. , MUELLER, W. S. , & SMITH, L. M. Attitudinal an behavioral effects of autonomous group working: A longitudinal field study . Academy of Management Journal, 1991, 34(2).
Google Scholar

CUMMINGS, T. G. Self-regulating work groups: A sociotechnical synthesis . Academy of Management Review, 1978, 3(3), 625-634 .
Google Scholar

GOODMAN, P. S. , RAVLIN, E. C. , & ARGOTE, L. Current thinking about groups: Setting the stage for new ides. In P. S. Goodman & Associates (Eds.), Designing effective work groups. San Francisco: Jossey-Bass & Associates , 1986, pp. 1-33.
Google Scholar

GOODMAN, P. S. , DEVADAS, S. , & HUGHSON, T. L. Groups and productivity: Analyzing the effectiveness of self-managing teams. In J. P. Campbell , R. J. Campbell , and Associates (Eds.), Productivity in organizations. San Francisco: Jossey-Bass & Associates , 1988, pp. 295-325.
Google Scholar

HACKMAN, J. R. , & MORRIS, C. G. Group tasks, group interaction process, and group performance effectiveness: A review and proposed integration. In L. Berkowitz (Ed.), Advances in experiential social psychology (Vol. 9). New York: Academic Press , 1975.
Google Scholar

MANZ, C. C. , & SIMS, H. P. Self-management as a substitute for leadership: A social learning theory perspective . Academy of Management Review, 1980, 5(3), 361-367 .
Google Scholar

# data[].attributes.attachment_link.title
The Effectiveness of Self-Managing Teams: A Quasi-Experiment - Susan G. Cohen, Gerald E. Ledford, 1994

# data[].attributes.attachment_link.description
This study used a quasi-experimental design to assess the effectiveness of self-managing teams in a telecommunications company. These teams performed customer service, technical support, administrative support, and managerial functions in a variety of locations. The balance of evidence indicates that self-managing teams were more effective than comparable traditionally-managed groups that performed the same type of work.

# data[].attributes.body
Park, H. and Kincade, D.H., 2010. Historical analysis of apparel marketer’s strategies: Evidence from a Nike case. Journal of Global Fashion Marketing, 1(3), pp.182-193.

# data[].attributes.attachment_link.readable_text
In the decades previous to 2010, many firms in consumer product industries, including the apparel industry, have experienced changes both in the structure of the industries (e.g., ownership, control and location of production) and the functions of the firms (e.g., production- or product oriented vs. marketing-oriented) within the industries (Su et al., ). One force in this change was the economic globalization that brought about shifts of production and trade from domestic production to off-shore sourcing of production and then of fabrication and design (Ha-Brookshire and Dyer, Ha-Brookshire, J.E. and 2009. Framing a descriptive profile of a transformed apparel industry: Apparel import intermediaries in the United States. 1994. "The organization of buyer-driven global commodity chains: How U.S. retailers shape overseas production networks". In 2009 1994 Commodity chains and global capitalism, Edited by: Gereffi, Gary and Miguel, Korzeniewicz . 95-122. Westport: Praeger. ) argued that the continued interest in this dynamic of globalization in the apparel industry may remain because global restructuring of the economy is historical in shifts of power across commodity chains. As a major industry in the global market, this argument emphasizes the importance of historical analysis for the apparel industry. However, a review of literature revealed that only a few academic studies (e.g., Kincade, ; Locker, ) have explored issues such as specific details about the growth and change in the apparel industry, and even fewer studies placed these issues in a historical context. Journal of Fashion Marketing & Management, 13(2): 161-178. ). Gereffi (

The current study is designed to fill this gap in the literature and provide insights into the historical perspective of the U.S. apparel firms' response changes in its business environment over the last four decades and the importance of the marketing function within this context. The objectives of this study were: (1) to analyze a U.S. apparel company's shift in organization and strategies in a historical perspective, and (2) to provide relevance of these findings within economic globalization.

To achieve the research objectives, a sample company's case with general industry trends from 1970s to 2009, were investigated. This time frame was determined based on the early 1970s as a beginning point of much of the apparel industry restructure in response to the economic globalization (Taplin, 1994. "Strategic reorientations of U.S. apparel firms". In 1994 Commodity chains and global capitalism, Edited by: Gereffi, Gary and Korzeniewicz, Miguel . 205-222. Westport: Praeger. ) and with the consideration of data availability for a sample company and its year of establishment. The sampling frame of this study consisted of the companies ranked in The Top Company lists (e.g., The Top 50, The Top 40) in Apparel magazine. Among the seven companies continuously listed during the period of study, Nike, Inc. was selected as the sample for this case study because of the extensive availability of historical data (e.g., data beginning in the 1970s) about Nike and its continued growth and leadership.

The case study covered four decades from the 1968, the year that Nike was incorporated, to 2009. In the first period from 1968 to 1975, Nike emphasized production and distribution related to imports and trade. By emphasizing principal business activities (i.e., designing, selling the products) and outsourcing all manufacturing in Japan and other countries, Nike initiated global sourcing in the late 1960s. This strategy became a trend that most other companies would eventually follow throughout the next several decades. In the second period between 1976 and 1983, Nike focused on product innovation with technology to exceed over other competitors (e.g., Adidas, Puma). Through this strategy a new product model, the Air shoe, was developed, which significantly contributed to a reversal in declining sales. In the third period, 1984 to 1996, Nike's innovative advertising started with the endorsement contract by Michael Jordan. The endorsement strategy built a strong foundation for Nike as a marketing company. Marketing strategies, developed for reinforcing the brand image with its Swoosh logo, the slogan "Just do it," and endorsements with famous athletes, strengthened Nike's strategic reorientation as it changed from a traditionally focused production-oriented company to a more flexible and reactive marketing-oriented company. Finally, from 1997 to 2009, Nike has diversified its marketing efforts and taken leadership in the industry. Nike's advanced marketing strategies include market diversification through branding and niche segmentation (e.g., casual apparel for women, golf shoes), further product innovation with technology (e.g., Nike Plus, Flywire, Swift Apparel), inventive advertising campaigns, and restructuring management operations.

Nike's business strategies exemplified efforts to become more consumer-reactive over the last four decades. Working with dedicated overseas contractors, Nike developed a flexible business structure and maintained a profitably reactive apparel firm. To function as the ultimate marketer and consumer-centric company, Nike readjusted its business strategy from a production-orientation to a marketing-orientation to the goal of being a consumer-oriented company during the decades from the late 1960s to 2009. The case study of Nike well supports the organizational learning theory literature for the apparel industry (e.g., Dyer and Ha-Brookshire, and Ha-Brookshire, J.E. 2008. Apparel import intermediaries' secrets to success: Redefining success in a hyper-dynamic environment. 2008 Journal of Fashion Marketing and Management, 12(1): 51-67. ; Kincade, ) in that Nike continuously redefined its business strategies as it achieved the key goals for each business period in response to the changing global environment. Following a basic organizational learning approach to strategic planning, Nike focused on what it could do best with available resources, what would be innovative, creative and market-leading, and what would sell to the consumer.

In conclusion, Nike's strategic business responses to a changing and global environment were varied and related to the company's structure and the business at the time. Historical analysis confirmed that although various strategies were stressed for each period of time, all strategies were implemented as a part of Nike's overall business planning and were market reactive. The examination of the Nike case study provides support for the general organizational learning theory (e.g., Jemison ; Ward and Duray, and 2000. Manufacturing strategy in context: Environment, competitive strategy and manufacturing strategy. 2000 Journal of Operations Management, 18(2): 123-138. ) and the apparel specific theory proposed by Park and Kincade (in press). Staying reactive with marketing-oriented strategy has become a key strategy for Nike and maybe useful for other firms to achieve competitiveness in a changing market.

# data[].attributes.attachment_link.title
Historical Analysis of Apparel Marketer's Strategies: Evidence from a Nike Case

# data[].attributes.attachment_link.description
In the decades previous to 2010, many firms in consumer product industries, including the apparel industry, have experienced changes both in the structure of the industries (e.g., ownership, control and location of production) and the functions of the firms (e.g., production- or product oriented vs. marketing-oriented) within the industries (Su et al., ).

# data[].attributes.headline
2.An article which describes the aim of the research reported as to "explain"

# data[].attributes.subject
2.An article which describes the aim of the research reported as to "explain"

# data[].attributes.body
Forgas, J.P., 2001. The Affect Infusion Model (AIM): An integrative theory of mood effects on cognition and judgments.

# data[].attributes.headline
1. An article which describe the aim of the research reported as to "explore"

# data[].attributes.subject
1. An article which describe the aim of the research reported as to "explore"

# data[].attributes.body
Jacoby, A., Smith, M. and Eccles, M., 2003. A qualitative study to explore influences on general practitioners' decisions to prescribe new drugs. British Journal of General Practice, 53(487), pp.120-125.

# data[].attributes.attachment_link.readable_text
Abstract

BACKGROUND: Ensuring appropriate prescribing is an important challenge for the health service, and the need for research that takes account of the reasons behind individual general practitioners' (GPs) prescribing decisions has been highlighted. AIM: To explore differences among GPs in their decisions to prescribe new drugs. DESIGN OF STUDY: Qualitative approach, using in-depth semistructured interviews. SETTING: Northern and Yorkshire Health Authority Region. METHOD: Participants were identified from a random sample of 520 GPs in a quantitative study of patterns of uptake of eight recently introduced drugs. Purposeful sampling ensured inclusion of GPs prescribing any of the eight drugs and working in a range of practice settings. Fifty-six GPs were interviewed, using a topic guide. Interviews were recorded on audiotape. Transcribed text was methodically coded and data were analysed by constantly comparing emerging themes. RESULTS: Both low and high prescribers shared a view of themselves as conservative in their prescribing behaviour. Low prescribers appeared to conform more strongly to group norms and identified a consensus among practice partners in prescribing and cost-consciousness. Conformism to group norms was represented by a commitment to practice formularies. High prescribers more often expressed themselves to be indifferent to drug costs and a shared practice ethos. CONCLUSIONS: A shift in the attitudes of some GPs is required before cost-effectiveness is routinely incorporated in drug prescribing. The promotion of rational prescribing is likely to be more successful if efforts are focused on GPs' appreciation of cost issues and attitudes towards shared decision-making and responsibility.

# data[].attributes.attachment_link.title
A qualitative study to explore influences on general practitioners' decisions to prescribe new drugs.

# data[].attributes.attachment_link.description
BACKGROUND: Ensuring appropriate prescribing is an important challenge for the health service, and the need for research that takes account of the reasons behind individual general practitioners' (GPs) prescribing decisions has been highlighted. AIM: To explore differences among GPs in their decisions to prescribe new drugs.

# data[].attributes.headline
12.An article which reports using focus groups as the method of data collection

# data[].attributes.subject
12.An article which reports using focus groups as the method of data collection

# data[].attributes.body
Patients’ Attitudes Regarding Lung Cancer Screening and Decision Aids. A Survey and Focus Group Study

# data[].attributes.attachment_link.description
This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Find out more.

# data[].attributes.headline
11.An article which reports using observation as the method of data collection

# data[].attributes.subject
11.An article which reports using observation as the method of data collection

# data[].attributes.body
Opportunity for Interaction? A Naturalistic Observation Study of Dual-Earner Families After Work and School

Belinda Campos , Anthony P. Graesch, Rena Repetti, Thomas Bradbury, Elinor Ochs

# data[].attributes.attachment_link.description
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336, VAT Registration Number GB 592 9507 00, and is acknowledged by the UK authorities as a " Recognised body " which has been granted degree awarding powers.

# data[].attributes.headline
10.An article which reports using interviews as the method of data collection

# data[].attributes.subject
10.An article which reports using interviews as the method of data collection

# data[].attributes.body
A methodological guide to using and reporting on interviews in conservation science research

# data[].attributes.headline
9.An article which mentions a null hypothesis

# data[].attributes.subject
9.An article which mentions a null hypothesis

# data[].attributes.body
Setting an Optimal α That Minimizes Errors in Null Hypothesis Significance Tests

Joseph F. Mudge ,Leanne F.Baker,Christopher B. Edge,Jeff E. Houlahan

# data[].attributes.attachment_link.readable_text
Joseph F. Mudge,

Leanne F. Baker,

Christopher B. Edge,

Jeff E. Houlahan

Subject Areas

For more information about PLOS Subject Areas, click here.

We want your feedback. Do these Subject Areas make sense for this article? Click the target next to the incorrect Subject Area and let us know. Thanks for your help!

Abstract

Null hypothesis significance testing has been under attack in recent years, partly owing to the arbitrary nature of setting α (the decision-making threshold and probability of Type I error) at a constant value, usually 0.05. If the goal of null hypothesis testing is to present conclusions in which we have the highest possible confidence, then the only logical decision-making threshold is the value that minimizes the probability (or occasionally, cost) of making errors. Setting α to minimize the combination of Type I and Type II error at a critical effect size can easily be accomplished for traditional statistical tests by calculating the α associated with the minimum average of α and β at the critical effect size. This technique also has the flexibility to incorporate prior probabilities of null and alternate hypotheses and/or relative costs of Type I and Type II errors, if known. Using an optimal α results in stronger scientific inferences because it estimates and minimizes both Type I errors and relevant Type II errors for a test. It also results in greater transparency concerning assumptions about relevant effect size(s) and the relative costs of Type I and II errors. By contrast, the use of α = 0.05 results in arbitrary decisions about what effect sizes will likely be considered significant, if real, and results in arbitrary amounts of Type II error for meaningful potential effect sizes. We cannot identify a rationale for continuing to arbitrarily use α = 0.05 for null hypothesis significance tests in any field, when it is possible to determine an optimal α.

Citation: Mudge JF, Baker LF, Edge CB, Houlahan JE (2012) Setting an Optimal α That Minimizes Errors in Null Hypothesis Significance Tests. PLoS ONE 7(2): e32734. https://doi.org/10.1371/journal.pone.0032734

Editor: Zheng Su, Genentech, United States of America

Received: June 20, 2011; Accepted: February 2, 2012; Published: February 28, 2012

Copyright: © 2012 Mudge et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Funding: The authors are supported by an NSERC Strategic Grant, the Canadian Rivers Institute at the University of New Brunswick Saint John, the Canadian Department of Defence, Natural Resources Canada-Canadian Forest Services, and Environment Canada. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

Introduction

A well-known problem associated with null hypothesis significance tests (NHST) is the arbitrariness of the chosen experimental significance level, alpha ( α). Yet the practice of comparing an observed p-value to an arbitrary α, usually α = 0.05, remains widespread as "methodological orthodoxy" in science [1], [2]. There is large body of literature that advocates for the abandonment of NHST [1]-[5], and although we recognize that NHST is frequently misused, we do not wish to contribute to the bashing that is unlikely to garner much attention or end such a deeply entrenched practice among scientists. Instead, we posit that " abuses non toll it sum", that abuse does not preclude proper use. The ease with which a correctly-interpreted null hypothesis significance test can be used as a decision-making tool causes it to continue to be favoured in most scientific fields. The goal of these tests should be to provide us with conclusions in which we have the highest possible confidence. Thus, the logical decision-making significance threshold, α, should be the value that minimizes the probability, or occasionally, the cost of making any relevant error. In the former case, this would make the goal of the statistical test to avoid making an erroneous conclusion, while in the latter it would make the goal of the statistical test to avoid making a costly erroneous conclusion. We feel that doing statistics for purposes other than these would be outside the realms of pure and applied science.

The Case for a New Approach to Setting α

In traditional NHST there are two types of errors, rejecting the null hypothesis when it is true (Type I error) and failing to reject the null when the alternate hypothesis is true (Type II error). Alpha is set to address the Type I error rate - it is the probability of making a Type I error that we are willing to accept in a particular experiment. The choice of an α level will determine the probability of a Type II error ( β) for a study with a given sample size and critical effect size. Decreasing the probability of Type I error increases the probability of Type II error and vice-versa ( Figure 1). Because α determines the power (1 - β) to detect effects of specified sizes, then using the standard α = 0.05 makes implicit decisions about the effect sizes a researcher will be likely to consider significant, if they exist. The standard α = 0.05 also arbitrarily determines the chance of Type II error relative to Type I error for meaningful potential effect sizes. These implicit decisions associated with using α = 0.05 are often both unrealized and unrealistic. Appropriate conclusions from statistical tests should involve explicit considerations about the magnitude of effect that would be important to be able to detect, if it were real, and whether the probabilities of Type I and Type II error reflect the relative seriousness of the consequences of a Type I vs. Type II error. Considering critical effect sizes and the relative consequences of Type I and Type II errors should not be perceived as an unnecessary extra step of statistical testing that can be avoided, because decisions about these factors are unwittingly made using the traditional approach of setting α = 0.05. For a given study, it will not be possible to detect some effect sizes at α = 0.05, and when the researcher simply fails to consider how setting α = 0.05 can affect β for a meaningful potential effect size, they are unknowingly providing a study-specific cost ratio that may be heavily weighted in favour of reducing Type I errors, at the expense of experimental power, or vice versa, depending on the sample size of the study. Turning a blind eye to Type II errors, in favour of controlling a Type I error rate, allows β to fluctuate as a function of the sample size and variability. Although the traditional approach of ignoring Type II error probabilities may be easy, it can result in poor decisions. Our goal is to improve upon an obviously flawed hypothesis-testing system while working under the constraints of that system, because we acknowledge that researchers are not very likely to abandon an approach that is so easy to use and so widely understood.

The relationship between α and β for an independent 2-sample, 2-tailed t-test with n ₁ = n ₂ = 10, and critical effect size = 1 σ.

Current practice involves a bias against accepting a falsehood (Type 1 error), where rejecting a truth (Type II error) is regarded as "healthy scepticism" [6], implying that there is much lower cost associated with Type II errors relative to the cost associated with Type I error [7]. We set α at a value other than zero because, while we want to limit our probability of falsely concluding there is an effect (Type I error), we also want to limit the probability of missing a real effect (Type II error). If one does not care at all about missing real effects (Type II errors), then a β level of 1 becomes acceptable and Type I error can be minimized by simply setting α = 0. The acceptability or magnitude of the potential for Type II errors should be an area of concern for all researchers, no matter the type of study performed. There are many cases where we believe that failing to detect a real effect, should be considered at least as serious, if not more serious than falsely detecting a non-existent effect. Should natural resource industries be encouraged to design low power environmental impact studies that would never have any chance of detecting anything but extreme environmental impacts using α = 0.05? In this case, a Type I error would result in loss of economic opportunity, while Type II error would result in environmental damage and associated economic costs [8]. We feel that, despite the difficulty in its quantification, the ratio of the importance of Type I vs. Type II errors should always be discussed, rather than disregarded.

Arguments for using a single, arbitrary value of α for all studies include: i) that it provides a consistent approach to statistical analysis and; ii) that it an objective approach that avoids value judgements; and iii) that it is easy to use. Arguments i) and ii) are incorrect because p-values are dependent on sample size. Setting α = 0.05 is not a truly consistent approach among studies because as sample size increases, the observed effect size required to produce a significant result using α = 0.05 decreases, so studies with different sample sizes require different observed effect sizes to yield a significant result. Because of this, consistently using α = 0.05 is not an objective approach. Subjectivity is merely shifted away from the choice of α to the choice of sample size, such that if a researcher wants to find statistical significance using α = 0.05 they should conduct a test with a large sample size (while small sample sizes could be used in the same way for cases in which the desired result is non-significance). While argument iii), the ease of use of α = 0.05 is certainly a benefit of using an arbitrary α, it is not necessarily good science. Given the absence of a strong rationale for continuing to use a single, arbitrary value for α among studies, what should be the basis for setting α in individual studies?

We argue that, in almost all contexts, the goal of statistical testing is to aid us in making conclusions that limit the probabilities of making mistakes, whether they be Type I or II errors. We think a strong case can be made that in most studies (and perhaps all) α should be set with the objective of either minimizing the combined probabilities of making Type I or Type II errors at a critical effect size, or minimizing the overall cost associated with Type I and Type II errors given their respective probabilities. This can be done rather simply if researchers explicitly consider the relative importance of avoiding Type I vs. II errors and estimate, a priori, what would constitute a relevant critical effect size. This makes it possible to set a study-specific optimal α level that can minimize either the average of the probabilities of Type I and Type II error at the critical effect size, or instead minimize the cost-weighted average of Type I and Type II errors. The optimal α approach can prevent the inflated overall error rates (combined probability of either a Type I or Type II error) that result from arbitrarily using α = 0.05, and requires that the user be transparent concerning detectability of a priori critical effect sizes and the relative importance of avoiding Type I and II errors. We describe a straightforward and flexible method for setting study-specific optimal α levels that is easily applicable in all pure and applied areas of scientific research and leads to optimal, evidence-based conclusions. We propose an approach to setting an optimal α level that has the flexibility to minimize either the combined probabilities of Type I errors and relevant Type II errors, or the minimize the relative cost of errors, and can incorporate specified or unknown critical effect sizes, relative costs of Type I and Type II errors or prior probabilities of hypotheses. This simple method can be used to derive an optimal α level that can be used by any researcher, setting NHST on a course of scientifically valid use and in a manner that leads to better decisions.

The optimal α level for any null hypothesis significance test depends on i) the prior probabilities that the null and alternate hypotheses are true, which are typically unknown but can therefore be assumed to be equal under Laplace's principle of indifference [9], ii) the relative costs of Type I and II errors, which are also often unknown but are assumed equal when the goal is to minimize combined probabilities of Type I and Type II error, iii) the critical effect size, iv) the sample size of the study, and v) the variability in the data. The remainder of this paper describes how the optimal α approach deals with prior probabilities, costs of Type I and II errors, and critical effect size estimation, provides a list of steps required to calculate an optimal α for any NHST. Case studies of the optimal α approach applied to previously published data [10]-[12] are also provided.

Optimal α - the basic approach

The combined probability of making a Type I error or a relevant Type II error for a particular study ( ω) is the average of α and β if we assume that the prior probabilities of the alternate and the null hypotheses to be equal (Equation 1). (1) In the absence of reliable information concerning prior probabilities (as is often the case in science), the assumption of equal prior probabilities is the only rational assumption [9]. However, if the probabilities of the alternate and null hypotheses are known and unequal, the probabilities of Type I and Type II error should still be averaged as in Equation 1, but with α and β each multiplied by their relative prior probabilities of the null and alternate hypotheses being true, respectively.

Due to the nonlinear but negative and monotonic nature of the relationship between α and β ( Figure 1), it is possible, through iterative examination of ω over a range of α values, to identify a unique combination of α and β that minimizes the combined probability of Type I and Type II error ( ω) for a desired critical effect size, in a study with a given sample size ( Figure 2).

α and ω (the average of Type I and Type II error) for independent, 2-tailed, 2-sample t-tests (n ₁ = n ₂). Data are for 3 (dotted line), 10 (solid line), and 30 (double line) samples per group, with critical effect sizes of 1 SD of either group. Drop lines indicate the minimum average of Type I and Type II error and its associated value of α.

Incorporating unequal costs of Type I and Type II errors

Although minimizing the combined probabilities of Type I and Type II errors is likely the main goal of statistical testing for situations of pure scientific inquiry, research in the applied sciences may be more interested in statistical testing for the purpose of minimizing the overall costs of error ( Figure 3). Overall costs of error can be minimized by choosing the α level associated with the minimum average of α and β weighted by the relative costs of Type I and Type II errors, ω c (Equation 2). (2) Where C I/II is the relative cost of Type I/Type II errors. In some cases it may be clear that one type of error is more serious than the other, either in terms of financial cost (e.g. when looking at the effects of a particular level of catch on an economically important fish population), human health (e.g. when screening for side effects of cancer treatment drugs), or research progress (e.g. any 'basic' science question). In these cases, optimal α level should be set to minimize the probability-weighted average of the relative costs of Type I and Type II errors (or equivalently, minimizing the cost-weighted average of α and β) instead of minimizing the combined probabilities Type I and Type II error. Setting decision-making thresholds in a way that accounts for the relative costs of Type I and Type II errors is an area of enquiry that has been largely unexplored in many scientific fields and it is particularly relevant to research where the costs of Type I and II errors are seen as potentially 'estimable' (e.g. environmental effects monitoring, pharmaceutical testing, or disease treatment efficacy). Minimizing overall cost of error implies that while making conclusions based on evidence is still a priority, some of the overall confidence in the conclusion should be sacrificed to make it more likely that when errors do occur, they are the least costly type of error (see Field et al. [13] for a discussion of this technique for environmental monitoring and management). If there are known financial costs associated with Type I and Type II errors, then calculating the relative cost of Type I/Type II error is simple (divide the cost of Type I error by the cost of Type II error), but we acknowledge that financial costs associated with one or both types of error are often unknown. In some cases it maybe be possible for consensus about the relative seriousness (i.e. cost) of Type I/Type II errors to be reached among stakeholders, and where dollar values cannot be used to quantify relative cost of error (e.g. an agreement that the consequences of Type I error are twice as serious as the consequences of Type II error). Methods for quantifying the relative costs of Type I/Type II error will therefore be study specific and will not be discussed further here, although we hope this work will generate more exploration of the relative costs of error in scientific research. Our opinion is that unless there is a strong justification for unequal costs of Type I and Type II errors, statistical testing in science should remain unbiased by costs of error and Type I and Type II errors should be treated as equally serious, allowing for the minimization of combined probabilities of Type I and Type II error.

The combined probabilities of Type I and Type II error, ω (a), and the cost-weighted probability of errors, ω c (b). The α level at i) minimizes average error (assuming a Type I/Type II error cost ratio of 1), while the α level at ii) minimizes the cost-weighted probability of errors at a Type I/Type II error cost ratio of 4.

Determining a critical effect size

A critical effect size is the magnitude of effect in the response variable that would be considered important to detect if it exists, and can be determined prior to conducting an NHST. All studies ought to include some discussion of what size of an effect would constitute a meaningful difference from the null hypothesis [14]. It has been argued that researchers should be far more interested in the size of the effect observed than in its statistical significance [15] and we would agree that effect sizes are often not given enough attention in science. There is a vast body of literature that argues for the use of an effect size statistic [16], and ∼60 different effect sizes measures have been described [17]. Calculating β requires an explicit decision about the 'critical' effect size that a researcher would want to be able to detect, if real, and this must be decided upon, a priori. We do not advocate a researcher use the observed effect size in a study to determine the optimal α and β, this is acknowledged as a misleading, and completely confounded use of power analysis [18]. Many researchers have, and will continue to, struggle with setting a critical effect size because in basic science (and often in applied science) there is no completely objective rationale for choosing a critical effect size, and there is often no consensus among researchers concerning what magnitude of effect would be considered meaningful. Nonetheless, most researchers do have opinions about what effect sizes would be too small to be considered significant and what effect sizes would be too large to be considered non-significant. Devising a study-specific critical effect size and incorporating it into the calculation of α enables researchers to be transparent about these opinions and lowers the probability that the choice of α does not result in relevant effect sizes leading to 'non-significant' conclusions and irrelevant effect sizes leading to 'significant' conclusions. If there is no clear rationale for a specific effect size, multiple critical effect sizes could be used and results could be presented for 'small', 'intermediate' and 'large' critical effect sizes.

Sample size and variability

This paper describes how to determine the optimal α after data have been collected, such that the sample size and estimated population variability are inherent and immutable characteristics of the experimental design. Contrary to the case with critical effect size, we think that it is most appropriate to use the observed variability and sample size of your data set, in order to yield the best estimate of power. This would be an appropriate and informative use of a power analysis; using the achieved sample size, a priori critical effect size and estimated variability, in order to most accurately determine Type II error probability. The optimal α can also play an important role in experimental design if used before data are collected (i.e. to determine the minimum sample size required to achieve a desired overall probability of error), in much the same way that one would use a prospective power analysis to design an experiment.

Results and Discussion

To provide examples of how the optimal α approach can be applied for actual scientific research, we decided to re-analyse the results of two simple, typical null hypothesis significance tests from high-profile journals and one null hypothesis significance test used as an example in a statistics textbook. These examples will show how the optimal α approach affects data interpretation for different types of statistical tests in different fields, under circumstances of high and low statistical power.

Case study 1: t-test with low power

First, consider a study comparing mountain yellow-legged frog ( Rana muscosa) tadpole densities between naturally fishless lakes and lakes that had undergone complete removal of previously introduced rainbow trout ( Oncorynchus mykiss) and brook trout ( Salvelinus fontinalis) three years before sampling the tadpole population [10]. One of the study objectives was to test whether the effects of previously introduced fish on amphibian abundance were still present 3 years after the removal of the introduced species. The author tested the hypotheses, H O: there is no difference in mean number of tadpoles/10 m of shoreline between naturally fishless and fish removal lakes ( μ ₁ =  μ ₂), and H A: there is a difference in mean number of tadpoles/10 m of shoreline between naturally fishless and fish removal lakes ( μ ₁≠ μ ₂). The number of tadpoles/10 m of shoreline was measured in 8 naturally fishless lakes and 3 lakes that had undergone removal of previously introduced fish (i.e. n ₁ = 8 and n ₂ = 3). The mean numbers of tadpoles/10 m of shoreline observed were 29.62 and 10.1 at fishless control lakes and fish removal lakes, respectively, with an observed pooled standard deviation of 17.27 tadpoles/10 m of shoreline. The author performed an independent, 2-sample, 2-tailed t-test on these data, reported a p-value of 0.14, and considered the result non-significant, presumably using the standard α of 0.05 (although their choice of α was not explicitly stated).

The conclusion of the t-test depends on the method used to choose α and the desired critical effect size ( Table 1). As there was no justification for unequal prior probabilities of hypotheses or costs of Type I and Type II errors apparent in the introduction of the paper, we assumed equal prior probabilities and costs of errors. In the absence of a strong rationale for any particular critical effect size, we chose to calculate optimal α levels for 3 different potential critical effect sizes; an effect size we consider to be 'large', representing a difference equal to 1.5 σ or a ±90% difference from the control mean, and 2 smaller effect sizes representing differences equal to 1 σ and 0.5 σ or ±60% and ±30% difference from the control mean, respectively. Using the traditional α of 0.05, the averages of Type I and Type II error at the critical effect size were 0.272, 0.394, and 0.474, for the large, medium and small effect sizes, respectively. Using the 'optimal α' approach, the averages of Type I and Type II errors were smaller, at 0.202, 0.319, and 0.443, for the large, medium and small effect sizes, respectively. The associated optimal α levels were 0.191, 0.266, and 0.323, so using optimal α has increased the probability of making a Type I error in all cases but dramatically lowered the probability of making Type II errors. In fact, for α = 0.05, using a small or medium critical effect size resulted in a >75% probability of failing to detect a true difference. Even when the critical effect size was ±90% difference from the control mean there was almost a 50% chance of missing a real effect of that size. This is a special concern when the major conclusion is one of 'no effect' as was the case in this paper. Using the optimal α method, we would conclude that the experimental design was not appropriate for detecting a small effect size - the minimum chance of making a mistake using optimal α is 44.3%, which is not substantially different than a coin toss (note: the probability of making a mistake using α = 0.05 was 47.4%). However, the minimum probability of making an error was 31.9% for a medium effect size and 20.2% for a large effect size and in both cases using the optimal α approach would have resulted in reaching the opposite conclusion (i.e. there was a significant difference in tadpole density between fishless lakes and fish-removed lakes). So, if the objective is to minimize the probability of making an error (Type I or II) - and there is rarely, if ever, a different and rational objective - then the author made the wrong conclusion regardless of his choice of effect size, and should have concluded that there was an insufficient number of samples to be able to make any strong conclusion, but that there was better evidence to for a difference between lake types than for no difference between lake types.

Case study 2: regression with high power

Now consider a study examining expression of a priori defined gene sets within human diabetic muscle tissue to determine whether there are sets of genes whose expression is correlated with insulin resistance and aerobic capacity [11]. One of the gene sets examined was a co-regulated set of genes involved in oxidative phosphorylation (OXPHOS-CR), and the authors wanted to test whether there was a relationship between the mean expression of this gene set in the muscle tissue of individuals and the total-body aerobic capacity of individuals (VO2max). The authors tested the hypotheses H O: there is no relationship between mean expression of OXPHOS-CR genes and VO2max of individuals, and H A: there is a relationship between mean expression of OXPHOS-CR genes and VO2max of individuals. The mean expression of OXPHOS-CR genes and the VO2max was measured in 43 age-matched male individuals with different levels of glucose tolerance (i.e. N = 43). The authors performed a simple linear regression analysis on the data, using the clinical variable (VO2max) as the dependent variable and mean gene expression as the predictor variable. They found that 22% of the variability in VO2max could be explained by mean OXPHOS-CR gene expression (R ² adj = 0.22), and the relationship had a p-value of 0.0012, which was considered significant by the authors, using α  = 0.05.

Despite the small p-value (0.0012), the significance of the regression depends on the method used to choose α and the desired critical effect size ( Table 2). As there was no justification for unequal prior probabilities of hypotheses or costs of Type I and Type II errors apparent in the introduction of the paper, we assumed equal prior probabilities and costs of errors. In the absence of a strong rationale for any particular critical effect size, we chose to calculate optimal α levels for 3 different potential critical effect sizes; an effect size we consider to be 'large' (R ²≥0.75), where the independent variable explains 75% of the variability in the dependent variable (representing a strong relationship between the dependent and independent variables), and 2 smaller effect sizes (R ²≥0.5 and R ²≥0.25) where 50% and 25% of the variability in the dependent variable could be explained by the variability in the independent variable (representing weaker relationships between the dependent and independent variables). Using the traditional α of 0.05, the averages of Type I and Type II error at the critical effect sizes were 0.0250, 0.0251, and 0.0568, for the large, medium and small effect sizes, respectively. Using the optimal α approach, the averages of Type I and Type II errors were smaller, (0.0000383, 0.00381 and 0.0567, for the large, medium and small effect sizes, respectively) and had optimal α levels of 0.0000286, 0.00378 and 0.0531 and for the large, medium and small critical effect sizes, respectively. The conclusions reached using the optimal α approach are consistent with the conclusions reached by the authors using α = 0.05, except in the case of the large critical effect size, which represents a situation where the authors would only be interested in detecting a strong relationship between the dependent and independent variables. If detecting only strong relationships between variables is important then for 43 samples, α can be set at a very low level and there will still be very high statistical power to detect a strong relationship. The use of α = 0.05 for this test in cases where only strong relationships would be important to detect if they exist results in an unnecessarily high level of Type II error and keeps the average of α and β unnecessarily high - holding α at 0.05 makes 0.025 the smallest possible average of α and beta, when beta approaches 0. If relationships as weak as R ² = 0.25 are important to be able to detect for this test, then the optimal α level is close to α = 0.05, and results in both methods concluding that the observed relationship of R ² = 0.22 is significantly different from no relationship. It would be unusual to set a critical effect size at a value as large as R ² = 0.75 without a very strong rationale and so optimal α would have lead to similar conclusions as using traditional α  = 0.05 in most cases.

Case study 3: analysis of variance with unequal cost ratios

Third, consider a study examining differences in crop yields among three soil types, sand, clay and loam [12]. The authors tested the hypotheses Ho: there are no differences in mean crop yield among the three soil types ( μ1 =  μ2 =  μ3), and Ha: there are differences in mean crop yield among the three soil types ( μ1≠ μ2≠ μ3). Crop yields were measured in 10 randomly selected fields for each of the 3 soil types (i.e. N = 30, k = 3). The mean observed crop yields were 9.9, 11.5 and 14.3 units for sand, clay and loam, respectively, and the observed within-group pooled standard deviation was 3.4. The researchers performed a one-way ANOVA on these data, and reported a p-value of 0.02495, which would be considered significant at α  = 0.05.

The significance of these data depend on the method used to choose α, the critical effect size and relative seriousness of Type I vs. Type II errors ( Table 3). For simplicity, we have assumed that the researchers would consider it important to be able to detect a standard deviation among group means that is at least as large as the standard deviation within-groups (i.e. critical effect size of SD among groups/SD within groups = 1). We assumed equal prior probabilities of null and alternate hypotheses and examined the influence of 3 different Type I/Type II error cost ratios, a situation where the cost of a Type I error is 4 times as serious as the cost of a Type II error; a situation where the relative cost of Type I and Type II error are equal, and a situation where the cost of a Type I error is 1/4 the cost of a Type II error. Using the traditional α of 0.05, β and the average of the probabilities of Type I and Type II error remain constant for the different Type I/Type II error cost ratios. This is not ideal because there is no logical rationale for maintaining the same willingness to make a Type I error as the relative costs of Type I and II errors change. However, using optimal α results in α, β and the cost-weighted average of the probabilities of Type I and Type II error changing under different cost ratio scenarios. For the critical effect size we assumed in this example, the optimal α results in the test being considered non-significant (contrary to the outcome using α = 0.05) except for the case where the cost of Type I error is considered to be one quarter the cost of Type II error. When costs of Type I and Type II error are considered equal in order to minimize the average of both Type I and Type II error, the optimal α is 0.0142, making the optimal conclusion 'non-significant' for this critical effect, with 98.8% power. For each scenario, the conclusion reached using an optimal α was most appropriate for the data, given the differing relative costs of Type I and Type II error.

Conclusion

There is a growing consensus that α = 0.05 is not an ideal method for statistical decision-making, and we have developed an approach to setting α that improves our ability to reach appropriate conclusions. Setting an optimal α does require the careful consideration of critical effect size, costs of Type I and II error and prior probabilities but these characteristics of any study always warrant consideration. Setting α at any particular value (including using α = 0.05) implies specific critical effect sizes and relative costs of Type I and II errors. The only difference between using an optimal α or a standard α is whether critical effect sizes and relative costs are thoughtfully considered and stated, or implied and unstated. The standard α approach has, therefore, not served science well; resulting in mistaken conclusions more often than necessary and allowing scientists to have the standard α level arbitrarily determine the size of effect that they are likely to consider significant, if the effect is real, and to have this arbitrary α level determine the relative chance of Type I vs. Type II error for meaningful potential effect sizes. We have described a rigorous approach to setting α that addresses both of these important (albeit, difficult) decisions and has an explicit and defensible objective, minimizing the combined probabilities or costs of making errors, and we recommend that this approach be applied in any field where null hypothesis significance tests are being used. More research is needed concerning how to estimate and properly incorporate appropriate critical effect sizes and relative costs of Type I and II errors into decision-making thresholds for pure and applied research questions. We are optimistic that the use of the optimal α approach will spur research in these areas and hope that these research gaps do not prevent the implementation of a technique that, even in its most basic form, offers strong improvements in probability of errors and decision-making transparency, over the arbitrary standard α approach. We acknowledge that the optimal α approach to null hypothesis significance testing is not without drawbacks and shortcomings, the traditional approach of setting α = 0.05 has these same drawbacks and shortcomings, but does a much worse job of addressing them. When the weaknesses of our approach are weighed against the weaknesses of the traditional α  = 0.05 approach, it is impossible to ignore that our approach is a dramatic improvement, even if it is not perfect. Our goal is to improve upon an obviously flawed hypothesis-testing system while working under the constraints of that system, because we acknowledge that researchers are not very likely to abandon an approach that is so easy-to-use and so widely understood.

Analysis

How to calculate the optimal α (post data collection, pre-NHST)

1. Determine critical effect size.

This must be determined a priori, and is the magnitude of the effect that would be considered important to detect, if real. The type of effect size will be dependent on the type of NHST to be conducted. We suggest a value that is meaningful for the study, and is based on knowledge of the system or what other studies have observed to be important in your system. For more information on choosing and calculating a critical effect size, we recommend the reviews [14], . If you choose an absolute (i.e. unstandardized) critical effect size, a measure of variability must also be provided.

2. Choose whether to minimize combined probabilities of Type I and Type II error (ω) or relative cost of errors (ωc).

If the average of the probabilities of Type I and Type II error is to be minimized, then the relative costs of Type I and Type II errors can be ignored and Equation 1 should be used. If cost of errors is to be minimized, then an estimate of the Type I/Type II error cost ratio is needed and Equation 2 should be used. Equations can also be weighted by the prior probabilities of null and alternate hypotheses at this stage, for special cases where prior probabilities are known with some degree of confidence.

3. Calculate optimal α.

Using the chosen equation of ω (average error) from step 2, calculate ω for a range of α levels. For each α level chosen, the associated β is 1 minus statistical power, calculated using the chosen α level, the study sample size and the critical effect size chosen in step 1. Look at the ω levels calculated for the range of α levels and find the value for α that results in the lowest resulting ω. This is the optimal α that minimizes the probability/cost of making a wrong conclusion. We have developed R code that will conduct this iterative process of determining the optimal α through examination of ω over a range of α levels ( Text S1, Text S2, Text S3).

4. Report all of the following values for each NHST.

Sample size, chosen critical effect size(s), chosen relative cost of Type I to Type II error (if applicable), optimal α, optimal β, and ω (average of Type I and Type II error).

Supporting Information

R code to calculate an optimal α for one-sample, two-sample, or paired t-tests for one or two tailed hypotheses is provided as supplementary information. This will perform the iterative optimization steps for choosing an optimal α based on sample size(s) and Cohen's d critical effect size provided by the user.
https://doi.org/10.1371/journal.pone.0032734.s001
(TXT)

R code to calculate an optimal α for two-tailed simple linear correlation and regression tests is provided as supplementary information. This will perform the iterative optimization steps for choosing an optimal α based on sample size and correlation coefficient critical effect size provided by the user.
https://doi.org/10.1371/journal.pone.0032734.s002
(TXT)

R code to calculate an optimal α for ANOVA is provided as supplementary information. This will perform the iterative optimization steps for choosing an optimal α based on the numerator degrees of freedom, denominator degrees of freedom and Cohen's f ² critical effect size provided by the user.
https://doi.org/10.1371/journal.pone.0032734.s003
(TXT)

Acknowledgments

This work arose out of conversations among the researchers in the Long-term Experimental Wetlands Area (LEWA), working to improve understanding of community-level conservation threats to wetland ecosystems. These ideas benefitted directly from discussions with Thijs Bosker, Kelly Munkittrick, Rémy Rochette, and many attendees of the 2010 Aquatic Toxicity Workshop in Toronto, Ontario, Canada. KM and RR provided comments on an earlier draft.

Author Contributions

Conceived and designed the experiments: JFM LFB CBE JEH. Performed the experiments: JFM LFB CBE JEH. Analyzed the data: JFM LFB CBE JEH. Contributed reagents/materials/analysis tools: JFM LFB. Wrote the paper: JFM LFB CBE JEH.

References

1. 6. Shrader-Frechette KS (1991) Risk and rationality: Philosophical foundations for populist reforms. Berkeley: University of California Press.

2. 12. Crawley MJ (2007) The R Book. Chichester: John Wiley and Sons.

# data[].attributes.attachment_link.title
Setting an Optimal α That Minimizes Errors in Null Hypothesis Significance Tests

# data[].attributes.attachment_link.description
Null hypothesis significance testing has been under attack in recent years, partly owing to the arbitrary nature of setting α (the decision-making threshold and probability of Type I error) at a constant value, usually 0.05.

# data[].attributes.body
Reimagining Alzheimer's Disease—An Age-Based Hypothesis

Karl Herrup

Journal of Neuroscience 15 December 2010, 30 (50) 16755-16762; DOI: https://doi.org/10.1523/JNEUROSCI.4521-10.2010

# data[].attributes.attachment_link.description
The historical roots of Alzheimer's disease provide a sound conceptual basis for linking the behavioral and neurological symptoms of the disease with the frequently associated pathology of amyloid plaques and neurofibrillary tangles. Out of these roots has grown the "amyloid cascade hypothesis"-a vision of the etiology of Alzheimer's that has spurred the discovery of many important insights into the neurobiology of the disease.

# data[].attributes.headline
7.An article which reports both qualitative and quantitative data

# data[].attributes.subject
7.An article which reports both qualitative and quantitative data

# data[].attributes.body
Methodological Reporting in Qualitative, Quantitative, and Mixed Methods Health Services Research Articles

Jennifer P Wisdom, Ph.D., M.P.H., Mary A Cavaleri, Ph.D., L.C.S.W., Anthony J Onwuegbuzie, Ph.D., and Carla A Green, Ph.D., M.P.H.

# data[].attributes.attachment_link.readable_text
Jennifer P Wisdom, Ph.D., M.P.H., Mary A Cavaleri, Ph.D., L.C.S.W., Anthony J Onwuegbuzie, Ph.D., and Carla A Green, Ph.D., M.P.H.

Abstract

Objectives

Methodologically sound mixed methods research can improve our understanding of health services by providing a more comprehensive picture of health services than either method can alone. This study describes the frequency of mixed methods in published health services research and compares the presence of methodological components indicative of rigorous approaches across mixed methods, qualitative, and quantitative articles.

Data Sources

All empirical articles ( n = 1,651) published between 2003 and 2007 from four top-ranked health services journals.

Study Design

All mixed methods articles ( n = 47) and random samples of qualitative and quantitative articles were evaluated to identify reporting of key components indicating rigor for each method, based on accepted standards for evaluating the quality of research reports (e.g., use of p-values in quantitative reports, description of context in qualitative reports, and integration in mixed method reports). We used chi-square tests to evaluate differences between article types for each component.

Principal Findings

Mixed methods articles comprised 2.85 percent ( n = 47) of empirical articles, quantitative articles 90.98 percent ( n = 1,502), and qualitative articles 6.18 percent ( n = 102). There was a statistically significant difference (χ ²(1) = 12.20, p = .0005, Cramer's V = 0.09, odds ratio = 1.49 [95% confidence interval = 1,27, 1.74]) in the proportion of quantitative methodological components present in mixed methods compared to quantitative papers (21.94 versus 47.07 percent, respectively) but no statistically significant difference (χ ²(1) = 0.02, p = .89, Cramer's V = 0.01) in the proportion of qualitative methodological components in mixed methods compared to qualitative papers (21.34 versus 25.47 percent, respectively).

Conclusion

Few published health services research articles use mixed methods. The frequency of key methodological components is variable. Suggestions are provided to increase the transparency of mixed methods studies and the presence of key methodological components in published reports.

Keywords: Health services, research methodology, mixed methods, qualitative methods

As the health services research field continues to evolve, so too does its methods. Mixed methods research capitalizes on the strengths of both qualitative and quantitative methodologies by combining approaches in a single research study to increase the breadth and depth of understanding ( Johnson, Onwuegbuzie, and Turner 2007). Mixed methods can be a better approach to research than either quantitative-only or qualitative-only methods when a single data source is not sufficient to understand the topic, when results need additional explanation, exploratory findings need to be generalized, or when the complexity of research objectives are best addressed with multiple phases or types of data ( Brannen 1992; Creswell and Plano Clark 2011). Rigorous mixed methods approaches require that individual components (qualitative or quantitative) adhere to their respective established standards ( Curry, Nembhard, and Bradley 2009; Creswell and Plano Clark 2011). Despite recent guidelines on frameworks for conducting mixed methods research (e.g., Curry, Nembhard, and Bradley 2009; Creswell and Plano Clark 2011), a critical challenge has been ensuring that reports from mixed methods studies transparently discuss the methodological components integral to the conduct of the studies. Health services researchers and reviewers need clear guidelines regarding research methodology, including methodological components that should be expected in mixed methods papers to indicate that they are sufficiently rigorous.

Mixed Methods in Health Services Research

Health services research is the study of how social factors, financing systems, organizational structures and processes, health technologies, and personal behaviors affect access to health care, the quality and cost of health care, and ultimately, health and well-being ( Lohr and Steinwachs 2002). As a result of the breadth of topics addressed, health services research draws upon methods and concepts from many fields, including medicine, epidemiological and economic studies, and the evaluation of services and interventions ( Field, Tranquada, and Feasley 1995). Health services researchers increasingly work in interdisciplinary partnerships (e.g., Aboelela et al. 2007) and use innovative methods, including mixed methods, to more fully understand health services phenomena. Mixed methods approaches are also consistent with suggestions to extend scientific and contextual health knowledge beyond randomized trials ( Berwick 2005).

Mixed methods research capitalizes on the strengths of both qualitative and quantitative methodology by combining both components in a single research study to increase breadth and depth of understanding ( Johnson, Onwuegbuzie, and Turner 2007). Qualitative and quantitative methods can be integrated for different purposes to provide a more comprehensive picture of health services than either method can alone. Mixed methods are appropriate in the following situations: (1) when researchers would like to converge different methods or use one method to corroborate the findings from another about a single phenomenon (triangulation); (2) when researchers would like to use one method to elaborate, illustrate, enhance, or clarify the results from another method (complementarity); (3) when researchers would like to use results from one method to inform another method, such as in creating a measure (development); (4) when researchers would like to use one method to discover paradoxes and contradictions in findings from another method that can suggest reframing research questions (initiation); and (5) when researchers seek to expand the breadth and depth of the study by using different methods for different research components (expansion) ( Greene, Caracelli, and Graham 1989). Bryman (2006) modified and expanded this list to add that mixed methods can also be useful in obtaining diversity of views, illustrating concepts, and developing instruments.

Quantitative and qualitative research can be distinguished by the philosophical assumptions brought to the study (e.g., deductive versus inductive), the types of research strategies (e.g., experiments versus case studies), and the specific research methods used in the study (e.g., structured survey versus observation) ( Creswell 2008). Qualitative health services research, for example, is a method in which the researcher collects textual material derived from speech or observation and attempts to understand the phenomenon of interest in terms of the meanings people bring to them ( Denzin and Lincoln 1994; Shortell 1999; Giacomini and Cook for the Evidence-Based Medicine Working Group 2000; Malterud 2001; Bradley, Curry, and Devers 2007). Certain characteristics are typical of qualitative research, including a naturalistic setting (as opposed to a laboratory), a focus on participants' perspectives and their meaning, the outcome as a process rather than a product, and data collected as words or images ( Padgett 2008).

Guidelines for Conducting Mixed Methods Research

The National Institutes of Health noted the need for rigor in combining qualitative and quantitative methods to study complex health issues in their recent publication, Best Practices for Mixed Methods in Health Sciences ( Creswell, Klassen, Plano Clark, and Smith for the Office of Behavioral and Social Sciences Research 2011). There are several frameworks to guide the rigorous conduct and evaluation of mixed methods research ( Collins, Onwuegbuzie, and Sutton 2006; Curry, Nembhard, and Bradley 2009; Tashakkori and Teddlie 2010; Creswell and Plano Clark 2011). Collectively, these frameworks recommend that the conduct of mixed method studies-and reports of mixed method research, including peer-reviewed publication-demonstrates explicit rationales for all decisions regarding study design, including the purpose of including both qualitative and quantitative methods. They specifically advise that each component (qualitative or quantitative) should be conducted with a level of rigor in accordance with established principles in its field, and that researchers be transparent in methodological reporting. For example, sampling design should be specified as identical, parallel, nested, or mixed ( Onwuegbuzie and Collins 2007); the level of mixing methods (fully versus partially) should be described, as should time orientation (sequential or concurrent components of research) and emphasis (equal importance of methodological approaches or one more dominant) ( Leech and Onwuegbuzie 2009).

Conducting and evaluating mixed methods research have unique methodological challenges, particularly related to rigor. Quantitative studies typically rely on quality criteria such as internal validity, generalizability, and reliability ( Campbell 1957; Campbell and Stanley 1963; Messick 1989, 1995; Onwuegbuzie and Daniel 2002, 2004; Onwuegbuzie 2003), whereas qualitative studies have roughly comparable quality criteria of credibility, transferability, and dependability ( Lincoln and Guba 1985; Guba and Lincoln 1989; Miles and Huberman 1994; Maxwell 2005; Pope and Mays 2006). For example, questions asked when evaluating a qualitative study might include the following: "Were participants relevant to the research question and was their selection well reasoned?" and "Was the data collection comprehensive enough to support rich and robust descriptions of the observed events?" ( Giacomini and Cook for the Evidence-Based Medicine Working Group 2000). In addition to determining whether methodological approaches unique to qualitative or quantitative research were employed, an evaluation of a mixed methods study should assess aspects unique to mixed methods, such as how multiple components are integrated and how consistency and discrepancy between findings from each method are managed ( Sale and Brazil 2004; O'Cathain, Murphy, and Nicholl 2007). Qualitative, quantitative, and mixed methodologists agree that study procedures should be reported transparently, including sufficient detail to allow the reader to make inferences about study quality ( Lincoln and Guba 1985; Giacomini and Cook for the Evidence-based Medicine Working Group 2000; O'Cathain, Murphy, and Nicholl 2007; Armstrong et al. 2008; Creswell 2008; Curry, Nembhard, and Bradley 2009; Leech et al. 2009; Teddlie and Tashakkori 2009).

Several researchers have proposed specific techniques to assess the overall methodology of mixed methods research and assess the methodological components of the qualitative, quantitative, and mixed portions of the studies (e.g., Pluye et al. 2009; O'Cathain 2010; Tashakkori and Teddlie 2010; Creswell and Plano Clark 2011; Leech, Onwuegbuzie, and Combs 2011). For example, O'Cathain (2010) assessed quality of mixed methods research by evaluating transparency and clarity in reporting planning, design, data, interpretive rigor, inference transferability, reporting quality, synthesizability, and utility. Others have suggested alternative methods for assessing quality, but criteria often are not elucidated or are vague. Further, those frameworks typically address quality of the study design as opposed to the characteristics provided in the published article. By contrast, Sale and Brazil (2004) proposed a structured framework for the evaluation of mixed methods publications by identifying key methodological components that should be included for both qualitative and quantitative portions of studies. Despite these advances, we found few published accounts of the rigor of published mixed methods research. Our article has three specific research questions: (1) How has the frequency of mixed methods studies published in health services journals changed over time? (2) How are mixed methods articles being used to elucidate health services? and (3) To what extent do mixed methods reports differ in methodological content compared to qualitative-only or quantitative-only articles?

Method

This systematic review assessed the frequency of mixed methods publications in top health services research journals and compared the frequency of key methodological components in qualitative, quantitative, and mixed method studies. We first reviewed articles in health services research journals to determine the prevalence of mixed methods designs and the presence of key methodological components. Then, we conducted statistical analyses of trends over time in the frequency of mixed methods articles and in the presence of key methodological components of those articles. Because this was an analysis of published data, no ethical oversight was required.

Identification of Mixed Methods Articles

We examined four journals: Health Affairs, Health Services Research, Medical Care, and Milbank Quarterly, which had 5-year impact factors of 2.94-4.71. Journals were selected by reviewing the Institute for Scientific Information (2007) rankings for the top 10 journals in health care sciences and services. Of these 10, we included all journals that focused generally on health services research and excluded journals with narrower foci ( Value in Health, Journal of Health Economics, Journal of Pain and Symptom Management, Statistical Methods in Medical Research, Quality and Safety in Health Care, and Quality of Life Research). Although 2001 marked a turning point in the proliferation of mixed methods studies published in major electronic bibliographic databases such as PubMED ( Collins, Onwuegbuzie, and Jiao 2007), we chose to examine articles from 2003 to 2007 because 2003 marks publication of the first edition of Tashakkori and Teddlie's landmark Handbook of Mixed Methods in Social and Behavioral Research, which provided the first comprehensive collection of mixed method theory, methodology, and application. Five years represents a sufficient period of time to examine trends of published articles following the publication of a landmark methodological work.

We reviewed empirical articles to determine whether each represented a quantitative, qualitative, or mixed methods study. This entailed using all the information presented in the abstract and the body of the article to identify the research design either as stated or implied by the author(s). We excluded nonempirical articles (book reviews, literature reviews, commentaries and opinion articles, letters to the editor, policy statements) and articles from a special issue of Milbank Quarterly (Volume 83, Number 4) that included only articles published between 1932 and 1998.

We classified articles as quantitative if they included (1) a primary goal of testing theories or hypotheses about relationships between/among variables, or (2) quantitative data and methodology, such as hierarchical linear modeling, multiple regression, or Markov modeling. We classified articles as qualitative if they included either (1) a primary goal of exploring or understanding the meaning ascribed to a specific phenomenon or experience, or (2) qualitative data such as observations, unstructured or semi-structured interviews, or focus group interviews or methodologies such as thematic analysis. Although more complex definitions of mixed method studies exist (e.g., Johnson, Onwuegbuzie, and Turner 2007; Creswell and Plano Clark 2011), we classified articles as mixed methods if they integrated or combined both quantitative and qualitative methods in a single study ( Sale and Brazil 2004). This definition reflects the general definitions of mixed methods and the lack of consensus on a specific definition across all multidisciplinary mixed methods researchers.

We used spreadsheets to track classifications, with cells containing articles' abstracts and our field notes. Two authors read and classified articles in batches of 50 according to type, conferring as needed until agreement was achieved ( n = 300 articles); the remaining articles ( n = 1,351) were each coded by one author. For the few articles for which methodology was ambiguous ( n = 58, 3.5 percent of all empirical articles), classification was resolved in consultation with a third author. Similar methods have been used in other evaluations of mixed methods articles ( Powell et al. 2008).

Assessments of Articles

We identified all mixed methods articles ( n = 47) and equal random samples ( n = 47) of quantitative articles (from 1,502 articles) and qualitative articles (from 102 articles) (total n = 141) in the four journals. Random samples of qualitative and quantitative articles were selected using a random number generator and did not adjust for journal or year. We assessed the frequency of key methodological components reported across articles, then compared rates by article type. The methodological components we focused on were drawn from two conceptual frameworks. The first included Sale and Brazil's (2004) criteria: (1) internal validity for quantitative findings and credibility for qualitative findings, (2) external validity for quantitative findings and transferability or fittingness for qualitative findings, (3) reliability for quantitative findings and dependability for qualitative findings, and (4) objectivity for quantitative findings and confirmability for qualitative findings (specific criteria are listed in ). The second was O'Cathain's transparency criteria for mixed methods studies ( O'Cathain, Murphy, and Nicholl 2007; O'Cathain 2010), which specify that mixed methods studies should state the (1) priority of methods (primarily quantitative, primarily qualitative, or equal priority), (2) purpose of mixing methods (e.g., triangulation, complementarity, initiation, development, or expansion), (3) sequence of methods (qualitative first, quantitative first, or simultaneous), and (4) stage of integration of both types of data (e.g., data collection, analysis, interpretation). We assessed four additional components of mixed methods studies: (1) whether qualitative and quantitative components were integrated, (2) whether limitations of design were detailed, (3) whether areas of consistency between qualitative and quantitative components were elucidated, and (4) whether areas of inconsistency between components were described.

Table 3

We assessed components using categories of 0 (not described), 1 (described), or not applicable (e.g., for criteria referencing control groups in a study that had none, or ethical review for a study with no human subjects data) ( O'Cathain, Murphy, and Nicholl 2007). We identified only whether the study contained or did not contain each methodological component and did not attempt to assess quality or appropriateness of each component within the context of the study. For example, we assessed whether the publication stated that missing data were addressed but not whether the methods to address missing data were the best methods for that particular research design. Similar to initial article classification, two authors read and coded articles to assess presence/absence of each criterion, with any ambiguity resolved in consultation with a third author.

Quantitative Analyses of Trends and Rigor

Once all articles were coded, we conducted a statistical analysis to determine whether there were trends over time in the prevalence of mixed methods articles. To assess this, we used linear regression to test the hypothesis that there would be an increase in the prevalence of the number of mixed methods articles over time. We also conducted chi-square tests to assess differences between mixed methods, qualitative, and quantitative articles on both quantitative and qualitative criteria. We tested whether each criterion was present in the same proportion of quantitative studies as in the quantitative portion of the mixed methods studies and in the same proportion of qualitative studies as in the qualitative portion of the mixed methods studies.

Results

In general, coders could easily categorize the type of study. Challenges arose when transparency about methods was inadequate ( N = 58, 3.5 percent of all empirical articles). For example, some papers indicated that data from interviews were included but did not provide details about who was interviewed, what was asked in the interviews, how the interview data were analyzed, or how the interview data were integrated into the overall study.

Research Question 1: How has the frequency of mixed methods studies published in health services journals changed over time?

presents a summary of the types of articles published in four major health services research journals from 2003 through 2007. Only 2.85 percent ( n = 47) of empirical articles were mixed methods studies; 6.18 percent ( n = 102) of empirical studies represented qualitative research. Quantitative research represented 90.98 percent ( n = 1,502) of empirical articles. The journal containing the highest proportion of empirical studies employing a mixed methods design was Milbank Quarterly (8.33 percent), followed by Health Affairs (6.91 percent), Health Services Research (4.03 percent), and Medical Care (0.78 percent). Chi-square test showed a significant difference in these proportions (χ ² = 34.67, df = 3, p < .0001).

Table 1

To detect temporal trends in the frequency of mixed methods research in the health services literature, articles were collapsed across journal and examined by publication year. presents the frequency of article type for each of the 5 years. All journals combined published an average of 10.8 mixed method articles per year, or 3.27 percent of empirical articles annually. A quadratic trend was seen across the 5 years ( R ² = 0.65), indicating a slight increase in mixed method articles in the first 2 years and then a decrease for the remaining years.

Research Question 2: How are mixed methods articles being used to elucidate health services research?

Mixed methods articles were categorized into four overlapping categories: Articles on organizational and individual decision making processes ( n = 18 studies) combined qualitative interviews with quantitative administrative data analyses to assess decision making about processes or impediments to processes. Examples include a study of formulary adoption decisions ( Dandrove, Hughes, and Shanley 2003) and states' decisions to reduce Medicaid and other public program funding ( Hoadley, Cunningham, and McHugh 2004).

Sixteen articles described outcomes or effects of policies or initiatives by combining administrative health record or performance data with interviews of health administrators, providers, or executives. Examples include papers describing outcomes of pay-for-performance changes to Medicaid ( Felt-Lisk, Gimm, and Peterson 2007; Rosenthal et al. 2007) and hospital patient safety initiatives ( Devers, Pham, and Liu 2004).

Thirteen measurement development articles employed mixed methods to create measurement tools to assess, for example, caregiver burden ( Cousineau et al. 2003), patient activation ( Hibbard et al. 2004), and the development of a Healthcare Effectiveness Data and Information Set (HEDIS) smoking measure ( Pbert et al. 2003). These studies typically examined qualitative data from individual or focus group interviews first to inform creation and testing of a survey.

Articles on experiences and perceptions were the least common category ( n = 8), typically combining surveys and interviews. These included family physicians' perceptions of the effect of medication samples on their prescribing practices ( Hall, Tett, and Nissen 2006); caregivers' experiences of the termination of home health care for stroke patients ( Levine et al. 2006); and consumer enrollment experiences in the Cash and Counseling program ( Schore, Foster, and Phillips 2007).

Only five mixed methods articles (10.64 percent) of the total mixed methods sample used the terms "mixed method" or "multimethod" in the abstract or text, although four articles (8.51 percent) referred to "qualitative and quantitative" data.

Research Question 3: Do mixed methods articles report qualitative and quantitative methodology differently than methodology is reported in qualitative-only or quantitative-only articles?

presents a summary of the frequency of key methodological components present in quantitative articles, qualitative articles, and mixed methods articles (each n = 47). For quantitative methodological components (32 items), mixed methods articles (M = 7.02 [21.94 percent], SD = 6.24) averaged statistically significantly fewer ( t(92) = −4.50, p < .00001, Cohen's d effect size = 0.93) components than did quantitative articles (M = 15.06 [47.07 percent], SD = 10.53). For qualitative methodological components (35 items), mixed methods articles (M = 7.17 [21.34 percent], SD = 6.36) did not average a statistically significantly different proportion of components ( t(92) = −1.10, p = .14, d = 0.23) than did qualitative articles (M = 8.91 [25.47 percent], SD = 8.83). No article met all criteria, and no criterion was met by all articles. For comparative analyses at a statistical significance level of α = 0.05, power to detect a medium difference (Cohen's h = 0.50) and a large difference (Cohen's h = 0.80) was 78 and 99 percent, respectively.

Of quantitative components, mixed methods studies were most likely to describe sources of data and data collection instruments (61.70 percent of studies), state the purpose/objective of the paper (59.57 percent), state the source of subjects (58.70 percent), and define/describe the study population (51.06 percent). Most mixed methods studies did not include control and intervention groups, which excluded related criteria. Quantitative studies tended to contain more key methodological components, with more than 90 percent of studies defining outcome measures (93.48 percent), defining/describing study population (91.49 percent), describing statistical procedures (95.74 percent), and stating hypotheses (97.87 percent). Quantitative studies were more likely than the quantitative portion of mixed methods studies to describe study characteristics (e.g., study design, subject recruitment), identify or control for confounding variables, provide probability values or confidence intervals, state hypotheses, or acknowledge both statistical and clinical significance (see ).

For qualitative methodological components, mixed methods studies were most likely to state the purpose/objective of the paper (72.34 percent), triangulate qualitative sources (e.g., use both individual and focus group interviews; 53.19 percent), and describe data-gathering procedures (53.19 percent). More than 50 percent of qualitative studies triangulated qualitative sources (57.45 percent), stated the purpose/objective of the paper (57.45 percent), and described the study setting (80.43 percent), how the setting was selected (63.04 percent), the participants (55.56 percent), and data-gathering procedures (76.60 percent). Qualitative studies were more likely than the qualitative portions of the mixed methods studies to describe the study setting, justify the sampling strategy, participants, and data-gathering procedures.

For criteria regarding method integration, few authors justified the use of mixed methods or clearly described the priority, purpose, and sequence of methods, and the stage of integration. Most articles, however, integrated qualitative and quantitative components (85.11 percent); examination of articles indicated components were most frequently integrated in the interpretation phase. Across all studies, few articles stated that informed consent was obtained, ethical review was undertaken, or that subjects' confidentiality was protected.

Discussion

Previous reports indicate mixed methods articles comprised <1 percent of empirical health articles examined in 2000 ( McKibbon and Gadd 2004). Since then, however, the National Institutes of Health has increased funding for mixed methods research, with the proportion of funded research projects up to 5 percent of studies in some institutes ( Plano Clark 2010). In the United Kingdom, the proportion of funded research that uses mixed methods is at 17 percent and continuing to increase ( O'Cathain, Murphy, and Nicholl 2007). We found that the use of mixed methods in articles published in top health services research journals was generally consistent between 2003 and 2007 at approximately 3 percent of all empirical articles, lower than would be expected given the complexity and depth of health services research questions for which mixed methods would be appropriate. The presence of key methodological components was variable across type of article, but the quantitative portion of mixed methods articles included consistently fewer methodological components than quantitative-only studies and the qualitative portion of mixed methods articles included about the same proportion of methodological components as qualitative-only articles. Mixed methods articles also generally did not address the priority, purpose, and sequence of methods or the integration of methods as suggested by experts in mixed methods (e.g., Creswell and Tashakkori 2008; O'Cathain 2010; Creswell and Plano Clark 2011).

Key methodological components that cut across qualitative and quantitative methodologies were often missing from mixed methods publications. Descriptions of sample selection and sampling procedures, the study context, and data-gathering procedures are essential aspects of interpreting study findings, and mixed methods studies should not be exempt from these basic research requirements. Many mixed methods studies did not include the level of detail that would likely be required for a qualitative or quantitative paper to be accepted in these high-ranking journals. Further, the studies appeared not to follow available guidance on the structure and components of mixed methods studies that discuss basic quality criteria, data collection strategies, methods of data analysis, procedures for integration of methods, processes of making inferences from text, and recommendations for adequate reporting of results (e.g., Giacomini and Cook for the Evidence-based Medicine Working Group 2000; Curry, Nembhard, and Bradley 2009; O'Cathain 2010; Tashakkori and Teddlie 2010; Creswell and Plano Clark 2011). In some ways this finding is not surprising because guidance on mixed methods standards is still emerging. We expect that the National Institutes of Health publication, Best Practices for Mixed Methods in Health Sciences (Creswell, Klassen, Plano Clark, and Smith for the Office of Behavioral and Social Science Research) will lead to increased standardization of mixed methods approaches.

Although they reported more key methodological components on average than the mixed methods articles, quantitative articles in this analysis had some surprising gaps as well, including low reporting of power analyses, how missing data were addressed, and descriptions of control/comparison groups. It should be noted, however, that quantitative articles with large sample sizes do not necessarily need power analyses. With regard to single-method qualitative articles, low proportions described the study context, coding techniques, or data analysis. Few articles with human subjects involvement included statements that the research was conducted with ethical oversight, promised confidentiality, or obtained consent. These findings suggest that the issue of poor transparency in reporting methodology is not limited to mixed methods studies.

Recommendations for Mixed Methods Reporting

The methodological components reported here are not optimal indicators of the quality of mixed methods publications; an article could conceivably have all of these components and yet still be a poor research study. These components are, however, a useful starting point for a systematic evaluation of the rigor of qualitative and quantitative portions of mixed methods studies. Some journals require inclusion of other criteria (e.g., Consolidated Standards of Reporting Trials 2010) to guide reporting of highly structured methodologies (e.g., randomized clinical trials); it would be useful to examine researchers' and editors' perspectives on the validity of the methodological components in this study for mixed method publications. It is difficult, however, to identify measurable criteria that capture the breadth of study designs in health services. Further, determination of what indicators of rigor would be appropriate might reasonably vary by study design, topic, scope, and even journal, and qualified judgment is required to determine which criteria are appropriate for each study. These findings suggest mixed methods researchers should provide enough detail on methodology and methodological decisions to allow reviewers to judge quality.

Researchers face challenges writing and publishing mixed methods articles, including communicating with diverse audiences who are familiar with only one methodological approach (i.e., quantitative research or qualitative research), determining the most appropriate language and terminology to use, complying with journal word counts, and finding appropriate publishing outlets with reviewers who have expertise in mixed methods research techniques and who are not biased against mixed methods studies ( Leech and Onwuegbuzie 2010; Leech, Onwuegbuzie, and Combs 2011). Our findings suggest that Sale and Brazil's (2004) criteria and existing guidance on conducting mixed methods research (e.g., Collins, Onwuegbuzie, and Sutton 2006; Tashakkori and Teddlie 2010; Creswell and Plano Clark 2011) might be useful frameworks for health services researchers as they work to improve methodological rigor. Journal editors might also encourage the publication of mixed methods projects by (1) publishing guidelines for rigor in mixed methods articles (e.g., Sale and Brazil 2004), (2) identifying experienced reviewers who can provide competent and ethical reviews of mixed methods studies, and (3) requiring transparency of methods for all studies so that (4) rigor and quality can be can be assessed to the same extent they are in quantitative studies. These modifications might require (5) some flexibility in word count or allowance of online appendices to allow mixed methods researchers to describe fully and concisely both qualitative and quantitative components, methods for integrating findings, and appropriate details.

Limitations

In this study, assessment was limited to only published articles. We did not contact authors to determine specific study activities, and studies may have included methodological components (e.g., consenting) not reported in publications. We assessed only whether publications reported the methodological component, but we did not evaluate whether each component was fully and appropriately implemented in the research.

Conclusions

Mixed methods studies have utility in providing a more comprehensive picture of health services than either method can alone. Researchers who use mixed methods techniques should use rigorous methodologies in their mixed methods research designs and explicitly report key methodological components of those designs and methods in published articles. Similarly, journal editors who publish mixed methods research should provide guidance to reviewers of mixed methods articles to assess the quality of manuscripts, and they must be prepared to provide adequate space for authors to report the necessary methodological information. Frameworks are now available to guide both the design and evaluation of mixed methods research studies and published works. Whatever frameworks are used, it is essential that authors who engage in mixed methods research studies meet two primary goals (developed by the American Educational Research Association 2006): Mixed methods researchers should (1) conduct and report research that is warranted or defensible in terms of documenting evidence, substantiating results, and validating conclusions; and (2) ensure that the conduct of research is transparent in terms of clarifying the logic underpinning the inquiry.

Acknowledgments

Joint Acknowledgment/Disclosure Statement: The authors appreciate funding from the National Institute on Drug Abuse (K23 DA020487) and comments and feedback on an earlier draft from the anonymous reviewers, John Creswell, PhD, Alicia O'Cathain, PhD, Hilary Vidair, PhD, Susan Essock, PhD, and Sa Shen, PhD. Portions of this manuscript were presented at the International Mixed Methods Conference in July 2010 in Baltimore, Maryland.

Disclosures: None.

Disclaimers: None.

SUPPORTING INFORMATION

Additional supporting information may be found in the online version of this article:

Appendix SA1: Author Matrix.

Please note: Wiley-Blackwell is not responsible for the content or functionality of any supporting materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.

# data[].attributes.attachment_link.title
Methodological Reporting in Qualitative, Quantitative, and Mixed Methods Health Services Research Articles

# data[].attributes.attachment_link.description
Jennifer P Wisdom, Ph.D., M.P.H., Mary A Cavaleri, Ph.D., L.C.S.W., Anthony J Onwuegbuzie, Ph.D., and Carla A Green, Ph.D., M.P.H. Methodologically sound mixed methods research can improve our understanding of health services by providing a more comprehensive picture of health services than either method can alone.