# data[].attributes.body
-significant does not mean important
-numbers can seem rational, but can sometimes confuse a situation
-you might be judging by extremes not the majority

# data[].attributes.body
You basically have to sum up all research to see where the gaps are. This may include, the author, location, date of study, purpose and research questions. Then you can show this rough a table or map. Then use this table to contribute to your literature review. 🎃😊

# data[].attributes.body
Read specific literature first
Google scholar
Create page on excel to compare literature (search for gaps)
Find out authors that worked specifically on ideas
Work systematically
Map

# data[].attributes.body
Fin

# data[].attributes.body
There are things we know, we know.
There are things we know we dont know.
There are things we dont know, we dont know.
Can create an excel page of articles and use maps and tables to demonstrate geographics and other information in order to find gaps.

# data[].attributes.body
-Go back to question after lit review
-explore a range of opinions and times and critique them (don't just pick literature that agrees with you)
-build review around own question
-be systematic
-make your own points from drawing on lit
-get to know key authors (read other sources from them)
-paper into context... wider to specific
-4 types of lit (theoretical, empirical, policy and practice)
- there are times when it is not interesting, but plod on!
- metholodological lit is separate
-don't bring in new lit during discussion
-question is most important thing. The way you ask it and your methodology is what gives you voice

# data[].attributes.body
•Read Widley
•Explore arguments for and against
•Include own take on findings, critique!
•Is there a Gap?
•Does the same response come up repeatedly?
•Wider context to specific context
•Think carefully about structure
•Explore different time frames
Explore findings in relation to lit. Review. Link back
•4 types of Lit.
•Be systematic
•Critical friends - similar view points
•Is it relevant?

# data[].attributes.body
-Be systematic
-"Swimming pool" put yourself in the middle and look for different themes.
-Don't be tempted to look just for your point of view. Challenge yourself and look for different sources.
-Keep critiquing with your voice supported by evidence.
-Specific context, everything must come back to your question.
-Critical friend
-Claim your findings in regards to literature

# data[].attributes.body
-Using resources from a range of times and views
-Your voice is from your critique
-Your critique stregthens your argument
-Use your literature review in your discussion of findings and do not add anything new in your discussion because you should have already mentioned them
-When you have done general literature review, go back to research questions to construct an argument
-Theoretical, empirical, policy and practice are all types of literature
-Get to know an author if they share your opinion or perspective, then you can use their work again
- Think carefully about structure
-Clear theoretical framework
-Do not only pick supportive literature, you will be penalised
-Show that you can find literature that does not support your argument

# data[].attributes.body
-Your choice of research is key- it gives you a voice.
- include literature that goes against what you're saying- recognise the challenges.
- Once you have found key authors (critical friends) explore them further.
- Put the paper into context: wider-specific-specific question.
- 4 types of literature as a starting point- 1. Theoretical 2. Empirical 3. Policy 4. Practice (tend not to use).
😊

# data[].attributes.attachment_link.readable_text
UNDERSTANDING RESEARCH: What do we actually mean by research and how does it help inform our understanding of things? Understanding what's being said in any new research can be challenging and there are some common mistakes that people make.
Have you ever tried to interpret some new research to work out what the study means in the grand scheme of things?

Well maybe you're smart and didn't make any mistakes - but more likely you're like most humans and accidentally made one of these 10 stuff ups.

1. Wait! That's just one study!

You wouldn't judge all old men based on just Rolf Harris or Nelson Mandela. And so neither should you judge any topic based on just one study.

If you do it deliberately, it's cherry-picking. If you do it by accident, it's an example of the exception fallacy.

The well-worn and thoroughly discredited case of the measles, mumps and rubella ( MMR) vaccine causing autism serves as a great example of both of these.

People who blindly accepted Andrew Wakefield's (now retracted) study - when all the other evidence was to the contrary - fell afoul of the exception fallacy. People who selectively used it to oppose vaccination were cherry-picking.

2. Significant doesn't mean important

Some effects might well be statistically significant, but so tiny as to be useless in practice.

Flickr/Frits Ahlefeldt-Laurvig, CC BY-ND

Associations (like correlations) are great for falling foul of this, especially when studies have huge number of participants. Basically, if you have large numbers of participants in a study, significant associations tend to be plentiful, but not necessarily meaningful.

One example can be seen in a study of 22,000 people that found a significant (p<0.00001) association between people taking aspirin and a reduction in heart attacks, but the size of the result was miniscule.

The difference in the likelihood of heart attacks between those taking aspirin every day and those who weren't was less than 1%. At this effect size - and considering the possible costs associated with taking aspirin - it is dubious whether it is worth taking at all.

3. And effect size doesn't mean useful

We might have a treatment that lowers our risk of a condition by 50%. But if the risk of having that condition was already vanishingly low (say a lifetime risk of 0.002%), then reducing that might be a little pointless.

We can flip this around and use what is called Number Needed to Treat ( NNT).

In normal conditions if two random people out of 100,000 would get that condition during their lifetime, you'd need all 100,000 to take the treatment to reduce that number to one.

4. Are you judging the extremes by the majority?

Biology and medical research are great for reminding us that not all trends are linear.

We all know that people with very high salt intakes have a greater risk of cardio-vascular disease than people with a moderate salt intake.

Flickr/JD Hancock, CC BY

But hey - people with a very low salt intake may also have a high risk of cardio-vascular disease too.

The graph is U shaped, not just a line going straight up. The people at each end of the graph are probably doing different things.

5. Did you maybe even want to find that effect?

Even without trying, we notice and give more credence to information that agrees with views we already hold. We are attuned to seeing and accepting things that confirm what we already know, think and believe.

There are numerous example of this confirmation bias but studies such as this reveal how disturbing the effect can be.

In this case, the more educated people believed a person to be, the lighter they (incorrectly) remembered that person's skin was.

6. Were you tricked by sciencey snake oil?

You won't be surprised to hear that sciencey-sounding stuff is seductive. Hey, even the advertisers like to use our words!

But this is a real effect that clouds our ability to interpret research.

In one study, non-experts found even bad psychological explanations of behaviour more convincing when they were associated with irrelevant neuroscience information. And if you add in a nice-and-shiny fMRI scan, look out!

7. Qualities aren't quantities and quantities aren't qualitites

For some reason, numbers feel more objective than adjectivally-laden descriptions of things. Numbers seem rational, words seem irrational. But sometimes numbers can confuse an issue.

For example, we know people don't enjoy waiting in long queues at the bank. If we want to find out how to improve this, we could be tempted to measure waiting periods and then strive to try and reduce that time.

But in reality you can only reduce the wait time so far. And a purely quantitative approach may miss other possibilities.

If you asked people to describe how waiting made them feel, you might discover it's less about how long it takes, and more about how uncomfortable they are.

8. Models by definition are not perfect representations of reality

A common battle-line between climate change deniers and people who actually understand evidence is the effectiveness and representativeness of climate models.

But we can use much simpler models to look at this. Just take the classic model of an atom. It's frequently represented as a nice stable nucleus in the middle of a number of neatly orbiting electrons.

While this doesn't reflect how an atom actually looks, it serves to explain fundamental aspects of the way atoms and their sub-elements work.

This doesn't mean people haven't had misconceptions about atoms based on this simplified model. But these can be modified with further teaching, study and experience.

9. Context matters

The US president Harry Truman once whinged about all his economists giving advice, but then immediately contradicting that with an " on the other hand " qualification.

Individual scientists - and scientific disciplines - might be great at providing advice from just one frame. But for any complex social, political or personal issue there are often multiple disciplines and multiple points of view to take into account.

To ponder this we can look at bike helmet laws. It's hard to deny that if someone has a bike accident and hits their head, they'll be better off if they're wearing a helmet.

Flickr/Petar, CC BY-NC

But if we are interested in whole-of-society health benefits, there is research suggesting that a subset of the population will choose not to cycle at all if they are legally required to wear a helmet.

Balance this against the number of accidents where a helmet actually makes a difference to the health outcome, and now helmet use may in fact be negatively impacting overall public health.

Valid, reliable research can find that helmet laws are both good and bad for health.

10. And just because it's peer reviewed that doesn't make it right

Peer review is held up as a gold standard in science (and other) research at the highest levels.

But even if we assume that the reviewers made no mistakes or that there were no biases in the publication policies (or that there wasn't any straight out deceit), an article appearing in a peer reviewed publication just means that the research is ready to be put out to the community of relevant experts for challenging, testing, and refining.

It does not mean it's perfect, complete or correct. Peer review is the beginning of a study's active public life, not the culmination.

And finally ...

Research is a human endeavour and as such is subject to all the wonders and horrors of any human endeavour.

Just like in any other aspect of our lives, in the end, we have to make our own decisions. And sorry, appropriate use even of the world's best study does not relieve us of this wonderful and terrible responsibility.

There will always be ambiguities that we have to wade through, so like any other human domain, do the best you can on your own, but if you get stuck, get some guidance directly from, or at least originally via, useful experts.
This article is part of a series on Understanding Research.
Further reading:
Why research beats anecdote in our search for knowledge
Clearing up confusion between correlation and causation
Where's the proof in science? There is none
Positives in negative results: when finding 'nothing' means something
The risks of blowing your own trumpet too soon on research
How to find the knowns and unknowns in any research
How myths and tabloids feed on anomalies in science

# data[].attributes.attachment_link.title
The 10 stuff-ups we all make when interpreting research

# data[].attributes.attachment_link.description
UNDERSTANDING RESEARCH: What do we actually mean by research and how does it help inform our understanding of things? Understanding what's being said in any new research can be challenging and there are some common mistakes that people make. Have you ever tried to interpret some new research to work out what the study means in the grand scheme of things?

# data[].attributes.attachment_link.readable_text
UNDERSTANDING RESEARCH: What do we actually mean by research and how does it help inform our understanding of things? With so much research already published it can be a daunting task for any new researcher to find out what's already done. So how do you find the knowns and unknowns in research?
Have you ever felt overloaded by information? Ever wondered how to make sense of claims and counter-claims about a topic? With so much information out there on many different issues, how is a person new to a subject supposed to find out what is known about a subject?

We often face the challenge of mastering new topics. This is the same for anyone be they students, journalists, activists or even researchers starting or changing research topics.

In a world of increasing information availability, finding ways to make sense of the literature on a topic is important.

We have developed a direct way for people to approach a chosen topic and to drill down through published literature to find out what is known, but also where there are knowledge gaps.

We call it a Systematic Quantitative Literature Review. This fancy term just means counting how many times papers on a chosen topic have been published, where, when and by who, and then thinking about results - what does this mean?

We have produced online support for those wanting to do these Systematic Quantitative Literature Reviews. This includes YouTube videos and papers outlining the method and a growing list of examples of many papers published using this technique.

What's most exciting about this method is that almost anyone can use it - not just students or academic researchers. No specialist equipment is required - just spreadsheet software such as Microsoft's Excel.

And almost any topic can be investigated: from climate change, to vaccination and even searching for better schools. Users can quickly master the technique to wade through claims and counter claims on any subject, discovering what's known and finding the gaps.

So how do you do it?

For a start you'll need to review the published literature on your chosen issue. But which literature and how much do we need to review?

Back in 2002, the then US Defence Secretary Donald Rumsfeld made the now famous quote about known and unknown information when he said:

[...] as we know, there are known knowns; there are things that we know that we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns, the ones we don't know we don't know.

Traditional narrative reviews conducted by experts tend to focus on Rumsfeld's "known knowns". These reviews tell a story about how knowledge has developed, where research has been concentrated and what it has found.

Other methods such as the famous Cochrane Reviews in medical sciences focus on what is known, but also create new knowledge: Rumsfeld's "unknown knowns". They collate data from published studies, reanalyse the data to provide more robust answers to important research questions, and identify knowledge gaps.

Both approaches are quite complex; they require technical knowledge and considerable familiarity with the literature before you can start, and definitely are not for the novice researcher or citizen activist.

There must be an easier way?

We use a third approach, which is potentially more useful for ordinary people wanting to navigate a seemingly endless mass of literature.

Our approach enables users to systematically identify what is known and then using quantitative methods they can assess what is unknown.

Our systematic quantitative reviews involve four steps:

1. First you systematically search the literature for original research (published reports and papers) on a specific topic. This can be done by anyone with an internet connection. Google Scholar has been proven to be a reliable tool to search for published literature. Just enter your keywords on the chosen topic and you're on your way.

2. Next, enter the different reports and studies that you find into a your spreadsheet (such as author, location, date of study, purpose, research questions etc).

3. Then you summarise and evaluate the papers by using tables and maps - showing what is known, but also what is not known. Google Maps can be used to display the countries where most research has been conducted. You can even create tables or graphs from your spreadsheet, showing who is being funded, or what findings are being reported about particular places and topics.

4. Finally, you can report the findings - in a newspaper, newsletter, chatroom, blog or even a scholarly journal.

What to look for

A Systematic Quantitative Literature Review can help answer important questions, such as:

1. who has published on the topic?

2. where has research been conducted?

3. when was most of the research undertaken?

4. what key research questions were addressed, and are any missing?

5. what methods have been used and have any been neglected?

6. what have researchers discovered?

7. are there any limitations (such as problems with methods, analyses and results)?

8. what is still unexplored?

There are some limits to this method. It may not be as appropriate for some areas in humanities, architecture, music, literature or dance-studies. It is important to be consistent and objective in how you search and select documents, and how you classify the data about each document in your database too.

This method can work for a wide range of subjects: from identifying the benefits of community gardens to selecting street trees, from assessing the impacts of tourism on birds and other animals to assessing performance in architecture, to evaluating the risk of cars carrying weeds, to finding out how your European holiday may threaten rare plants.

Using this method, any one from academic researchers to teachers, lawyers, activists and parents can become more knowledgeable on a subject, contribute to knowledge-creation and identify limits to current knowledge.

They can also prioritise where we need to go to gain new knowledge, turning more of Rumsfeld's unknown unknowns into known unknowns.
This article is part of a series on Understanding Research.
Further reading:
Why research beats anecdote in our search for knowledge
Clearing up confusion between correlation and causation
Where's the proof in science? There is none
Positives in negative results: when finding 'nothing' means something
The risks of blowing your own trumpet too soon on research
How myths and tabloids feed on anomalies in science
The 10 stuff-ups we all make when interpreting research

# data[].attributes.attachment_link.title
How to find the knowns and unknowns in any research

# data[].attributes.attachment_link.description
UNDERSTANDING RESEARCH: What do we actually mean by research and how does it help inform our understanding of things? With so much research already published it can be a daunting task for any new researcher to find out what's already done. So how do you find the knowns and unknowns in research?

# data[].attributes.body
Adeela Shafi on the writing process, temptations and the role of literature

# data[].attributes.attachment_link.entities[].name
Teach Toddlers Colors Numbers Preschool Race Cars & Magnet Truck

# data[].attributes.body
Hazel Bryan discusses the role of literature in research, with advice for those starting out.

# data[].attributes.attachment_link.entities[].name
Teach Toddlers Colors Numbers Preschool Race Cars & Magnet Truck