# data[].attributes.attachment_link.description
The Public Engagement Evaluation Toolkit is designed to make it easy to evaluate the design, execution and impact of public engagement with research. Find out how to design an evaluation and discover tools that will enrich your engagement whilst providing you with opportunities to evaluate what works.

# data[].attributes.headline
Diversity and Inclusion Survey Question Guidance

# data[].attributes.subject
Diversity and Inclusion Survey Question Guidance

# data[].attributes.attachment_link.readable_text
This updated guide for 2022 gives recommendations when asking diversity questions. It includes suggestions for questions you could ask and why you might choose to ask them in certain ways. This replaces the DAISY Guidance Version 1 published in January 2020 and has been developed in collaboration between EDIS and Wellcome.
The DAISY Guidance Version 2 is available to read as a PDF here. It is available as a downloadable Word document here.

# data[].attributes.attachment_link.title
Diversity and Inclusion Survey (DAISY) Question Guidance (Version 2) - EDIS

# data[].attributes.attachment_link.description
This updated guide for 2022 gives recommendations when asking diversity questions. It includes suggestions for questions you could ask and why you might choose to ask them in certain ways. This replaces the DAISY Guidance Version 1 published in January 2020 and has been developed in collaboration between EDIS and Wellcome.

# data[].attributes.headline
Making a Difference: A guide to evaluating public participation in central government 

# data[].attributes.subject
Making a Difference: A guide to evaluating public participation in central government 

# data[].attributes.attachment_link.description
This guide sets out how to evaluate public participation in central government. It is intended to help those involved in planning, organising or funding these activities to understand the different factors involved in creating effective public participation. It helps planners set and measure attaina…

# data[].attributes.headline
Auditing Community Participation: An Assessment Handbook

# data[].attributes.subject
Auditing Community Participation: An Assessment Handbook

# data[].attributes.attachment_link.readable_text
A new study exploring the experiences of residents involved in urban regeneration projects suggests that there is still much to learn about involving local people in area regeneration initiatives. The study, by a team from Goldsmith's College, found:

Communities are diverse and local interests may conflict with each other. If the community is seen as homogenous then only the most powerful voices will tend to be heard.

Residents felt there was a gap between the rhetoric that demands community participation in area regeneration programmes and the realities of work on the ground.

The study did find examples of good practice, but residents also had major criticisms to make. Too often, in their view, the mechanisms for effective community involvement had been inadequate, with too little time for effective consultation. Many commented that there had been insufficient support and not enough training (a conclusion shared by many professionals).

There was broad agreement between officers and local community representatives that official funding for bids should be conditional on guarantees that there are adequate resources for community involvement.

The study piloted 'Audit Tools' for area regeneration programmes - which could be used by communities themselves. Community representatives expressed their interest in using these as part of an obligatory system of monitoring community participation.

The researchers conclude that:
- debate should focus upon the most effective forms of dialogue between different structures of governance and different community interests, rather than simply polarising distinctions between 'top-down' and 'bottom-up' approaches;
- mainstream policies and resource allocation decisions need to take account of the lessons of locally based programmes and policies;
- community participation should be subjected to an auditing process that echoes the monitoring of 'hard outputs' and the financial accountability of major public spending programmes.

Background

The study is based on community participants' own views on their experiences of participating in area regeneration programmes, drawn largely from case studies and a series of workshops bringing together community representatives.

The case study areas
The research was based principally around four case study areas: two areas in London characterised by major property development pressures (Greenwich Waterfront and King's Cross), an area characterised by the collapse of coal mining (Barnsley) and one metropolitan area outside London (Aston in Birmingham).

The research highlighted the differences between the case study areas and the diversity and complexity of community interests within each locality.

In Barnsley, the local political culture is dominated by the mining industry and conventional structures of local government: "In Barnsley people always used to say that the council do it for you and the Coal Board do it to you. To a large extent that mentality still prevails." The region also faces massive economic pressures. Nevertheless, major changes which attempt to enhance community participation are being brought forward; for example, the council has created innovative local Community Partnerships, autonomous companies intended to draw together local councillors and community interests.

Regeneration programmes such as City Challenge had been the subject of considerable criticism from local residents in inner city Birmingham. As a result of determined efforts, there were examples of previously excluded minority ethnic groups gaining access to regeneration resources, however. Overall, communities had learned valuable lessons through these experiences, they felt. "Decentralisation has been working better in Aston/Newtown as a result of the experiences, knowledge and skills developed through City Challenge and subsequently via the SRB."

In Greenwich, the model has succeeded in attracting major government funds. However, it has been less successful in reflecting the diversity of local communities: "The partnership board wants one view [to negotiate with]. It's more convenient for them to 'do business' with a unified view, whilst the reality is that there is a diversity of views."

In King's Cross, the overwhelming significance of the parties that own key land sites in setting the agenda for regeneration questions the expectations that can be raised in some programmes: "We have the paradoxical situation that King's Cross is awash with money, there has never been so much money in the area - build anything you like - but there isn't the money to run a service."

Both London case studies revealed a higher level of antagonism between local authorities and local communities than in either the Midlands or in South Yorkshire: "I think that there is a feeling that the money is on one side and you are on the other" (Greenwich). In both Birmingham and London the issue of race and the nature of contemporary racism provide particularly strong challenges to representative participation.

Residents' views

Despite this diversity of background, community representatives shared a number of views on the process. Residents felt that too often regeneration agendas are set in advance of community participation. There were comments about the pressures to respond to official agendas even when these distracted community organisations' attention away from their own agendas. Groups felt that they were being defined as 'acceptable' in terms of how far their agendas coincided with council agendas and/or with private sector interests (such as property development interests). Black/minority ethnic groups were particularly concerned about this.

Residents often considered that they were being put under too much pressure, with insufficient technical support. Programmes were felt to 'make' and conversely to 'break' 'community stars': individuals and groups whose contributions were valued on some occasions and then denigrated on others, depending upon how closely they reflected official agendas:

"The local establishment see professionalisation as a good thing; they like to see individuals becoming effectively full-time unpaid community professionals ... One reason why the senior officers and members like to see the development of individual community reps into community professionals is so they can develop one-to-one relationships, so they have a single person to go to when they want to negotiate with a community. For lower officers, it's more about ... developing local people who can speak their language." Council officer

The report also echoed the conclusions of previous research demonstrating the importance of community owned structures, such as Community Development Trusts, if regeneration is to be sustainable.

Those running small programmes argued for the importance of taking account of the realities of the regional economies in which they were situated, and of feeding the lessons of small area programmes back into regional and national planning and decision-making structures. As one resident explained, the shortage of affordable housing was a key problem in the area, "but that's not an issue for some reason" in terms of wider policy discussions. "Housing," she concluded, "is off the agenda".

In terms of technical support, experiences varied greatly. There were excellent examples of training in some areas - which residents valued - but not in others. Similarly, the impact of community participation varied greatly from place to place. It was clear that over time community activists felt that they had been learning important lessons about how to be most effective in these regeneration partnerships: "People have learnt a lot over the years ... the officers have learnt too."

Both community participants and professionals emphasised the importance of resources for training and community work support being made available, right from the start. Without this, as one professional commented, many groups, including black and minority ethnic groups, "do not even realise that the [partnership] board is giving money out".

Through networking, community representatives were also sharing their understanding of how they could 'make a difference'. But, workshop participants concluded that there were few, if any, ways in which community perspectives on the lessons from one-off projects could be fed back into mainstream policies and spending programmes.

Conclusion

Evidence from the case studies indicates that community representation has been problematic. In the workshops, participants shared local experiences and raised a number of issues about area regeneration policies and practice at regional and national as well as at local levels. They also raised questions about the relationship between small area policies and mainstream policies. The researchers also identified a number of issues which present continuing problems.

Recommendations for improving practice
The researchers conclude that the following could enhance practice:
Possible action by central government:

Provide increased time and resources to enable communities to develop their own agendas.

Build in increased resources for participation and capacity building as part of the requirements for bidding.

Develop a national framework for supporting learning, technical aid and training.

Require that appropriate and effective systems for monitoring and evaluating participation be developed (including processes as well as outputs) before bids can be agreed.

Provide resources to enable community representatives to network regionally and nationally.

Develop structures at both regional and national levels to disseminate findings and recommendations.

One of the purposes of the study was to develop Audit Tools, to provide a mechanism by which communities would be enabled to evaluate the effectiveness of community participation, for themselves. Community participation needs to be evaluated with transparency and rigour. The Audit Tools provide ways of mapping the context for participation, the quality of participation structures, the capacity of partners and communities to participate, and the overall impact of their participation. Regeneration agencies need to ensure that there are effective mechanisms for auditing community participation right from the outset.

Community representatives expressed their interest in using these 'Audit Tools' as part of an obligatory system of monitoring of community participation.
Possible action for government regional offices and Regional Development Agencies and partnerships:

Adopt a consistent approach to evaluating community participation in area regeneration programmes; the Audit Tools (see box) could be one way forward.

Ensure that sources of technical advice, independent consultancy, community support and learning support are available to local communities.

Ensure that there are coherent links between partnerships at neighbourhood, city and sub-regional levels.

Possible action for local authorities and other agencies involved in regeneration partnerships:

Ensure that all bids include appropriate provision for community participation - including independent advice, support and training - and monitor and evaluate performance (again the Audit Tools provide a possible model).

Ensure that practical support is available to enable people to participate, valuing people's time and paying for this where appropriate.

Disseminate examples of good practice, supporting initiatives to enable communities to develop their own projects and their own community-based partnership bids.

Take shared responsibility with Partnership Boards to ensure that the range of community views are effectively being heard, taken into account and incorporated into action and that structures are both representative and democratically accountable.

Co-operate with requirements that the impact of area regeneration on the community sector is effectively monitored and that appropriate action is taken where necessary to ensure the strengthening of the community sector and its independent voice.

Provide support for the community sector to facilitate participatory monitoring and evaluation and to facilitate networking at local and regional levels to share experiences and findings.

Develop strategies to ensure that the changing role of local councillors takes account of these requirements for the community sector.

Possible action for the community sector:

Share responsibility for ensuring that community participation structures are genuinely inclusive, representative and democratically accountable, taking account of minority as well as majority interests.

Share experiences and learning within their own organisations and with organisations and groups with common interests in their areas and beyond.

Share responsibility for participatory monitoring and evaluation, taking account of the wider impact on the community sector.

Strengthen networks between community sector organisations locally and regionally, sharing experiences and building alliances around shared interests and concerns.

About the study

As well as work in the case study areas, the research also involved a survey of 110 regeneration partnerships and programmes and an extensive literature review that drew on the lessons of community participation in 'third world' development programmes as well as 'first world' social welfare programmes. The literature review also took into account the reports of the Social Exclusion Unit's Policy Action Teams and the final report of the National Strategy for Neighbourhood Renewal. The research in the four case study areas was as participatory as possible. Interim findings from the first rounds of interviews were fed back and discussed in focus groups. Representatives from the four case study areas then took part in a joint workshop to consider the draft report.

# data[].attributes.attachment_link.title
Auditing community participation: An assessment handbook

# data[].attributes.attachment_link.description
A new study exploring the experiences of residents involved in urban regeneration projects suggests that there is still much to learn about involving local people in area regeneration initiatives. The study, by a team from Goldsmith's College, found: Communities are diverse and local interests may conflict with each other.

# data[].attributes.headline
Creative & Credible supports arts and health organisations and practitioners to:

# data[].attributes.subject
Creative & Credible supports arts and health organisations and practitioners to:

# data[].attributes.body
– engage with evaluation creatively

– improve your practice

– make well-informed spending decisions

– strengthen the evidence base around the benefits and impacts of arts and health projects

# data[].attributes.attachment_link.readable_text
Creative & Credible supports arts and health organisations and practitioners to:

- engage with evaluation creatively
- improve your practice
- make well-informed spending decisions
- strengthen the evidence base around the benefits and impacts of arts and health projects

Use this website to understand why you might need to evaluate, what approaches might be appropriate, and to plan and implement evaluation for your project. You will also find downloadable handouts, examples and links to other evaluation resources here.

For news of Creative & Credible Training Programmes like us on Facebook and follow us on Twitter.

Preparing to Evaluate

There is a growing interest in the role that the arts can play in addressing health and social care priorities, and a need to demonstrate their impact effectively. A variety of strategic programmes and policy initiatives are helping both to highlight the contribution that the arts can make to health and wellbeing improvements and to develop the evidence base. Arts organisations and practitioners will want to consider how they fit into this strategic landscape, how they can forge partnerships with health professionals and policy makers, and how then to prepare their own plans for evaluation.

Evaluation is the process we use to find out whether a project can meet, or has met its aims. It will be led by the core values of the organisation or project. It should be informed by a 'theory of change', that is, an understanding of how an activity may be linked with its outputs and outcomes, or why what is done has the effect you observe that it does.

It may be useful to think about what evaluation is not. It involves more than just collecting and reporting monitoring information about a project. Although evaluation findings can be powerful when advocating for a project, it is important not to confuse the two, because evaluation will ask and answer questions and reveal impacts that may be both positive and negative. And evaluation and research are also different. Research usually seeks to generate new knowledge, while evaluation generally judges an existing service.

We evaluate for various reasons. Top amongst them has to be a desire to find out whether what we are doing has an effect and whether that effect is the one we thought it would. Without this knowledge we cannot develop or refine a service or respond effectively to the needs of those people who hold a stake in the work - such as participants, project funders or commissioners.

Do any of the following apply to you? You want to know how many people took part in a project and whether they were the people you had hoped to reach. You are interested to know about how the artistic outputs of a project have been displayed or received. You are concerned to know whether participants enjoyed the experience of taking part. You want to establish what practical challenges were involved in delivering the project and how they were overcome, what did not go well and what you could improve on next time. You are interested in unexpected outcomes, and whether the project worked in ways you had not anticipated. You want to understand whether the project represented a worthwhile financial investment. You are interested in understanding the benefits the project has delivered for participants in terms of their health, wellbeing or quality of life. Evaluation can help in all of these areas.

Arts and health organisations often look outwards when evaluating because they feel they don't have the skills or the capacity to evaluate internally. External evaluation can appear more credible, avoiding issues around areas where bias may creep in. An external evaluator provides specialist skills and knowledge and may be able to disseminate findings more widely. However, these benefits may come at a significant financial cost or mean that evaluation is 'one off' and that learning is not embedded within the project team. In reality, many successful projects develop by using an iterative process involving both internal and external evaluation. It may be useful to think through carefully where you need to be on a continuum between simple monitoring and academic research and the organisation AESOP has developed a number of tools designed to help think through this process. And sometimes, it is useful to accept that your evaluation just needs to be 'good enough' for the context in which you are working.

Approaches to Evaluation

The approach you take to an evaluation will depend on what you want to find out. You will choose the approach that suits your project, your stakeholders and the resources and skills available to you. You may feel that you need to evidence your work using the language and methods used by those who are commissioning or funding you. However, it may be that the outcomes you want to demonstrate cannot usefully be addressed in this way. Co-producing an evaluation with commissioners and funders and involving them in identifying aims and suitable outcomes, will help to make sure that you understand each other from the outset.

Evaluation frequently relies on the collection and analysis of data in number form. Approaches which involve this are termed 'quantitative'. They range in complexity from simple monitoring to the randomised control trial. Most arts and health project evaluations will benefit from using a mix of simple quantitative and qualitative methods.

Even the smallest scale project evaluation will involve some kind of monitoring of attendance or number of activities delivered. Quantitative data are also often collected at the end of a project or activity using surveys or questionnaires. Reporting on these can be a good way to describe impacts on participants. However, they cannot tell us much about whether a project has actually had a measurable effect. To find out whether a change might have taken place, you will need to measure at both the beginning and end of a project.

Use of a measure developed by researchers and 'validated' or tested on similar participants in a similar context, for example, the Warwick-Edinburgh Mental Well-being Scale, can give more reliable data. Validated scales also allows what you find for participants in your project to be compared with those of similar projects elsewhere.

It is always good for those delivering arts and health projects to clearly understand what the project costs and how this might compare to the cost of delivering similar work. However, as well as demonstrating outcomes, project providers sometimes feel they need to show that they can deliver specific economic benefits for commissioners and other stakeholders. If this is the case for your project, it will be important to develop a collaborative relationship with the commissioner to ensure that you both understand what is required.

The 'value' of an arts and health project may be measured in various ways. Formal approaches such as 'cost benefit analysis' value outcomes of a project in monetary terms but may not easily capture many important more subjective aspects of an arts project. Economic evaluation that seems better suited to the context might seek to assess the benefits of an intervention in terms of Quality of Life Years or QALYs. Another commonly-used tool in arts and health is 'Social Return on Investment' (SROI). SROI seeks to establish what the cost and impact to society would have been if an intervention had not taken place.

Economic evaluation of any kind requires significant skills and resources to conduct. Results will only be reliable if a project has first established a clear understanding of why the work it does has the effect it does - a theory of change. Further information about a variety of approaches to measuring value can be found by downloading Economic Evaluation.

Qualitative evaluation approaches focus on exploring the experiences, perspectives and opinions of those involved. They can help us understand what an arts activity or process 'means' to participants. They can be illuminating about other subjective elements such as project delivery. These approaches frequently rely on interviews and focus groups to collect data which is then analysed to identify themes and patterns within participants' words.

It is important to make a distinction between 'anecdote' and qualitative evaluation. Personal testimonial or case studies are often used in advocacy by arts and health organisations but these 'anecdotes' are often chosen because they tell a good story, usually positive. This process lacks credibility, and is unlikely to be helpful in developing a project in the longer term. In contrast, balanced reporting of qualitative data, carefully collected from a sample of participants, can produce rich, detailed evidence and stories. This can inform advocacy and give really meaningful information to support improvement of a project.

Arts and health evaluations may benefit from using a mix of qualitative and quantitative methods to both identify specific outcomes for participants and explore their subjective experience.

An evaluator demonstrates 'creativity' every time they view an evaluation problem from a fresh perspective, or devises an evaluation approach tailored to a particular context. While creativity is very definitely not just the province of artists or the arts, there is a growing interest in the use of evaluation methods that use film and visual arts, poetry and creative writing, music, drama and performing arts.

Arts-based methods can be particularly powerful in uncovering hidden perspectives and in empowering participants. They may also be less intrusive than more clinically-based evaluation tools as they can be inspired by and modeled on the intervention itself. They also involve a number of challenges. For example, the results you get from them (pictures, performances or poems for example) are by nature difficult to interpret and you may require technical skills that are not a part of standard evaluation.

It is particularly important to make sure that any creative evaluation approach you use 'fits' the project and the people who are taking part in the evaluation.

There is power in a good story. You may have used case study stories for many reasons, including when advocating for a project. Personal testimonial from participants who have had a positive experience can be very persuasive. However, this is not what is meant by 'case study' in evaluation. For an evaluator, the quality of a case study hinges on the methods used to collect and analyse the data and the writing or presentation of the story. The evaluator will select case studies that are relevant to the outcomes or issues the evaluation seeks to explore. You can have a case study of a single individual, groups of individuals, an object, even a place or a situation. A range of different kinds of data may be pieced together in a case study to tell a story relevant to the questions that inform the evaluation.

It is important to remember that a case study story is particular to an individual case; because one participant experienced something during a project, this does not necessarily mean that all participants did, or that any future participants will. This is why it is vital to select your cases carefully, so that they can genuinely help to illuminate the particular issues you are investigating.

Evaluation Cycle

Any evaluation you undertake should be part of an iterative process, with your learning being used to help improve and develop the next phase or project. If you look at it in this way, rather than as a costly and time-consuming 'add-on', it becomes obvious that evaluation is an essential part of the process of delivering a project itself. The Evaluation Cycle offers an overarching framework in which to understand evaluation as this iterative process. It was developed by Willis Newson with researchers at the University of the West of England and will fit any kind of evaluation approach you adopt.

This cycle has three main phases: project planning; implementation; and reporting and dissemination. It is worth noting that the implementation phase (including data collection and analysis) is only one part of the cycle and that project planning and reporting and dissemination demand just as much, and possibly more, time and thought.

A published version of this cycle was developed by Norma Daykin, Meg Attwood and Jane Willis and its use should be credited to them wherever it is used.

What can you learn from the work of others? In any evaluation, the first step should be to find out what other evaluation and research can tell you about the likely impacts of a project like yours.

An evidence review summarises the available evidence, revealing what is known about the impacts, outcomes and delivery of similar projects. It should also tell you about what is still not known, or difficult to establish. This is all critical information for evaluation planning. It will help you to decide what to evaluate and what approach might be appropriate. It may also be useful in planning the project itself. Crucially for an arts and health evaluator with limited resources and time, an evidence review may identify research which - if relevant - can then be used to support or illuminate limited, but promising findings from a smaller scale evaluation.

Consulting with the people involved in or affected by your project when planning an evaluation will tell you how they perceive the evaluation and what information they might need from it. Consultation is particularly important at an early stage for informing both project aims and identifying appropriate evaluation aims. It can also provide an opportunity to test out ideas and methods and to discuss whether a particular kind of evaluation activity might be appropriate.

Setting aims is a crucial part of the evaluation cycle. A clear set of evaluation aims should inform the questions that the evaluation seeks to answer. Many arts and health projects have a wide range of project aims, not all of which can be easily measured, particularly given limited evaluation resources. Evaluators will need to decide which aims to focus on, based on stakeholder priorities, an understanding of what can actually be measured, and the funds and resources available.

When you have decided on your evaluation aims and questions, you can plan how you will go about answering them. An evaluation framework or protocol describes what you are interested in evaluating and how you will go about doing it. At this point you will also consider what resources you might need, how you will manage the data, what the ethical issues are, and how you are going to report on and disseminate your findings.

An outcomes framework may underpin all this planning. In arts and health terms we generally understand 'outcomes' to mean the changes, benefits, learning or other effects that can be attributed to a particular service or activity. These are what you are seeking to measure. An outcomes framework will clearly map each outcome to a set of indicators that will establish whether or not it has happened and a method of measurement. This is a clear and manageable way to understand how and what you will be evaluating.

The data you collect for an evaluation may come in a number of forms. It may be quantitative, including numbers from monitoring or collected through closed questions on questionnaires. This information might have been collected at the end of a project or activity, or throughout. Data can also come in the form of transcriptions of interviews or focus group discussion, open feedback, meeting minutes, photographic documentation of activities, even artworks created by participants. Things you will need to consider, whatever kind of data you collect, include how you will ensure that the confidentiality, and maybe also the anonymity of respondents and participants will be ensured, and how you are going to avoid bias. The way in which you design and administer forms and questionnaires or run interviews and focus groups can introduce bias, and will need careful thought. In addition, you will want to think about how to use sampling to make your evaluation encompass as wide a range of participants' experience as possible.

The techniques used to analyse quantitative and qualitative data are quite different. For quantitative data you are likely to want to describe the patterns you observe in the numbers you have collected. If your analysis needs to go beyond the descriptive - and you are seeking to infer meaning from the data - it is best to seek advice. Qualitative data will generally be analysed using content or 'thematic analysis', a method which identifies patterns within the data, usually (but not always) participants' words collected in interviews or focus group discussion.

You may want to explore how to collect and analyse data using arts-based methods such as film or video. Some examples of ways in which researchers have gone about doing this can be browsed from this section.

Evaluation findings need to be recorded so that they can be shared. This is usually achieved through a written report describing the project, the evaluation and highlighting what has been learned. A report is useful however brief, and sometimes brief is best. It should present a balanced account of the project, commenting on its impact, the strengths and weaknesses of its delivery and the learning that has been captured that might inform future projects.

Reporting does not have to be confined to a written document. You could explore using film, animation or infographics to bring your findings to life. The important thing to consider is who you are addressing with the report and how this audience will want to access the findings.

Resources

The resources in this section are those that came to light during the Creative & Credible project, either during the literature review phase, or through discussion and interview with members of the Stakeholder Reference Group. This is not intended to be an exhaustive list of resources relating to arts and health evaluation, instead it provides background and references for all the elements discussed on the website. Weblinks (URLs) change. Please contact Willis Newson if you find a broken link so we can fix it.

Willis Newson, in collaboration with Professor Norma Daykin, Tampere University, Finland and other partners including the Royal Society of Public Health and the University of Winchester Centre for the Arts as Wellbeing, is offering a number of training and professional development programmes based on Creative and Credible.

For news of Creative & Credible Training Programmes follow the links below.

Helpful links for planning and implementing data collection and analysis.

There are a range of open-access repositories and collections of Arts and Health reports and literature. Please also see the evidence review section above which contains some significant literature reviews.

Some general resources that explain and contextualise the sector.

# data[].attributes.attachment_link.description
Arts and Health evaluation resource, tools, tips, guidelines for evaluating Arts & Health projects

# data[].attributes.body
Measures Toolkit