AECoP: Data analysis by

Treatment or intervention fidelity

helencrosby2 — 2024-03-07 11:30:17 UTC

Useful for singular interventions but does require resource to do this

2024-03-07 11:35:01 UTC

Pilots often set up to test different models for delivering an intervention. Local adaptations often made by design or otherwise due to local circumstance (e.g. can't recruit planned roles/band). Need to be pragmatic about where to make trade-offs - depth v breadth; timings to maximise learning

Question 1

2024-03-07 11:35:59 UTC

Capture as much detail of each service to provide context of future evaluation and measurement

Metrics

2024-03-07 11:36:43 UTC

Not having all of the right metrics or variables in place before the evaluation starts, or not having enough longitudinal data to look at change or trends over time. Having to look at alternative metrics or extend the time period of the evaluation.

2024-03-07 11:37:50 UTC

understanding intervention/expected theory of change - then exploring how this is delivered locally to identify the key features of delivering the ToC in different contexts

2024-03-07 11:37:58 UTC

What works in one setting may not work in another

Need for broader culture change towards evaluation

helencrosby2 — 2024-03-07 11:40:05 UTC

How to get involved from the start and not 'fire fight'

How do we get people to embed evaluation in the planning of the interventions?

Stakeholder involvement and engagement to have a robust evaluation

2024-03-07 11:41:29 UTC

Capturing as much detail / context as possible

Fall back on national measures (where possible)

Triangulation of qual and quant

Dealing with other confounding factors (happens anyway)

Difficulties measuring longitudinal impact

Managing commissioner expectations

comparator(s)

2024-03-07 11:42:21 UTC

programmes/intervention often rolled-out nationally so no counterfactual - compounded by lack of sufficient data for synthetic controls, or for comparator programmes in other areas when local initiatives. Timescales often too short. All in all - very difficult to provide robust analysis of impact including economic consequences - within policy/decision-maker's timescales

Q2

2024-03-07 11:42:48 UTC

Q2

2024-03-07 11:46:31 UTC

Difficult to influence evaluation design once the service is under way for eg. capturing patient and user experience at the beginning and end of the intervention

Focus on those not adopting

2024-03-07 11:47:33 UTC

Focus no those who are not adopting interventions -what are their barriers...? Trish Greenhalgh NASSS framework helpful

Flexibility

2024-03-07 11:47:52 UTC

Being pragmatic about what data can be collected in the time available. For qualitative interviews this can mean shortening the interview time to make it easier for busy participants to take part, and having flexibility in scheduling interviews. This approach may involve prioritising key questions and also deciding in advance which types and how many stakeholders to interview to be able to get the information you need.

2024-03-07 11:49:33 UTC

Understand where the pressure to show instant results comes from and try and influence

2024-03-07 11:50:22 UTC

Too often asked to magic up some data to show attribution

2024-03-07 11:51:33 UTC

Where there is not enough (quality) data available or enough data points, making a decision with the key stakeholders about what data can be used and whether it will be able to answer the evaluation questions or if a different analysis approach is needed.

2024-03-07 11:53:47 UTC

Logic modelling still helpful no matter what stage brought in

2024-03-07 11:54:31 UTC

Simulated retrospective analysis can be helpful

2024-03-07 11:56:53 UTC

Difficult to anticipate any delays in programme and extra time needed to start evaluation (e.g. delays in implementation which may be caused by factors outside of participants' control) which shortens or postpones the timeframe available for evaluation

2024-03-07 11:58:02 UTC

Use data to illustrate activity happening within different models supported with qualitative data