2.   Acquiring: systematically searching for and retrieving the evidence


3.   Appraising: critically judging the trustworthiness and relevance of the   evidence

4.   Aggregating: weighing and pulling together the evidence


5.   Applying: incorporating the evidence into the decision-making process


6.   Assessing: evaluating the outcome of the decision taken

  to increase the likelihood of a favorable outcome.’

Intervention: Use of the new screening tool to inform care decisions

Comparison: Treating without use of the new screening tool

Outcome: Patient care outcomes and patient flow

Context: Patients aged 70+ admitted by ED

Question: Does the new screening tool improve outcomes for patients aged 70+ admitted via ED without impacting patient flow?

Evidence:

• Published literature/validation

  • What biases?

  • Is it peer reviewed?

• In-house audit/data collection

  • Waiting times

  • Clinical outcomes

  • Patient satisfaction

  • Length of stay

  • Staff satisfaction

  • Financial impact

Barriers:

• Staff reluctance

• Cultural resistance

• Training

• Digital literacy

• Time constraints

• Financial costs

• Infrastructure

• Patient perception

Engaging colleagues:

• Engagement sessions

• Early training

• Demos/roadshows prior to implementation

• Champions/early adopters

• Good dissemination of evidence

• Observation of good practice

• Small pilot sessions at quiet/manageable times

• Assess effectiveness after initial trial

Acquiring: Have AI run alongside human for "ground truth" period (including minorities for diverse population data), implement AI alongside human, perform comparative study.

Aggregating: Evaluate A&E triage and outcomes for N patients. Score and rank each outcome and compare human vs AI.

Applying: Review data and evaluate validity against own population/common A&E visits.

Assessing: Decision on AI implementation, especially regarding instead of or alongside human.

Types of evidence - Look at literature and also other Trusts - have they implemented this procedure. Understand the current state, relevance of evidence

Barriers - staff resource, patient flow, funding, culture change, training. Aggregation of evidence may indicate further barriers,

Colleague engagement - workshops, understand why it's necessary / important. Local champions - identify who is for the adoption. Local validation / pilot - look for the early adopters. Outcomes - tool improves patient outcomes - pilot may provide evidence for or against

I - Remote consultation

C - In person consultation

O - Emergency admissions, patient experience, demographics, equity of access, incidence of disease between I/C populations

C - Clinic outpatient consultations

Asking Is remote consultation platform safe and do patients like it?

Acquiring - Pilot data, other published data, expand trial locally, include patient experience, EQ5D scores.

Appraising - Does the pilot need to be expanded, does it cover range of demographics, is it relevant outside of COVID, sufficient sample size, does published data reflect our patient population

Aggregating - Systematic review of published data. Collection of pilot data, follow up patients in Trust

Applying - Review with stakeholders, continue to collect data

Assessing - Risks. Efficiency vs patient satisfaction vs. outcome vs. increase in waiting times vs. patient lost to follow-up

Acquiring: waiting times /prioritisation of patients in ED. Studies / review data in similar settings

Appraising: Comparison of clinical decision making versus AI decision / waiting times. Clinical safety evaluation.

Aggregating: meta-analysis of data

Applying: continue double-running of clinical and AI interpretation; select low risk group of patients, limited time frame at lower capacity ED times.

Assessing: Have waiting times improved? Is the AI clinical decision appropriate? What are the patient outcomes e.g improved clinical outcomes?

Can decommissioning the diabetes service reduce costs whilst quality of care is maintained for patients with diabetes via other services?

Evidence:

Costings for both scenarios, cohort of patients and characteristics, QALY, cost effectiveness, patient outcomes, look at impact of other Trusts who have closed their local services, looking at capacity of alternative services, resources, change process, alignment with NHS strategies.

Stakeholders - patients, ICS, finance, GP, primary care, diabetes centre employees, public, secondary care, A&E

Potential biases:

Finance team - cost saving

GP, secondary care, A&E - increase workload, ambulance services

Diabetes centre - think they will provide better specialist care

Patients - worried about getting worst care and outcomes, might have to travel, waiting times

Independent review? Small pilot study? PPIE

Mobilising evidence:

Communicating, QA, frameworks, policies, guidelines, routine review, sharing learning with other ICS.

Remote monitoring of patients – are they as effective or more effective than traditional pathway model.

P – patients being referred to specific speciality (e.g. patients under care of respiratory services)

I – Does remote monitoring improve patient outcomes

C – comparing outcomes for patients who are seen in clinical

O – determine if remote monitoring is beneficial or not

C – the follow up of patients under care of the clinic

Asking:

Research question: does remote monitoring provide the same or better-quality service and does it have a role in healthcare in the future? What is the patient perspective on remote monitoring?

Acquiring:

·         Literature review, guidelines, original research in the area (quantitative)

·         Patient feedback – questionnaire for patients who are monitored remotely/in clinic (qualitative)

·         Review of patient outcomes (quantitative) – comparison between remote vs face-to-face

·         Evaluation of cost efficiency – health economics (quantitative) – more work in the same amount of time

Appraising:

·         Evaluation of performance of remote monitoring systems from a variety of specialisms to generate comparator data.

·         Patient feedback – bias in self reporting

·         NHS trusts deliver care in very different settings – could more rural hospitals be biased towards wanting to introduce a remote solution? Could less IT competent staff members be reluctant to introduce change.

Aggregating:

·         Accumulating data from both quantitative and qualitative sources.

Applying:

·         Review data and make recommendations based on if this change could be applied to your own service.

Assessing:

·         Review and audit change – monitor patient outcomes over time including a risk benefit analysis and PDSA cycle.

P – Remote Consultation Platform

I – Remote platform

C- not using the platform

O – number of DNA’s, number admissions, patient satisfaction, number of appointments, successful consultation , incident reporting numbers

C – Clinic management

Does the use of the consultation platform improve patient management whilst maintaining clinical safety?

Appraising – look for bias, patient population and criteria for remote consultation

Aggregative: qualitative – satisfaction data and quantitative metrics: number of DNA’s, number admissions, number of appointments, incident reporting numbers

Applying: Depending on the metrics, communication and dissemination of findings, education, apply to clinics locally/nationally, business continuity,

Assessing outcome: Review evidence from other sites

Question:

Does the implementation of the new frailty screening tool (I) for all patients >70 years of age admitted via ED cause increased workload and delay (P) in patient flow compared to the previous screening tool (C)?

Evidence to look at:

Delays in patient flow

Direct comparison of old tool and new tool

Staff questionnaire on workload

Patient satisfaction

Waiting times

Barriers:

Already large delays in ED already

Unaware of environment it was trialed in would it be similar to our ED

Is it comparable

IT infrastructure

Finance

Staff are already disengaged

How to engage:

Focus groups

Staff education on improved outcomes

Case studies

Audit and publish results

Staggered roll out with sufficient training provided

In patients presenting to the ED, how does the HealthTech AI triage tool compare with standard clinical triage in terms of accuracy, speed, cost and patient safety in a UK hospital setting?

Forms of evidence:

• Peer-reviewed studies e.g RCTs on AI triage

• Pilot trials from similar settings or other locations if no similar settings available.

• NHS/NICE/MHRA or Royal College of Emergency Medicine/Physicians guidance on use of AI tools

• HealthTech company validation data

Local small pilot studies e.g. random selection of patients for AI triage.

Mitigating concerns:

• Begin with a small-scale pilot of low risk patients e.g. no sepsis, major trauma, chest pain or children

• Ensure clinical staff are trained and understand how to interpret the AI tools recommendations.

• Clarify that the AI is a decision-support tool, not a replacement for their clinical judgment.

Collaborate with IT, governance, and frontline staff during rollout and have a forum for concerns.

Evaluate and adapt tool in real time:

• Look at Accuracy of AI recommendations vs clinician assessment

• Time savings in triage or improved flow of patients

• Frequency of Safety incidents

• User satisfaction (clinicians and patients)

• Equity of access and outcomes across different patient groups

• Use dashboards and regular reporting to highlight performance trends or issues and be ready to discontinue if adverse events are severe or frequent.

Carried out in 3 phases

Acquiring/Appraising: (i) Retrospective study of large-scale, de-identified data to determine if tool works for our population, by comparing to "gold standard" or clinical results. (ii) Interoperability test, with participating clinicians - feasibility of fitting AI into existing structure

Applying/Assessing: (iii) Operating in Shadow mode - running AI in parallel to existing system and compare results, then once confident in AI performance, switch to operating AI as primary, with human assessment running in shadow mode.

Aggregating: Done throughout each phase, collecting the data and evaluating to determine whether appropriate to move to next phase, and ultimately whether to implement clinically

Safety concerns and bias: Observational research method outlined above provides robust evidence. Consider stakeholder management - how to best communicate results, risks and benefits to clinicians and other system users

Evaluate and adapt tool in real time: See phase (iii) Shadow mode

Acquiring: What is the current clinician-led sensitivity and specificity of decision making? What level of risk is acceptable to take on when using an AI tool? Survey clinician experiences and views of the risks (e.g. deskilling or errros in judgement)

Appraising: Side by side trial using local data, as AI trained at a non trauma centre may not be suitable for trauma centre etc. and other biases in population.

Aggregating: Compare to human, check if outcomes, try on low-risk cases, ask if this would affect human decision making at the end.

Applying and Assessing - ran out of time in the breakout room!

Acquiring: Retrieving evidence on current triage rates, waiting times, diagnoses and outcomes. Also collating any existing evidence of similar tools and the effectiveness of it

Appraising : Evaluating the amount and statistical significance of available evidence on current practice and existing AI tools

Aggregating: Meta-analysis or systematic review of all available evidence

Applying: Comparing any existing data for current practice against available data for AI triage tools (health technology assessment)

Assessing: Real world trial of the AI tool in ED. As tool is currently unproven, it would need to be tested alongside current clinical practice so in the first stages of assessment it is not being used for clinical decision making, but its outputs can be reviewed against the decisions that were made

Acquiring: Qualitative and quantitative evidence using PPI and patient data/numbers

Appraising: Evaluating evidence

Aggregating: Looking at evidence and biases, and weighing up the pros and cons

Applying: Using this for form a decision on whether to decommission the service or not

Assessing: Look back at the data and use case studies over a certain period of time to understand the outcome data / long term effects of decommissioning the service.

• Frame a PICOC question to assess impact.

P patient effect on decommissioning

I decommissioning

C not decommissioning

O cost saving

C community diabetes patients

• What evidence is needed to make a sound decision? qual and quant (PPI), modelling patients, patient outcomes

• What biases or assumptions could influence stakeholders?

?cognitive bias / dissonance

• How can evidence be mobilised to ensure transparency and legitimacy?

Appraising

1. Ask

PICOC question:

In adults with diabetes, how does management via community diabetes services vs general practice alone impact clinical outcomes and patient satisfaction within an ICS aiming to optimise resources?

2. Acquire

Collect data on:

HbA1c, complications, hospital admissions, retinopathy

Service usage (DNA rates, wait times, referrals)

Patient satisfaction surveys

Cost analysis (direct + indirect)

3. Appraise

Evaluate quality and relevance of:

Clinical audits and outcome studies

Patient feedback tools (validated PROMs/PREMs)

Economic models and service evaluations

Check for biases, gaps, and generalisability

4. Apply

Integrate evidence with:

GP capacity and skill

Needs of diverse patient populations

Goals for efficiency, quality, and equity

Balance short-term savings vs long-term risks

5. Act

Share findings with stakeholders transparently

Use independent reviews to support decisions

Link choices to NHS values

Pilot changes in one locality before system-wide rollout

6. Assess

Monitor outcomes post-implementation:

Clinical results, patient satisfaction, GP workload

Use dashboards for tracking

Review and adapt decisions based on real-world impact