# data[].attributes.body
1.   Asking: translating a practical issue or problem into an answerable question

2.   Acquiring: systematically searching for and retrieving the evidence

3.   Appraising: critically judging the trustworthiness and relevance of the   evidence

4.   Aggregating: weighing and pulling together the evidence

5.   Applying: incorporating the evidence into the decision-making process

6.   Assessing: evaluating the outcome of the decision taken

to increase the likelihood of a favorable outcome.’

# data[].attributes.body
Problem: Screening tool potentially beneficial but may lead to delays in patient care

Intervention: Use of the new screening tool to inform care decisions

Comparison: Treating without use of the new screening tool

Outcome: Patient care outcomes and patient flow

Context: Patients aged 70+ admitted by ED

Question: Does the new screening tool improve outcomes for patients aged 70+ admitted via ED without impacting patient flow?

Evidence:

Published literature/validation

What biases?

Is it peer reviewed?

In-house audit/data collection

Waiting times

Clinical outcomes

Patient satisfaction

Length of stay

Staff satisfaction

Financial impact

Barriers:

Staff reluctance

Cultural resistance

Training

Digital literacy

Time constraints

Financial costs

Infrastructure

Patient perception

Engaging colleagues:

Engagement sessions

Early training

Demos/roadshows prior to implementation

Champions/early adopters

Good dissemination of evidence

Observation of good practice

Small pilot sessions at quiet/manageable times

Assess effectiveness after initial trial

# data[].attributes.body
1. In patients aged over 70, who are admitted to ED, does using a validated frailty score improve early identification and improved outcomes, without affecting workload or patient flow?

2. Patient outcomes, length of stay, waiting times, mortality rates, readmission, cost, datix (or number of incidents reported), patient satisfaction, staff satisfaction. Published peer reviewed papers, how was the tool validated? Guidelines, other Trusts and stakehodlers who are using the tool.

3. Staff resistance, Trust buy-in, financial implications, impact on flow, safety/risk assessments, ethical considerations, long-term impact, impact on other departments.

4. Need clear and reliable evidence base, with Trust buy-in, consider confirmation bias, possibly would need local ICB support, senior manager buy in, quality and improvement support, good patient feedback and clearly demonstrable benefit in pilot study. Auditing outcomes with regular feedback and changes according to findings. Use SPC charts to assess outcomes.

# data[].attributes.body
Asking: Does AI triage help reduce A&E waiting times and/or outcomes?

Acquiring: Have AI run alongside human for "ground truth" period (including minorities for diverse population data), implement AI alongside human, perform comparative study.

Aggregating: Evaluate A&E triage and outcomes for N patients. Score and rank each outcome and compare human vs AI.

Applying: Review data and evaluate validity against own population/common A&E visits.

Assessing: Decision on AI implementation, especially regarding instead of or alongside human.

# data[].attributes.body
Clinical question - How would the implementation of the frailty screening tool affective current resource and patient workflow?

Types of evidence - Look at literature and also other Trusts - have they implemented this procedure. Understand the current state, relevance of evidence

Barriers - staff resource, patient flow, funding, culture change, training. Aggregation of evidence may indicate further barriers,

Colleague engagement - workshops, understand why it's necessary / important. Local champions - identify who is for the adoption. Local validation / pilot - look for the early adopters. Outcomes - tool improves patient outcomes - pilot may provide evidence for or against

# data[].attributes.body
P - Is remote consultation platform safe and do patients like it?

I - Remote consultation

C - In person consultation

O - Emergency admissions, patient experience, demographics, equity of access, incidence of disease between I/C populations

C - Clinic outpatient consultations

Asking Is remote consultation platform safe and do patients like it?

Acquiring - Pilot data, other published data, expand trial locally, include patient experience, EQ5D scores.

Appraising - Does the pilot need to be expanded, does it cover range of demographics, is it relevant outside of COVID, sufficient sample size, does published data reflect our patient population

Aggregating - Systematic review of published data. Collection of pilot data, follow up patients in Trust

Applying - Review with stakeholders, continue to collect data

Assessing - Risks. Efficiency vs patient satisfaction vs. outcome vs. increase in waiting times vs. patient lost to follow-up

# data[].attributes.body
Asking: Can AI reduce waiting times in ED?

Acquiring: waiting times /prioritisation of patients in ED. Studies / review data in similar settings

Appraising: Comparison of clinical decision making versus AI decision / waiting times. Clinical safety evaluation.

Aggregating: meta-analysis of data

Applying: continue double-running of clinical and AI interpretation; select low risk group of patients, limited time frame at lower capacity ED times.

Assessing: Have waiting times improved? Is the AI clinical decision appropriate? What are the patient outcomes e.g improved clinical outcomes?

# data[].attributes.body
PICOC question:

Can decommissioning the diabetes service reduce costs whilst quality of care is maintained for patients with diabetes via other services?

Evidence:

Costings for both scenarios, cohort of patients and characteristics, QALY, cost effectiveness, patient outcomes, look at impact of other Trusts who have closed their local services, looking at capacity of alternative services, resources, change process, alignment with NHS strategies.

Stakeholders - patients, ICS, finance, GP, primary care, diabetes centre employees, public, secondary care, A&E

Potential biases:

Finance team - cost saving

GP, secondary care, A&E - increase workload, ambulance services

Diabetes centre - think they will provide better specialist care

Patients - worried about getting worst care and outcomes, might have to travel, waiting times

Independent review? Small pilot study? PPIE

Mobilising evidence:

Communicating, QA, frameworks, policies, guidelines, routine review, sharing learning with other ICS.

# data[].attributes.body
Scenario 2:

Remote monitoring of patients – are they as effective or more effective than traditional pathway model.

P – patients being referred to specific speciality (e.g. patients under care of respiratory services)

I – Does remote monitoring improve patient outcomes

C – comparing outcomes for patients who are seen in clinical

O – determine if remote monitoring is beneficial or not

C – the follow up of patients under care of the clinic

Asking:

Research question: does remote monitoring provide the same or better-quality service and does it have a role in healthcare in the future? What is the patient perspective on remote monitoring?

Acquiring:

·         Literature review, guidelines, original research in the area (quantitative)

·         Patient feedback – questionnaire for patients who are monitored remotely/in clinic (qualitative)

·         Review of patient outcomes (quantitative) – comparison between remote vs face-to-face

·         Evaluation of cost efficiency – health economics (quantitative) – more work in the same amount of time

Appraising:

·         Evaluation of performance of remote monitoring systems from a variety of specialisms to generate comparator data.

·         Patient feedback – bias in self reporting

·         NHS trusts deliver care in very different settings – could more rural hospitals be biased towards wanting to introduce a remote solution? Could less IT competent staff members be reluctant to introduce change.

Aggregating:

·         Accumulating data from both quantitative and qualitative sources.

Applying:

·         Review data and make recommendations based on if this change could be applied to your own service.

Assessing:

·         Review and audit change – monitor patient outcomes over time including a risk benefit analysis and PDSA cycle.

# data[].attributes.body
Scenario 2

P – Remote Consultation Platform

I – Remote platform

C- not using the platform

O – number of DNA’s, number admissions, patient satisfaction, number of appointments, successful consultation , incident reporting numbers

C – Clinic management

Does the use of the consultation platform improve patient management whilst maintaining clinical safety?

Appraising – look for bias, patient population and criteria for remote consultation

Aggregative: qualitative – satisfaction data and quantitative metrics: number of DNA’s, number admissions, number of appointments, incident reporting numbers

Applying: Depending on the metrics, communication and dissemination of findings, education, apply to clinics locally/nationally, business continuity,

Assessing outcome: Review evidence from other sites

# data[].attributes.body
Scenario 1

Question:

Does the implementation of the new frailty screening tool (I) for all patients >70 years of age admitted via ED cause increased workload and delay (P) in patient flow compared to the previous screening tool (C)?

Evidence to look at:

Delays in patient flow

Direct comparison of old tool and new tool

Staff questionnaire on workload

Patient satisfaction

Waiting times

Barriers:

Already large delays in ED already

Unaware of environment it was trialed in would it be similar to our ED

Is it comparable

IT infrastructure

Finance

Staff are already disengaged

How to engage:

Focus groups

Staff education on improved outcomes

Case studies

Audit and publish results

Staggered roll out with sufficient training provided

# data[].attributes.body
PICOC Question:

In patients presenting to the ED, how does the HealthTech AI triage tool compare with standard clinical triage in terms of accuracy, speed, cost and patient safety in a UK hospital setting?

Forms of evidence:

Peer-reviewed studies e.g RCTs on AI triage

Pilot trials from similar settings or other locations if no similar settings available.

NHS/NICE/MHRA or Royal College of Emergency Medicine/Physicians guidance on use of AI tools

HealthTech company validation data

Local small pilot studies e.g. random selection of patients for AI triage.

Mitigating concerns:

Begin with a small-scale pilot of low risk patients e.g. no sepsis, major trauma, chest pain or children

Ensure clinical staff are trained and understand how to interpret the AI tools recommendations.

Clarify that the AI is a decision-support tool, not a replacement for their clinical judgment.

Collaborate with IT, governance, and frontline staff during rollout and have a forum for concerns.

Evaluate and adapt tool in real time:

Look at Accuracy of AI recommendations vs clinician assessment

Time savings in triage or improved flow of patients

Frequency of Safety incidents

User satisfaction (clinicians and patients)

Equity of access and outcomes across different patient groups

Use dashboards and regular reporting to highlight performance trends or issues and be ready to discontinue if adverse events are severe or frequent.

# data[].attributes.body
Asking: Can an AI-based triage tool reliably and safely add value to ED patients and healthcare providers?

Carried out in 3 phases

Acquiring/Appraising: (i) Retrospective study of large-scale, de-identified data to determine if tool works for our population, by comparing to "gold standard" or clinical results. (ii) Interoperability test, with participating clinicians - feasibility of fitting AI into existing structure

Applying/Assessing: (iii) Operating in Shadow mode - running AI in parallel to existing system and compare results, then once confident in AI performance, switch to operating AI as primary, with human assessment running in shadow mode.

Aggregating: Done throughout each phase, collecting the data and evaluating to determine whether appropriate to move to next phase, and ultimately whether to implement clinically

Safety concerns and bias: Observational research method outlined above provides robust evidence. Consider stakeholder management - how to best communicate results, risks and benefits to clinicians and other system users

Evaluate and adapt tool in real time: See phase (iii) Shadow mode

# data[].attributes.body
Asking: What is the sensitivity and specificity of an AI triage tool in an ED context? What are the concerns of the clinicians in deployment of an AI tool?

Acquiring: What is the current clinician-led sensitivity and specificity of decision making? What level of risk is acceptable to take on when using an AI tool? Survey clinician experiences and views of the risks (e.g. deskilling or errros in judgement)

Appraising: Side by side trial using local data, as AI trained at a non trauma centre may not be suitable for trauma centre etc. and other biases in population.

Aggregating: Compare to human, check if outcomes, try on low-risk cases, ask if this would affect human decision making at the end.

Applying and Assessing - ran out of time in the breakout room!

# data[].attributes.body
Asking: Can an AI based triage tool increase efficiency of triage for patients presenting to emergency departments?

Acquiring: Retrieving evidence on current triage rates, waiting times, diagnoses and outcomes. Also collating any existing evidence of similar tools and the effectiveness of it

Appraising : Evaluating the amount and statistical significance of available evidence on current practice and existing AI tools

Aggregating: Meta-analysis or systematic review of all available evidence

Applying: Comparing any existing data for current practice against available data for AI triage tools (health technology assessment)

Assessing: Real world trial of the AI tool in ED. As tool is currently unproven, it would need to be tested alongside current clinical practice so in the first stages of assessment it is not being used for clinical decision making, but its outputs can be reviewed against the decisions that were made

# data[].attributes.body
An innovation team wants to introduce AI triage in ED: how would you review and apply

the evidence?

Asking - How effective is the AI tool at triaging patients in the ED?

Acquiring - Evidence supplied from manufacturer - we are the trial dataset. Are there comparable systems or is this novel? Patients viewpoint from this technology?

Appraising/Aggregating - Caution to be used as there are are no real world data sets. Standard precautions with critiquing journals. E.g. dataset independence, independent analysis, peer-review in respected journals. Correlating patient experience and manufacturer claims.

Applying - pilot under review 'service evaluation'/'user specification review'. Risk assessment.

Assessing - Real time evaluation. Revert to manual in failure event.

# data[].attributes.body
Asking: Will cost savings by decommissioning diabetes services negatively affect patient outcomes?

Acquiring: Qualitative and quantitative evidence using PPI and patient data/numbers

Appraising: Evaluating evidence

Aggregating: Looking at evidence and biases, and weighing up the pros and cons

Applying: Using this for form a decision on whether to decommission the service or not

Assessing: Look back at the data and use case studies over a certain period of time to understand the outcome data / long term effects of decommissioning the service.

• Frame a PICOC question to assess impact.

P patient effect on decommissioning

I decommissioning

C not decommissioning

O cost saving

C community diabetes patients

• What evidence is needed to make a sound decision? qual and quant (PPI), modelling patients, patient outcomes

• What biases or assumptions could influence stakeholders?

?cognitive bias / dissonance

• How can evidence be mobilised to ensure transparency and legitimacy?

Appraising

# data[].attributes.body
Scenario 3

1. Ask

PICOC question:

In adults with diabetes, how does management via community diabetes services vs general practice alone impact clinical outcomes and patient satisfaction within an ICS aiming to optimise resources?

2. Acquire

Collect data on:

HbA1c, complications, hospital admissions, retinopathy

Service usage (DNA rates, wait times, referrals)

Patient satisfaction surveys

Cost analysis (direct + indirect)

3. Appraise

Evaluate quality and relevance of:

Clinical audits and outcome studies

Patient feedback tools (validated PROMs/PREMs)

Economic models and service evaluations

Check for biases, gaps, and generalisability

4. Apply

Integrate evidence with:

GP capacity and skill

Needs of diverse patient populations

Goals for efficiency, quality, and equity

Balance short-term savings vs long-term risks

5. Act

Share findings with stakeholders transparently

Use independent reviews to support decisions

Link choices to NHS values

Pilot changes in one locality before system-wide rollout

6. Assess

Monitor outcomes post-implementation:

Clinical results, patient satisfaction, GP workload

Use dashboards for tracking

Review and adapt decisions based on real-world impact