DIA/FDA Statistics Forum: Visual Overview of The Evaluation Plan by Mihaela C Munteanu

3. Purpose of the evaluation

mihaelamunteanu — 2014-11-09 04:31:36 UTC

The proposed evaluation hopes to identify strengths and weakness of the learning objectives of the forum. These findings could assess to what extent the program met its outcomes, and the needs and expectations of the target audience. Additionally, it is sought to identify the appropriate topics for the next training to refine the forum. It is a pilot evaluation at first that could be the basis for a larger evaluation plan later.

6. Data Collection Methods

mihaelamunteanu — 2014-11-09 04:31:36 UTC

The data collection methods were chosen to fit the evaluation questions. There are mixed methods of qualitative and quantitative data to help with multiple sources of information (i.e., triangulation). For a brief summary and at-a-glance view, please refer to the Evaluation Matrix below that shows how the sources of information, and data collection methods attempt to serve the key evaluation questions. A sample Survey is also attached as Appendix B.

Concerns of the Plan?

mihaelamunteanu — 2014-11-09 04:31:36 UTC

Some of concerns I have are as follows:

(a) respondents will not fill in the survey

(b) respondents will not fill in entirely the survey

(d) not confident that respondents (i.e attendees and speakers) will be entirely truthful in their answers

(e) not able to identify trends from the focused interviews

7. Evaluation Matrix

mihaelamunteanu — 2014-11-09 04:31:36 UTC

How the Data Collection Methods and the Sources of Information Support Each Key Evaluation Question Visually

4. Key Evaluation Questions

mihaelamunteanu — 2014-11-09 04:31:36 UTC

The key evaluation questions are related and focused to the overall rationale of the evaluation. The outcome of the evaluation plan is to use the information gathered from the data collection methods to improve the program and ultimately make decisions. Additionally, the stakeholders’ input, was taken in consideration in the development of these questions. Consequently the questions are as follows:

1. To what extent are the participants satisfied with the program and what recommendations for the next year do they have?

2. In what ways do the participants use the knowledge or awareness learned in the program?

3. To what extent were activities (i.e., the planned communication exchanges among participants, speakers, and FDA speakers and FDA participants) described in the program helpful and allowed?

1. Introduction

mihaelamunteanu — 2014-11-09 04:31:36 UTC

This evaluation plan is for the program called DIA/FDA Statistics Forum organized by Drug Information Association (DIA). DIA is a non-profit international organization. There are other associations for non-profit and for profit that conduct these types of workshops. Some of these training programs differ in their costs and some differ in how often they are offered to keep the costs low. Due to increasing pressure from the industry, academia, private sector, and government on knowing which ones to attend evaluation designs of these educational and professional programs are extremely warranted and sought out.

The learning objectives for the forum were given as the following:

Recognize statistical solutions to issues associated with the evidence and regulatory review
Describe application of statistical methodologies and thinking to the development of new products
Assess the impact of regulations and guidance on statistical practice
Discuss ideas for improving the communication between industry statisticians and FDA reviewers

2. Program Description

mihaelamunteanu — 2014-11-09 04:31:36 UTC

The DIA/FDA Statistics Forum is given every year in April. FDA stands for Food and Drug Administration and it is an agency within the U.S. Department of Health and Human Services responsible to approve safe and efficacious drugs. It is important to note each time the presentations are different. The presenters are different as well but there are exceptions. The forum facilitates open discussions of topics of practical and theoretical interest to statisticians, clinical trialists and clinical leaders involved in research and development of drugs and medical devices.

The program takes place over two days:

The forum allows for FDA and attendees to interpret together statistical guidelines used in clinical development, and present on “how to” rationales and proposals to theories that do not have real answers for clinical development in statistics.
Statistical methodologies and data standards regarding the clinical study design, analysis of data, and reporting of data from clinical trials are presented by topic selected by global speakers across the industry. Presentations with Questions and Answers (Q and A) are available and at the end of each session the speakers answer additional questions from the audience. At this time, the applicable FDA teams join the presenters of a module (usually no more than 3 presentations on a topic).
The participants are from the pharmaceutical companies, statisticians from academia, FDA statisticians, and consulting firms so that participants learn regulatory and statistical solutions associated with appropriate answers to the design and analysis of clinical trial data in a consensus way if possible.
In parallel, FDA discusses and shows how the regulations around the statistical methodologies needed for approval of drugs are applied are discussed.

Note: Logic Model is available but not presented here.

5. Stakeholders

mihaelamunteanu — 2014-11-09 04:31:36 UTC

8. Survey Template

mihaelamunteanu — 2014-11-09 04:31:36 UTC

Appendix B

9. Analysis of Data -Preliminary Ideas

mihaelamunteanu — 2014-11-09 04:31:36 UTC

Descriptive statistics with frequency and percentage from the survey
Provide a pie chart with results from the survey if possible
Report on the number of surveys distributed and the number of surveys answered/completed to see if there is a good data pool to analyze; report on non responses and possible biases
For focus interviews: understand what data trends or themes evolve that will help with key evaluation question 3, and also set up questions in more detail for the next program evaluation.
Report if these interviews took place as designed and shown in Evaluation Matrix.

Note: This evaluation plan is used first as pilot.

Brief Narrative

mihaelamunteanu — 2014-11-09 04:31:36 UTC

DRAFT, need to be less than 5-too long now

Michele Westphal

michelewestphal — 2014-11-09 19:51:40 UTC

One comment for you in terms of trying to decrease the length of your audio is that you have posted a lot of text, and then you are repeating a lot of this in the audio portion. I would recommend finding a blend, so that the audio supports and provides clarifications of the items you have typed. I do not think you need to duplicate your work.

Michele Westphal

michelewestphal — 2014-11-09 19:57:05 UTC

You have a very comprehensive plan, and I do not have many recommendations. I am attaching my comments in audio form for your consideration.

Art Campbell

artcampbell — 2014-11-09 21:24:09 UTC

I agree with Michele. Your plan is very comprehensive and you do repeat a lot of information. Perhaps lose some of the wording that you already provided in your voice threads and maybe provide graphics that represents the wording.

I really like your design and detail.

Perhaps have an online survey component that you could send afterward in order to have a better chance of people responding