https://padlet.com/tnsbrooks/x1b8mx6xxju0 Tamara Brooks EDUC804 en-us 2017-11-09 20:42:16 UTC 2023-01-28 07:42:50 UTC hello@padlet.com tnsbrooks https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205503178 "Evaluation is the systematic process of gathering and analyzing data and other objective information on learning, performance, and change processes and outcomes within the context of the business or organizational setting to determine the quality, value, and/or effectiveness of the intervention or solution" (Combs & Falletta, 2000).

In selecting an evaluation method you should take into   consideration "how intrusive the method is and how readily available the information already is.  Collecting reliable, valid and readily available data may be the most desirable approach due to low cost, the lack of interference in the organizations operations and the ability to represent historical data" (Combs, 2017).

Prior to conducting an evaluation, you should consider resource requirements and stakeholder preferences.  Your stakeholders should be consulted when determining the appropriate format of your evaluation results.

Keep in mind the key evaluation questions where determining the data collection method.

The ADDIE model is the generic process traditionally used by instructional designers.

]]> 2017-11-09 20:45:32 UTC https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205503178 tnsbrooks https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205510413 1.  Surveys/Questionnaires (e.g. Likert Scale, program documents, interviews, policy documents)

2.  Interviews or Focus Groups

3. Program archives

4.  Case Studies

5. Observations

6.  Existing Metrics/Indicators/Benchmarks

7.  Tests and Performance Assessments]]> 2017-11-09 21:08:03 UTC https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205510413 tnsbrooks https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205514746 - Formative evaluation is typically conducted during the development or improvement of a program or product and it is conducted, often more than once, for in-house staff of the program with intent to improve.

-Summative evaluation provides information on the program or products efficacy (or effectivness/impact (i.e. Does it have the ability to do what it was designed to do?
(Combs, 2017)]]> 2017-11-09 21:22:46 UTC https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205514746 tnsbrooks https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205516766 2017-11-09 21:29:56 UTC https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205516766 tnsbrooks https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205520389 Background
The Office of the Comptroller of the Currency (OCC) was established in 1863 as a bureau of the Department of the Treasury. It is responsible for regulating and supervising the national banking system. The OCC charters, regulates and examines approximately 2,200 national banks and 52 federal branches and agencies of foreign banks in the United States. The OCC's nationwide staff of examiners conducts on-site examinations of national banks and provides sustained supervision of bank operations.  Its mission is to ensure a safe, sound and competitive national banking system that supports the citizens, communities and economy of the United States.

The OCC hires approximately 150 entry-level bank examiners (also known as Assistant National Bank Examiners (ANBE) every year in two recruitment cycles, one in the fall and one in the spring for two lines of business, Midsize and Community Bank Supervision (MCBS) and Large Bank Supervision (LBS).  Applications are received during set dates in the campaigns - in September (fall) and February (spring).  

The goal of the ANBE program is to attract, hire, and retain the best and brightest students with degrees in accounting, banking, finance, business administration, economics and marketing across the nation to ensure the OCC remains a preeminent prudential supervisor of the national banking system.  Our current entry-level recruitment program’s quality per hire is relatively average to high with more than 65% of our hires still on board after 5 years – consistent with external benchmarks.  However, it is within the first year of hire in which we determine whether our entry-level hires are a “good fit” for the bank examination career by placing them on a training team of approximately 5 trainees, a Training Team Leader, and a Training Team Assistant for on-the-job training.  Based on the geographic location of the training team, we have noticed a disproportionate rate of performance related separations during the first year of hire.  Our internal stakeholders have anecdotally connected performance related separations with a trainee’s weakness in our major competency areas:  organizational skills, written communications skills, technical skills, analytical skills, and oral communication/interpersonal skills.  It is the belief of our internal stakeholders that our assessments should predict with the highest degree of certainty a trainee’s ability to adequately perform in our major competency areas on day one.
Purpose
The purpose of this evaluation is to determine whether the OCC, recruits and onboards talent with the appropriate knowledge, skills, and abilities (KSAs) to be successful entry-level bank examiners. The evaluation will be used to determine what factors in addition to KSAs contribute to a trainee's success as well as barriers to successfully completing the trial period (first year on the job).

Key Program Evaluation Questions:

1.  Is the OCC assessing the right pre-hire KSAs for our trainees to be successful on day one?

2.  What factors other than KSAs contribute to a trainee's success?

3.  What barriers are there for a trainee to successfully complete the training team (simultaneous with trial period)?]]> 2017-11-09 21:44:00 UTC https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205520389 tnsbrooks https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205526254 Tashakkori and Creswell (2007) define mixed methods as “research in which the investigator collects and analyzes data, integrates the findings, and draws inferences using both qualitative and quantitative approaches or methods in a single study or a program of inquiry (p.4)” (Russ-Eft & Preskill, 2009, 206).  

Due to the robust nature of the OCC’s recruitment, onboarding and development of entry-level Assistant National Bank Examiners, the program of inquiry will be delivered in a two phase design, collecting the quantitative data first followed by the qualitative data.  “The purpose is to use the qualitative results to further explain and interpret the findings from the quantitative phase” (Creswell, 2013).

]]> 2017-11-09 22:09:22 UTC https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205526254 tnsbrooks https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205532805 Mixed methods data analysis will involve analyzing the data from both the qualitative and quantitative research collected in the study.    "Once the data from the qualitative and quantitative portions of the study have been analyzed, the data will need to be integrated in an appropriate way depending on the purpose of the study. The integration of the data will maximize the strengths of each approach, while minimizing the weaknesses. Effective integration of the data is the key to utilizing the mixed methods approach to draw more comprehensive conclusions from a project" (Creswell, 2013).]]> 2017-11-09 22:41:41 UTC https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205532805 tnsbrooks https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205535869 1.  Most programs are initiated with political influence.  They are proposed, defined, funded, etc. through political processes.

2.  Evaluation findings feed into decision making and resources.

3. Evaluations may call attention to waste, misuse, abuse, legitimacy of program goals, and controversy.]]> 2017-11-09 22:59:37 UTC https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205535869 tnsbrooks https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205536515 1.  Will interview participants provide open and honest feedback?  Lack of candor could skew the study's results.

2. Will the agency be receptive to the study's results and implement recommendations for improvement.

3. There could be other external factors that have not been considered that are impacting entry-level attrition other than KSAs.]]> 2017-11-09 23:04:17 UTC https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205536515 tnsbrooks https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205537967 Russ-Eft, D., & Preskill, H. (2009).  Evaluation in organizations. (pp. 209-228).  New York, NY: 
Basic Books.

Combs, W., & Falletta, S. (2000).  The targeted evaluation process.  Alexandria, VA: ASTD.

Creswell, J. (2013).  What is mixed methods research?  Retrieved from https://www.youtube.com/watch?v=1OaNiTlpyX8 

]]> 2017-11-09 23:14:26 UTC https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205537967 tnsbrooks https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205540225 2017-11-09 23:29:26 UTC https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205540225 tnsbrooks https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205551773 2017-11-10 01:01:44 UTC https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205551773 tnsbrooks https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205687531 Voice Recording]]> 2017-11-10 14:01:32 UTC https://padlet.com/tnsbrooks/x1b8mx6xxju0/wish/205687531