1. Background
When a new graduate nurse enters the hospital for the first time the feeling is intense, and intimidation is at an all-time high. The fact that everyone in the unit knows there is a new nurse is worrisome for often the phrase “nurses eat their young” is a thought that overwhelms so many new nurses. The hospital located in central Florida has a 16-week orientation program that has been in effect for three years, but is facing negative results. The job turnover rates in the hospital from new graduate nurses is 70% in just one year [1]. The new nurses’ voice that it is leadership and disorganization that is driving them away [1]. These new graduate nurses are newly hired licensed registered nurses who have just graduated nursing school within a three-month period. The purpose of this evaluation plan for this orientation program was to stimulate an effective strategic plan to decrease the new graduate job turnover rates in the hospital. Possible theories for high turnover rates could be attributed to inadequacies in leadership, mentoring, or other organizational issues. The structure of this evaluation plan integrated research and practice skills that enhanced the current practices in the hospital [2]. The material in this study covered the program, including its goals, objectives, the stakeholders involved, and the evaluation process that was conducted in the program.
2. Objective
The hospital stakeholders have observed a significant increase of new graduate nurse job turnover rates of 70% which is above the expected performance gap of 5%. This poses a significant issue for the hospital in terms of revenue and patient care. Without enough nurses, there is a decrease in the quality and quantity of patient care and lowers the degree to which the hospital receives revenue from Medicare and Medicaid is affected. It is a continuous cycle that if not properly fixed can turn into a downward spiral and lead to severe consequences for the most significant population, patients. The new graduate nurses have filed numerous complaints about the orientation program and feel that the process involved in its administration needs to be revised and modified [1]. It was hoped that evaluating this program would not only help with new graduate retention rates but also increase revenue and satisfactory care for patients. An evaluation program is a systemic investigation that is used to address the success of a program and make a difference in the lives of stakeholders [2]. The evaluator utilized the evaluation plan to determine whether or not the orientation program had caused any actual outcomes or differences. Not only is it the duty of the hospital to evaluate the program but also it is their ethical duty to ensure that all staff are given fair treatment to succeed [2]. Currently, there have been no previous evaluations conducted on the program due to poor awareness of the effect this program has on the hospital by stakeholders.
3. Methods
3.1. Evaluation Model
The evaluation plan conducted was formulated to specifically target the key stakeholders and audience within the organization. The objective was to identify the problem and conclude which parties were contributing factors to this problem. Data was then collected from past evaluation efforts and staff to better understand where the issue began. Interviews, surveys, and questionaries were methods used when communicating with the stakeholders to understand their perspectives of the problem within the orientation program.
Table 1. Evaluation plan.
Steps |
Duration (weeks/months) |
Evaluation Task 1. Identify the issue, the stakeholders involved, and the resources needed to gather data Who Evaluator Where Onsite at the organization |
1 week |
Evaluation Task 2. Gather data via human resources on retention and turnover rates of new graduate nurses over the last 5 years. Create evaluation plan outline Who Evaluator, human resource personnel Where Onsite at the organization |
3 weeks |
Evaluation Task 3. Conduct a collaboration meeting with stakeholders to explain reason for evaluation plan with the supporting data Who Human resource personnel, CFO, CEO, unit leaders, nurse educators Where Meeting onsite at the organization |
2 days |
Evaluation Task 4. Gain permission to conduct interviews Who Human resource personnel Where Onsite at the organization |
1 week |
Evaluation Task 5. Conduct interviews on the perceptions of the program Who Key stakeholders (new graduate nurses who are still currently working at organization and those who have left, unit leaders, preceptors, human resource personnel, nursing educators, CFO, and CEO Where Interviews onsite at the organization |
8 weeks |
Evaluation Task 6. Analyze data and draw conclusions Who Evaluator Where Onsite at the organization |
2 weeks |
Evaluation Task 7. Conduct collaboration meeting to present the data found from the interviews to show the status of the interviews and overall results. Who Key stakeholders (new graduate nurses who are still currently working at organization and those who have left, unit leaders, preceptors, human resource personnel, nursing educators, CFO, and CEO Where Meeting onsite at the organization |
2 days |
Evaluation Task 8. Conduct collaboration meeting to problem-solve the issues in the orientation program Who Key stakeholders (new graduate nurses who are still currently working at organization and those who have left, unit leaders, preceptors, human resource personnel, nursing educators, CFO, and CEO Where Meeting onsite at the organization |
1 week |
3.2. Total Duration of Evaluation Plan: 4 Months and 4 Days
In the following Table 1, a process step outline is presented to describe how the evaluator organized and produced the deliverable to the organization. There was a total of eight evaluation tasks the evaluator conducted to gather and analyze data. Key stakeholders participated in the tasks and the overall collaboration on the final conclusion of the results. The total duration of the plan consisted of four months and 4 days for completion.
3.3. Stakeholders, Participants, and Target Audience
The evaluator of the program evaluation focused on a significant population of stakeholders and participants in the hospital. Key stakeholders and participants who participated, benefitted, and were impacted by the evaluation were human resources, leadership and management on the units, new graduate nurses, preceptors, and non-precepting nurses. The breakdown of each individuals’ role in the evaluation began with human resources. The human resources personnel group provided data on the percentage of new graduates who had left the hospital and the reasons why. Information in regards to what plan was in place to retain these nurses was also obtained to further evaluate what was going on to either fix or contribute to the issue. The human resource department benefited from the evaluation in that they retained more nurses and thus create more revenue for the hospital.
Leadership, management, preceptors, and non-preceptors of the units were also key contributors to understanding the turnover rates. Data was obtained to better understand the mental models and other significant events that had led to the new graduates leaving. It also showed if they had put any new measures into fixing the issue themselves. The data of the evaluation plan helped the unit benefit by successfully retaining staff to work and thus limiting the shortage of nurses and burnout rates. The new graduates were the pivotal key in understanding why so many of them were leaving. When the issue was clear, the evaluation helped the new graduates want to stay on their units. Overall, each individual stakeholder and participant had something to gain with the program evaluation whether it was revenue, decrease in nurse shortage, or keeping a job.
3.4. Sample
The manner in which the sampling of the population occurred was in ongoing evaluations that increased the level of confidence in the project and overall report attempting to be developed. Ideally, the aim was to draw a random sample, in which each individual in the population sample had an equal probability of participating in the project [3]. The way in which the evaluators sampled and grouped the sample population increased the validity and reliability of the outcomes, in the event they were to be tested in a different environmental setting. The tactic of stratification is the method of sampling and processing the evaluators used, to require characteristics of the population members before grouping. This allowed for each sample to be represented in the same manner [3]. This method allowed for each sample to give valid and conclusive results even though the characteristics were different. It also showed differences in how the orientation program affected each sample size differently which could pose promising data in future experimentation [3]. Bias was not shown upon creating the sample size. The random sampling in the project approach began with the evaluator choosing 40 random past new graduate nurses from a list and then 40 random current new graduate nurses from another list. The 80 participants on the list had similar characteristics of having worked at the organization in the past five years and had also been placed with the same preceptors. In each group, the evaluator ensured that there were 20 men and women in each list so that bias would be further eliminated in terms of gender perspectives. The purpose of choosing this method of sampling the population for the evaluation was to gain as many perspectives from the nurses as ultimately the goal was to retain more nurses in the hospital. This retention could only occur if the issue of understanding why they were leaving was discovered from their own words.
3.5. Evaluation Questions
The following evaluation questions were used to guide this study; they are also located in Appendices I, II, III, and IV. The chart below further breaks down the evaluation program that was being implemented to improve the outcomes of the orientation program. The focus areas that were being looked into specifically were measuring the effectiveness, magnitude, and satisfaction of the orientation program and the quality of the program and its leadership. Key stakeholders that were impacted by the orientation program were the intended audience for the evaluation: new graduate nurses unit leaders, preceptors, and human resources. Each of these individuals had detailed questions about the program and these were presented in the evaluation questions column. Each question was specific to understanding the program, its issues, and the interventions that needed to be implemented to have favorable outcomes. The fourth column, Use, is where the evaluator analyzed and configured what the data collected from the audiences’ questions had either contributed to or formed in terms of interventions, outcomes, and impacts (See Table 2).
Table 2. Evaluation questions.
Evaluation
Focus Area |
Audience |
Evaluation Question |
Use |
Measuring effectiveness, magnitude, and satisfaction of orientation program |
New graduate nurses |
Is the program reaching its desired retention rates? What does the organization think of the program? Does leadership see there is an issue? What is the evaluation method to assess preceptors and their criteria to precept? |
Assess the mindset and concerns that new graduates have about the program. Assess the effectiveness and satisfaction the program is imposing on its intended participants. Analysis of most prevalent reasons why retentions rates are high-use to create interventions or revisions to programs to lower rates and improve both patient and new graduates’ satisfaction. |
Unit leaders |
How satisfied are the new graduates with the unit and program makeup? What are the concerns the new graduates have? |
Measure the accountability leadership has for issues in the program. Assess the support of leadership in the program. Analysis of what leadership is doing to contribute to the issue-use to create an education for unit leaders to better have them understand the issues the program is enduring on each of their units. |
Assessing the quality of the program and its leadership |
Unit leaders |
What are the benchmarks for evaluating how effective the program is? What new education do experts say is necessary for preceptors? |
Assess if preceptors are qualified accountable in the program. Analysis of how dedicated leaders are and willing to accept their fault in issues: use to create an action plan to have leaders more involved in the program. |
Preceptors |
Is the program achieving its intended goals? What do the new graduates think about the program? |
Assess the mindset of preceptors and their support in the program. Analysis of preceptor accountability in issues: use to develop an education class to better train preceptors. |
|
Human
Resources |
Are we reaching our retention rate goals? Are our new graduates satisfied with the program? Is the program being run efficiently? How can we improve our program? |
Analysis of the structure of the program and what is contributing to the issues. Analysis of how willing the organization is to improve the program and their knowledge of what is going on: use to formulate an action plan with their collaboration. |
3.6. Data Collection and Procedures
The data was collected over a period of four months. The key stakeholders were interviewed separately in the same setting but asked different questions. The three objectives followed when collecting the data included: To document retention and turnover rates within the past five years, document the stakeholders’ perceptions of the purpose and satisfaction of the orientation program, and document activities conducted in the past and current orientation programs. There was a total of 80 new graduates interviewed with 40 being past employees and the other 40 being current employees. There was a total of 80 preceptors interviewed who had participated in the orientation program over the past five years. In total, there were 20 unit leaders who were interviewed as well. Each participant was interviewed and asked to complete a survey on their perceptions and experiences of working in the current orientation program (See Table 3).
Table 3. Data plan.
Program Goals or
Objectives* |
Data
Collected** |
Data Source+ |
When
Collected++ |
Evaluation Objective 1: To document retention and turnover rates within the past 5 years |
Tracking data reports |
Human resource personnel |
March |
Evaluation Objective 2: Document the stakeholder perceptions of the purpose and satisfaction of the orientation program |
Interviews, collaborative meetings, surveys |
40 Previous new graduate nurses, 40 current new graduate nurses 80 preceptors, 20 nursing leaders |
April-May |
Evaluation Objective 3: To document activities conducted in the past and current orientation program |
Interviews, data reports, surveys |
40 Previous new graduate nurses, 40 current new graduate nurses 80 preceptors, 20nursing leaders |
June |
3.7. Data Analysis Methods (See Table 4)
Table 4. Data analysis.
Data Source/Type |
Data Analysis Procedures |
Interviews |
The data collected during the interview process was coded and categorized into three main sections: organizational, substantive, and theoretical. The rationale for this method was to sort the descriptive data collected so that the material specific to the topic or issue could be physically sorted and separated from other data (Maxwell, 2013). The evaluator used an organizational category to investigate for board areas or issues within the problem attempting to be solved. The reason for categorizing and coding data into substantive and theoretical data was that evaluators could explicitly identify the content of the person’s statement or actions (Maxwell, 2013). As this research was based on a qualitative form of study, understanding the interviewees’ statements and actions was key to implementing a strategy for change. |
Data Reports |
Compared hospital employee records on the turnover and retention rates of new graduate nurses who participated in the orientation program in the past 5 years. |
Collaborative Meetings |
During the analytical process, the methodological approach of Elo and Kyngas was used. Elo and Kyngas were content scholars who analyzed data in qualitative science studies in the nursing profession. This method was where the evaluator analyzed the conversations between stakeholders in the collaborative meetings for their meanings (Creswell & Creswell, 2018). The evaluator focused the analysis on the single words, sentences, or complete paragraphs the stakeholders said to each other. While identifying these meanings coding was used to process and categorize them into lists and themes. The themes were grouped in high-order headings with the purpose of reducing their numbers and identifying how they could be further separated. When necessary, subthemes were used to reduce complexity. The data was than compared and discussed in an analytical triangulation form for understanding of the takeaways each stakeholder had from the results presented by the researcher. |
3.8. Interviews and Data Reports
The data collected during the interview process was coded and categorized into three main sections: organizational, substantive, and theoretical which can be found in Appendices VI and VIII. The rationale for this method was to sort the descriptive data collected so that the material specific to the topic or issue could be physically sorted and separated from other data [4]. The first category was based on statistical information that was found within the organization such as policies, goals and mission of the hospital, retention, and turnover rates of nurses. The evaluator used this information as the basis or foundation to understand the severity of the issue and which aspects were contributing to it. The substantive data was coded based on the responses and perspectives of the nurses being interviewed. For example, the question of what their views on orientation programs were gave significant information on the perspectives of the nurses. The nurses gave responses such as, “It is an essential part of caring for patients”, “It is unnecessary and time-consuming”, “It seems impossible in times to do this because of all the distractions”. The keywords of “essential”, “unnecessary”, and “distractions” showed that there are different views on the importance of implementing bedside report. The third category was the theoretical segment which looked at literature and evidence-based studies previously conducted on effective new graduate orientation programs in hospitals. This information showed that there is significant evidence supporting that orientation programs do produce competent and safe nurses.
3.9. Collaborative Meetings
The evaluator focused the analysis on the single words, sentences, or complete paragraphs the stakeholders said to each other. While identifying these meanings coding was used to process and categorize them into lists and themes. The themes were then grouped in high-order headings with the purpose of reducing their numbers and identifying how they could be further separated. The results showed that 80% of the stakeholders used the words “preceptor”, “feedback”, and “satisfaction” throughout the entire meeting. These single words were categorized based on frequency to establish a hierarchy of priority on exactly which concepts were pertinent during the collaborative meetings. The words were categorized in this order: preceptor, feedback and satisfaction. These words were then broken down further into sub-themes to determine what aspects were of importance with these words.
4. Limitations
The potential limitations of the evaluation plan that posed issues in the strength of the data were specific to the type of population being interviewed. When interviewing current and past participants in the orientation program there was hesitancy due to fear of being fired for speaking the truth of their experiences. This lack of trust could have posed weaknesses in the responses by the participants and made the data collected false or invalid. The subjective nature of the interviews may have also allowed bias such as favoritism and politics to enter the evaluation. The ways in which the evaluator mitigated these weaknesses was by creating an interview environment that was neutral for all the participants. The setting was in a well-lighted room with two chairs adjacent to a coffee table. The intended depiction of the room was to make it feel like a casual conversation between two peers instead of an interview. The interviewee was also someone who did not work for the organization or the healthcare field. The idea was to have an unbiased interviewee to hinder any possible bias when the questions were being asked to the interviewer. The comfortable setting and ensuring each participant that their identity would remain anonymous were also potential ways the evaluator utilized to lower the limitations of the evaluation. Credibility refers to the confidence in the accuracy of the data being reported by the evaluation. Credibility is assessed by how well a researcher demonstrates their understanding of their research methodology and how well they applied this to their data collection and analysis plan [3]. When the data is presented in a clear and concise manner it assures participants and stakeholders that the results are fair and valid. Validity is drawn if there are meaningful and useful results from the experiment. The evaluator ensured that effective and congruent literature was included in the evaluation to demonstrate that the results were conclusive with past experiments. This made the results being presented more credible to the participants as they could see that prior experiments were done and showed the same or similar results. The evaluator also ensured that the results were transparent in their delivery. The participants were able to easily trace the steps the researcher took to arrive at their results in terms of data collection, analysis, and coding with this method. The use of triangulation was another method the evaluator used to conclude that the results were credible and valid. Triangulation involves the evaluator using multiple methods, data sources, observers, or theories in order to gain a more complete understanding of the topic being evaluated [3]. The evaluator included different learning theories and data collection methods to pose the most credible results to the participants and stakeholders.
Dependability is defined as the stability of data over time and conditions. Consistency and repeatable use of the instruments in a study determine how reliable and dependable the study is [3]. The strategy the evaluator conducted to ensure that dependability was seen amongst the results to participants was an inquiry audit. An inquiry audit involves the evaluator having the data collection and data analysis methods examined by an outside researcher who has no relevance to the study. This method was done to confirm the accuracy of a study’s findings. When the evaluator implemented this method, it allowed them to determine whether their conclusions and interpretations were being supported by the presented data.
When the results of a qualitative research study can be generalized and transferred to other contexts and settings, this is referred to as transferability. The evaluator enhanced transferability by doing a thorough job of describing the research context and the assumptions that were central to the evaluation plan. From the perspective of the qualitative research standpoint, the act of transferability is the primarily responsibility of the one doing the generalizing. The individual who desires to “transfer” the results to a different context or setting has the decision of making the judgment of whether results are appropriate to be applied. The evaluator ensured that the results were articulated so that the participants could better decide if the results could be applied in future setting or situations concerning orientation programs.
5. Ethical Considerations
Ethics are the moral principles that govern a person’s behavior or the conducting of an activity. Often ethical issues arise when either an individual or group fails to follow these moral principles in decision making. In the evaluation of the orientation program, there were a few ethical and risk aspects of the plan that could have imposed harm, which included: participant risk, potential coercion, conflicts of interest, confidentiality issues, and biases. Each was unique in its impact on the evaluation and required specific mitigation efforts for prevention.
When interviewing current and past participants in the orientation program there was hesitancy due to fear of being fired for speaking the truth of their experiences. This risk to the participants was that they could lose their job or bring fault to other active members in the program which could have resulted in disciplinary action for them. The potential for coercion was also a possible ethical risk in the evaluation plan. Key stakeholders or leadership could have imposed certain views or forced participants to only speak of certain topics in specific ways to prevent disciplinary action towards them, leadership. Often, threats of termination could have been made if the participant were to go against the coercion efforts. Conflicts of interest were other risks that could have occurred during the evaluation plan phase for opinions and experiences may not have matched up with everyone. This mismatch could have imposed conflict in the sight of what each individual believed or viewed what the problem was or who it was. Division could have occurred between participants and stakeholders involved in the plan which then could have led to ethical issues of imposed biases and confidentially issues. Another issue during the evaluation plan could have been that certain individuals could have felt targeted or discriminated against based on the questions and refused to answer or participate in the evaluation. Confidentiality was also a possible ethical issue the evaluation faced as well. Many participants were fearful to participant due to the risk of not remaining anonymous when speaking about specific issues on particular events that had happened. Bias was a major issue in which the evaluation plan faced due to the organization being involved in the plan. Their bias of not wanting to fault the organization or its leadership could have imposed serious ramifications in the evaluation plan by seeing the new graduate nurses as the only issue.
The way to assure that the hospital did not create ethical issues was to have the evaluation program be evaluated by an institutional review board. These boards were established to review the investigator’s research objectives, methodology, and protocols with special emphasis on the plans for recruiting the desired participants and gaining their consent [2]. The measures in which the organization practiced to prevent ethical risks were to strongly follow the three ethical principles for research: beneficence-maximizing good outcomes for humanity and research subjects, respect-protecting the autonomy of all persons and treating them with courtesy and respect specifically those who are the most vulnerable, and justice-ensuring reasonable, nonexploitative, and well-considered procedures are administered fairly. Other mitigation efforts to prevent these risks and issues was to have consent forms signed by all participants, ensuring that all information whether good or bad was kept anonymous including their names, and allowing the participants to have access to sufficient information about the study to determine any possible risks or discomforts as well as benefits they may experience [2]. Participants were also informed that they had the right to withdraw from the plan whenever they wanted without fear of persecution of losing their job or other disciplinary actions. Following these measures ensured that participant risk, potential coercion, conflicts of interest, confidentiality issues, and biases were limited.
6. Results
The evaluator’s findings revealed that all the stakeholders involved agreed that an orientation program is key to developing a competent nurse. The stakeholders agreed on the importance of the program but their rational as to what was the main issue resulted in diverse answers. Seventy-eight percent of nurse leaders who were in charge of the nursing units and oversaw the preceptee-preceptor relationships voiced that the reason the program was failing was due to a lack of competent preceptors and education. The new graduate nurses who did and were still currently working at the organization gave an 89% response that the issues were a result of poor communication, cohesiveness, and training by the organization. The preceptors who precepted the new graduate nurses reported that 70% of the issue was centered on the obstacle that the preceptors were not educated properly on how to instruct novice nurses properly. The data showed that the issue the program was experiencing was centered on a lack of education among all the key stakeholders. The preceptors or nursing leaders did not understand how to instruct or provide proper feedback to new graduate nurses, thus resulting in the high turnover rates due to poor job satisfaction.
7. Discussion
After the careful evaluation of the orientation program the conclusion revealed that a strong preceptor-preceptee relationship is the key to having a new graduate nurse successfully transition into the hospital setting. Preceptorship played an invaluable role and was associated with multiple benefits for the organization as well, such as lower adverse events and higher retention rates of new graduate nurses. The key findings also resulted in the new graduate nurses reporting that they appreciate the continuous follow-up from leadership and their preceptor. The new graduate nurses also voiced that it instilled a sense of belonging, unity, and acceptance. When the new graduate nurses felt that they belonged in the organization, the likelihood of them staying working for the organization increased.
8. Conclusions
The orientation program for new graduate nurses at the capstone study site is failing in its mission to empower new nurses with confidence and competence to deliver care in an environment where they can trust with consistent education throughout their every stage journey. The program is exceeding the expectations of turnover rates at an alarming rate over the past three years. Revenue and costs for the orientation program are being lost and wasted due to the failure of evaluating the program. Stakeholders use orientation programs as blueprints to lay the foundation for a new employee’s entire career within an organization and provide the new employee with concise and accurate information to make him/her more comfortable in their job and gain the confidence to perform to the desired expectations [1]. The evaluator utilized the proposed evaluation program to address effective communication, promote a productive workplace, and help new graduate nurses adapt to the transition of clinical practice more easily. Evaluations are tools in which evaluators use to promote and support good management practice as it helps managers adjust, revise, modify and implement new practices in which to improve program designs [4]. Evidence has revealed that orientation programs are effective means of training and educating new graduate nurses. Though this method is appropriate and brings about positive outcomes, the action of continuous evaluation is needed for these positive results to continue. The results showed that an adequate preceptor-preceptee relationship and connection is crucial for the success of novice nurses on any unit.
A recommended evaluation method that revealed successful in orientation programs for new graduate nurses was the five-stage approach developed by the University of Colorado Hospital in their SICU. This program was divided into stages that provided structure and permitted better tracking of the status of where each individual graduate nurse was in the orientation program process [1]. This approach also allowed for evaluation of the new graduate during each stage compared to the traditional orientation program which only allowed for evaluation twice throughout the entire program. Each individual stage had a checklist in which specified certain skills and content the new graduate nurse was to acquire, demonstrate, and learn in a given period of time. The checklists were designed for focused content and evaluation in which would be more user-friendly for preceptors as well [2]. These checklists also provided a guideline for how the preceptors were to train and educate their preceptees each week. The results of this new approach improved consistency between preceptors and provided more structure for the new graduate nurses. Compared to graduate nurses who completed the traditional orientation program, the nurses who experienced the stage program seemed to have an easier transition to typical unit patient assignments along with a higher build in their skills and confidence levels [1]. The continuous evaluation methods utilized by the stage program also showed higher preceptor-preceptee relationship outcomes as well. Seventy-eight percent of the preceptor-preceptee relationships reported improvement in communication both on and off the clinical setting [1]. Charge nurses and unit leaders also reported higher compliance rates with preceptors with the orientation program, seeing there was a difference in commitment and passion than before. These are possible methods and measures that should be implemented to enhance and allow for continuous evaluation for the current new graduate nursing program at the hospital of study in this report.