Journal of Nursing Management, 2016, 24, 184–191
Pre-implementation studies of a workforce planning tool for nurse
staffing and human resource management in university hospitals
CATHARINA J. VAN OOSTVEEN R N , M S c 1, DIRK T. UBBINK M D ,
EDWIN A. POMPE R N , M S c 4 and HESTER VERMEULEN R N , P h D 5
PhD
2
, MARIAN A. MENS
RN, MSc
3
,
1
PhD candidate, Departments of Surgery and Quality Assurance and Process Innovation, 2Principal Investigator,
Department of Surgery, 3Nursing director, Department of Internal Medicine, Academic Medical Centre (AMC),
4
Nursing director, Department of Surgery, Free University VU Medical Centre (VUmc), 5Associated professor,
Department of Surgery, Academic Medical Centre (AMC) and Amsterdam School of Health Professionas (ASHP),
Amsterdam University of Applied Sciences, Amsterdam, the Netherlands
Correspondence
Catharina J. van Oostveen
Department of Quality Assurance
& Process Innovation and Surgery
Room G4–141
Academic Medical Centre
PO Box 22700
1100 DE Amsterdam
the Netherlands
E-mail: c.j.vanoostveen@amc.nl
VAN OOSTVEEN C.J., UBBINK D.T., MENS M.A., POMPE E.A. & VERMEULEN H.
(2016) Journal
of Nursing Management 24, 184–191.
Pre‐implementation studies of a workforce planning tool for nurse staffing
and human resource management in university hospitals
Aim To investigate the reliability, validity and feasibility of the RAFAELA
workforce planning system (including the Oulu patient classification system –
OPCq), before deciding on implementation in Dutch hospitals.
Background The complexity of care, budgetary restraints and demand for highquality patient care have ignited the need for transparent hospital workforce
planning.
Methods Nurses from 12 wards of two university hospitals were trained to test
the reliability of the OPCq by investigating the absolute agreement of nursing
care intensity (NCI) measurements among nurses. Validity was tested by assessing
whether optimal NCI/nurse ratio, as calculated by a regression analysis in
RAFAELA, was realistic. System feasibility was investigated through a
questionnaire among all nurses involved.
Results Almost 67 000 NCI measurements were performed between December
2013 and June 2014. Agreement using the OPCq varied between 38% and 91%.
For only 1 in 12 wards was the optimal NCI area calculated judged as valid.
Although the majority of respondents was positive about the applicability and userfriendliness, RAFAELA was not accepted as useful workforce planning system.
Conclusion and implications for nursing management Nurses’ performance using
the RAFAELA system did not warrant its implementation. Hospital managers
should first focus on enlarging the readiness of nurses regarding the
implementation of a workforce planning system.
Keywords: nursing, nursing care intensity, patient classification system, personnel
staffing and scheduling, workforce planning, workload
Accepted for publication: 12 February 2015
Background
Present-day developments in care complexity, budgetary restraints and demand for safe and high-quality
patient care have exacerbated the need for systematic
184
and transparent workforce planning in hospitals (Welton et al. 2010).
Many studies show an association between nurse
staffing levels (in quantity and skill mix) and patient
outcomes (Aiken et al. 2002, 2014, Kane et al. 2007).
DOI: 10.1111/jonm.12297
ª 2015 John Wiley & Sons Ltd
Pre-implementation studies of a workforce planning tool
In hospitals with low nurse-to-patient ratios (NPRs),
adverse events occur more frequently, and patients
experience higher mortality and failure-to-rescue rates
(Aiken et al. 2002, 2014, Kane et al. 2007). Furthermore, nurses in hospitals with low NPRs are more
likely to experience burnout and job dissatisfaction
(Aiken et al. 2002, Rafferty et al. 2007). Thus, it is
important for nurse managers and policy makers to
know what determines the optimal number and skill
mix of nurses required to deliver high-quality and
cost-effective patient care.
While NPRs are easily intelligible for politicians, the
public and policy makers to understand, nurse managers have to guarantee sufficient staffing to meet the
patients’ demand for care. Therefore, appreciating the
impact of the demand for care on nursing care intensity
(NCI) would help managers to plan the optimal number
and skill mix of nurses. For this purpose, a uniform and
valid measurement and communication tool is lacking
(Morris et al. 2007). Such a tool would enable nurse
managers and nurses to balance NCI and nurse staffing
levels not only on the tactical management level (i.e.
hospital directors and policymakers) for determining
optimal nurse staffing levels but also on the operational
level in terms of admission planning (with planners and
physicians), daily nurse allocation and nurse-to-patient
assignment (with nursing colleagues) (van Oostveen
et al. 2014a,b). The need for a tool to quantify NCI is
especially high in some European countries where legislation or a national policy on nurse staffing is lacking,
unlike for example in California (NPR) (Serratt 2013)
and Australia (NHPPD) (Twigg & Duffield 2009).
Nursing care intensity is defined as ‘patient-related
workload’, as measured with a wide range of patient
classification systems (PCSs) (Fasoli & Haddock
2010). However, in the way these systems are commonly used the resulting NCI is not considered an
objective measure because of reliability and validity
problems (Fasoli & Haddock 2010). This has rarely
been investigated because methods for validating these
instruments, for example time and motion studies, are
time-consuming. At present, a variety of unreliable
and invalidated PCSs are used in hospitals, which
causes difficulties in comparing nursing intensity
scores among wards and hospitals.
A positive exception on these common but unreliable
PCSs is a workforce planning tool based on NCI, called
the ‘RAFAELA patient classification system’ which was
developed and introduced in Finland by Fagerstr€
om
and Rainio in the late 1990s (Fagerstr€
om 1999, Fagerstr€
om & Rainio 1999). The validity and feasibility of
the different parts of this system have been assessed in
ª 2015 John Wiley & Sons Ltd
Journal of Nursing Management, 2016, 24, 184–191
many clinical studies (Fagerstr€
om 1999, Rauhala 2008,
Frilund 2013). Furthermore the RAFAELA system
offers a fully Information and Communication Technology (ICT)-supported and uniform system for all
clinical nursing wards, which facilitates a clear communication about nursing care intensity on all management levels throughout the hospital, and even on
regional and national levels (Fagerstr€
om et al. 2014).
Given these purported merits, we investigated the
reliability, validity and feasibility of the RAFAELA
system in two university hospitals in the Netherlands,
before a final decision could be made on a broad
implementation of this system.
Methods
For the proper conducting and description of this
study the Standard of Quality Improvement Reporting
Excellence (SQUIRE) checklist was used (http://squirestatement.org).
Ethics
Our local medical ethics review board (Academic
Medical Centre, Amsterdam, the Netherlands)
approved the study but waived the need for written
informed consent, as the study had no effect on the
patient’s treatment or psychological wellbeing. Furthermore, the authors state they have no conflicts of
interest in implementing and evaluating RAFAELA.
Setting
Two Dutch university hospitals, each with approximately 700–1000 beds, contributed to the study: the
Academic Medical Centre (AMC) and the Free University VU Medical Centre (VUmc) in Amsterdam.
These hospitals were represented by at least five wards
of different specialties per hospital (Table 1). Each of
these wards had 20–47 operational beds and
employed 11–49 full time equivalent (FTE) nurses at
both licensed vocational nurse (LVN) and Bachelor
Science nurse (BSN) levels and working 8-hour shifts.
Staffing policies in both hospitals did not differentiate
between LVN or BSN levels.
Intervention of interest
The RAFAELA system consists of three subsystems:
(1) the Oulu Patient Classification qualisan (OPCq),
(2) a database of available nursing resources, in which
one resource unit is equal to eight nursing hours per
185
C. J. van Oostveen et al.
Table 1
Agreement based on the parallel measurements for total nursing care intensity points (NCI) per ward
Hospital A
Specialty
Neurology
Neurosurgery
Neurosurgery/orthopaedics
Gastrointestinal surgery/haematology
Vascular surgery/urology
Cardio-thoracic surgery
Short-stay surgery
Internal medicine
Pulmonology/gastrointestinal medicine
Kidney transplantation
Cardiology
Paediatrics > 1–10 years
Hospital B
Ward
Period 1
consensus (%)
Period 2
consensus (%)
Ward
Period 1
consensus (%)
Period 2
consensus (%)
1
48
75
1
62
44†
2
3
56
50
67‡
76
4
5
69
50
62‡
38†
2
3
4
5
6
7
X*
67
73
59†
X†
78
91
82
40
45
41
60
X, no data.
*Unable to participate in the parallel period.
†
Performed less than 50 parallel measurements.
‡
To continue the consensus proportion was set at 60%.
day, and (3) the Professional Assessment of the Optimal Nursing Care Intensity Level (PAONCIL) tool.
The OPCq instrument determines the individual
patients’ caring needs (NCI) per 24 hours and is based
on nursing experiences and the patient reports documented by the nurses of each contributing ward. The
OPCq consists of six subsections, or nursing areas,
regarding patient care that are to be scored; (1) planning and coordination of nursing care, (2) breathing,
blood circulation and symptoms of disease, (3) nutrition and medication, (4) personal hygiene and secretion, (5) activity, sleep and rest, and (6) teaching,
guidance in (follow-up) care and emotional support.
Each subsection is scored on a four-point scale ranging from 1 (‘slight’) up to 4 (‘very demanding’ or ‘continuously’). Therefore, the total NCI score can vary
between 6 and 24 points per patient.
The PAONCIL tool is used to calculate the optimal
daily NCI/N for each ward by dividing the total NCI
by the available nursing resources. As input for the
PAONCIL calculation, nurses have to assess the optimal NCI on a scale from –3 (below optimal), to 0
(optimal) and up to +3 (above optimal).
To determine an estimate of the optimal NCI/N per
ward, the daily NCI/Ns are compared with the average PAONCIL values by means of a regression analysis, which is integrated into the system. The resulting
estimate is used to determine the optimal NCI range
(i.e. optimum value 15%). Comparing this area
with the daily NCI/N provides information about the
adequacy of the current nurse staffing level and facilitates solutions for (ad hoc) staff (re-)allocation (Fagerstr€
om et al. 2014).
186
The PAONCIL instrument also includes 12 additional non-patient factors to assess ward processes and
aspects that may affect nurses’ workload during a shift
(i.e. organizational issues, planning issues, managerial
roles, staff situations, meetings, trainings or other
absences, students, collaboration among nursing team
members, collaboration with physicians, collaboration
with other disciplines, nurses’ own physical and mental state, and other factors) (Rauhala & Fagerstr€
om
2007).
Several conditions must be met to enable calculation
of the optimal NCI area: (1) assessment of the OPCq
should be reliable (i.e. the agreement between OPCq
measurements by two nurses of the same patient
should be at least 70%) (Frilund & Fagerstr€
om 2009),
(2) the available resources must be recorded completely, and (3) At least 70% of the nurses must assess
their NCI by means of the PAONCIL tool during the
measurement period. Finally, the regression analyses
have to find an explanation degree of at least 25% for
PAONCIL explaining the NCI/N (Rauhala & Fagerstr€
om 2004).
Design
This study contained three parts, based on the three
study questions:

A reliability study, investigating the agreement
among nurses when scoring NCI using the OPCq of
the RAFAELA system;
A validity study, in which head nurses were to
assess whether these NCI scores, together with the
nurses’ appreciation of their workload, would result
ª 2015 John Wiley & Sons Ltd
Journal of Nursing Management, 2016, 24, 184–191
Pre-implementation studies of a workforce planning tool
in a realistic NCI/N score, as calculated and presented graphically by the RAFAELA system. This
would supply them with valuable information
regarding staff allocation and benchmarking.
A feasibility study (in terms of user-friendliness,
applicability and acceptability) of the whole RAFAELA system, as judged by all nurses involved.
Each of these study parts would result in a ‘go’ or
‘no-go’ outcome regarding a hospital-wide implementation of the RAFAELA system. Criteria for a ‘go’
were a 70% agreement regarding the reliability (Frilund & Fagerstr€
om 2009), a ‘realistic’ verdict as to its
validity and 50% of the nurses should appreciate the
RAFAELA system as ‘feasible’. Feasibility was justified
if the median Likert score was ≥ 5 and less than 25%
of the scores were below the 25% percentile (Fitch
et al. 2001).
The results of the different study parts had no consequences for current ward processes or policies, as
this study was a pre-implementation study.
The study
The OPCq and the PAONCIL instruments were translated into Dutch by the researchers based on forward
and backward translation.
Data collection for the three study parts took place
from December 2013 to June 2014. All nurses on the
12 participating wards were to measure their patients’
NCI once per 24 hours, 7 days a week, between
December 2013 and June 2014.
To facilitate the introduction of RAFAELA on the
nursing wards, a users’ support team was created in
each hospital. These teams consisted of one ‘super-user’
(a researcher) for conducting the study and at least two
‘key-users’ (nurses) per participating hospital ward for
teaching and motivating the nursing team involved. All
super-users and key-users attended three dedicated RAFAELA trainings conducted by an associate of the Finnish supplier of RAFAELA, FCG International Ltd,
Helsinki. During a 3-month practicing period, nurses
gained experience in measuring the nursing care intensity by the OPCq, while members of the support team
practiced recording nursing resources.
Part 1: reliability study
After the training period, the NCI of at least 50
patients per ward were scored using the OPCq in the
RAFAELA system by two nurses independently. These
parallel measurements were taken once per 24 hours
during a 1-week period. The RAFAELA system
ª 2015 John Wiley & Sons Ltd
Journal of Nursing Management, 2016, 24, 184–191
provides an absolute measure of agreement between
two parallel OPCq measurements in the same patient.
Agreement was defined as a difference between the
nurses’ scores of less than two NCI points (Frilund &
Fagerstr€
om 2009).
Part 2: validity study
Nurses were to score the PAONCIL every shift and
the NCI once per 24 hours during a 6-week period.
Based on these data, the RAFAELA system generates
output about the NCI/N for each ward.
Subsequently, these management reports regarding
the optimum NCI/N would be presented to the head
nurses of each contributing ward to assess face validity of the RAFAELA system.
Part 3: feasibility study
Nurses of all contributing wards were asked to evaluate the user-friendliness (functionality), applicability,
and the acceptability of RAFAELA by means of a digital questionnaire (https://www.surveymonkey.com).
The questionnaire contained 17 questions, each with a
10-point Likert scale, and four open questions about
the use of RAFAELA (see the Supporting Information,
Appendix S1), which were analysed descriptively.
Nurses were given 2 weeks to complete the questionnaire and received two reminders if needed.
Results
From December 2013 until June 2014, 38 819 and
26 261 OPCq measurements and 1441 and 405 PAONCIL measurements were performed in hospitals A
and B, respectively, by 443 nurses, totalling 66 926
measurements.
Part 1: reliability study
Agreement for the OPCq measurements ranged from
40% to 67% for hospital A and 50% to 69% for hospital B. Given these low agreement results, it was
decided to allow a second measurement after another
1 months’ training and motivation period for the
nurses involved, and accepting an agreement of at
least 60%. This resulted in agreements between 59%
and 91% for hospital A, and between 38% and 76%
for hospital B (Table 1). The number of measurements
performed increased by 20% (Figure 1). A total of
eight wards scored a sufficient agreement to continue
on to Part 2 of the study: in hospital A five out of
seven and in hospital B three wards out of the five
passed.
187
C. J. van Oostveen et al.
100%
% Measurements performed
80%
60%
40%
20%
0%
Ward 1
Ward 2
Ward 3
Ward 4
Ward 5
Ward 6
Ward 7
Period 1 A
77%
0%
15%
62%
82%
72%
56%
Period 1 B
94%
90%
83%
85%
49%
Period 2 A
95%
85%
52%
0%
92%
94%
79%
Period 2 B
91%
93%
84%
77%
95%
Figure 1
Mean percentages Oulu patient classification system (OPCq) measurements performed by hospitals A and
B per ward. For wards 1–7 see
Table 1.
percentile (Table 2), for the questions (Q) 9–17
(Appendix S1). The respondents perceived the OPCq
as a suitable instrument to measure all aspects of the
nursing care intensity (Table 2: Q9, 10% < 25th percentile). However, the OPCq was not perceived as a
correct reflection of the nursing care intensity
(Table 2: Q10, 20% < 25th percentile). The respondents were positive about the usability of RAFAELA
(Table 2: Q13 and Q15, 7% and 15% < 25th percentile), but did not see RAFAELA as an improvement
(Table 2: Q16 and Q17, 30% and 34% < 25th percentile). In the open questions about half of the
respondents were positive and appreciated the benefits
of RAFAELA on the operational level – i.e. (ad hoc)
allocation of nursing resources and nurse–patient
assignment.
Part 2: validity study
Only the neurology/neurosurgery ward in hospital A
reached the threshold of 70% of their nurses measuring the PAONCIL (77%; Figure 2) required to calculate an optimal NCI area for their ward (Figure 3).
The variance explained by the regression model was
29.4%. The head nurse involved studied the output
and judged it to be valid and valuable for staff planning and benchmarking.
Part 3: feasibility study
The response rate for the questionnaire was 30%.
Median scores for each question varied between 4 and
8, while 2–34% of the scores were below the 25th
% Measurements performed
100%
80%
60%
40%
20%
0%
188
Ward 1
A
B
Ward 2
Ward 3
Ward 4
Ward 5
Ward 6
Ward 7
39%
0%
46%
69%
40%
77%
64%
0%
51%
0%
22%
0%
Figure 2
Mean percentages Professional
Assessment of the Optimal Nursing
Care Intensity Level (PAONCIL)
measurements performed by hospitals A and B per ward. For wards
1–7 see Table 1.
ª 2015 John Wiley & Sons Ltd
Journal of Nursing Management, 2016, 24, 184–191
Pre-implementation studies of a workforce planning tool
21
20
19
18
17
16
15
Above the optimum
14
13
Optimum
12
11
Below the optimum
10
9
8
7
6
5
4
3
2
1
01.04.2014
02.04.2014
03.04.2014
04.04.2014
05.04.2014
06.04.2014
07.04.2014
08.04.2014
09.04.2014
10.04.2014
11.04.2014
12.04.2014
13.04.2014
14.04.2014
15.04.2014
16.04.2014
17.04.2014
18.04.2014
19.04.2014
20.04.2014
21.04.2014
22.04.2014
23.04.2014
24.04.2014
25.04.2014
26.04.2014
27.04.2014
28.04.2014
29.04.2014
30.04.2014
01.05.2014
02.05.2014
03.05.2014
04.05.2014
05.05.2014
06.05.2014
07.05.2014
08.05.2014
09.05.2014
10.05.2014
11.05.2014
12.05.2014
13.05.2014
14.05.2014
15.05.2014
16.05.2014
17.05.2014
18.05.2014
19.05.2014
20.05.2014
21.05.2014
22.05.2014
23.05.2014
24.05.2014
25.05.2014
26.05.2014
27.05.2014
28.05.2014
29.05.2014
30.05.2014
0
Figure 3
Optimal Nursing Context Index
(NCI)/N area.
Hospital A – ward 4
Table 2
Main results from the digital survey
Question
Median
%
Purchase answer to see full
attachment