Original article

Validity and Reliability of Neonatal Infant Pain Scale (NIPS) in Neonatal Intensive Care Unit in Vietnam

Hai Thanh Ngoa, Kathleen Fitzsimmonsb, Kien Gia Toc,*
Author Information & Copyright
aVINMEC General Hospital; 208 Nguyen Huu Canh, Ward 22, Binh Thanh District, Ho Chi Minh City, Vietnam
bGreenville County Schools, Greenville, South Carolina, USA
cFaculty of Public Health, University of Medicine and Pharmacy at Ho Chi Minh City; 217 Hong Bang, Ward 11, District 5, Ho Chi Minh City, Vietnam
*Address correspondence to Kien Gia To at the Faculty of Public Health, University of Medicine and Pharmacy at Ho Chi Minh City; 217 Hong Bang, Ward 11, District 5, Ho Chi Minh City, Vietnam; Tel/Fax: (+84) 91 95 11 121; E-mails: kiengiato@gmail.com or togiakien@ump.edu.vn

ⓒ Copyright 2019 MedPharmRes. This is an Open-Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Received: Jul 14, 2018; Revised: Jun 11, 2019; Accepted: Jul 04, 2019

Published Online: Sep 30, 2019

Abstract

Background:

The study aimed to culturally adapt and validate Neonatal Infant Pain Scale (NIPS) for use in Vietnamese settings.

Methods:

The original NIPS was translated into Vietnamese using a standard protocol. Registered nurses of Neonatal Intensive Care Unit (NICU), Tien Giang General Hospital, Vietnam used the Vietnamese NIPS for assessing neonatal pain and then provided feedback on acceptability of the scale. Five registered nurses of NICU were randomly selected and used NIPS for assessing neonatal pain while watching thirty videos at two times, two weeks apart from each other. Pulse rates per minute and oxygen saturation (SpO2) were also recorded for validity evaluation. Intraclass correlation coefficients (ICC) with two-way random effects were applied to assess intra-rater and inter-rater reliability. Multilevel linear regression was applied to assess the association between NIPS score with pulse rates and SpO2 adjusting for raters, three periods and two assessments.

Results:

The Vietnamese NIPS was accepted and valued by nurses at the NICU. ICCs between the first and second assessments were from 0.53 to 1.00 for five raters before, during and after clinical procedures showing moderate to excellent intra-rater reliability. ICCs among five raters were moderate to good before and after, but poor (ICC<0.4) during clinical procedures. NIPS score was not associated with SpO2, but with pulse rates per minute.

Conclusions:

The preliminary results showed that the Vietnamese version of NIPS is reliable and should be used. However, it is recommended that further research should be conducted to confirm its reliability and validity.

Keywords: NIPS; reliability; validity; NICU; Vietnam

1. INTRODUCTION

Pain is defined as “an unpleasant sensory and emotional experience associated with actual or potential tissue damage or described in terms of such damage” [1]. Pain in neonates has been extensively studied in recent years and evidence suggests that neonates experience pain much more severe than adult or older children do as neonatal pain seriously affects development of nervous system and growing of babies [2]. Literatures showed that neonates experience many painful and stressful procedures in Neonatal Intensive Care Unit (NICU) [3-5].

Pain should be assessed for optimal care of neonates in NICU [5, 6]; however, assessing pain is very difficult and complicated, particularly in neonates as they are unable to verbally communicate [5, 7]. Although some validated and reliable pain scales are available for assessing neonatal pain, they are rarely used in clinical practice and most of the scales are in English which require translation and cross-cultural adaptation for use in local contexts [4, 5, 8-11].

Neonatal Infant Pain Scale (NIPS) was developed by Lawrence et al. at Children’s Hospital of Eastern Ontario to assess pain in neonates based on behaviors. Their study included videotaping 90 medical procedures of 38 neonates before, during and after procedures [12]. The Cronbach’s Alpha coefficients before, during and after procedures were 0.95, 0.87 and 0.88, respectively. The Pearson correlation was 0.92 to 0.97 showed good inter-rater reliability. NIPS includes six items assessing six states of neonates including facial expression, cry, breathing patterns, movement of arms, movement of legs and state of arousal. Each item is scored 0 to 1, except state of cry is scored 0, 1 and 2, for a possible total score of 7. Neonates have no pain if NIPS score is 0-2, moderate pain if NIPS score is 3-4 and severe pain if NIPS score is more than 4.

NIPS has the advantage that it does not require users have additional skills or equipment to assess pain and would be a good tool for assessing and improving pain management of neonates [2, 4]. Moreover, it is valid and reliable and has been used worldwide [10, 13-15]. A study conducted in a southern California hospital to assess the inter-rater reliability of NIPS using 27 neonates with 100 medical procedures showed high Cronbach’s Alpha coefficients of 0.9, 0.9, and 0.86 before, during and after medical procedures [14]. Moreover, Pearson’s correlation coefficients of 0.82, 0.75 and 0.81 showed good inter-rater reliability between three raters including a nursing faculty, a senior resident doctor and a post-graduate neonatal nurse [14]. NIPS was successfully translated into Persian with good inter-rater reliability of 0.87 and excellent intraclass correlation coefficient of higher than 0.9 [15]. The Brazilian version of NIPS provided a Cronbach’s alpha of 0.76, a kappa score of 0.93, inter-rater reliability of 95% and intra-rater reliability of 90% [10].

In Vietnam, assessing neonatal pain in NICU has not been a routine practice as few studies have been done to provide supporting evidences. Moreover, valid and reliable scales were not available. Therefore, the aim of this study was to cross-culturally adapt NIPS for use in Vietnamese hospitals and assess its validity and reliability. The study is a useful reference for practitioners to manage neonatal pain in NICU, and for students and researchers to do their studies.

2. METHODS

2.1. Study setting

The study was conducted at Tien Giang General Hospital located in My Tho City, Tien Giang province, Mekong Delta, the South of Vietnam. Tien Giang has an area of 2,367 km squared and a population of 1.7 million. The hospital was founded in 1921 and is responsible for healthcare of people from Tien Giang and nearby provinces. The NICU of Tien Giang General Hospital has 14 nurses who are responsible for caring for under-one-month old neonates.

2.2. Translation process

A registered nurse and an English teacher independently translated NIPS into Vietnamese. Two Vietnamese copies were compared, differences noted, and were then synthesized by one researcher. The final Vietnamese version of NIPS was independently back translated into English by another registered nurse and English teacher. The English-backward translated versions of NIPS were compared with the original NIPS. Again, differences were discussed among a pediatrist and two registered nurses who are native English speakers until a consensus was reach. The final Vietnamese version of NIPS was reviewed and accredited by an expert panel of Tien Giang General Hospital. The study process is presented in Figure 1.

mpr-3-2-1-g1
Figure 1: The study process
Download Original Figure
2.3. Pilot study to assess the characteristics of the Vietnamese version of NIPS

Registered nurses, who were working at Neonatal Intensive Care Unit (NICU) of Tien Giang general hospital, were invited to participate in the pilot study. Inclusion criteria were nurses working at the NICU of Tien Giang General Hospital and responsible for caring newborn babies. Participants were excluded if they were a probationer or experienced less than one working year at the hospital. A researcher approached all nurses of NICU and screened for their eligibility. If they were eligible, they were informed about the purpose of the study and invited to participate in it. If they agreed, they were asked to sign an informed consent before joining the study. All 14 nurses were eligible and were trained to use the Vietnamese NIPS as suggested by Gallo [2]. They were then asked to use the tool to assess pain level of newborn babies in the NICU as well as answer-seven questions to assess whether the NIPS was 1) clear and easy to understand, 2) easy to use, 3) convenient for use, 4) time-consuming, 5) helpful for nurses to decide when to treat pain, 6) practical to use in clinical practice, and 7) able to classify levels of pain. The seven questions were Likert-style scales and successfully used in previous studies [16, 17]. Each item was rated from 1 (strongly disagree) to 5 (strongly agree). Moreover, the length of time needed by nurses to complete the assessment was recorded and reported. The mean and standard deviation of 7 items and the duration of assessment were calculated.

2.4. Main study to assess validity and reliability of NIPS

Thirty neonates hospitalized in the NICU and prescribed an intramuscular injection (15 neonates) and an intravenous catheter insertion (15 neonates) were recruited for the study. A total of 30 videos was recorded. The neonates having Apgar scores of at least seven were selected based on consultation with medical doctors. These neonates were monitored for pulse rates per minute and oxygen saturation (SpO2) before, during and after intramuscular injection and catheter insertion. Literature shows that pulse rates were positively associated with painful level whereas SpO2 was negatively associated [18-20]. All the above clinical interventions were normal procedures in the care of neonates in the NICU of the hospital. Neonates who were prescribed painkillers, had congenital or nervous abnormalities, used respiratory support-device, or had birth by Caesarean section were excluded from the study. A researcher screened for the eligibility of neonates. He approached and explained the study to parents of eligible neonates and asked if they allowed their neonates participating in the study. If they agreed, they were asked to sign an informed consent before their neonates participating in the study.

Five nurses were randomly selected from all nurses of the NICU. The inclusion and exclusion criteria were the same as those had been used in the pilot study. These five nurses were asked to use NIPS to assess pain of neonates. After two weeks, the same five nurses were asked to re-assess the videos using the same tool. The two-week interval was considered appropriate to assess the intra-rater reliability [10]. The sample size of 30 videos and five raters was acceptable for reliability study [21].

2.5. Statistical analysis

All statistical analysis was performed using STATA13. All nurses were asked to watch 30 videos and use NIPS to assess the pain of recorded neonates. Two weeks after the first assessment, they were asked to re-assess the videos. Intra-rater reliability was calculated to assess a difference in NIPS mean score of five raters between two assessments (first and second assessment) at three periods (before, during and after clinical intervention) using group-average intra-class correlation coefficient (ICC) with two-way mixed-effects absolute agreement model [21]. Intra-rater reliability of each of five raters between two assessments at three periods was also calculated using individual ICC with two-way mixed-effects absolute agreement model [21]. Inter-rater reliability was assessed using individual ICC with two-way random-effect absolute agreement model [21-25]. ICC was classified as poor (ICC < 0.5), moderate (0.5 ≤ ICC ≤ 0.75), good (0.75 < ICC ≤ 0.9), and excellent (ICC > 0.9) reliability [21].

As pulse rates per minute and SpO2 predicted level of pain [18-20], they were used to assess NIPS validity. Associations between pulse rates per minute and SpO2 with NIPS score were checked using multilevel linear regression. Three levels were raters (five nurses), three periods of the clinical procedures (before, during, and after), and two assessments (first or second). The total number of observations of 30 videos of five raters in three periods after two assessments is 900.

2.6. Ethical issues

The study was approved by the ethics committee of the University of Medicine and Pharmacy at Ho Chi Minh City (55ÐHYD-HÐ/2017); and accepted by the Executive Board of Tien Giang general hospital.

3. RESULTS

The translation of NIPS into Vietnamese appeared to be successful as shown in Table 1. The English back translation version kept the original meaning despite some minor differences in wording.

Table 1: The difference between the original NIPS and English-back translation of its Vietnamese version
English-back translation of Vietnamese NIPS The original NIPS (Lawrence et al., 1993)
Facial Expression Facial Expression
0 - Relaxed muscles
1 - Grimace
Restful face, neutral expression
Tight facial muscles, furrowed brow, chin, jaw
0 - Relaxed muscles
1 - Grimace
Restful face, neutral expression
Tight facial muscles, furrowed brow, chin, jaw
Cry Cry
0 - Not crying
1 - Whimper
2 - Vigorous cry
Quiet, not crying
Mild moaning, intermittent
Loud scream, shrill, continuous (Note: Silent cry may be scored if the infant is intubated, when there is evidences by obvious mouth, facial movements)
0 - No cry
1 - Whimper
2 - Vigorous cry
Quiet, not crying
Mild moaning, intermittent
Loud scream, shrill, continuous
(Note: Silent cry may be scored if baby is intubated, as evidenced by obvious mouth, facial movement)
Breathing Patterns Breathing Patterns
0 - Relaxed
1 - Change in breathing
Usual breathing pattern
Indrawing, irregular, faster than usual, gagging, breath holding
0 - Relaxed
1 - Change in breathing
Usual breathing pattern for this baby
Indrawing, irregular, faster than usual, gagging, breath holding
Arms Arms
0 - Relaxed /restrained
1 - Flexed /extended
No muscular rigidity, occasional random movements of arms
Tense, straight arms, rigid and/or rapid extension/flexion
0 - Relaxed /restrained
1 - Flexed /extended
No muscular rigidity, occasional random movements of arms
Tense, straight arms, rigid and/or rapid extension/flexion
Legs Legs
0 - Relaxed /restrained
1 - Flexed /extended
No muscular rigidity, occasional random movements of legs
Tense, straight legs, rigid and/or rapid extension/flexion
0 - Relaxed /restrained
1 - Flexed /extended
No muscular rigidity, occasional random leg movement
Tense, straight legs, rigid and/or rapid extension/flexion
State of Arousal State of Arousal
0 - Sleeping /awake
1 - Fussy
Quiet, peaceful, sleeping or alert and stabilized
Alert, restless, and thrashing
0 - Sleeping /awake
1 - Fussy
Quiet, peaceful, sleeping or alert and settled
Alert, restless, and thrashing

* Total score from 0 to 7

0-2: no pain; 3-4: moderate pain; >4: severe pain

Download Excel Table

Table 2 shows the characteristics of the Vietnamese version of NIPS assessed by 14 nurses at the NICU, Tien Giang General Hospital. On average, assessment points for the characteristics ranged from 3.4 points for “helps nurses give decision on medical care” to 4.4 points for “clear and understandable”. The mean duration NIPS took nurses was 1.6 minutes ranging from one to two minutes.

Table 2: The characteristics of Vietnamese version of NIPS were assessed by 14 nurses at Neonatal Intensive Care Unit (NICU), Tien Giang General Hospital
Characteristics of NIPS Mean± SD Min Max
NIPS is clear and understandable 4.4± 0.6 3 5
NIPS is easy to use 4.1± 0.6 3 5
NIPS is convenient for nurse 4.0± 0.4 3 5
NIPS is less time-consuming 3.9± 0.5 3 5
NIPS helps nurse give decision on medical care 3.4± 0.5 3 4
NIPS is feasible and applicable to clinical context 3.6± 0.5 3 4
NIPS is able to classify pain level 3.5± 0.5 3 4

SD: standard deviation; Min: Minimum; Max: Maximum

Download Excel Table

Group-average ICCs of five nurses showed excellent intra-rater reliability before (ICC=0.92, 95%CI: 0.88-0.94) and after (ICC=0.99, 95%CI: 0.98-0.99) clinical procedures, but good (ICC=0.81, 95%CI: 0.73-0.86) during clinical procedure (p<0.01) (table 3). All individual ICCs showed good to excellent agreements (ICC ≥ 0.84) with the exception of rater 1 before clinical procedures, and rater 2, 3, 5 during clinical procedure just showing moderate agreement (ICC=0.53 to 0.72, p<0.01).

Table 3: The group-average and individual intra-rater reliability of NIPS before, during and after clinical procedures
Before During After
ICC (95%CI) p ICC (95%CI) p ICC (95%CI) p
Overall intra-rater 0.92 (0.88; 0.94) <0.01 0.81 (0.73; 0.86) <0.01 0.99 (0.98; 0.99) <0.01
  Rater 1 0.72 (0.50; 0.86) <0.01 0.85 (0.72; 0.93) <0.01 0.94 (0.88; 0.97) <0.01
  Rater 2 0.84 (0.70; 0.92) <0.01 0.53 (0.22; 0.75) <0.01 1 (1; 1) <0.01
  Rater 3 0.84 (0.70; 0.92) <0.01 0.67 (0.42; 0.83) <0.01 1 (1; 1) <0.01
  Rater 4 0.84 (0.70; 0.92) <0.01 0.88 (0.77; 0.94) <0.01 1 (1; 1) <0.01
  Rater 5 0.93 (0.85; 0.96) <0.01 0.66 (0.40; 0.82) <0.01 0.97 (0.93; 0.98) <0.01

ICC: intraclass correlation coefficient; 95%CI: 95% confidence interval

Five nurses assessed 30 videos in first and second assessment for group-average intra-rater reliability using two-way mixed- effects absolute agreement model.

For each rater, individual intra-rater reliability was reported using two-way mixed-effects absolute agreement model.

Download Excel Table

At first and second assessments, almost 100% videos were rated no pain before clinical procedure at the first and second assessments. Five nurses showed good agreements (ICC=0.77, 95%CI: 0.66-0.87) before clinical procedures, poor agreements (ICC=0.28, 95%CI: 0.14-0.48 and 0.33, 95%CI: 0.18-0.53) during clinical procedures, and moderate agreements (ICC=0.60, 95%CI: 0.45-0.75 and 0.63, 95%CI: 0.48-0.77) after clinical procedures (p<0.01) at the first and second assessments (table 4).

Table 4: The inter-rater reliability of NIPS between five nurses using NIPS to assess pain score before, during and after clinical procedures at the first and second assessment
Before (n=30) During (n=30) After (n=30)
ICC (95%CI) p ICC (95%CI) p ICC (95%CI) p
First assessment 0.77 (0.66; 0.87) <0.01 0.28 (0.14; 0.48) <0.01 0.60 (0.45; 0.75) <0.01
Second assessment 0.77 (0.66; 0.87) <0.01 0.33 (0.18; 0.53) <0.01 0.63 (0.48; 0.77) <0.01

ICC: intraclass correlation coefficient; 95%CI: 95% confidence interval

Five nurses assessed 30 videos, two-way random-effects absolute agreement model was used to assess inter-rater reliability

Download Excel Table

After adjusting for raters, periods and assessments, the multilevel linear regression showed that NIPS score was not significantly associated with SpO2 (p>0.05). However, for every point increase in NIPS score, pulse rate per minute increased three beats (p<0.01) (Table 5).

Table 5: Multilevel linear regression assessing the association of NIPS score with SpO2 and pulse rates per minute adjusting for raters, period (before, during and after clinical intervention) and time (first vs. second assessment) (n=900)
SpO2 (%) Pulse rates per minute
Coef. p-value 95%CI Coef. p-value 95%CI
NIPS score -0.15 0.40 -0.51; 0.20 3.25 <0.01 1.19; 5.31
Rater 1 1 1
Rater 2 -0.02 0.95 -0.68; 0.63 0.45 0.82 -3.34; 4.24
Rater 3 -0.01 0.98 -0.66; 0.65 0.14 0.94 -3.64; 3.92
Rater 4 -0.00 0.99 -0.66; 0.65 0.05 0.98 -3.72; 3.83
Rater 5 0.01 0.97 -0.64; 0.67 -0.27 0.89 -4.05; 3.51
Before clinical intervention 1 1
During clinical intervention -15.09 <0.01 -17.50; -12.69 25.54 <0.01 11.62; 39.47
After clinical intervention -0.14 0.58 -0.65; 0.36 0.05 0.98 -2.90; 2.99
Assessments (second vs. first) 0.01 0.97 -0.41; 0.42 -0.15 0.90 -2.54; 2.24

Multilevel linear regression model; Coef.: Beta coefficient; 95%CI: 95% Confidence Interval.

Download Excel Table

4. DISCUSSION

This study translated and adapted the original NIPS into Vietnamese using the process guided by World Health Organization [26]. English-Vietnamese and Vietnamese-English translations of NIPS is highly consistent. The Vietnamese version of NIPS were also reviewed and accredited by experts and experienced NICU nurses. In order to apply the NIPS, it is important that hospitals’ nurses accept and are willing to use it. Our study indicated that all nurses agreed that the Vietnamese version of NIPS is applicable in the local hospital context.

The NIPS mean score indicated that neonates have almost no pain before and after procedures but severe pain during them. Neonatal pain is highly frequent in NICU because neonates suffered seven invasive procedures per day on average and a half of them had at least pain once during their hospitalization [27]. Compared to other neonatal pain assessment tools, NIPS is easy to use and not time-consuming; and does not require extra equipment [5]. Therefore, NIPS is applicable in NICU.

The Vietnamese version of NIPS had good test-retest reliability as its average ICCs of five raters were above 0.75 before, during and after clinical procedures, and individual ICCs were from moderate to excellent (0.53 to 1). Five raters showed good agreements (ICC=0.77) before clinical procedures and moderate agreements (ICC≥0.60) after clinical procedures at first and second assessment. However, poor agreement (ICC<0.4) were detected between the five raters during clinical procedures at the two assessments. As rater 2 had lower ICCs during clinical procedures, this rater was removed from the calculation. ICCs were re-calculated among the other four raters which improved to 0.51 at the first assessment and 0.49 at the second assessment. A possible explanation was that four raters had different level of clinical experiences from the rater 2 as an experienced pediatric nurse tends to rate lower pain score [28]. However, the explanation could not be confirmed as personal data of raters were not collected in this study. More training on how to identify different aspects of pain may be provided to nurses to help increase inter-rater reliability.

This study used SpO2 and pulse rates per minute which were objective measures to assess validity of the NIPS. Although the NIPS score was not significantly associated with SpO2, it was positively associated with pulse rates. This finding may reflect the inconsistent results in previous studies. While some showed that heart rate increased and SpO2 decreased during painful procedures [15, 18], the other found that heart rate and SpO2 were not sensitive and not associated with pain scores [29].

Our study has some limitations. Firstly, due to very low variability, it was unable to calculate internal consistency for the entire scale. Secondly, although there are many clinical procedures conducted in the NICU, only two clinical procedures were observed in this study, including intramuscular injection and intravenous catheter insertion. Thirdly, the sample size is small and the study was only conducted in one hospital in the South of Vietnam. Therefore, the results may not be generalizable to other areas of the country.

In conclusion, the preliminary results showed that the Vietnamese version of NIPS showed acceptable reliability for use in clinical settings. As there was currently no validated tool for assessing neonatal pain available in Vietnamese, this tool should be used although it is recommended that further research should be conducted to confirm its reliability and validity.

LIST OF ABBREVIATIONS

ICC:

intra-class correlation coefficient; NICU: Neonatal Intensive Care Unit; SpO2: Oxygen saturation.

Notes

CONFLICT OF INTEREST The authors declare that they have no competing interests.

ACKNOWLEDGEMENTS

We would like to thank Dr. Quyen Gia To, Queensland University of Technology, for his helpful advice on statistics. Our special thanks should go to nurses of Neonatal Intensive Care Unit and the Executive Board of Tien Giang Hospital for accepting the study. We also thank Dr. Deborah Kupecz for her helpful comments on translation; the reviewers for their useful comments on the manuscript; and all participants for participating in this study. All other contributions to the study should be acknowledged.

REFERENCES

1.

Loeser JD, Treede RD. The Kyoto protocol of IASP Basic Pain Terminology. Pain. 2008; 137(3):473-7

2.

Gallo AM. The fifth vital sign: implementation of the Neonatal Infant Pain Scale. J Obstet Gynecol Neonatal Nurs. 2003; 32(2):199-206

3.

Carbajal R, Rousset A, Danan C, Coquery S, Nolent P, Ducrocq S, et al. Epidemiology and treatment of painful procedures in neonates in intensive care units. JAMA. 2008; 300(1):60-70

4.

Polkki T, Korhonen A, Axelin A, Saarela T, Laukkala H. Development and preliminary validation of the Neonatal Infant Acute Pain Assessment Scale (NIAPAS). Int J Nurs Stud. 2014; 51(12):1585-94

5.

American Academy of Pediatrics. Prevention and management of pain in the neonate: an update. Committee on Fetus and Newborn and Section on Surgery, Section on Anesthesiology and Pain Medicine, Canadian Paediatric Society, and Fetus and Newborn Committee. 2006 Nov; Report No.: 1098-4275 (Electronic), 0031-4005 (Linking) Contract No.: 5.

6.

Aukes DI, Roofthooft DW, Simons SH, Tibboe D, van Dijk M. Pain Management in Neonatal Intensive Care: Evaluation of the Compliance with Guidelines. Clin J Pain. 2014

7.

Bhalla T, Shepherd E, Tobias JD. Neonatal pain management. Saudi J Anaesth. 2014; 8(Suppl 1):S89-97

8.

Guedj R, Danan C, Daoud P, Zupan V, Renolleau S, Zana E, et al. Does neonatal pain management in intensive care units differ between night and day? An observational study. BMJ Open. 2014; 4(2)e004086

9.

Medicine COFNSOAP. Prevention and Management of Procedural Pain in the Neonate: An Update. Pediatrics. 2016; 137(2)e20154271.

10.

Motta Gde C, Schardosim JM, Cunha ML. Neonatal Infant Pain Scale: Cross-Cultural Adaptation and Validation in Brazil. J Pain Symptom Manage. 2015; 50(3):394-401

11.

Stapelkamp C, Carter B, Gordon J, Watts C. Assessment of acute pain in children: development of evidence-based guidelines. International journal of evidence-based healthcare. 2011; 9(1):39-50

12.

Lawrence J, Alcock D, McGrath P, Kay J, MacMurray SB, Dulberg C. The development of a tool to assess neonatal pain. Neonatal Netw. 1993; 12(6):59-66.

13.

Hudson-Barr D, Capper-Michel B, Lambert S, Palermo TM, Morbeto K, Lombardo S. Validation of the Pain Assessment in Neonates (PAIN) scale with the Neonatal Infant Pain Scale (NIPS). Neonatal Netw. 2002; 21(6):15-21

14.

Malarvizhi G, Manju V, Roseline M, Nithin S, Sarah P. Inter-rater reliability of neonatal infant pain scale (NIPS) as a multidimensional behavioural pain tool. Nitte University Journal of Health Science. 2012; 2(1):26-30

15.

Sarhangi F, Mollahadi M, Ebadi A, Matinzadeh ZK, Tadrisi SD. Validity and reliability of Neonatal Infant Pain Scale in Neonatal Intensive Care Units in Iran (2010). Pak J Med Sci. 2011; 27(5):1087-91.

16.

Suraseranivongse S, Kaosaard R, Intakong P, Pornsiriprasert S, Karnchana Y, Kaopinpruck J, et al. A comparison of postoperative pain scales in neonates. Br J Anaesth. 2006; 97(4):540-4

17.

Taddio A, Hogan ME, Moyer P, Girgis A, Gerges S, Wang L, et al. Evaluation of the reliability, validity and practicality of 3 measures of acute pain in infants undergoing immunization injections. Vaccine. 2011; 29(7):1390-4

18.

Taksande AM, Vilhekar KY, Jain M, Chitre D. Pain response of neonates to venipuncture. Indian J Pediatr. 2005; 72(9):751-3

19.

Gokulu G, Bilgen H, Ozdemir H, Sarioz A, Memisoglu A, Gucuyener K, et al. Comparative heel stick study showed that newborn infants who had undergone repeated painful procedures showed increased short-term pain responses. Acta paediatrica (Oslo, Norway : 1992). 2016; 105(11):e520-e5

20.

Porter FL, Wolf CM, Miller JP. Procedural pain in newborn infants: the influence of intensity and development. Pediatrics. 1999; 104(1):e13

21.

Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med. 2016; 15(2):155-63

22.

StataCorp LLC. ICC - Intraclass correlation coefficients. Available from: https://www.stata.com/manuals/ricc.pdf#riccRemarksandexamples.

23.

Yoon PJ, Starr JR, Perkins JA, Bloom D, Sie KCY. Interrater and Intrarater Reliability in the Evaluation of Velopharyngeal Insufficiency Within a Single Institution. Archives of Otolaryngology–Head & Neck Surgery. 2006; 132(9):947-51

24.

Eliasziw M, Young SL, Woodbury MG, Fryday-Field K. Statistical methodology for the concurrent assessment of interrater and intrarater reliability: using goniometric measurements as an example. Physical therapy. 1994; 74(8):777-88

25.

McGraw KO, Wong SP. Forming Inferences About Some Intraclass Correlation Coefficients. Psychological Methods. 1996; 1(1):30-46

26.

World Health Organization. Process of translation and adaptation of instruments. 2017.Available from: http://www.who.int/substance_abuse/research_tools/translation/en/.

27.

Sposito NPB, Rossato LM, Bueno M, Kimura AF, Costa T, Guedes DMB. Assessment and management of pain in newborns hospitalized in a Neonatal Intensive Care Unit: a cross-sectional study. Revista Latino-Americana de Enfermagem. 2017; 25e2931

28.

Gomez RJ, Barrowman N, Elia S, Manias E, Royle J, Harrison D. Establishing intra- and inter-rater agreement of the Face, Legs, Activity, Cry, Consolability scale for evaluating pain in toddlers during immunization. Pain research & management. 2013; 18(6):e124-e8

29.

Backus LA. Validation of the Neonatal Infant Pain Scale. Michigan, United States: Grand Valley State University. 1996.