Development of a Quality Measure for Adults with Post-Traumatic Stress Disorder. E. Inter-Rater Agreement Results

05/01/2019

Inter-rater reliability assesses the extent to which clinicians, supervisors, and clients agreed on whether the clinician delivered the survey element. We used the AC1 statistic, a measure of agreement adjusted for chance, to quantify agreement for the overall survey and at the item level.[7]

Inter-rater agreement between clinicians, supervisors, and clients. All three raters completed the survey on 76 therapy sessions and at least two raters completed it on 97 therapy sessions. The weighted agreement for the whole survey ranged from 0.39 to 0.58 across different rater pairs (Table V.4), which is considered fair-to-moderate agreement (Gwet 2014). Supervisors and clinicians had the highest weighted inter-rater agreement; supervisors and clients and clinicians and clients had comparable inter-rater agreement.

TABLE V.4. Inter-Rater Reliability for Clinicians, Supervisors, and Clients
Raters AC1 SE CI Significance Level
Supervisor-clinician-client 0.43 0.005 (0.34-0.53) <0.01
Supervisor-clinician 0.58 0.04 (0.51-0.65) <0.01
Supervisor-client 0.39 0.07 (0.25-0.54) <0.01
Client-clinician 0.39 0.07 (0.26-0.51) <0.01
NOTE: AC1 values above 0.80 suggest high agreement; 0.61-0.80 substantial agreement, 0.41-0.60 moderate agreement, 0.21-0.40 fair agreement, and 0-0.20 slight agreement.

In addition to calculating inter-rater agreement for the whole measure, we also calculated it at the item level. Across the three raters, item percentage agreement ranged from 39 percent to 90 percent and the AC1 statistic ranged from -0.09 to 0.86 (Table V.5). Items for which there was only slight agreement included two homework-related items, one therapeutic technique item, and one item on managing the therapy session. Similar trends occurred when examining item-level agreement between each rater pair (clinicians/supervisors, clinicians/clients, supervisors/clients) with high agreement in ratings of some survey items and low agreement in others (see Appendix K).

TABLE V.5. Item-Level Inter-Rater Agreement Between Clinicians, Supervisors, and Clients
  Overserved Agreement AC1 CI Significance Level
Did you and your therapist discuss an agenda or plan for your session? 85.48% 0.42 <0.01 (0.24-0.60)
Did your therapist talk about or check-in on your expectations of how therapy will go? 62.50% 0.38 <0.01 (0.20-0.56)
Did your therapist work with you to set goals you both agreed on? 65.45% 0.52 <0.01 (0.35-0.69)
Did your therapist help you become aware of or realize feelings, views or thoughts in your life that have been influenced by your traumatic experience?
These might include feelings, views, or thoughts about being safe in the world, the presence of danger, trust, and self-esteem.
76.71% 0.69 <0.01 (0.57-0.82)
Did your therapist ask you several direct questions to make you think critically about or examine your thoughts, feelings, or beliefs?
For example, your therapist might ask:
  • How do you know this? Can you give me an example?
  • What are some other ways of viewing this? What are the pros and cons to your way of thinking about this?
  • How did you come to this conclusion? What evidence do you have to justify this?
58.62% 0.48 <0.01 (0.31-0.64)
Did your therapist offer other ways of thinking about your issues (e.g., problem areas or areas you want to work on) related to the trauma?
For example:
  • Thought: "I can't trust anyone."
  • Thought suggested by therapist: "Some people can't be trusted, but there are other people who are trustworthy."
64.81% 0.41 <0.01 (0.23-0.60)
Did you and your therapist discuss people, events, or places you now avoid or stay away from because of your traumatic experience? For example, someone in a car accident might avoid driving on the freeway. 61.67% 0.20 <0.01 (0.01-0.39)
Did your therapist do any of the following things to help you deal with fear, anxiety or things you now avoid because of your trauma?
  1. Ask you to imagine or retell your traumatic experience for longer than 10 minutes.
  2. Ask you to write about your traumatic experience.
  3. Ask you questions to make you think critically about or examine your thoughts, feelings, or beliefs related to your fear, anxiety, and avoidance of things (i.e., "How do you know this? Can you give me an example?").
  4. Ask you to do real world experiments like visiting a place related to the traumatic experience for longer than 10 minutes.
49.30% 0.22 <0.01 (0.06-0.38)
After you described your traumatic experience, did you and your therapist discuss the details of what happened to you, how it impacted your life, or your emotions about the event? 62.86% 0.19 <0.01 (0.03-0.35)
Did your therapist make good use of your session time today? 76.19% -0.09 <0.01 (-0.23-0.05)
Did your therapist ask for your opinion on how your treatment is going? 66.07% 0.50 <0.01 (0.34-0.66)
Did your therapist ask for feedback on how she/he is doing in helping you recover from your PTSD? 45.10% 0.25 <0.01 (0.07-0.44)
Did your therapist assign homework or practice assignments (to be completed by the next session) to work on your PTSD symptoms or problem areas? 58.82% 0.33 <0.01 (0.16-0.50)
Did your therapist make sure you understood how to complete your homework for the next session? 65.52% 0.45 <0.01 (0.29-0.61)
If you had problems completing your previously assigned homework, did your therapist work with you to come up with solutions to these problems? 66.07% 0.09 <0.01 (-0.08-0.27)
Did your therapist review and discuss your homework from the previous session? 54.55% 0.17 <0.01 (-0.01-0.35)
When reviewing the homework from the previous session, did your therapist encourage or provide you with constructive feedback? 60.00% 0.22 <0.01 (0.03-0.40)
My therapist and I have built mutual trust. 10.67% 0.85 <0.01 (0.80-0.90)
I am confident in my therapist's ability to help me. 9.33% 0.76 <0.01 (0.70-0.83)
I believe my therapist likes me as a person. 17.33% 0.86 <0.01 (0.81-0.91)
Has your therapist ever asked you if have had thoughts about committing suicide? 90.00% 0.79 <0.01 (0.68-0.89)
During this session, did your therapist ask you if you had thoughts about committing suicide? 39.39% 0.61 <0.01 (0.47-0.76)
Has your therapist ever asked you to answer questions about your PTSD symptoms? This might include completing a form before or after therapy. 78.69% 0.28 <0.01 (0.12-0.43)
During this session, did your therapist ask you about your PTSD symptoms? This might include completing a form or survey before or after therapy. 61.54% 0.44 <0.01 (0.29-0.59)
Has your therapist ever provided information about PTSD and PTSD symptoms? 84.75% 0.86 <0.01 (0.77-0.95)
During this session, did your therapist provide information about PTSD and PTSD symptoms? 60.94% 0.26 <0.01 (0.09-0.43)
Has your therapist ever provided with specific education on the nature of the traumatic event (i.e., facts about the type of trauma)?
  • For example, this might include education on the nature of sexual assault, or how sexual assault generally influences your viewpoints and beliefs.
68.00% 0.48 <0.01 (0.31-0.65)
During this session, did your therapist ever provide you with specific education on the nature of the traumatic event (i.e., facts about the type of trauma)?
  • For example, this might include education on the nature of sexual assault, or how sexual assault generally influences your view points and beliefs
56.67% 0.22 <0.01 (0.06-0.38)
Has your therapist ever explained how your particular treatment will work? 78.18% 0.77 <0.01 (0.65-0.89)
During this session, did your therapist explain how your particular treatment will work? 67.21% 0.21 <0.01 (0.05-0.36)

Implications for survey revisions. Although there was high agreement between raters for several survey items, the inter-rater agreement results suggest that several items may benefit from further investigation and potential revision. Examples of items with low agreement and/or poor AC1 values include:

  • Two questions regarding Socratic discussion methods.

  • Two questions about therapeutic techniques to deal with avoidance.

  • One question about emotional reprocessing regarding the emotions surrounding the traumatic event.

  • One question regarding the psychoeducation about the nature of the traumatic event.

It is possible that these and other items with low agreement could be revised by further simplifying the questions or providing more detailed examples; however, further cognitive interviewing may be needed to better understand how stakeholders interpret them. Alternatively, the items may need to be deleted.