Overview. The literature on evaluating emergency risk communications is “fraught with challenges” (Thomas, Vanderford, and Quinn, 2008) and our literature review and site visits revealed that evaluation studies of risk communication for at-risk populations were also limited. However, there were a few examples that stood out, including the Latino program in Montgomery County, Maryland, and other evaluations to map at-risk populations used in California and Oklahoma (described below). Given the dearth of effectiveness evaluations, we also asked site visit informants whether they conduct any kind of vulnerability assessments to guide their approach to risk communication with at-risk populations because these approaches can either facilitate evaluation or, in the case of exercising, can provide feedback for improving future activities. Specifically, we inquired about whether they collect information on the size and location of at-risk populations to gauge the communication needs of a specific population during an emergency. We also asked informants about whether they conduct any formal (or informal) evaluation of the impact of communication activities that have been conducted. For example, do they survey their at-risk constituents to assess whether communication efforts were successful at increasing preparedness behaviors and response following actual emergencies?
Vulnerability assessments. The literature points to vulnerability assessments as a key part of formative research in the pre-event phase. Vulnerability assessments can include geographic information systems (GIS) as a method to map the location of at-risk populations so that communication campaigns can be targeted accordingly. Use of GIS to plan communication strategies is already underway in one state. In addition to GIS mapping, many states are using community-based participatory approaches to foster preparation, response, and recovery. As described previously, most of the states we studied employed community partnerships and networks to build capacity by better understanding local concerns and identifying ways to best address them (Quinn, 2008). Assets mapping can be used to elicit perspectives of at-risk populations through the process and can engage communities in identifying key strengths, assets, and partners that may be useful in risk communication activities. Moving beyond GIS, it would enable health departments to also have a comprehensive picture of at-risk communities including key natural leaders, important community locations that could serve as gathering places, critical partners such as specific churches or CBOs, and non-traditional communication channels. Other methods of assessment (e.g., telephone focus groups with professionals representing at-risk populations) are in use as well. Additionally, though more challenging, formative research is still possible at the time of an event. In fact, rapid assessment that can help to identify any hidden audiences, identify specific environmental factors that may increase risk, uncover critical audience questions and concerns, and identify any potential trusted spokespersons or partners is proposed by the literature (Quinn, Thomas, and McAllister, 2005).
Oklahoma’s health department conducted a study in August 2004 to better identify their at-risk populations and determine their needs. They employed a consulting firm to run focus groups by telephone with professionals representing their key groups: Native Americans, immigrants and refugees, minorities, homeless and low-income populations, PWD, and senior citizens. That assessment concluded that lack of proficiency with English, cultural differences, and limited literacy were the greatest barriers faced by the state’s at-risk populations. Consistent with the literature, these professionals highlighted the importance for at-risk populations to receive reliable information delivered by trusted spokespersons.
Effectiveness evaluation. As noted previously, evaluation studies to assess the impact of risk communication are limited. Nonetheless, we did learn about several notable evaluation activities from both a systemic and programmatic level (Thomas et al., 2008). For example, California’s emergency services recently surveyed county and city emergency managers about the use of registries for tracking at-risk populations. The survey highlighted the need to increase manager awareness about the utility of registries for enhancing emergency preparedness, response, and recovery efforts. Survey findings also revealed concerns about privacy among constituents, which may explain the limited use of registries to track PWD in the community.
In addition, the California Department of Health uses a very rigorous message development methodology that incorporates evaluation. It begins with CDC risk communication messages, which are then adapted to the needs of particular at-risk populations and, subsequently, sent to CDC technical and medical personnel, who check the adapted versions for accuracy. When the risk communication product is both medically and technically correct and understandable for the relevant population and at the appropriate reading level, they translate the product into 12 languages. They then conduct focus groups to make sure that the translation actually conveys the intended message. However, this process takes about 6-12 months, so it cannot be used to develop messages about new events as they emerge (for example, as with the sudden wave of fires in 2008). Some chapters of the state Red Cross conduct periodic telephone surveys with members of the community about preparation to inform future program design.
Oklahoma has exercised most aspects of its response plans, including risk communication. Combining exercises to test different aspects of response plans with after-action evaluation provide Oklahoma with insight into what works, what does not, and what needs to be modified for future response planning and response efforts. As an example of learning from an exercise, Oklahoma conducted an influenza clinic exercise during flu season and learned from the effort that they need to repeat messages many times and in many different formats to get the target populations to come to the clinics. To make risk communication most effective in the future, public health officials will put messages in the newspaper every day for a week up to the start of the flu clinic; they will also broadcast messages on the radio every day at different times to ensure that the messages reach the widest possible audience. They also learned that it helps to distribute flyers at the places where people frequent (e.g., Wal-Mart). These strategies are consistent with recommendations from the literature to offer frequent messages in multiple modes that are locally accessible and personally relevant. This example also demonstrates the value of a multi-modal risk communication strategy, which was identified in the compendium as potentially increasing attention and comprehension. This practice provides an example of how evaluation can be incorporated into regular activities to improve preparedness and response.
The Metropolitan Washington Area has evaluated several of its programs. For example, in Montgomery County, Maryland, the faith-based programs had several committee members perform outreach activities with their own organizations. One organization conducted a follow-up survey six months after the outreach, but the response was poor. Most of the participants had not yet started preparing a kit, although they reported knowing that they should do so. In addition, homebound care training was evaluated with a survey of aides at multiple assessments to determine whether clients had obtained the core items in the Plan 9 list, and case manager training is evaluated through required reporting every six months. Last year, Montgomery County informants evaluated a program for pregnant women by reviewing records to assess whether case managers were engaging pregnant women in preparedness planning. The reviewers saw increases in the number of women who included supplies (such as formula) for their child in their preparedness kits.
Another example of evaluation is for the Latino program in Montgomery County. There, informants conducted a small pilot evaluation focusing on health promotores and developed a curriculum to train health promoters based on their research. They have worked with six promotores who are active in a variety of venues (e.g., through parent-teacher associations and schools, churches, neighbors) in order to encourage their creativity.
The evaluation, which was performed at two sites, revealed that health promotores do affect recipients’ actions with regard to emergency planning. Informants learned that “one-shot” interventions do not work well. Efforts must include repetition as well as precise and simple messages. A structured training that includes follow-up is required to ensure that outreach workers are communicating the right message, and to provide incentives (such as food at trainings or gift certificates) for doing the work since they are volunteers.
Use of health promotores is consistent with recommendations from the literature to enlist community members as partners in message development and dissemination. This approach leverages existing community resources and capitalizes on the willingness (as suggested by the literature) of community members to be actively involved in emergency preparedness, response, and recovery efforts. It also augments the resources available to achieve a core recommendation from the literature review: communicate early, communicate often, and communicate in accessible and personally relevant ways.