No formal evaluation of Florida's quality improvement programs has been performed to date. AHCA staff reported that they are interested in evaluating the success of the programs, particularly the TA component. However, because the programs have been operating only a short time, it is not yet possible to evaluate their impact. Because many of the programs were implemented simultaneously, it will be difficult to measure improvement or to attribute improvement solely to any one program. Uncertainty about appropriate measures also makes the evaluation complicated. A decrease in the number of deficiencies cited, a decrease in overall scope and severity, or a decrease in the number of citations have been considered as possible measures by AHCA, but none is yet considered to be reliable. AHCA has been tracking liability claims and reported that they have been tapering off since they peaked in October 2001 (which was the deadline for all claims). They produce an annual report on adverse events and survey citations, which was due to be published in December 2002. They stated that they have not seen big changes in the aggregate of deficiencies, but that it is too early to see changes especially those that would be related to the passage of SB1202. Agency staff are also aware of the impact staff turnover both at facilities and within the TA program has had on program effectiveness and sustainability, making them hesitant to begin an evaluation that does not take turnover into account. Facility staff turnover was described as being particularly concerning, with some QMs reporting that they were seeing a new Director of Nursing at each facility visit, and finding that QM reports and recommendations were often lost in the transition.
Dr. Acker stated that anecdotal evidence indicates the TA program is having positive effects, however. As described in the previous section, many providers we spoke to noted that they felt that the quality of care in their facilities had improved as a direct result of the visits. AHCA also has received positive feedback from surveys and feedback forms used to gauge the success of the Quality Monitoring program. They have conducted two surveys--one with field office managers on the relationship between monitors and field office staff, and one with providers on the value of the monitor program. AHCA also receives feedback from facility staff in the form of a paper questionnaire given to facility staff at the end of a visit, asking facilities to provide information rating the performance of the TA staff and how helpful the visit had been. Most comments have been complimentary, with observations such as the visits were helpful and that staff at facilities were pleased to have someone to ask when questions arose. However, at the time of our visit, AHCA was revising the form and hadn't used it for six months. Some providers we spoke with also said that they are reluctant to offer criticism on the questionnaire for fear that there could be negative repercussions from a Quality Monitoring staff that increasingly has ties with the survey process.
Regarding the Gold Seal Program, many comments we heard from providers and consumer advocates indicated they thought the program probably was unlikely to affect quality. Some stakeholders voiced the opinion that the award was primarily a marketing tool which may become increasingly relevant when bed occupancy is lower. They felt that the greatest impact may be on those facilities on the cusp of providing higher quality care which are deciding whether to make the investments that quality improvement requires. For those facilities, the Gold Seal program could make a positive difference.
Assessing the value of the Alzheimer training program, most stakeholders said they thought it provides good information, and that it is was most likely to have a positive impact for nursing aides and for facilities that do not have a specific dementia care unit. But some expressed the opinion that facilities would benefit more from being able to choose for themselves the training that would most benefit their facility. And some said mandatory training felt more like a "big brother is watching" regulatory approach than a valuable educational program that improved quality of care.
Florida's web-based public reporting program was considered sufficiently valuable by consumer representatives that they said they thought that every state should have one. But a number of stakeholders stated their belief that a several factors were currently limiting its impact on quality improvement. They thought that consumers frequently do not know that the Guides and Watch List exist, may not have internet access, or may not be proficient in navigating the internet. Some provider representatives also noted that some facilities have been on the Watch List many times, and that this does not appear to have provided sufficient motivation for those facilities to do a better job. Stakeholders said that they believe public reporting of deficiencies can improve quality of care provided by stimulating competition and sparking change in facility culture. However, one provider representative stated that since 90 percent of admissions come from the hospital, the discharge planner has the greatest influence on where patients go, rather than a family member who had taken the opportunity to review quality ratings. He posited that as consumers become more computer savvy, interest and impact will increase--and that would make facilities be more concerned about how they look on the public reporting website.
Opinions varied about whether the mandated increases in staffing had impacted quality. Some providers said they spent a huge amount of time and money on this issue and it had not made any impact on quality. Another said the belief that by increasing staffing, turnover will be decreased, and that increased staffing creates more flexibility, increases the ratio of staff to residents and improves the quality of life for the residents by allowing staff able to spend more time with them.
As with all of Florida's quality initiatives, the impact of the risk management program has not yet been formally evaluated. AHCA staff and provider representatives reported that the number of lawsuits has declined, but it is impossible to know whether this is due to improved quality processes, or whether the number of facilities "going bare" (operating without liability insurance) has made the state's facilities less attractive targets for litigation. Regardless, several providers expressed the belief that the risk management program had been one of the quality initiatives that had the greatest impact on nursing home quality of care. They reported that at first there was resistance to changes such as monthly meetings of the risk management committee, but they now see it as very useful. "It forces us to keep focused." One provider reported that they now do a lot of education around risk management with staff. When staff understand the goals, they stated that their participation and openness increases and they are less defensive. Another provider said that the way that they investigate bruises has changed dramatically since the risk management program was instituted and that how they do their investigation has impacted quality on each nursing unit in their facility.