Advancing States' Child Indicators Initiatives:
Thursday, May 31, 2001
What follows is a transcript of remarks made by Christine Ferguson and Elizabeth Burke Bryant of Rhode Island and by James Dimas of the Annie E. Casey Foundation. Also included, following her remarks, is Christine Ferguson's Powerpoint presentation.
Fred Wulczyn, Research Fellow, Chapin Hall
The challenge with indicator data is that it provides a lot of information about the past. As people in state agencies gather such data, the question becomes, what do these data tell us about the future? How can you best gather and use indicator data for guiding policy decision-making in the future? How can we engage the underlying policy-making processes proactively so that policy and programmatic decisions guide the future to a more desirable point?
This morning we will be discussing the challenges of creating that forward-looking view, the political challenges of selling that view, and the conceptual challenges of understanding the right policy lever for moving that view towards a conclusion that is desirable for families, children, and the community. To address these topics we have from Rhode Island "Ms. Inside" and "Ms. Outside." Christine Ferguson is director of Rhode Island's Department of Human Services. She has served in the highest levels of government for two decades. Her record reflects a clear commitment to using information indicators to make policy decisions. Elizabeth Burke Bryant is director of Rhode Island Kids Count. They're not exactly "Ms. Inside" and "Ms. Outside," though one looks in to and the other out from state government. One of the benefits of their respective positions is that they get to work together to create consensus around their viewpoints so that they can move forward together.
James Dimas will then speak with us on these issues. He has worked in public health in Washington, D.C., and for the Illinois Department of Public Aid in its welfare-to-work program. And he's now joining the Annie E. Casey Foundation, which is in the process of forming a group that will provide technical assistance to states in this very area of technical information, with its links to public policy and practice.
Christine Ferguson, Director, Rhode Island Department of Human Services
"Agree on some measures; publish what you have; perfect the measurements over time. We don't have time for perfect measurements. People like me are only in the positions we're in for a very short time." (Ms. Ferguson's Powerpoint presentation follows the text of her remarks.)
Rhode Island has developed a vision, and people in leadership roles in and out of government have been able to craft and implement that vision. I'd like to give you some perspective on what we've done. In Rhode Island in 1995-96, we identified four outcomes for all children that we want to work toward:
Depending on what kind of thinker you are, these outcomes are either really great or way too big. But what they did was to begin to narrow people who are very fragmented in their thinking and to broaden people who are narrow in their thinking. And so this consensus gave us a way of saying in any one instance that, "This is an initiative around one of these four things." In response to these outcomes, we were able to implement three programs critical to kids.
RIte Care. RIte Care is Rhode Island's comprehensive health care program for uninsured low-income families, enacted in 1993. This is our Medicaid managed care program in Rhode Island.
The Family Independence Act. The Family Independence Act, Rhode Island's welfare reform legislation, was enacted in 1996. This act included an entitlement to childcare. In other words, if someone comes to the door and meets the eligibility criteria, we have to provide them with the subsidy. There is no waiting because it is an entitlement.
Starting RIte. Starting RIte, Rhode Island's early education and childcare initiative, was enacted in 1998. It's not one piece of legislation, but, rather, lots of concepts combined, including quality health insurance for childcare providers and stepped up eligibility criteria for the entitlement. Under these provisions, a licensed childcare provider taking one subsidized kid for six months receives 100 percent free health insurance.
So our expectation was to begin to really invest in kids. One of the things we did with respect to our health insurance was to have incremental expansions every few years, and we're now up to coverage for adults at 185 percent of the poverty level and for kids at 250 percent. Those benefits are the Medicaid benefits that include Early and Periodic Screening, Diagnosis, and Treatment (EPSDT). EPSDT benefits are wrapped around any private coverage that we subsidize. And we've been aggressive with respect to providing EPSDT coverage because it's connected to early education, early childhood kinds of issues. The result: improved primary care services, and a lot of other things.
But the reason I bring this up is this chart. The reason that we continue to have RIte Care in Rhode Island is this chart (slide 6). What this chart says is that those who got their insurance through Medicaid pre- and post-managed care, and those who got it pre- and post-RIte Care, virtually became the same in terms of the outcome of waiting eighteen months between children. That's important because it's an indicator of both maternal health and child health.
Usually I say the following things. What we found is that women who are low-income — working poor and very poor families — access health care in the same way and make the same decisions middle-income and upper-income women make when they have consistent medical services. I'm basing my conclusion on this and other data and on focus groups, and frankly on the observations of people I meet. What the data tell me is that when low-income women have a doctor or nurse practitioner or nurse midwife throughout pregnancy, they talk about contraceptives. When they deliver a baby in the emergency room, they don't, and probably none of you would either.
But my whole point is this: this is not peer review science. We've tried to publish some of these data in very strict journals and they wouldn't let it in because it's not drawn from large enough samples. So if you were pure you'd say to me, "But you can't draw those conclusions from this data," and I would say, "You're right, but you want to wait ten years to draw any of those conclusions and, guess what, ten years from now, I might be dead and this program might be dead." The reality is that when you're dealing with states or at the federal level, if you want to maintain a program or expand it, you have to be able to show either that nothing's gotten worse-because we guarantee the amount of money spent got worse — and ideally that something's improved.
The Health Care Numbers and Charts Are Great, Although the Data Aren't Perfect
We've also been able to show adequacy of prenatal care. Again the gap gets closed. If you take health insurance status as a proxy for income, this indicator shows that the working poor and poor women are responding to health care in very encouraging ways.
And ultimately, we say we're number one in the country for health insurance coverage now. Only 6.9 percent of our people are uninsured, compared to everyone else. According to the GAO data, we're second, but it doesn't really matter because the drop in uninsured has been tremendous in the past few years. The change is what's important. Universal coverage is not so much the indicator, as what's happened as a result of it.
So, over the past eight years we've shown progressive improvements in health status with RIte Care. I always say to people you can't take this to a scientist and have them say to you that this is perfect data. It's not perfect data. We cobble it together from health department data and we have very credible researchers doing this at Brown University. It's outside evaluation. But it's not perfect and it's never going to be perfect. There's no way we could ever collect enough data to be perfect.
The Childcare Numbers Are Also Good
The childcare numbers are also good, but expenditures for childcare have increased 270 percent and the data are not that useful yet. It's too soon to use these data. They do not explain the financial impact of the childcare measures for legislators.
The Family Independence Act and Starting RIte pulled together a series of provisions related to health insurance for family childcare providers. We provide health insurance to licensed family childcare providers on an entitlement basis. We have a series of interventions in effect around quality and eligibility. We have wrap-around childcare services.
The number of kids served has increased by 100 percent since 1997 from 6,000 to 12,000, roughly. That's a lot because we've got a population of about one million. There are about 150,000 kids under age eighteen. Our childcare and after-school care goes up to age sixteen. So subsidies are available to kids up to age sixteen. Our childcare placements have increased by 27 percent in two years. Our childcare providers accepting DHS subsidies are up from 61 to 76 percent in two years. Availability has increased and we have almost 1,000 people who are receiving health insurance because they're family childcare providers or related to family childcare providers. But here's the rub: our childcare subsidy expenditures have increased by 270 percent since 1997, from 18.6 to 68.8 million dollars. And it's all state dollars. The legislature has flipped out on a number of occasions and the budget officer in the governor's officer has flipped out on more than a number of occasions.
Our rate of increase has been steady, luckily. So now I can give you all these great outcomes: who accepts subsidies, how many kids of which ages receive the subsidies and how that's increased over time, our investment, administration (what we've spent has gone down at the same time the program's increased), how much we pay for each slot, all of those things.
But you'll notice I could do all this with health care, too, but I never give those slides because they're not particularly interesting. All it is is claims and financial data mixed around a couple different ways, but it doesn't tell me anything about whether what kids are getting is helping in any way.
Knowing whether kids know their letters and numbers will help ensure the future of these programs. If the only thing you worried about with childcare was helping families go back to work, the data we have now is okay. But it doesn't tell me the important thing, the critical thing for the future of these programs — whether kids enter school ready to learn. And I know this is a big issue but I want to know how many letters they know. I'm sorry, but that's important to me. It's important to the grandfathers that are on the finance committee, it's important to the governor — that's what they value. Can you tell me, are the kids aggressive or not aggressive and all those things? Yeah, I want to know those things too, but I really want to know, do they know their numbers? Can they read? Do they have books?
I need to be able to say, oh yeah, childcare is going to continue to grow at a rate of 100 percent per year, and next year it's going to be 68 million dollars and we're going up to 74 million and yeah, I know you want a car tax repeal and I know you want capital gains and I know you're really worried about the expenditure in health care. But these kids are in a better position to be reading in the fourth grade and to be graduating from high school because they have this after-school enrichment program and these Head Start-like comprehensive services, because they have childcare.
And it's not a bad thing for these moms to be working. You don't have to feel guilty because you've put the moms back to work because the kids are doing well. That's what we need to tell them because the things that mobilize interest in children's issues are usually all the negatives, all the bad cases. You'll always be able to find a bad case of lead poisoning or a bad childcare situation or something that'll send somebody off in that direction. It's really hard to focus on the whole picture. We really ought to be fixing this in the context of what we already have, so don't set up a new program, please.
What our governor and our legislature have done is to really decide to make the investment in children and families. But the last two years have been really hard because our childcare spiked up one year by 25 million dollars and next health care expenditures went up by 80 million dollars. And they're looking at me going, "Hello, we really want to do these tax cuts and you're really preventing us from doing them." And what we were able to say back to them, is, "These are not our decisions. These are your decisions. Here are the choices. There's progress that's been made here and you can cut it back but you've got to understand there's probably going to be some impact." They've chosen not to cut back. So we've gone through three years of them making the decision not to cut back.
And the only reason is because, on the childcare side, we took all those pretty charts but we didn't show them until after we showed them the health care charts and they just assumed the childcare charts were as impressive as the health care charts. But I can't keep that up for very long, because I really am honest. And when they ask that question about childcare I say "I don't have it, I can't tell you, it's too soon anyway. It's only been three years."
So what I say in Rhode Island is, I know you guys are working real hard but it's not fast enough! Because I know I'm going to get these questions pretty soon. They're going to say, "It's been four years. Did the kids we helped four years ago enter kindergarten knowing their letters and their numbers? Can you tell me that, Christine?" And if I can't, and I say, "You know, well, there's a real disagreement over whether that's a good indicator of good childcare." They'll say to me, "So the answer is you don't know." And I'll say, "Yup." And they'll say, "So, how much money will it take before you do know." And I'll say, "Well, we really haven't decided. We can't decide whether it's knowing their letters or social adaptability." So take them all. Pick 100 indicators and put them all together. Come up with some way of providing decision-making kinds of information, because the backlash is coming.
This is not an unthinkable cycle. This is what happens every time. You pull people back to work. They focus on upper-income women who go back to work and they have all the mommy wars in the suburbs. And I'm one of the women who work and I know the mommy wars and now we get: children shouldn't be in childcare and women should stay home. But we're not willing to pay people to stay home, so what should we do?
We've always missed one part of the cycle. We've missed the opportunity to say, it's okay for some kids to be in childcare and it's probably not great for other kids — it depends. But we like to have some basis to say those things and to have some sense that, even if they're not willing to say it, the reality is that the way to appeal to most of these people is: if you want a good work force twenty years from now you have to invest in these kids now — in health care, education, and childcare.
And as long as money's the most important thing in the US — and it is, that's what we value as a society, that's how we've evolved — as long as that's true, then you've got to talk about it in that language. You've got to say, these are investments worth making, not because the kid's a better social citizen, but because the kid's a better worker. You may not like it but maybe you can also ensure the kid has better social skills at the same time. And the latter's beginning to reflect in workforce training. You've all been through the Myers-Briggs test and teamwork and systems thinking. All of that is really based on the things the kids learned in kindergarten — you know, don't hit your friend over the head when he has the truck.
Track As Many Indicators As You Can. Understand That The People Who Make Decisions Have To Be Able To Justify Them To People Who Don't Care
So track as many indicators as you can. Understand that the people who make decisions have to be able to justify them to people who don't care about all the things that you care about, to the person in the street who is upset because they paid thirty percent of their income in taxes. They have to be able to demonstrate that their thirty percent went to something that really improves the future of the state.
It's Harder To Get Childcare Data Than Health Care Data
We have a lot of health care data to extrapolate from. A lot of people collect it. That's not true of early childhood data. We need to get data from childcare providers and schools. A critical issue for us is that we don't have a single student identifier. That makes it very difficult for us. And while we have three major health insurers to get data from, we have umpteen hundred childcare providers. That makes it more difficult.
We're working on all those things. People should be working on all these things at the state level. And regardless of which indicators you pick, you still have to be able to get the data once you pick them. So don't just focus on which indicators to pick. Once you tell me what you want, it's going to take me two years to figure out how to implement it — if I'm lucky. So we need to figure out what we need to put into place in state government, regardless of what kinds of indicators you pick, to be able to get information back from a childcare provider, from a school.
The Kinds Of Comparisons We'll Need To Make: Subsidized Versus Non-Subsidized Kids
We also need to be able to compare subsidized kids' outcomes with non-subsidized kids' outcomes. We need to compare by income levels. Our reimbursement rate is very high. Three years ago it caused a thirty million dollar increase in one year. They want to know that what they're paying for is a return that is giving them something other than a babysitter, which is how many grandparents think of childcare. (We try to focus on early education, but they still call it daycare in many places.)
Have Faith! If It's Worth Fighting For, It's Worth Measuring!
All these things are really scary if you're frightened of ending up with comparisons that are not so great. But get them! You have to have some faith! If you believe the services are worth fighting for, you have to believe the supporting data are worth measuring. What will be shown will be used by you to improve the situation — or to make a correction. If you don't believe they're worth measuring, then why are you fighting for them? So you have to be able to take the leap that what will be shown will be used by you to improve the situation or to make a correction. We may all be wrong. Maybe we should be back in caves. But those are kinds of value discussions that need to occur and they can't occur without some kind of reasonable information.
In Sum, They Need to Know: Are Poor Kids Reading Better, etc. (And ABCs ARE Important)
In sum, here's what they need to know. Are poor kids reading better? (And I use the word "poor" strategically, usually I say "working families," "the working poor" etc.) But the reality is, the guy sitting up there making the decision on the finance committee, the talk show radio host, the guy in his car sitting and listening to the talk show host, and most of your neighbors, think this way. And you have to be able to think this way in order to give them the information they need. You can change the way they think, but only gradually over time.
Do poor kids read better in the third grade? Are poor kids reading as well as suburban kids? Do poor kids know more when they go to school? Do they know as much as wealthier kids when they go to school? Those are the kinds of answers they want.
They're also interested in aggressive/nonaggressive, but frankly they are probably more interested in ABCs and 1-2-3-4. Maybe they don't care if we have smart kids that are mean. Maybe that's not worth it. Maybe the guy they work for is mean and maybe he's successful and they're not, so maybe meanness is also a value that we have. I don't know, but it's interesting to have the conversation. And if you have some data to talk about, it makes it even more interesting.
We're Looking At Everything We Can
In Rhode Island we have started the process of putting some questions into our SALT (School Accountability for Learning and Teaching) survey of schools, of looking at everything we have access to, and meeting my need of knowing whether the kids know their alphabet when they go to kindergarten and their need of making sure they're good kindergarten citizens and have good social skills and family situations. So we're getting there. I'm worried it's not fast enough.
The Pressure For Tax Cuts is Great
They want to see it fast. I don't know how much longer they'll continue to do this. The guy leading the finance committee, who is a great guy, has put off for one or two years his big centerpiece legislation, which is to give back property tax on cars. I don't understand the economics of that, but he's absolutely committed to it. He's delayed it for two years because of health and childcare expenditures! How many more years is he going to delay this? He can cut childcare and no one's going to think about it — except the childcare providers. Health care's a different story because he can see the results and it makes him nervous. But childcare? There's always this back and forth, back and forth cycling.
Don't Let The Perfect Be The Enemy Of the Good
So agree on some measures. Don't think you're going be published in peer review articles. Most people don't read them anyway. Publish in magazines. And talk about the results and perfect them over time. Don't let the perfect be the enemy of the good. Do longitudinal studies to verify what we're learning. But we don't have the time for perfect measurements. The reality is, people like me are in the positions we're in for a short period of time. Six years is a short period of time, but for someone in my position, that's a long period of time. Legislators need answers. And they need information translated for them in a way they can use. Otherwise it all comes down to, how much of an increase did we give this program last year and isn't it time for another program to get it.
The basic thing is, I want to be able to say that your investment in health insurance and early education results in less money spent to correct other problems. I want to be able to say because you have health insurance and early education, your schools systems have less money going to special education. If I can say that, we will never have those programs cut. OK? So, get to work! Keep plugging. I feel like I'm holding up a dam and there's a whole bunch of people behind it just waiting to tear down the dam. And I'm just waiting for the people behind me to get the work done so I can hold up the chart and say, "Here! Look! This is what happens! You don't want to come through here!"
If you can do that for me, I will be eternally grateful.
Slide 1
Slide 2
Slide 3
Slide 4
Slide 5
Slide 6
Slide 7
Slide 8
Slide 9
Slide 10
Slide 11
Slide 12
Slide 13
Slide 14
Slide 15
| FY99 Actual | FY00 CEC Estimate | FY01 Gov Rec Estimate | FY00 DHS 3/00 Estimate | FY01 DHS 3/00 Estimate | |
|---|---|---|---|---|---|
Cash Assistance |
3,566 | 4,064 | 4,304 | 4,137 | 4,304 |
Low Income |
4,432 | 5,135 | 5,264 | 5,501 | 5,854 |
Starting Right |
208 | 701 | 738* | 786 | 1,025* |
Total |
8,206 | 9,900 | 10,306 | 10,424 | 11,183 |
Starting Right included above: |
|||||
Youth care: |
26 | 73 | 110 | 70 | 87 |
200/225 FPL: |
182 | 628 | 628 | 716 | 938 |
Not included above: |
|||||
DCYF/JTPA |
451 | 527 | 540 | 527 | 540 |
Child care disregard is based on the number and ge of the
children. Children under age 2, maximum of $200 per month;
other children up to $175 per month. No total maximum. Client
must provide proof of expenses. Child care providers can
be unregulated. |
|||||
Slide 16
Charts depict the growth in licensed child care placements in Rhode
Island - subsidized and unsubsidized since Starting Right.
This chart describes growth in individual licensed placements - previous
chart shows the increase in licensed providers
Outcome: provider rate increases, health care subsidies for providers, employees,
development grants have been effective in building the system.
Slide 17
Percent growth from 1999
DHS Center: 33.2%
DCYF Center: 26.%
DHS family : 57.9%
DCYF family: 24%
This chart shows growth in number of licensed providers (not placements).
Also shows growth in number of providers who accept DHS children
Outcome: Starting Right has built more capacity for subsidized children
and non- subsidized children
Total certified/licensed equal 1404: 988 certified family CC homes
+ 416 licensed center based programs
Accept DHS subsidy: 740 certified family CC homes + 317 licensed
center-based programs
Since Feb. 1999 reimbursement rate increases took effect, 26.l4% of
providers have increased the Number of slots within their programs that are
available to DHS children-so since Starting Right Impact-increased number
of programs and increased number of slots within programs
Slide 18
Slide 19
Slide 20
Slide 21
Slide 22
Slide 23
Slide 24
Slide 25
Slide 26
Slide 27
Slide 28
Slide 29
Slide 30
Slide 31
Slide 32
Slide 33
Slide 34
Slide 35
Slide 36
Slide 37
Elizabeth Burke Bryant
Rhode Island Kids Count
What's allowed for this very strong inside-outside partnership? We don't make a move at Rhode Island Kids Count without making sure that the senior level data policy staff at the state's Department of Human Services at least knows what we're doing and understand what approach we might be taking. We don't always necessarily need their sign-off, but we always know we need a professional coming-together. We always communicate about what we're doing with data that belongs to their department and about how we're releasing it.
About six or seven years ago when we released our first fact book, Dr. Barry Zuckerman came into Rhode Island as our first keynote speaker, which was a little dangerous because it meant bringing a doctor from Boston across state lines. But we decided we'd go for it because we really loved Barry Zuckerman. And Dr. Zuckerman said, "As a health professional, I'm here to tell you that the number one indicator of child well-being is fourth grade reading scores. And what we have to do is keep our eyes on the prize. We have to understand that children's health and a lot of other things going wrong in their lives can be measured by whether or not they're reading at a fourth grade level, and we have to absolutely focus like a laser beam on that."
It was a really strong message. His notion that we need world-class fourth graders in Rhode Island became a kind of rallying cry. And from that the four indicators were developed, and the one that we're talking about today is that all Rhode Island children shall enter school ready to learn. We were able to use our indicators work through Kids Count to become the clearinghouse for identifying, "How will we know if we got there?"
So we started with our first fact book with the ten indicators the Casey Foundation required, plus thirteen more, and then over the last six years we upped them to about forty-three indicators, all about children's health, education, economic well-being, and safety, plus an initial section on demographics. But this whole ready-to-learn opportunity, incredibly enhanced by the Chapin Hall work on the ASPE grant we're all in, has been absolutely essential to the research work going on behind the scenes.
The inside/outside magic also involves the press. Politicians pay attention to what's in the newspaper. We don't consider our role just in terms of a data function. That's the foundation of what we do — we're a research and educational organization. But if we didn't have an incredibly savvy media component, it wouldn't matter. Because, if it's not in the Providence Journal, it hasn't happened. If it's not on Channel 10 and Channel 12, it didn't occur. So what we've tried to get better at doing every year is to make sure everything we release is very pick-upable, very colorful, something that would be very appropriate in the highest corporate-level board room. Because for too long kids have had to exist with their advocacy materials on cheap-looking handouts, and they deserve better. And that got people's attention.
We also had to be there with all the facts and figures whenever the press needed them. As a result we've cultivated a very positive relationship with the press. They've put childcare on the front page. We have resources from a Starting Point grant from Carnegie enabling us to do more than we would have been able to afford to do ourselves. That has been an incredibly important thing. Because when politicians read these things on the front page of the paper, it makes them want to act. When they read that Parents and Working Mother Magazine are recognizing Rhode Island's policy gains, it makes them even happier.
We're almost over the finish line. We were absolutely committed to the multidimensional view about what is a ready-for-school child. We absolutely understand Christine's point about numbers and letters. I think she's absolutely right. We sometimes, I think, get drawn into a politically correct sort of thing where we think, gee, that's really putting kids in a pigeon hole with that one measure. But frankly, we're not talking about little whiz kids. We're talking about what would we expect our own kids to feel comfortable with when they get into that first formal setting. Why should we expect less from poor kids?
It's time to kiss that way of thinking goodbye. We have enough of these general areas that everyone's satisfied with, like mental health, access issues, numbers, and letters. As long as you're looking at categories and are going to end with how we categorize things, really, as long as you touch all those bases, it's okay to pick a few indicators and have that be your picture.
The other thing that's hung us up and that takes some guts to talk about is the need to professionalize early education. It's kind of been going along in a mediocre way, with people saying, "We're doing the lord's work — don't ask us about our outcomes. All kids are different." But, as Christine says, as the numbers go up from $12 million to $68 million dollars, as our rates for childcare providers go up as they should be going up every two years so that there was a 67 percent increase in rates in one year, I've lost patience with friends saying, "You've got to wait, we're back-filling from ten years of flat rates and we can't show you program changes yet."
As advocates we are going up there every day fighting for these programs, and if we can't have some accountability, then the very best early child educators and care providers are going to be brought down with the very worst. All you need is an exposé of a really bad childcare provider or a really bad center. And we've had staff go out to centers to consider them for their own kids and they've not liked what they've seen. We'd better be able to correct that, to institute some four-star rating systems. We better be able to follow that Starting RIte money and be able to say, "They did increase wages for those childcare providers who were underpaid; the money didn't go into an untrackable hole." So we're really pushing all of those accountability things with Christine's department for the next two years. I think that all of these expectations are sort of fair and square exchanges for going from $12 million to $68 million dollars.
As we strive for accountability, we need to remember to use a few tools that are already in place. What really hung us up, what was really frustrating, was what happened after we got past the access data, which is administrative data that you have access to from your state departments of human services or childcare agencies. We got past the early child health data but then we got to those hard things — how do you track those major state expenditures for childcare down to the individual child and individual center? Without the student identifier number and with specific strings attached, we couldn't do that.
But what we did do, and what I'm so pleased to be able to tell Christine about, is that on June 13 at the Children's Cabinet meeting, the presentation that you've all been waiting for is going to happen. We have been able to tap our SALT (School Accountability for Learning and Teaching) survey, which is the survey the Department of Education invested in. Every parent, every public school teacher, every public school kid is interviewed. It's an enormous reservoir of data that sometimes doesn't see the light of day. We've been working for a year and a half and we've been successful thanks to Cathy Walsh, our program director, in getting them to take K-through-3 aggregate data for the past three years and "disagg it" down to K. So we're not going to be able to give Christine everything she wants but I think it's very exciting to let you know that by "disagging" out K and having it for three years prior to this one, we've been able to create some preliminary charts.
One chart that we have created shows children working independently, being self-directed, age appropriate pre-literacy skills, numerical skills, and reasoning and problem solving skills. [This chart is not included in this working paper.] As Christine says, we're not going out for a long walk on a short pier for causality, but you can bet your life that, when we give that presentation, we're going to say that, in 1997, when the childcare investment was $12 million, this is what the SALT data showed about those what kids knew as they enter kindergarten. And we'll be able to say three years later, this is what that they know now. Hopefully the trend will be going in the right direction. We're going to report it as we see it, obviously. At least that will be the beginning of something that is going to get at what we really want. So that's how we're trying to overcome the hurdles a lot of you are experiencing.
The ASPE group from Rhode Island has been able to go through and categorize things in what I think Christine would agree are common sense areas. Children in kindergarten should have the language or literacy skills they need and the knowledge and cognitive skills. There are mental health indicators and child health indicators. So that's the road we're going down.
There have got to be lots of different ways of getting this information in people's hands. And now we have courage to have a press conference for every issue brief we release because the press is really starting to come to our events. So we're very excited about that. Plus CBS is underwriting some of the costs of producing the issue briefs, which shows you can get some businesses involved.
We've partnered with Brown University to create "Ideas that Work," a quick-and-dirty two-pager on what promotes early school success. It incorporates a little bit of data and explains the program description and why it's working to promote early school success. Our trademark is a bee and on our web site, the bee spins.
I just want to close by saying what a privilege it's been to work with Christine and that we really absolutely need to keep going. We haven't given up family time, family life, blood, sweat, and tears only to see this stuff at this critical moment suddenly seem like it's too much. And I totally agree with Christine, if you have some common sense answers in progress — that's not something you're going to want to undo.
Q. Were there any up-front commitments made to get the entitlements passed?
A. Christine: The only guarantee that I had to make to the governor's office and the legislature was that it would be budget neutral. In other words, the reduction in the caseload would offset the increase in childcare. Which it did until the legislature passed the increased rates and we had a deficit of $28 million. But I've been able to get through that on the legislative side. The administrative side's ticked at me. They feel it wasn't budget neutral. But things change. And that was the only thing I had to comply with.
We didn't have a lot of these health care outcomes until after welfare reform, interestingly. I didn't expect them, frankly. I thought it would take a lot longer to get the outcomes on the health care side. It wasn't until after we got the outcomes on the health care side and saw that the response after every crisis was not to cut that I began to really understand the magnitude of the importance of having those indicators, though nobody was really focused on it back then.
Q. What is the percentage of kids in unlicensed care?
A. Christine: A very small percentage — only about 10 percent.
Q. How do you make that so small?
A. Christine: In Rhode Island, 69 percent of the subsidized kids are in licensed centers, 20 percent in family childcare providers, and 10 percent are with relatives. We have more reliance on licensed centers than other places do. A family childcare provider who takes one subsidized kid for at least six months out of the year receives 100 percent free health insurance. But they have to be licensed and they have to take subsidized kids. So that's a trade off a lot of them make.
Q. What was the nature of the market response to the entitlement? Did you get a lot of small providers, for-profits providers, existing centers ramping up their slots? What was the response to entitlement and to the availability of health care?
A. Christine: A little bit of everything. When I came home to Rhode Island, I had a four-year-old. I could not find childcare. It just wasn't there. Over the past six years, there's been a tremendous growth in capacity. I would say the pick-up rate has been slow because a lot of family care providers have had awful problems with the department. Not because they're bad people, but because if you look at the administrative line, at the same time we've had this tremendous growth, we've had fewer and fewer people doing it. And we've had to do everything manually. There are people who didn't get paid for six months. I have people who live near me who won't take subsidized kids because they wouldn't get paid for six months. In one case it was a year.
What we've done is, we've worked with the childcare community to change the way we do business. So in mid-July we're going to have Web enrollment. The childcare provider and the individual will be able to go right to the Web. So we won't be involved directly anymore.
That could cause another spike in growth. I'm a little concerned about that, frankly. We are so good at outreach. The inside/outside operation is just so awesome. We brought 25,000 people into our health insurance program in a year, which almost broke everybody's back. I'm afraid the Web enrollment could have the same impact. So we're going to monitor that pretty closely. More than anything else, it's that trade-off: is the health insurance option worth the hassle of waiting to get paid by the state. It depends on how sick you are. And that probably makes a difference in whether you are able to do the job or not.
Q. Are you using indicators to track the impact on youth? And what can the childcare subsidy be used for?
A. Christine: Here's my next big problem. I've been trying to separate out the licensing requirements so that the after-school programs can include a basketball program, theater program, and an art or computer program — three different programs in a week. That's the goal. We haven't gotten there yet. One of the biggest problems is getting parents to apply for the subsidy, ironically. Even if you get the program licensed, the kids and the parents get a little bit uncertain around whether or not they want to apply for this. And there are things we need to do. We need to get attendance taken. We need to know if the person was there. So it's a question of culture change all the way around. The growth in that age group has been steep but in the context of the number of all the kids that are out there, it's probably relatively low. So we have a lot of work left to do.
My goal was first, get the entitlement in place because once you have the spikes starting, you're not going to get more eligibility expansion. Get the eligibility expansion in place while programs really aren't that good. People won't want to take advantage of them. Then start to improve the programs. Try to do it so you don't have these huge peaks. But you can't always control that. We are doing indicators for older kids. We're getting there. By no means is it perfect. We have the foundation in place; now we're building the house.
Q. A question about the SALT survey: are you able to link child outcomes at kindergarten age to children's status pre-kindergarten? Are you assigning the identifier that lets you link kids who received these Starting RItes?
A. Christine: No, it's going to have to be done on an aggregate basis, by income or some other factor. We don't have a way to connect a specific child with a specific kindergarten. We're working on it. I would say that in general Rhode Island is like a river. We go around the boulders. Nothing stops us. We couldn't get the single child identifier so we tried an approach that was a little bit different. The idea is, don't let the perfect be the enemy of the good. Just do the best you can. Ultimately, you may get to perfection. But you need to keep moving to get there.
So I think it's a testament to the Children's Cabinet and the interdepartmental team and the outside groups that every time we reach something that really ought to be a dam, they kind of take it apart piece by piece and eventually they get through it and on to the next thing. And that's what's really important in Rhode Island. There's no one person or group that's really trying to keep that dam up. They all can understand where we need to get. They may not agree with each bend, but they know that ultimately there's enough good will and intention and purpose and they know the risks and stakes are high so they're willing to make the kinds of compromises that they may not be willing to make elsewhere. And I think all of our agencies and Kids Count have been really good about that. Compared to a lot of other places, we're light years ahead.
We never say, "Oh no, this is perfect, there isn't a problem." We always say, "Oh yes, that's a good question, that's a big problem, we haven't solved that yet." But it's all out in the open in terms of what has to be done.
James T. Dimas,
Senior Associate,
Casey Strategic Consulting
If policy makers question your veracity or the validity of your analysis, and if you develop that reputation, you might as well be driving a cab for all the influence you'll have on policy. That was the situation I saw first-hand in Illinois. I was credited recently as being from the Department of Public Aid. Not that they're bad people, but I was from the Department of Human Services. And the Department of Public Aid, in contrast, had a huge problem in the state house and in the governor's office with respect to the quality of information that they use and try to influence policy with. It made it very very hard for them to be effective. So I offer you three things to keep in mind on the subject of maintaining credibility on data.
Be Proactive. If you're in the role of data provider, especially if you're state agency staff, don't wait for the secretary or the director to call you after a story's in the paper to say, "What do we have on that?" If you want to have influence it's all about relationships, just like in business or any other walk of life. You've got to establish a relationship with the policy makers you're interested in influencing, and it has to be based on credibility and trust.
And a way to get there is by hustling. When you see a story in the paper, go back without anybody asking you to and figure out what you have that can shed some light on that, and how you can help with the response. I can tell you from personal experience that nine out of ten times what you work so hard to produce isn't going to get used. But maybe the tenth time it will. And when people start to use that you'll be recognized as someone who's an asset and someone who has value to add, and then it begins to snowball. So you have to just soldier through the lean times, the nights you stay up putting together a letter to the editor in response to a negative story. Even if it doesn't get used, you have to still be content that you're on the right track. And just keep on doing this, because eventually you'll break through and that begins a pattern of credibility with policy makers.
Don't overcook the numbers. Anyone with any data savvy can see the telltale sings of a statistic that's been tortured too long to tell the right story. It's not worth it. You're debasing your own credibility. A good rule of thumb is, think about a neighbor or a family member who is just not that interested in the information. And if it takes more than five minutes to explain to them what an index or table or graphic means, it's probably overcooked, and you're probably better off not having anything on that particular issue than rushing in with something that's been on the rack all night.
Check to ensure there is no credible, contradictory data. Do an internal check against overcooking. Before you send something up the chain of command, ask yourself, "Is there a chance a credible source of information might offer something that contradicts this?" If there is, you're probably better off not doing it. Because all it takes is another group or agency coming up with data on the same issue. If yours is more attenuated because you've worked so hard to get something relevant to this issue, you end up being the loser.
You'll know you're on the right track when your data gets the benefit of the doubt. And that feels good. When there is an issue where somebody else has an opposing point of view that's supported by data, and your secretary or senate finance committee chairman trusts your data over theirs, you've established the credibility you need to be effective.
My premise is that, whether we like it or not, in making public policy what we're looking for is insight telling us whether a reasonable hypothesis is on target and whether it's worthy of being acted upon. We're not trying to prove causality. There's a role for that. That's important work, but it's not our work. We need to be careful that we don't get confused about that or refrain from acting on information that does provide insight.
Let me show you an example of how that can work. This was something from 1997. You remember that back in the early nineties the federal jobs program was created. This was the precursor to the whole welfare-to-work movement. Illinois operationalized this new funding source by concluding that this was separate work. It established separate offices for people who wanted to go to work. So when a TANF or AFDC client expressed an interest in going to work, we would have to send them to wherever that office was. On its face that didn't make sense. The reasonable hypothesis was, if we really want to have people go to work, wouldn't it make sense to have jobs and income maintenance in the same place? And another thing you need to know is that from the beginning of the program the staff that ran the jobs program always set targets for local offices regarding the number of job slots they were expected to fill. No local office in the five years that program was operating had ever met its monthly target.
So we decided to try something different based on that reasonable hypothesis and based on very little more than, gee, it makes more sense to have those things integrated. So we took twenty-four local offices, some of them in Cook County and some of them downstate, and divided them into two groups which we called blitz offices and non-blitz offices. We physically moved the jobs staff in with the income maintenance staff in the blitz offices, and within the space of one week. And within four weeks, we did a briefing for the secretary, and showed him a graphic. Within one month most of the blitz offices had achieved their target for the very first time, and conversely the others fell short as usual. We didn't have a p value, but that was enough for our secretary to say, okay, let's do this statewide — which we did.
And within about one month, a lot of the non-blitz offices started hitting their targets or coming close to it, which validated that step. But an important point is, the secretary didn't wait and we didn't wait to take the step until we had something more conclusive. I think that if you're working off a reasonable hypothesis, that's a responsible way to go in human services. There's too much at stake and the window is so short for acting, that you'll have a change of leadership if you don't act on less-than-conclusive evidence.
I almost forgot the most important point of the story. I had the dubious pleasure two years later in 1999 of attending a conference in Washington, D.C. It was sponsored by ASPE in part. A colleague from Illinois in a division that ran the separate jobs program presented the results that MDRC had gotten with two other states and Illinois. They concluded it didn't make sense to have the offices separate. And they demonstrated it to a .0001 level of significance. And that was great, but if we had waited for a study before we made this move, there would have been no way we would have won two awards, totaling almost $40 million. This willingness to act sooner ended up being an important step in the success that we had.
I've seen young folks especially start out with a spreadsheet and try to mine it to see what's there. That's not real productive without a hypothesis to work against. A hypothesis provides a context that helps turn the data into information and if you don't have that, you won't be as focused as you should be.
Another really important way to have an effect with your data is to engage policy makers in the formulation of those hypotheses. That's what we did in Illinois. It gives them some "skin in the game" and allows them to couple their accumulated experience, wisdom, and insight with your ability to provide data and information. And it also has the ancillary benefit, that, well, it's their hypothesis, they've got some ownership of it, and they're also the people who happen to be providing you with the authority and resources you need to pilot test interventions that are driven by that hypothesis. It's an easier proposition if you're trying to get someone who has some ownership of the hypothesis to give you the time and money and talent available to test interventions that relate to it.
Using indicators isn't a spectator sport. There's not much payoff in being an observer. You can get a lot farther if you can engage a policy maker in the formulation of reasonable hypotheses that are testable.
Once you have a reasonable, testable hypothesis, to continue to maintain your credibility and influence you need to be able to mobilize pretty quickly to test interventions driven by that hypothesis. What you're looking for there is to either substantiate the hypothesis or disprove it. I usually go at it from the perspective of trying to disprove it if it's really inherently feasible. Disproving is easier. If you can't come up with something that challenges the credibility of the hypothesis, then it's probably worth going ahead and doing a pilot test on an intervention that responds to it.
And then it becomes a recursive cycle. You can use your indicators to help with the formulation of a hypothesis. You can work with program and leadership people to design interventions that respond to that hypothesis and then use your indicators to get in the back way to evaluate whether the intervention had the desired effect. And it's a real good way of closing that book and making your indicators real and vital to policy makers.
And that leads me to the second and last part of this presentation. I want to show you three techniques useful to developing influential information.
Pilot Testing
One of these is pilot testing; I showed you something on that earlier. But I wanted to show you another thing that's more on point with respect to family and child outcomes and the work ASPE is doing with respect to service integration. When we started DHS in Illinois in 1997, the whole concept was that we were going to do integrated services delivery. That was going to help us achieve our federal work requirements and improve outcomes for families and children. The reality is that in the first couple of years that we did this, we got so consumed with the burning platform that those work requirements represented that we didn't really attend as carefully as we said we would to service integration.
When we reached a point where we had a little breathing room on work requirements, we decided we needed to do a pilot test on a service integration model that would feature co-located staff — substance abuse, mental health, domestic violence counselors. This involved co-locating them in our local offices and seeing how that improved the rate at which people were referred for treatment and follow-up services.
We didn't want caseworkers that had come out of an income maintenance background and eligibility determination trying to make those calls. We weren't trying to do treatment in our offices. We were just trying to do case findings and get referrals made and we knew that wasn't happening very well at the site. So we did a pilot test using eleven local offices, six of them in Chicago and five of them downstate. And we tracked what happened in the offices before we did co-location and what happened afterwards and made a chart. There's no statistical test here. But the important thing is, if you've taken the steps to establish credibility with policy makers, if you've worked with policy makers on the formulation of hypotheses, they'll trust their eyes, they'll trust their instincts and common sense and they'll know when a picture is telling them something. In this case this picture said, yeah, there were clearly more referrals happening, especially in Chicago in substance abuse, after we did the co-location. Likewise the same kind of relationship held for mental health. There was no real debate that something seemed to have been changed and it seemed to have improved things and it was improving the rate at which referrals were made.
We learned something else interesting on domestic violence. And that was that the same relationship held but look what happened in downstate Illinois: the rate remained really flat. And that gave us a pregnant opportunity to mine that data further and find out what was going on in downstate Illinois that was different from what was going on in Chicago.
Field Work
The second technique I want to recommend to you is fieldwork. In Georgia we are working with folks in their Department of Family and Child Services to try to help them with some poor child welfare outcomes they're getting there. We are looking at the substantiated cases of abuse and neglect as a percent of all cases reported. Georgia has 159 counties, which is actually a nice large number if you want to do this kind of quasi-experimental approach. There are counties with 40 percent or greater rates of substantiated abuse among all cases reported. You've also got a number that are at 10 or 15 percent or below. And that doesn't tell you anything specifically except that it tells you where to look for some answers.
We don't know if this difference lies in people's propensities to report or in the quality of the investigations done. Both of these are reasonable and interesting hypotheses. And what it sets us up to do is to send in a team on the ground to poke around in local offices in the counties and develop a more robust hypothesis based on that kind of field work. What we see on the ground will help us formulate a hypothesis that we then can test with the right kind of pilot structure, and then we can test different interventions that we hope will produce a different result.
An Epidemiological Approach
I started out in public health and that's where a lot of my perspective on this comes from. I encourage you to think about a kind of epidemiological approach where you just look for clusters.
For example, looking at the percentage of substantiated abuse or neglect per thousand population plotted against the percentage of children in poverty shows a couple of interesting clusters. Some counties have a very high percentage of poverty, but relatively low substantiated abuse. That suggests that these counties are doing something right, something that we need to learn more about. Other counties have a lower rate of poverty but much higher substantiated abuse. And so one of the things we're proposing to do is again, send a field team in on the ground to look at those counties to review case records and interview case workers using an assessment protocol. We may find something about the different practices or the different characteristics of those counties that might support the development of a testable hypothesis that could perhaps shed some light on this and suggest some promising interventions.
The second cluster that's worth looking at is the counties that have a comparable rate of poverty, but are really all over the board in their rates of substantiated abuse. And I'm dying to know what it is that people are doing differently in different counties. Again we would like to sharpen our hypotheses and identify promising interventions that we can test to see how they might impact the outcomes.
This is a process that is pretty easy to engage policy makers in because they want to know, too. And if you lay it out this way, they get "skin in the game" and then getting the resources you need to do pilot testing of interventions becomes a much less daunting proposition.
So I'd encourage you to use your indicators to gain insight and also to engage policy makers in the formulation of reasonable hypotheses suggested by what you can tease out of the indicators. And then work with program folks and policy staff to identify pilot interventions that you can test to respond to those reasonable hypotheses. And you can then close the loop by using indicators to assess whether those pilot interventions had the desired impact on the outcomes you're interested in.
Q. Do you have experience using GPS mapping data?
A. We've done a little bit of that and I think it's really useful. It's especially useful for focusing on the mal-distribution of resources. If, for instance, you can plot where community health centers are located and compare that to the incidence of preventable diseases or teenage pregnancy, that's something that would make a mal-distribution of resources glaringly apparent.
Q. I struggle with the language we use to talk about this. My understanding of social indicators is that the language is very broad. When you use "test," "prove," "hypothesis," it becomes confusing to folks without that background.
A. I concur. When I talk with policy makers, I don't ever use the word "hypothesis." Then they think they're in for a discussion about data. I try to keep it more focused on "hypothesis," but I don't say that to them. I talk about "insight into what the right work is." That's the way I like to talk about it. That's something with greater resonance with people. I use "hypothesis" with people in this room. I'd be very careful using that term outside this room. To borrow a phrase, I encourage you to "Think quantitatively and act qualitatively." By which I mean, the indicators are good for zeroing in on a possible hypothesis, but the real knowledge comes from going in on the ground and looking through case records and talking with case workers and trying to figure out in the real world what seems to be contributing to the disparities.
Q. When you present data like this, do you discriminate between population-based data and service-driven data for policy makers? Second: when I saw that graph on referrals, two questions came to mind: 1) was there a difference in resource availability in terms of who would be referred and 2) when you pop up with early data, what's your response in terms of being prepared to answer those kinds of questions?
A. You kind of have to stay light on your feet. I use whatever works that doesn't run afoul of the credibility issue. Sometimes mixing administrative data and population based data gets you there. Sometimes you can just go with population-based data to establish an insight. If I'm trying to get back to the real world that a policy maker's involved with, I sometimes use administrative data. I kind of mix and match as I need to, but I try not to leave myself vulnerable to criticism or do it in a way that would cost me credibility.
On the referrals point: yeah, we concluded that part of what we saw was the result of fewer providers downstate and also of transportation barriers. But those were things we didn't really know until we looked at it that way. And then we raised the next questions.
Q. Do you share data across departments in your state, and if so, how?
A. I do and I have the scars to prove it. This is a good opportunity to put in a plug for our hosts. The right thing is to say people should share information. I pounded my head against that wall a lot and only moved it imperceptibly. So I finally decided to, like the Rhode Island folks, flow a different way around that rock. I go to the governor's office and others and tell them to provide data to Chapin Hall. What we run up against is, people raise concerns about confidentiality and privacy that are legitimate but when we look at this hard in Illinois, the 80/20 rule applies. Eighty percent of the reasons they don't share information is related to about 20 percent of the reality about what the law requires. The laws aren't nearly as restrictive as our culture and folklore make them out to be. When you dig into that one exception that's provided even in that most restrictive setting — substance abuse data — you can use that information without identifiers to support research. We used Chapin Hall as a repository that would get data with identifiers and then give it back to us without identifiers. And we were then able to use the data in an integrative way that we wouldn't have been able to otherwise.
Q: I've found that some of the confidentiality requirements can be gotten around, but that some of the political people don't want to even be compared across departments.
A: You're absolutely right.
Q: In Hawaii we had the opposite result with respect to integrated offices. We found that if you tell staff you want more referrals, you'll tend to get them. It doesn't really matter if they're integrated or not. But the other factor that enters into it is you have to do the work on the treatment side. You can get these higher referrals but then the question is, what percentage of those cases is accepted for treatment? And in our case we had to work out arrangements with the managed care providers who provided substance abuse counseling as well as medical treatment and get the directors of treatment within those plans to agree that yes, they were going to accept those patients and allow them to be treated under the plan so there wasn't a separate cost. So sometimes I think you can get what appear to be the results you're looking for by hypothesizing that if you do this you'll get that. But if you look a little deeper, or try different results, you may discover you'll get the same thing without doing it.
A. That's absolutely true that the stuff that gets measured gets done. So you have to be careful about what it is that you're measuring, because people will do the wrong thing. It's really clear that you need to retain your focus on what is the right work. When I know that what we're about is doing the right work and I know that we're going to do the case work behind it, I'm not above using something even if I have to hold my nose a little bit to get a policy maker to say, "Okay, yeah, we should do that." If you're not a person of good conscience or you're not prepared to remain focused on what the right work is, you're absolutely right, it can be dangerous.
Home Pages:
Advancing States' Child Indicators Initiatives
Human Services Policy (HSP)
Assistant Secretary for Planning and Evaluation
(ASPE)
U.S. Department of Health and Human Services
(HHS)