Tess Nicholson Powers and Alex Newell 2023-02-16 04:59:19
In central California lies Fresno Unified School District (FUSD), a K-12 urban school district serving over 76,000 students across 106 schools. Reflecting the changing demographics of the community, FUSD’s student population has been growing increasingly diverse; currently, about two-thirds of the district’s students identify as Latino and the remaining one-third identify primarily as Asian, Black, or White. Over 80% of FUSD’s students are considered “socieconomically disadvantaged” by California’s Department of Education.
FUSD’s Superintendent Bob Nelson had long been concerned about the wide variation in academic performance across the district—a trend that was further exacerbated by the pandemic. Superintendent Nelson wanted to ensure that supports were being provided equitably to the diverse student population so that each and every student would receive what they need to be prepared for college and career opportunities. FUSD had turned to other similar school districts to learn what they were doing to successfully support the needs of a broad range of students. Though FUSD made investments in initiatives that seemed to be working well elsewhere, the district did not seem to be getting the hoped-for results. Naturally, FUSD’s Board of Trustees began asking two key questions:
• With all that we are doing, why are we still getting the same results for our kids?
• How do we know what is working for our kids and what isn’t?
Understanding exactly how district resources (e.g., time, people, and money) are allocated and what really works for students is no easy task, especially across a large urban school district with an operating budget of over $1 billion. Around the time FUSD trustees began asking these questions, Superintendent Nelson heard about the strategic budgeting approach of District Management Group (DMGroup), which includes a data-driven methodology for measuring what works, using academic return on investment (A-ROI) analyses. Superintendent Nelson was specifically interested in seeing if DMGroup could help FUSD examine how equitably resources were being distributed across the district and assess whether resources were being effectively allocated to best support the district’s strategic plan.
"This process has been integral for every single one of us as staff in Fresno Unified to critically analyze what we are doing, and to make certain that all of our resources—our time, our money, and our efforts—are all having a substantive positive impact on our kids and our city."
- Bob Nelson, Superintendent, Fresno Unified School District (CA)
A year after working in partnership with DMGroup, FUSD is transforming how they approach budgeting to take a more data-driven, equitable lens to resource allocation. Deputy Superintendent Misty Herr shared, “Now, before investing in an initiative, we know the right questions to ask: What outcomes will we measure? How will we monitor progress? How are we going to know if something worked or didn't? Before, we did ask these questions, but now we have a process to actually get the answers.” FUSD is now moving forward to develop the systems and structures to expand and sustain this work long term so that they can continuously evaluate their work and ensure that resources are being used to deliver the best outcomes for students. Superintendent Nelson shared, “Part of the stewardship of the district is making absolutely sure that every taxpayer dollar is being utilized in initiatives that have a direct and meaningful impact on the kids whom we serve. This process has been integral for every single one of us as staff in Fresno Unified to critically analyze what we are doing, and to make certain that all of our resources—our time, our money, and our efforts—are all having a substantive positive impact on our kids and our city.”
After speaking with Superintendent Nelson and his team to gain a more detailed understanding of FUSD, its challenges, and its objectives, DMGroup proposed that the district begin by taking stock of all the programs and initiatives already in place and gathering key, actionable information about them. This information would then allow FUSD to conduct A-ROI analyses to ascertain what was working best, for which students, and at what cost—information key to allocating resources effectively and efficiently as well as aligning the investment of resources to support strategic objectives.
Building an Initiatives Inventory
As in virtually all districts around the country, FUSD has added many new programs over the years. But, as is the case for the majority of districts, the central office lacked a centralized management system and therefore had difficulty tracking all the programs that were in place. Even more challenging was trying to track how successful each had been and how different student subgroups were being served by these programs. In the absence of this type of information, budgeting too frequently defaults to maintaining the status quo or acquiescing to the loudest voices in the room.
In partnership with DMGroup, FUSD began to tackle the work of creating what DMGroup calls an Initiatives Inventory—a single document that lists all the programs and initiatives in place across the district, along with key information about each of them. FUSD decided to focus their Initiatives Inventory on those initiatives operated by the central office. An initiative is often defined by school districts as an investment in people, programs, or processes that is intended to meet a specific goal that lies outside of day-to-day operations.
As expected in a large, urban school district, there were many. Because FUSD was particularly interested in examining the degree to which resources were being equitably distributed across the district and understanding how resources were being allocated in relation to the district’s strategic plan, DMGroup proposed gathering the following data points:
Program Data
• Objective of the program
• Metrics for success
• Student outcome data
• Student populations served, including segment data
• Alignment to existing strategic plan
Fully Loaded Cost Data
• Direct: cost category, budget-line items
• Indirect: staff time
The DMGroup team created for the district staff an easy-to-use prototype in Excel to collect the specific data desired on each initiative. Superintendent Nelson engaged multiple stakeholders in the Initiatives Inventory work to help gather all the needed information. The main focus was engaging stakeholders who were involved at the systems level. Involving these stakeholders from the start would allow these team members to more easily use and maintain the inventory. It also had the added benefit of developing understanding and alignment among these stakeholders. The following stakeholders from across all 16 of FUSD’s departments were engaged in the work:
• A4 (African American Academic Acceleration)
• College and Career Readiness
• Communications
• CIPL (Curriculum, Instruction, and Professional Learning)
• Early Learning Department
• English Learner Services
• G2 Student Engagement
• Human Resources
• Information Technology
• Leadership Development
• Parent University
• Prevention and Intervention
• School Leadership
• Special Education
• Teacher Development
Each department head was responsible for working with their team to collect the needed information. DMGroup supported teams throughout the process by providing examples, hosting office hours to support data collection, and reviewing data entry to identify any gaps. This was a heavy lift over the course of a month for FUSD, but engaging so many stakeholders in the work was critical to gathering a comprehensive list of initiatives across the district and to start building a shared understanding of academic return on investment at a variety of levels across the district.
Developing Capacity to Capture Fully Loaded Costs
Capturing fully loaded cost information is not work that is typically done in school districts. Districts do typically track direct costs, such as the cost of purchasing curriculum. But the largest investment for districts is in staff salaries and benefits, and the cost of staff time devoted to specific programs and initiatives is not well-tracked in the majority of districts. The DMGroup team shared with FUSD staff a formula to capture the cost of staff time for each initiative (Exhibit 1), and then worked with them to put it into action. While the formula appears relatively simple and straightforward, this cost calculation is both a science and an art, built on estimates of the amount of time being spent and by whom. The underlying assumptions and rules for making these estimates must be applied with consistency and rigor across the board to allow for fair comparisons between programs.
To ensure shared understanding and consistency of approach, DMGroup first held a training session for all department heads. DMGroup asked department heads to consider their own investment of time in various initiatives and to estimate staff time invested in specific programs. FUSD department leaders then engaged with their department teams to address in finer detail the investment of staff time.
Analyzing Initiatives to Understand the Current State
At the end of the data collection process, FUSD ended up with a comprehensive inventory of 345 initiatives operated by the central office across 16 departments totaling roughly $153 million in ongoing costs. This is not an uncommon number of initiatives in a district of FUSD’s size.
Upon examining the data, DMGroup found that not all data was available for every initiative; this is typical for virtually all districts just launching this type of work. Many of FUSD’s initiatives did not have measurable objectives or systems to collect and analyze data. And actual student outcomes data was often not readily accessible or simply not available (Exhibit 2).
Because much of this data is needed to calculate the academic return on investment of an initiative, understanding the gaps in the data was useful information. It provided department heads a detailed understanding of what information they needed to capture in the future and where they might need better data systems to set themselves up to understand the academic return on their investments.
Gaining Insights into What Works Through Principal Feedback
Because outcomes data for many programs was not actually collected (or collected inconsistently), DMGroup suggested as a proxy gathering feedback from the school sites themselves. DMGroup often recommends this practice to districts where outcomes data is lacking or incomplete in the inventory.
DMGroup sent a survey with the following questions to the district’s principals to provide insight on the use and impact of initiatives at school sites:
• Use of Initiatives
- Is this effort happening at your school?
• Impact of Initiatives
- If yes:
- How effective is this initiative?
- What is this initiative’s value and importance?
- If no:
- Would it be valuable to add?
With the answers to just these few questions, FUSD had access to a wealth of information about where programs were or were not operating and which programs seemed to actually be working day-to-day for students. The survey also had the added benefit of making sure that principals and other site leaders had a chance to add their voice and perspective to this important conversation.
The results of the survey provided some interesting insights (Exhibit 3):
• Only about half of the initiatives that were operated by the central office were occurring across individual school sites.
• About three-fourths of principals felt initiatives that were being implemented at their school site were important but also expressed that about half of those important initiatives needed either some or significant improvement.
While this data represents principal perceptions and is not a full program evaluation, it provides insight into specific initiatives that may be working well and others that may need improvement—actionable information to help guide the district’s decisions.
Gaining Insights from the Initiatives Inventory
Once DMGroup consolidated the Initiatives Inventory data and summarized findings from the principal survey, DMGroup team members analyzed all the information. To address questions initially raised by FUSD district leaders and trustees, DMGroup analyses included specific focus on how resources were allocated in relation to the district’s strategic plan and across specific student populations.
• Alignment to Strategic Priorities
FUSD’s strategic plan specifies goals related to supporting students, families, and staff, and includes corresponding objectives and key results (OKRs) which outline how the district will measure progress toward these goals. FUSD prioritized five priority OKRs that they identified would drive impact in the district.
DMGroup mapped each of the 345 district initiatives to these five OKRs (Exhibit 4). Analyses showed that the majority of initiatives and ongoing costs were associated with Student Goal 1.
• A Look at Equity: Resource Allocation Among Specific Student Populations
While FUSD had placed great emphasis on equity, DMGroup found that almost half of the district’s initiatives were not targeted specifically toward any student population. Of the half that were targeted, a majority of these initiatives focused on students receiving special education services and tailored SEL and mental health supports; far fewer initiatives were targeted to English language learners despite FUSD’s large EL population (Exhibit 5).
This information and analysis helped FUSD understand and sense-check their commitment to equity. Deputy Superintendent Herr shared: “This will impact the way we do budgeting—I am sitting down with the CFO and using this information to appropriately allocate resources as we talk about equity. Usually we just say ‘everyone’ gets this [resource], but having this information is making us think about how we become better decision makers around equitable allocation of resources.”
FUSD leaders gained a great deal of insight through the Initiatives Inventory process and recognized the power of compiling and maintaining this document. Deputy Superintendent Herr commented, “By having all of this information in one place, we work better together. I think that one thing that really helped is we started to see where there was alignment between our priorities, like equity, and our spending. We’re now not looking at the budget as isolated departments but seeing how all of our work impacts and supports each other. We realized we had some initiatives doing the same thing, so being able to identify that allows us to allocate resources in a more cohesive, streamlined way to ensure we are supporting each and every student.”
Since reviewing the Initiatives Inventory, FUSD has been working to ensure that every large initiative has the necessary data points to be able to monitor impact against cost. This lays the foundation for better decision making and the ability to conduct more in-depth analyses in the future.
FUSD’s Initiatives Inventory offered a high-level understanding of how resources were allocated across the district, but district leadership’s primary goal was still to learn what was working for their students and at what cost. Conducting an A-ROI analysis was the next step. From the Initiatives Inventory, it was evident which initiatives had sufficient information to proceed with the analysis.
District leaders selected three large initiatives to take through this analysis, among which was the Clinical School Social Work Interventions (CSSW) program. The CSSW program provided licensed social workers offering robust mental health supports to students. The program had expanded in recent years, both in scope and in mission, and district leadership wanted to learn more about the impact of this change.
Here we will share how FUSD conducted an A-ROI analysis of the CSSW initiative. We will discuss the plan for the analysis, how data was collected and evaluated, and then how district leaders drew insight from the evaluation that empowered them to take action.
Planning for an A-ROI Analysis
Defining Success
Defining success sounds straightforward, but it can be a surprisingly difficult process for many programs. If the definition of success was not clearly articulated from the beginning, it is tempting to define success as whatever results the program ends up achieving.
"One of the big effects of our work was that we’ve become better decision makers when it comes to ensuring equitable allocation of resources."
- Misty Herr, Deputy Superintendent, Fresno Unified School District (CA)
FUSD’s Executive Cabinet brought in program leaders to align on the definition of success for CSSW. The initiative had established three clear program objectives but needed to decide how these would be measured. Together, the CSSW team worked to create a measurable definition of success for each program objective (Exhibit 6).
With support from DMGroup, however, the CSSW team saw that only one of these definitions was measurable using existing data-collection systems: the second definition of success required more frequent gathering of academic outcome data in order to show the impact of the mental health interventions, while the third definition relied on survey data that could not be accessed on a student-by-student level. The first definition, however, was able to be measured and provided a clear benchmark by which the CSSW program could be evaluated. It stated that 70% of the students served by the six-to-eight-week interventions would see some degree of improvement on their self-assessment taken after the interventions (when compared to the self-assessment they took before the interventions), a metric which both program leaders and district leaders felt accurately reflected a successful outcome for the program. The CSSW team resolved to focus on the first definition of success and then to return to their second and third definitions once they could measure them more clearly.
Collecting and Analyzing Data to Understand Student Outcomes
After the CSSW initiative leaders had planned for their analysis, FUSD was ready to supplement the data gathered during the Initiatives Inventory with more granular information that would help them understand the fundamental questions of A-ROI: Was the initiative working, for which students, and at what cost?
1. Gathering Student and Background Data
CSSW program leaders had already identified what student characteristic data they wanted to collect and had little difficulty in gathering this data from their systems. Because some of the data was specifically related to student mental health, additional coordination was required to make sure that student privacy was guaranteed, but CSSW staff were experienced in managing these requirements. The data fields gathered here were:
• Dates of first and last intervention sessions
• Number of sessions
• Type of counseling the student received
• Focus of the intervention the student received
In addition, central office data personnel provided background data on all the students in the CSSW program, including what grade and school level students were in. Assembling this data would allow for insightful analysis into which types of students were seeing the most success from the CSSW interventions.
2. Analyzing Student Outcomes
The CSSW data team was also responsible for providing outcomes data, since the main metric being used to determine the success of the program was a mental wellness assessment administered by CSSW. This proved easy to gather for the experienced CSSW data staff, who pulled the following information:
• What type of evaluation each student took
• What their scores were on entry to the program
• What their scores were after receiving intervention
Initiative leaders had hoped to see whether interventions resulted in an increase in GPA, so central-office data personnel pulled student GPA as well. DMGroup also conducted extensive interviews and focus groups with staff from all levels of the program, as well as students, parents, and principals, in order to understand how the initiative’s outcomes and implementation were viewed.
To see whether the initiative was meeting the definition of success that program leaders had set, DMGroup looked at the change in individual student self-assessments after they had participated in the CSSW interventions and found that, overall, the program was meeting its goal: 71% of all students in the program improved on their self-assessment after receiving intervention, passing the 70% threshold that leaders had agreed to.
3. Understanding Resource Investment
Having seen that CSSW met its definition of success, DMGroup analyzed the cost data collected during the Initiatives Inventory to understand the sum total of the initiative’s direct and indirect costs (Exhibit 7). The direct costs were relatively low, as the full expense of social worker licensing, training costs, and facilities used amounted to about 5% of the full cost of the initiative.
By far the biggest expense was the time invested by the 44 social workers employed as part of the program. Not all of their time was spent on these particular interventions, but taking into account their salaries and the share of time they spent each week working on this initiative, DMGroup found that their time was worth roughly $1.2 million to the district. Program and district leaders were surprised, having expected something closer to the $64,000 spent on materials, but found it helpful to be able to fully understand the extent of the resources they were allocating to student mental health supports.
A-ROI Yields Actionable Insights
As discussed above, the CSSW program overall was meeting its goal: 71% of all students in the program improved on their self-assessment after receiving intervention, passing the 70% threshold that leaders had agreed to. DMGroup conducted further analysis to examine how well various populations of students were meeting the program’s definition of success.
Upon investigating how successful the program was for students based on their reason for referral or the focus of their intervention, DMGroup found that while some student populations fared better than others (78% of students who were referred for Personal/Emotional reasons improved on their assessments compared to 60% of students referred for reasons related to being in the Foster Care program), most student populations were very close to seeing the 70% improvement rate targeted by initiative leaders (Exhibit 8).
When investigating students by grade level and type of counseling, DMGroup found more variation (Exhibit 9). In particular, DMGroup noticed two actionable opportunities based on student performance:
1. Elementary and high school students saw more consistent improvement
Elementary and high school sites saw fewer interventions than middle schools, but their students were more likely to improve on their mental health assessments after receiving support. While this finding suggests that interventions should be expanded at both types of school sites, elementary schools provided a particularly actionable opportunity, as interviews revealed that four social workers were spread across all the elementary school sites (in contrast to a dedicated staff person at each of the middle and high school sites).
2. Group counseling was less costly, but just as effective as individual counseling
In interviews, DMGroup heard from many stakeholders (including students) that increased time in group counseling would be preferable for students. As a result, the team was curious to learn how the intervention quality differed between individual counseling and group counseling in order to gauge how feasible this change would be. Surprisingly, the team learned that students in group counseling were essentially just as likely to improve as students in individual counseling sessions—meaning that a single social worker could reach more students without making it any less likely that those students would improve.
Beyond these specific considerations, DMGroup also shared other findings from the interviews and focus groups, including feedback from staff who said they spent a considerable amount of time recording the same data in numerous locations for the sake of compliance—time they could be spending with students. DMGroup also recommended investigating ways to gather metrics related to the other objectives of the program so that the full extent of the program’s impact could be measured.
By working through the Initiatives Inventory and the A-ROI analyses, the stakeholders who participated realized the power of A-ROI and the potential it held to help the district ensure its resources were being used effectively and efficiently. Deputy Superintendent Herr identified three primary takeaways from the work:
• A New Outlook on Equity: Ensuring equitable resource allocation was a priority at the start of the work and FUSD now has a clear understanding of how their resources are allocated across student populations. The high-level view provided by the Initiatives Inventory combined with the focused, student-level A-ROI analysis equipped FUSD leaders with a new set of tools for understanding how they were allocating resources and what effect those resources are having on specific student populations’ outcomes. “One of the big effects of our work was that we’ve become better decision makers when it comes to ensuring equitable allocation of resources.”
• Moving Forward with a Strategic Budgeting Lens: Following this work, FUSD wanted to make sure that the A-ROI mindset and approach becomes embedded into district processes at all levels. “The Superintendent and I sat down with the CFO to talk about our budget for the coming year, and we all reminded each other of the work we’d done with DMGroup: we needed to make sure that as we were setting aside funds, we were clear on how we would measure success.” This approach sets the foundation for any new initiative: whenever the district starts a new initiative, understanding how success will be measured ensures that FUSD can understand the return on investment and sustain this type of work.
• Cross-department Collaboration: As a result of the strategic budgeting work, FUSD leaders saw how some programming had become isolated and siloed in separate departments, preventing productive collaboration or consolidation of resources. “One thing that really helped was realizing all the work taking place across the district that was trying to achieve similar goals and impacting the same groups of students.” Now that the district has a holistic view of all initiatives, it’s much easier to see where that overlap exists. FUSD can now group those similar initiatives and determine which are working well for students and which can potentially be sunsetted or refined to remove duplicative efforts.
The FUSD team was energized by the work and gained a deep understanding of the power of A-ROI analysis and the potential of strategic budgeting. The district finished this work eager to expand it and embed it into the way things are done at FUSD.
By developing a comprehensive Initiatives Inventory and conducting A-ROI analyses in partnership with DMGroup, FUSD has moved toward a more strategic budgeting approach to ensure resources are distributed equitably and are showing results for students. Superintendent Nelson shared, “I think perhaps the most jarring element of this entire body of work was for us to realize how many of our initiatives are funded by the district without regard to specific metrics being captured, or any specific connection back to our existing strategic plan. It’s hard to move in any specific direction when you are taking a scattered approach to accomplishing your goals.” Now, Superintendent Nelson and the FUSD team have a clear understanding of their initiative investments and a plan to measure success.
To continue the work of building capacity, shifting mindsets, and embedding the strategic budgeting approach across the district, FUSD is enrolling multiple teams in DMGroup’s A-ROI Institute. Through the A-ROI institute, FUSD teams will receive professional development and coaching while they work as teams to conduct additional A-ROI analyses for their district.
“It’s really about the learning journey we’ve gone on. The conversations I am having with people right now around how we allocate resources are very different from when we started this work,” said Deputy Superintendent Herr. “This work has not only made us think about our current investments, but it is going to help us think through what we continue to invest in.”
FAST FACTS
76,050 students*
• 68.5% Hispanic
• 8.9% White
• 10.4% Asian
• 8.2% Black
• 2.6% Multiracial
• 0.6% Native American
• 12.9% Special education
• 18.7% English language learner
• 82.3% Eligible for Free/Reduced Lunch
Source: CASPP (California Assessment of Student Performance and Progress) Scores, 2020-21.
Total Schools: 106 (69 elementary schools, 14 middle schools, 12 high schools, 11 other schools)
Total Staff: 2,953
Per Pupil Expenditure: $17,046
Operating Budget: $1,140,425,580
Graduation Rate: 85.2%
Source: http://www.ed-data.org/district/Fresno/Fresno-Unified
©District Management Group. View All Articles.