The bottom lines that drive non-profits are often numerous and disparate, making it challenging to develop useful data science solutions in the non-profit sector.
We were very grateful to recently team up with The Graduate! Network (TGN) – a group that collaborates with stakeholders to build, nurture, and assess structures and systems that support adult ‘Comebackers’ to attend and complete their college degrees. The effort began in Philadelphia and has now been adopted by communities across the country.
For this project, we analyzed data on 8,401 Comebackers in 12 communities nationally. TGN collects these data to understand the most impactful practices that support Comebacker success.
Urban Spatial was tasked with using these data to predict Comebacker outcomes including enrollment and graduation – work we framed around a fundamental question: How can a more data-driven approach to TGN advising provide value to Comebackers? This question was addressed in two ways. First, we sought to understand if Comebacker outcomes like college re-enrollment, could be predicted. Second, we made some broader recommendations to help TGN find even more value in the administrative data it collects daily.
Prediction is only possible if there are some shared TGN ‘experiences’. In other words, do Comebackers move systematically through the TGN ecosystem? We spent weeks working with the data and with TGN to understand the TGN advising process, how advisors or “Navigators” interact with Comebackers, how those interactions are coded in the data, and ultimately, how Comebackers move through this system.
The figure below is our attempt to illustrate these systematic patterns which are highly non-random. This gave us the sense that it should be possible to make predictions in this system.
To better understand the efficacy of a ‘dose’ of TGN advising, we set out to estimate the effect of an additional High Touch Contact (HTC), effectively an advising session, on the propensity for a Comebacker to re-enroll in school. This is a critical outcome for TGN.
Our approach was to estimate a statistical model that identifies the HTC effect on enrollment, conditional on demographic and pre-TGN factors. On average, the model estimates that TGN Advising has a small but statistically significant and positive effect on a Comebacker’s propensity to re-enroll.
As some Comebackers receive relatively few HTC doses, and others, many more, we wanted to use the model to gauge success for the ‘average’ Comebacker. Two comparable personas were created from the data – one older and one younger. Imagine fastening an ‘HTC knob’ to the statistical model, subjecting it to these two personas across three TGN communities, and recording the predicted enrollment probability while iteratively dialing up the dosage of HTC.
The results are above. Each line is positive, suggesting that as the dose of HTC increases, so does the probability of enrollment. On average, younger Comebackers have a higher probability of enrollment than their older peers. The steeper slope of the lines in certain communities suggest that an additional HTC has a greater marginal effect in some places than others.
To predict the probability that a Comebacker would enroll in school beyond identifying the HTC effect, we limited ourselves to the relatively small amount of data TGN has on clients as they begin working with the organization. A set of ‘goodness of fit’ indicators contextualize the predictive model’s ability to help Navigators counsel Comebackers.
We found that it is possible to predict enrollment. The figure below shows the distribution of predicted enrollment probabilities for different groups in the TGN data. Some demographic groups exhibit a higher propensity for success, and success again, varies by community. An alternative goal of this analysis is to test the extent to which our predictions generalize across different protective groups – as our past work has focused on.
Finally, by interviewing various TGN stakeholders and studying TGN’s existing tools, we developed a vision for an enhanced ‘Comebacker Tracker’ dashboard to enable TGN Navigators to manage their clients.
A Navigator can see how a Comebacker is progressing across time; basic demographic factors; recent Navigator correspondence; some critical indicators for success; and a predicted probability of enrollment success. This is just one example we offered about how TGN could continue to capitalize on the value of the administrative data they collect.
Ken Steif, Ph.D is the founder of Urban Spatial. You can follow him on Twitter @KenSteif. He is the Program Director of the Master of Urban Spatial Analytics program and a Professor of Practice in City Planning, both at the University of Pennsylvania.