In the marketing world, big data is used to answer ostensibly minute questions every day: are computer mouse movements predictive of purchasing? Does an orange background increase user engagement? In every place with Silicon in its name, there are teams of data scientists asking these questions.
In the social sector, by contrast, answering helpful questions is more difficult. For instance, is our program reducing homelessness? How is health spending distributed across the state? Part of the difficulty is the nature of the problems we are trying to solve. Lowering rates of recidivism is far more complex than driving ad clicks.
But there is another reason that governments, nonprofits, and philanthropists find it difficult to answer basic questions about social services: missing data. We simply do not have the infrastructure to collect and analyze data that reveals true social outcomes.
We at the Sorenson Impact Center are excited to help address this problem. Alongside Raj Chetty and some excellent organizations, we received a federal grant to help free up, merge, and analyze data to create better social policies. As part of the Pay for Success initiative, the White House has challenged us to assist service providers who will be paid only if they produce good outcomes.
The exciting part, from our perspective, is the possibility of using the same tools that were created to optimize things like ad performance in a totally different context. Silicon Valley’s emphasis on data science has led to a proliferation of free, open-source software, a great deal of which can be repurposed in support of Pay for Success projects. The trick is to know how to work with it.
For instance, data scientists at Google, Twitter and Airbnb have released software that can help estimate causal impacts, detect anomalies in time series data, and interactively visualize large datasets, respectively. This is part of a broader movement to share code and create tools that others can use. Ultimately, this movement towards open source could give cash-strapped governments and nonprofits access to the same software resources as billion-dollar tech companies.
The Bazaar
Open-source software has been described as a metaphorical bazaar. As opposed to a Cathedral — where top-down decisions drive proprietary software — the Bazaar is a place where people from all over the globe can help fix bugs and generate a final product that is free thereafter. In our proposal to the federal government, we volunteered to serve as “guides to the Bazaar,” helping governments and nonprofits identify and learn how to utilize these rich resources.
At the risk of sounding like techno-utopians, we think this approach of using big data tools has the potential to help solve some of our biggest social problems. One can imagine employing Artificial Neural Networks to identify the parolees who are most in need of social work, or the students who need advanced tutoring. At the very least, we can use packages in the programming languages Python and R to transfer and merge data so that it is useful to service providers.
If this work sounds appealing to you, please reach out. We are hiring data scientists.