ON AUGUST 15, 2017

Making Data Play Nice

How to find value in a mess of open data

When we started working on Namara a few years ago the idea propelling it forward was clear.

In order to live up to its potential, all the open data in the world needs to be made available on a single platform

It was a pretty easy problem to diagnose, all things considered. Governments of every size were starting to release open data, but they were doing it all over the place. Not only did you need to know where to look for the data, once you found it you had to know how to start working with one of the dozens of different formats it came in. Open data was supposed to be the fuel for innovative thinking, but in reality it was like a hundred people in different rooms trying to solve a problem together.

Namara was our solution. It would sit on top of every data portal, standardize to a common format, update and monitor changes, and in so doing create a standard space for all open data to exist. Ultimately, we thought it would be everything open data needed in order to finally live up to its potential. Two years later, we launched the platform and waited for the world to change.

When it didn’t, we looked a little harder.

To be fair, we’d solved one problem very effectively. We’d tracked down hundreds of data sources and brought their data (over a hundred thousand data sets) onto a single platform. We’d done what we set out to do. We’d put all the problem-solvers in the same room.

Turns out none of them spoke the same language.

Not only is the data that’s offered on these portals wildly different from source to source (I’ve written about this problem here), when you do find similar data sets there’s never an easy way to tie them together. We’d aggregated all the data and standardized it to a common format, but now we were confronted by a different problem:

The data sets were as idiosyncratic as the data sources.

To solve this new problem, we realized we had to build a tool that was capable of merging, refining, and enriching open data to create data sets more valuable than the sum of its parts.

The feature that came out of this realization is called Unity, and we’ve been amazed at the kinds of insight it’s provided. We’ll be making Unity available on Namara in the coming months, but before we do we wanted to showcase a few of the ways that we’ve used it.



Unity Case Study: Geocoding Traffic Accidents

World of Angus

World of Angus wanted to build an app that would tell people what they needed…

Yukon — Optimized File Deduplication and Merge

This was on our mind when we first discussed developing a strategy for removing and…