ON FEBRUARY 15, 2017

Open Data in the Age of Trump

As the new administration finds their way around the new digs in Washington, they’ve been unpacking their bags and claiming top bunk in much the same way that every administration has done before them. If you ignore various entanglements with the media, a cavalier sense on objective truth, an immigration ban, and the leader of the free world taking a hard stance on a department store, it’s all been pretty ho-hum on Pennsylvania Avenue.

In the first hundred days they’re doing their best to make good on several of their campaign promises, a systematic and total overhaul of the government chief among them. “Drain the swamp” was a popular rallying cry during the waning months of the election, and it spoke to a popular opinion that everything in Washington was fundamentally corrupt and horrible. The problem with draining a swamp is that it’s an ecosystem, you see, and while you might be trying to get rid of a crocodile, you’re also going to be getting rid of those adorable little birds that pick the stuff out of their teeth. It ignores the effects, and because of that I assumed it was just a soundbite.

So it was a surprise, to say the least, when I checked in on the White House’s Open Data Portal the other day and saw that every data set in it had been removed. This, no doubt, is at least partially due to a website redesign. The portal’s homepage now proudly boasts the shining face of a new Commander-in-Chief and says that we should “check back soon for new data” (huge data, no doubt). But it also speaks to something problematic: while it’s all well and good to change the look of the site to reflect the current administration, the data that was available on the site wasn’t about President Obama’s favorite movies — it was a window into what the government was doing. Here are a few of the data sets that went missing:

It is entirely possible that these and the few dozen other data sets on the White House’s open data portal will be restored after they finish the redesign, but what’s worrying is that they were removed at all. These are pieces of data that are supposed to represent a commitment to transparency from the most powerful nation on the planet. The data didn’t lose the election; we shouldn’t have lost the data.

But wait, we didn’t! A bunch of clever, forward-thinking people at the US National Archives archived the data. You can find the archive here.

And a different, equally clever, group of people built a platform that hosts all the open data portals in North America. It’ll be there, just in case anything else goes missing. Check out the catalogue of data on Namara, and the White House Data here.

Addendum: Last month a House bill was introduced that would effectively scuttle the Affirmatively Further Fair Housing (AFFH) initiative. The bill states that

“No Federal funds may be used to design, build, maintain, utilize, or provide access to a Federal database of geospatial information on community racial disparities or disparities in access to affordable housing”

While the disruption of service to the White House open data portal is problematic for a litany of reasons, Congress defunding open data initiatives is the biggest risk to a transparent and accountable government.

Note: It has been pointed out that a previous statement regarding the National Archives “deciding in the wake of the election to start… archiving data” is false. Of course the National Archives has since 1934 been committed to the preservation and care of the records of the US government and any statement to the contrary would be inaccurate.

“Micro-optimizing” Parallelization
in JRuby using Sidekiq workers

I’ve been reading up on improving JRuby/Java performance and I keep running into sentiments like…

Streaming Large Files in JRuby

Before the world’s Open Data ends up on Namara, it’s pumped through a multi-component pipeline that…