Wednesday, November 26, 2014
3

Living Big Data

CAMBRIDGE – Big data is made from the digital trail that we leave behind when we use credit cards, mobile phones, or the Web. Used carefully and accurately, these data give us unprecedented scope to understand our society, and improve the way we live and work. But what works in theory may not translate well in the real world, where complex human interactions cannot always be captured, even by the most sophisticated models. Big data requires us to experiment on a big scale.

My own laboratory, for example, is building a Web site which, based on Google maps, uses society’s digital trail to map poverty, infant mortality, crime rates, changes in GDP, and other social indicators, neighborhood by neighborhood – all of which will be updated daily. This new capability allows viewers to see, for example, where government initiatives are working or failing.

But, while such impressive visualization tools can dramatically enhance transparency and public knowledge, they are surprisingly limited when applied to solving society’s problems. One reason is that such rich streams of data encourage spurious correlations.

Even the use of the normal scientific method no longer works; given so many measurements, and so many more potential connections among what’s being measured, our standard statistical tools generate nonsensical results. Without knowing all possible alternatives, we cannot form a limited, testable set of clear hypotheses. And if we can no longer rely on laboratory experiments to test causality, we must test it in the real world, using massive volumes of real-time data. This involves moving beyond the closed, question-and-answer process typical of the lab, and applying our ideas in society, earlier and more frequently than ever before.

To see how things work in reality, we must construct living laboratories – that is, communities willing to try new ways of doing things (to be blunt, to act as guinea pigs). An example of such a living lab is the “open data city,” which I launched with the city of Trento in Italy, along with Telecom Italia, Telefónica, the research university Fondazione Bruno Kessler, the Institute for Data Driven Design, and local companies. Importantly, this living lab has the approval and informed consent of all involved; they understand that they are participating in a gigantic experiment whose goal is to create a better way of living.

One major challenge for a living lab is to protect individual privacy without diminishing the potential for better government. The Trento lab, for example, will pilot my proposed “New Deal on Data,” which gives users greater control over their personal data through trust-network software such as our open PDS (Personal Data Store) system. We hope that the ability to share data safely, while protecting privacy, will encourage individuals, companies, and governments to communicate their ideas widely, and so increase productivity and creativity across the entire city.

But the biggest difficulty in using big data to build a better society is being able to develop a human-scale, intuitive understanding of social physics. Although dense, continuous data and modern computation allow us to map many details about society, and to explain how communities might work, such raw mathematical models contain too many variables and complex relationships for most people to understand.

What is needed is some kind of dialogue between human intuition and the compelling reality of big data – a dialogue that is currently absent from management and government systems. If big data is to be deployed effectively, people must be able to understand and interpret the relevant statistics.

This calls for a new understanding of human behavior and social dynamics that goes beyond traditional economic and political models. Only by developing the science and language of social physics will we be able to make a world of big data a world in which we want to live.

Read more from the "The Innovation Revolution"

  • Contact us to secure rights

     

  • Hide Comments Hide Comments Read Comments (3)

    Please login or register to post a comment

    1. CommentedNathan Coppedge

      I am the originator, or at least one of the originators, for a method called categorical deduction. This may have a promising role in relation to big data as described in the article, through the use of coherent data-points and deductions performed upon objective tables of correspondence or coherent sets such as truth-tables (and similar operati).

      The general method of categorical deduction is explained in my book, The Dimensional Philosopher's Toolkit (2013, 2014). But, basically, it follows the method of A-B : C-D and A-D : C-B using polar opposites or typologically and cyclically related category sets. Because all words are potentially opposites, according to my theory, some level of coherent detail is possible about any given data-point that can be explained as a category-set.

      I advise that those wishing for an integration of virtual reality with social media, wishing to investigate the future of public interface or intelligent A.I. look into my book. Although it is unassuming in its message, the technique of categorical deduction simplifies data by a factor of ^2, what I call exponential knowledge. So, therefore, it is strange to call it a development upon previous methods. It is more like a revolution. This is like quantum computing without quantum technology. Big stuff. And apparently, it applies to big data as well.

    2. CommentedDan Kronstadt

      First of all - WAY too hard to register for comments. Why such a strong password? And give the rules at the START OFF THE REGISTRATION. And the image captcha is too small for a mobile device.

      Moving on : it says "this living lab has the approval and informed consent of all involved." Really? ALL involved? You got 100% approval from the community? And second: it occurs to me this sounds a lot like the social engineering that Isaac Asimov described in the Foundation sci-fi trilogy. If only he had published the DETAILS of how it worked!

    Featured