…there are vastly more possible comparisons than there are data points to compare. Without careful analysis, the ratio of genuine patterns to spurious patterns – of signal to noise – quickly tends to zero.

Tim Harford in the Financial Times has a great article called Big data: are we making a big mistake?.

5 thoughts on “Statistical Validity in Big Data

  1. Isn’t the spirituality of data more important than the science of data? Is the illusion of separation between our biological selves, 1’s and 0’s intentionally predatory?

    Does anyone else believe that zeros and ones aren’t enough to communicate the complexity and value of the human spirit? As we become aware, our tools must also become aware.

    To become truly meaningful, we must first understand the big data between our ears. Otherwise, the human value proposition of big data is going to remain fuzzy at least to me.

    Maybe data can help us measure our collective awareness / consciousness?

    1. Interesting post. I’ve never thought of data connected with spirituality, but more as an electronic result (in the form af 1’s and zeros) produced by human(s)pirits.

      But, yeah, in the sense of unity (being on board with the 111) in the sense of, we are all one, do exist in tHIS reality outside-and/or-bside the illusion of seperation in harmony with the complexity of your bRAINs/MINds, body & soul …
      big data gone thru some measurement and brought in-formation could be a way to show (a mirror?!) the mass of collective awareness / consciousness on this planet.

      BTW hows the weather at your side of the world … sun or rain? 🙂
      If you know what I mean?

  2. To be honest, I think the “BIG” data ITself is the problem, not leading to more insight, answers, etc. because it’s just too big and gettin’ bigger every day. (BTW … wasn’t the answer 42? And, yeah, what was the question in the beginning?!)

    Or in a more non-fictional context:

    What do the data analyst’s want ->
    (most of them surely know never to trust a statistic that isn’t doctored by themselves; first lesson in every statistic lecture)
    -> to present, to convine, to sell us data consumers, again and a:gain?
    Is the aim to help the needy or just serving the greedy?
    Is it a higher goal to lift the human soul or contributing just to the cosmic black hole?
    I do love the zero, my personal hero!
    Look at zero and you see nothing, but look through IT and you will see the WOR[L]D.
    0ne can even look beside IT and you can see something, just not NOTHING.
    Ohh, Eastern Wisdom and (Wild) Western (Monkey) Mind – You could collide! What can you find?

    Ach00!
    Sorry, it’s not the flu … just pollen or dust to give you a clue 🙂

SHARE YOUR THOUGHTS