What Does Data Think?

In most of the media companies that employed me in the early years of the web, one of my responsibilities was supervising the metrics department.

In case that sounds like a big deal, this was well before the days of data scientists and multi-variable analysis; in most cases the metrics department in those media companies consisted of a lone individual.

And that person often felt like no one was listening to them.

After all, much more significant than the actual numbers he or she gathered was figuring out how to interpret that data. In and of themselves, of course, the numbers were neutral. But the people we worked with had a wide variety of opinions over what those numbers actually meant.

Was our audience growing? Which types of content were most successful? What was success in this type of media environment anyway? Which metric mattered most?

Occasionally, especially in the early years, we would publish a story that “broke the servers,” i.e., generated more traffic than our system could handle. There was little debate on those occasions over whether we had a winner, particularly because additional things tended to happen to support the data.

Things like attention from other media outlets, tons of email from subscribers and a boost to whatever financial metric we were tracking.

But these experiences caused me to eventually draw a few conclusions about people in general:

• Most people are not very good at math.

• Most of us see what we want to see in the numbers and don’t see what we don’t want to see.

• Most of us don’t change our behavior or opinions even when the numbers say we should.

In the end, I wondered, what did the data itself think about all of this human frailty? That is one reason I have long been curious about the coming of generative artificial intelligence — we may find out the answer to that question.

Addendum 2025; I guess we’re finding out now.

(This is originally from 2023.)

Meanwhile, a family note:

Congratulations James!