I realized how deeply seated some cognitive biases are.
In fact, we often dont even consciously realize when our thinking is being affected by one.
As data scientists, our job is to make sense of the facts.
In carrying out this analysis, we have to make subjective decisions though.
So even though we work with hard facts and data, theres a strong interpretive component to data science.
It’s free, every week, in your inbox.
In this piece, I want to point out five of the most common types of cognitive biases.
They needed to reinforce the militarys fighter planes at their weakest spots.
To accomplish this, they turned to data.
Based on that information, they recommended that the planes be reinforced at those precise spots.
Do you see any problems with this approach?
Lets think about how this might apply to our work in data science.
Say you begin working on a data set.
You have created your features and reached a decent accuracy on your modeling task.
But maybe you should ask yourself if that is the best result you could achieve.
Have you tried looking for more data?
Sometimes, hard as it is, the best thing to do is to let go.
This happens often with data science projects.
You have to develop the habit, hard as it is, of ignoring the previous cost information.
Of course, it is never easy for us data scientists to just disregard data.
For myself, I have found that a methodical way works best in this case.
If the cost part of the task seems overly significant, then it is time to move on.
False causality
As data scientists, we are always in search of patterns.
The tendency means that sometimes we even find patterns where none really even exist.
Those five words are like the hammer of the data science toolbox without which you cant accomplish anything.
Just because two variables move in tandem doesnt necessarily mean that one causes the other.
This principle has been hilariously demonstrated by numerous examples.
Thus, you might infer that more firemen are causing more damage.
Since this makes little sense, we should obviously suspect that there was an unobserved variable causing both.
Ice cream sales dont cause crime, nor does crime increase ice cream sales.
In both of these instances, looking at the data too superficially leads to incorrect assumptions.
As data scientists, we need to be mindful of this bias when we present findings.
Often, variables that might seem causal might not be on closer inspection.
We should also take special care to avoid this bang out of mistake when creating variables of our models.
If you have, then youve suffered from availability bias.
You are trying to make sense of the world with limited data.
As a result, we limit ourselves to a very specific subset of information.
This happens often in the data science world.
A way to overcome availability bias in data science is to broaden our horizons.
Commit to lifelong learning.
Then read some more.
Discuss your work with other data scientists at work or in online forums.
Be more open to suggestions about changes that you may have to take in your approach.
With enough work, you might distort data to make it say what you want it to say.
We all hold some beliefs, and thats fine.
Its all part of being human.
We can see this tendency in our everyday lives.
We often interpret new information in such a way that it becomes compatible with our own beliefs.
We read the news on the site that conforms most closely to our beliefs.
We talk to people who are like us and hold similar views.
Ive seen people clinging to the data that confirms their hypothesis while ignoring all the contradictory evidence.
Obviously, doing this could have a negative impact on the benefits section of the project.
Sometimes it is useful to be able to make some sense out of the world based on limited information.
In fact, we make most of our decisions without thinking much, going with our gut feelings.
The potential harm of most of our day-to-day actions is pretty small.
Allowing our biases to influence our work, though, can leave us in an unfortunate situation.
Knowing how our brain works will help us avoid these mistakes.
