Outliers

What are outliers and how do you deal with them ?

We are living at an unprecedented time, where none of us in our life time would have anticipated a pandemic. In fact, most of us would not even be aware that the words ‘pandemic’ and ‘quarantine’ could be applied in real life situations. In the last six months we are trying to adjust to the new realities, with masks becoming new norms.

After the COVID-19 outbreak, I started to look into the data to learn about the history of pandemics. We see that there were quite a number of infectious diseases in the past. Even in this modern era of data analytics, we missed predicting a pandemic in advance and are being left unprepared to deal with it.

Looking from my experience I think, a pandemic at the scale of COVID-19 is not a regular event. Can we term it as an outlier?

Outliers are irregular, clearly different data from the rest of the data points. In the process of data analysis and interpretation of the results, a good understanding about outliers is very essential. Outliers may negatively bias the result of data set.

Example:

New born babies generally weigh between 5.5 lb (2.5 kg) and 10 lb (4.5 kg).

In the given data set below, baby Tracy’s weight is an outlier, most possibly a typo.

In this case, the outlier data point can be dropped from data analysis or imputed using mean of the variable (baby’s weight at birth).

Visualization helps us understand the data better. Box and whiskers plot or box plot is generally used to visualize the outliers.

Any datapoint that lies below the first quartile and more than third quartile is called the outlier. If we mark our data, baby Tracy’s weight 85.7 will be outside the box, i.e., an outlier.

One factor is dataset size. In a large dataset, each individual point carries less weight, so an outlier carries less weightage than the same data point would be in a smaller dataset. In the above example, data set size is 4. Each data carries more weightage. In a case where there are 40,000 or 400,000 data points, each data point would then carry a lesser weightage.

Another factor is: how far out of line is the outlier with respect to the rest of the data points. In the above example, data point 85.7 is way farther than maximum 10. If there was a baby weighing 12 lbs, it is also an outlier but the data point would be pretty close to other data points compared to 85.7.

With all these understanding about outliers what do we feel when we come across people who are different. Wondering how a recruiter will react when they come across a candidate with a different background than rest of the team, or with a break in their career? Would love to hear your thoughts on how your organization collects data points about the candidates and hire or reject them? How many companies have data analyst publishing these data points out in public? Are they dropped out of opportunities or given a chance to get them inclusive?

In my views, when people think different than others rather than completely ignoring it we need to give a deeper thought. In terms of business before a decision is taken the data needs to be explored in all possible ways, even outliers need to be analyzed.

1) Business decisions needs to be data driven.

2) Outliers — data that is different from rest of the data points.

3) Outliers cannot be completely ignored, they need to be analyzed.

4) When outlier is a risk, they should definitely be addressed.

One of my favorite Thirukkural — shortly the Kural, is a classic Tamil language text couplets of seven words.

thottanaithu oorum manarkaeni maantharku
kattraianthu oorum arivu

Meaning: In sandy soil, when deep you delve, you reach the springs below; The more you learn, the freer streams of wisdom flow.

Related posts:

Something that I took for granted before quarantine was simple human interaction. It seems so trivial, however, sitting next to a friend, hip to hip, in a photo booth without masks or a fear of…

Lori Carlson writes Poetry, Fiction, Articles, Creative Non-Fiction and Personal Essays. Most of her topics are centered around Relationships, Spirituality, Life Lessons, Mental Health, Nature, Loss…

The retrospective post we shared last year included a roadmap that spans multiple years. It included three focus areas: Flow is built for huge codebases: we do types at scale. But Flow wasn’t holding…