[Spectator@simileoluwa]

Descriptive Statistics is an integral part of any Data Analysis procedure, its concepts are necessary because they help make better business decisions. Generally, Statistics is defined as the study of the collection, analysis, interpretation, presentation, and organization of data. When raw data is presented, it would be hard to instantly make inferences from such data, this is where Descriptive Statistics comes in. It provides methods of presenting data in a meaningful way and eases the processes of interpretation. We can classify its importance into two parts:

1. To provide basic information about the characteristics of the variables in a dataset.
2. To identify existing relationships amidst variables in a dataset.

In a world of Big Data, it is becoming increasingly important for organizations to derive meaning from there voluminous datasets and one of the very basic procedures to derive meaning is Descriptive Statistics. There are different measures used for Descriptive Statistics which includes some visual methods, aimed at providing insights into a dataset:

1. Measures of Central Tendency: These measures are the most widely used and perhaps the most informative methods existing in describing a population’s characteristics. There are three notable Measures of Central Tendency:
   • Mean: This is a sum of the variables in a dataset, divided by the total number of variables
   • Median: This is the middle value of a variable in a dataset
   • Mode: This is the most occurring value existing in a variable
2. Visualization Techniques: The most prominent are Histograms and Density Plots, Boxplots. Visualizations are very efficient methods of giving summaries of data, identification of patterns, outliers, etc.
3. Measures of Dispersion and Frequency: These measures are used to show how spread-out data is and how often a response is given in a dataset. Some of the prominent measures include:
   • Range: This is the difference between the lowest and highest values.
   • Frequency: This refers to the number of times an event occurs or response is given.
   • Standard Deviation: This is a measure of the amount of variation or dispersion of a set of values.
4. Measures of Position: These include Percentiles and Quantile Ranges.

Implementing Descriptive Statistics in Python

We will address the four different Descriptive Statistics Measures; the dataset can be downloaded from Kaggle. The “MELBOURNE_HOUSE_PRICES_LESS.csv” dataset contains 13 columns and 63023 rows, it provides details of houses in Melbourne and how prices differ based on the location and characteristics of the house. Implementing the above-listed techniques in Python, we will need to first set up our libraries:

#Import the necessary libraries import pandas as pd import seaborn as sn import matplotlib.pyplot as plt

1 2 3 4

#Import the necessary libraries import pandas as pd import seaborn as sn import matplotlib.pyplot as plt

Lastly, for our setup, we want our Jupyter Notebook to show all results, therefore we will run the code below:

from IPython.core.interactiveshell import InteractiveShell InteractiveShell.ast_node_interactivity = "all"

1 2	from IPython.core.interactiveshell import InteractiveShell InteractiveShell.ast_node_interactivity = "all"

The downloaded dataset needs to be loaded into our console, we will also examine the contents of the dataset, once we have it loaded as shown below:

#Load dataset and examine the structure df = pd.read_csv(r"C:\Users\SIMIYOUNG\Downloads\MELBOURNE_HOUSE_PRICES_LESS.csv") #Examine structure of the dataset print(df) #Print the column names df.columns #Examine the tail of the dataset df.tail()

1 2 3 4 5 6 7 8

#Load dataset and examine the structure df = pd.read_csv(r"C:\Users\SIMIYOUNG\Downloads\MELBOURNE_HOUSE_PRICES_LESS.csv") #Examine structure of the dataset print(df) #Print the column names df.columns #Examine the tail of the dataset df.tail()

Examining the dataset shows us the existing columns, number of rows and columns. An important step in any Analytical process is to determine if there are missing variables in the dataset, this is easily achievable in Python:

#Examine missing values in each column df.isnull().sum() #Visualizing missing variables sn.heatmap(df.isnull(), cbar=False)

1 2 3 4

#Examine missing values in each column df.isnull().sum() #Visualizing missing variables sn.heatmap(df.isnull(), cbar=False)

Examining the dataset reveals that we have 14590 missing only in the price column, there are lots of ways to deal with missing values, however, that is not the focus of this tutorial. We will adopt the simple method of dropping all rows of data containing missing values in out dataset using the function provided in the pandas library:

#Drop all columns containing Nan (Not a number) df = df.dropna()

1 2	#Drop all columns containing Nan (Not a number) df = df.dropna()

We have dropped the rows containing missing prices. A practical implementation of Measures of Central Tendency is implemented as shown in the code below:

#Mean df["Price"].mean() #Median df["Price"].median() #Mode df["Price"].mode()

1 2 3 4 5 6

#Mean df["Price"].mean() #Median df["Price"].median() #Mode df["Price"].mode()

The results for each measure is given as 997898.2414882415, 830000.0, 600000.0 respectively. We can now move on to implementing visual methods which includes but not limited to Histograms and Densityplots and Boxplots for Descriptive Statistics:

#Visualizing price using Histogram sn.distplot(a = df["Price"], kde = False) plt.title("Histogram for Prices") plt.show()

1 2 3 4

#Visualizing price using Histogram sn.distplot(a = df["Price"], kde = False) plt.title("Histogram for Prices") plt.show()

#Visualizing using a density plot sn.kdeplot(data = df["Price"], shade = True) plt.title("Density Plot for Prices") plt.show()

1 2 3 4 5

#Visualizing using a density plot sn.kdeplot(data = df["Price"], shade = True) plt.title("Density Plot for Prices") plt.show()

A Boxplot is usually very efficient in visualizing the relationship between a categorical and a numeric data type. The Type column is a categorical type and the Price column is a numeric type, we can implement a Boxplot to help determine how the Price of houses varies across the Type of houses available.

#Implement boxplot to see how price varies across housing types sn.boxplot(x = "Type", y = "Price", data = df, orient = "v") plt.title("Type vs Price")

1 2 3

#Implement boxplot to see how price varies across housing types sn.boxplot(x = "Type", y = "Price", data = df, orient = "v") plt.title("Type vs Price")

From the image above, we can easily determine the housing type having the highest pricing. The third method of implement Descriptive Statistics is the Measures of Dispersion and Frequency, some of the procedures can be implemented as shown below:

#Standard deviation df["Price"].std() #Assessing the range. df["Price"].max() df["Price"].min() #We can use frequency statistics to visualize the most common type of rooms rooms = df["Rooms"].value_counts() #Extract the Rooms variable and count

1 2 3 4 5 6 7 8

#Standard deviation df["Price"].std() #Assessing the range. df["Price"].max() df["Price"].min() #We can use frequency statistics to visualize the most common type of rooms rooms = df["Rooms"].value_counts() #Extract the Rooms variable and count

From the code chunk above, we have easily assessed the standard deviation, range using the min and max functions and also executed a frequency count which helps us assess the number of rooms most prominent across the numerous houses in Melbourne. We can also visualize the rooms variable created using a bar graph for ease of inference:

#Visualizing the most prominent rooms rooms.plot.bar() plt.title("Most common room types") plt.xlabel("Room Numbers") plt.ylabel("Count")

1 2 3 4 5

#Visualizing the most prominent rooms rooms.plot.bar() plt.title("Most common room types") plt.xlabel("Room Numbers") plt.ylabel("Count")

Another robust way to implement a frequency count on the dataset is by assessing the region most represented in the dataset, in this case, we will find out which region in Melbourne is most represented in the dataset:

sn.set(style='darkgrid') plt.figure(figsize=(10,5)) ax = sn.countplot(x='Regionname', data=df) plt.setp(ax.get_xticklabels(), rotation=20)

1 2 3 4

sn.set(style='darkgrid') plt.figure(figsize=(10,5)) ax = sn.countplot(x='Regionname', data=df) plt.setp(ax.get_xticklabels(), rotation=20)

The plot above, produced by using a frequency count and a barplot helps to visualize which region is largely represented. The last method mentioned for Descriptive statistics is the Measure of Position, we can easily implement on the dataset a function that helps know the quantile rankings in the dataset:

#Examining the quantile ranges import numpy as np df["Price"].quantile(np.linspace(.1, 1, 9, 0))

1 2 3

#Examining the quantile ranges import numpy as np df["Price"].quantile(np.linspace(.1, 1, 9, 0))

After implementing the code above, the result is displayed in the image. It shows us the quantile which divides the distribution of the data into ten equal parts.

Conclusion

This tutorial has covered the basic procedures existing for Descriptive Statistics. The main objective is to discuss the various ways of summarizing a dataset and also implementing these procedures using Python. This isn’t to say that this tutorial covers all there is to Descriptive Statistics, it, however, gives an introduction to Descriptive Statistics.

[Author]

Posts