kopia lustrzana https://github.com/animator/learn-python
Descriptive Statistics with Python and Pandas
rodzic
406004d9c9
commit
a478e70c4b
|
@ -0,0 +1,599 @@
|
|||
## Descriptive Statistics
|
||||
|
||||
In the realm of data science, understanding the characteristics of data is fundamental. Descriptive statistics provide the tools and techniques to succinctly summarize and present the key features of a dataset. It serves as the cornerstone for exploring, visualizing, and ultimately gaining insights from data.
|
||||
|
||||
Descriptive statistics encompasses a range of methods designed to describe the central tendency, dispersion, and shape of a dataset. Through measures such as mean, median, mode, standard deviation, and variance, descriptive statistics offer a comprehensive snapshot of the data's distribution and variability.
|
||||
|
||||
Data scientists utilize descriptive statistics to uncover patterns, identify outliers, and assess the overall structure of data before delving into more advanced analyses. By summarizing large and complex datasets into manageable and interpretable summaries, descriptive statistics facilitate informed decision-making and actionable insights.
|
||||
|
||||
|
||||
```python
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
```
|
||||
|
||||
|
||||
```python
|
||||
df = pd.read_csv("Age-Income-Dataset.csv")
|
||||
df
|
||||
```
|
||||
| | Age | Income |
|
||||
| --- | ----------- | ------ |
|
||||
| 0 | Young | 25000 |
|
||||
| 1 | Middle Age | 54000 |
|
||||
| 2 | Old | 60000 |
|
||||
| 3 | Young | 15000 |
|
||||
| 4 | Young | 45000 |
|
||||
| 5 | Young | 65000 |
|
||||
| 6 | Young | 70000 |
|
||||
| 7 | Young | 30000 |
|
||||
| 8 | Middle Age | 27000 |
|
||||
| 9 | Young | 23000 |
|
||||
| 10 | Young | 48000 |
|
||||
| 11 | Old | 52000 |
|
||||
| 12 | Young | 33000 |
|
||||
| 13 | Old | 80000 |
|
||||
| 14 | Old | 75000 |
|
||||
| 15 | Old | 35000 |
|
||||
| 16 | Middle Age | 29000 |
|
||||
| 17 | Middle Age | 57000 |
|
||||
| 18 | Old | 43000 |
|
||||
| 19 | Middle Age | 56000 |
|
||||
| 20 | Old | 63000 |
|
||||
| 21 | Old | 32000 |
|
||||
| 22 | Old | 45000 |
|
||||
| 23 | Old | 89000 |
|
||||
| 24 | Middle Age | 90000 |
|
||||
| 25 | Middle Age | 93000 |
|
||||
| 26 | Young | 80000 |
|
||||
| 27 | Young | 87000 |
|
||||
| 28 | Young | 38000 |
|
||||
| 29 | Young | 23000 |
|
||||
| 30 | Middle Age | 38900 |
|
||||
| 31 | Middle Age | 53200 |
|
||||
| 32 | Old | 43800 |
|
||||
| 33 | Middle Age | 25600 |
|
||||
| 34 | Middle Age | 65400 |
|
||||
| 35 | Old | 76800 |
|
||||
| 36 | Old | 89700 |
|
||||
| 37 | Old | 41800 |
|
||||
| 38 | Young | 31900 |
|
||||
| 39 | Old | 25600 |
|
||||
| 40 | Middle Age | 45700 |
|
||||
| 41 | Old | 35600 |
|
||||
| 42 | Young | 54300 |
|
||||
| 43 | Middle Age | 65400 |
|
||||
| 44 | Old | 67800 |
|
||||
| 45 | Old | 24500 |
|
||||
| 46 | Middle Age | 34900 |
|
||||
| 47 | Old | 45300 |
|
||||
| 48 | Young | 68400 |
|
||||
| 49 | Middle Age | 51700 |
|
||||
|
||||
|
||||
|
||||
```python
|
||||
df.describe()
|
||||
```
|
||||
|
||||
|
||||
|
||||
| | Income |
|
||||
|-------|-------------|
|
||||
| count | 50.000000 |
|
||||
| mean | 50966.000000 |
|
||||
| std | 21096.683268 |
|
||||
| min | 15000.000000 |
|
||||
| 25% | 33475.000000 |
|
||||
| 50% | 46850.000000 |
|
||||
| 75% | 65400.000000 |
|
||||
| max | 93000.000000 |
|
||||
|
||||
|
||||
### Mean
|
||||
|
||||
The mean, also known as the average, is a measure of central tendency in a dataset. It represents the typical value of a set of numbers. The formula to calculate the mean of a dataset is:
|
||||
|
||||
$$ \overline{x} = \frac{\sum\limits_{i=1}^{n} x_i}{n} $$
|
||||
|
||||
* $\overline{x}$ (pronounced "x bar") represents the mean value.
|
||||
* $x_i$ represents the individual value in the dataset (where i goes from 1 to n).
|
||||
* $\sum$ (sigma) represents the summation symbol, indicating we add up all the values from i=1 to n.
|
||||
* $n$ represents the total number of values in the dataset.
|
||||
|
||||
|
||||
|
||||
```python
|
||||
df['Income'].mean()
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
50966.0
|
||||
|
||||
|
||||
|
||||
|
||||
```python
|
||||
def mean_f(df):
|
||||
for col in df.columns:
|
||||
if df[col].dtype != 'O':
|
||||
temp = 0
|
||||
for i in df[col]:
|
||||
temp = temp +i
|
||||
print("Without pandas Library -> ")
|
||||
print("Average of {} is {}".format(col,(temp/len(df[col]))))
|
||||
print()
|
||||
print("With pandas Library -> ")
|
||||
print(df[col].mean())
|
||||
|
||||
mean_f(df)
|
||||
```
|
||||
|
||||
Without pandas Library ->
|
||||
Average of Income is 50966.0
|
||||
|
||||
With pandas Library ->
|
||||
50966.0
|
||||
|
||||
|
||||
### Median
|
||||
|
||||
|
||||
The median is another measure of central tendency in a dataset. Unlike the mean, which is the average value of all data points, the median represents the middle value when the dataset is ordered from smallest to largest. If the dataset has an odd number of observations, the median is the middle value. If the dataset has an even number of observations, the median is the average of the two middle values.
|
||||
|
||||
The median represents the "middle" value in a dataset. There are two cases to consider depending on whether the number of observations (n) is odd or even:
|
||||
|
||||
**Odd number of observations (n):**
|
||||
|
||||
In this case, the median (M) is the value located at the middle position when the data is ordered from least to greatest. We can calculate the position using the following formula:
|
||||
|
||||
$$ M = x_{n+1/2} $$
|
||||
|
||||
**Even number of observations (n):**
|
||||
|
||||
When we have an even number of observations, there isn't a single "middle" value. Instead, the median is the average of the two middle values after ordering the data. Here's the formula to find the median:
|
||||
|
||||
$$ M = \frac{x_{n/2} + x_{(n/2)+1}}{2} $$
|
||||
|
||||
**Explanation:**
|
||||
|
||||
* M represents the median value.
|
||||
* n represents the total number of observations in the dataset.
|
||||
* $x$ represents the individual value.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
```python
|
||||
df['Income'].median()
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
46850.0
|
||||
|
||||
|
||||
|
||||
|
||||
```python
|
||||
def median_f(df):
|
||||
for col in df.columns:
|
||||
if df[col].dtype != 'O':
|
||||
sorted_data = sorted(df[col])
|
||||
n = len(df[col])
|
||||
if n%2 == 0:
|
||||
x1 =sorted_data[int((n/2))]
|
||||
x2 =sorted_data[int((n/2))+1]
|
||||
median=(x1+x2)/2
|
||||
else:
|
||||
median = sorted_data[(n+1)/2]
|
||||
print("Median without library ->")
|
||||
print("Median of {} is {} ".format(col,median))
|
||||
print("Median with library ->")
|
||||
print(df[col].median())
|
||||
median_f(df)
|
||||
```
|
||||
|
||||
Median without library ->
|
||||
Median of Income is 49850.0 \
|
||||
Median with library ->
|
||||
46850.0
|
||||
|
||||
|
||||
### Mode
|
||||
|
||||
|
||||
The mode is a measure of central tendency that represents the value or values that occur most frequently in a dataset. Unlike the mean and median, which focus on the average or middle value, the mode identifies the most common value(s) in the dataset.
|
||||
|
||||
|
||||
```python
|
||||
def mode_f(df):
|
||||
for col in df.columns:
|
||||
if df[col].dtype == 'O':
|
||||
print("Column:", col)
|
||||
arr = df[col].sort_values()
|
||||
|
||||
prevcnt = 0
|
||||
cnt = 0
|
||||
ans = arr[0]
|
||||
temp = arr[0]
|
||||
|
||||
for i in arr:
|
||||
if(temp == i) :
|
||||
cnt += 1
|
||||
else:
|
||||
prevcnt = cnt
|
||||
cnt = 1
|
||||
temp = i
|
||||
if(cnt > prevcnt):
|
||||
ans = i
|
||||
|
||||
print("Without pandas Library -> ")
|
||||
print("Mode of {} is {}".format(col,ans))
|
||||
print()
|
||||
print("With pandas Library -> ")
|
||||
print(df[col].mode())
|
||||
mode_f(df)
|
||||
```
|
||||
|
||||
Column: Age
|
||||
Without pandas Library -> \
|
||||
Mode of Age is Old
|
||||
|
||||
With pandas Library -> \
|
||||
0 Old \
|
||||
Name: Age, dtype: object
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
### Standard Deviation
|
||||
|
||||
Standard deviation is a measure of the dispersion or spread of a dataset. It quantifies the amount of variation or dispersion of a set of values from the mean. In other words, it indicates how much individual values in a dataset deviate from the mean.
|
||||
|
||||
$$s = \sqrt{\frac{\sum(x_i-\overline{x})^{2}}{n-1}}$$
|
||||
|
||||
* $s$ represents the standard deviation.
|
||||
* $\sum$ (sigma) represents the summation symbol, indicating we add up the values for all data points.
|
||||
* $x_i$ represents the individual value in the dataset.
|
||||
* $\overline{x}$ (x bar) represents the mean value of the dataset.
|
||||
* $n$ represents the total number of values in the dataset.
|
||||
|
||||
|
||||
```python
|
||||
df['Income'].std()
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
21096.683267707253
|
||||
|
||||
|
||||
|
||||
|
||||
```python
|
||||
import math
|
||||
def std_f(df):
|
||||
for col in df.columns:
|
||||
if len(df[col]) == 0:
|
||||
print("Column is empty")
|
||||
if df[col].dtype != 'O':
|
||||
sum = 0
|
||||
mean = df[col].mean()
|
||||
for i in df[col]:
|
||||
sum = sum + (i - mean)**2
|
||||
|
||||
std = math.sqrt(sum/len(df[col]))
|
||||
print("Without pandas library ->")
|
||||
print("Std : " , std)
|
||||
print("With pandas library: ->")
|
||||
print("Std : {}".format(np.std(df[col]))) ##ddof = 1
|
||||
|
||||
std_f(df)
|
||||
```
|
||||
|
||||
Without pandas library ->
|
||||
Std : 20884.6509187968 \
|
||||
With pandas library: ->
|
||||
Std : 20884.6509187968
|
||||
|
||||
|
||||
|
||||
```python
|
||||
df['Income'].count()
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
50
|
||||
|
||||
|
||||
|
||||
### Minimum
|
||||
|
||||
|
||||
```python
|
||||
df['Income'].min()
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
15000
|
||||
|
||||
|
||||
|
||||
|
||||
```python
|
||||
def min_f(df):
|
||||
for col in df.columns:
|
||||
if df[col].dtype != "O":
|
||||
sorted_data = sorted(df[col])
|
||||
min = sorted_data[0]
|
||||
print("Without pandas Library->",min)
|
||||
print("With pandas Library->",df[col].min())
|
||||
|
||||
min_f(df)
|
||||
```
|
||||
|
||||
Without pandas Library-> 15000 \
|
||||
With pandas Library-> 15000
|
||||
|
||||
|
||||
### Maximum
|
||||
|
||||
|
||||
```python
|
||||
df['Income'].max()
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
93000
|
||||
|
||||
|
||||
|
||||
|
||||
```python
|
||||
def max_f(df):
|
||||
for col in df.columns:
|
||||
if df[col].dtype != "O":
|
||||
sorted_data = sorted(df[col])
|
||||
max = sorted_data[len(df[col])-1]
|
||||
print("Without pandas Library->",max)
|
||||
print("With pandas Library->",df[col].max())
|
||||
|
||||
max_f(df)
|
||||
```
|
||||
|
||||
Without pandas Library-> 93000
|
||||
With pandas Library-> 93000
|
||||
|
||||
|
||||
### Percentile
|
||||
|
||||
|
||||
```python
|
||||
df['Income'].quantile(0.25)
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
33475.0
|
||||
|
||||
|
||||
|
||||
|
||||
```python
|
||||
df['Income'].quantile(0.75)
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
65400.0
|
||||
|
||||
|
||||
|
||||
|
||||
```python
|
||||
def percentile_f(df,percentile):
|
||||
for col in df.columns:
|
||||
if df[col].dtype != 'O':
|
||||
sorted_data = sorted(df[col])
|
||||
index = int(percentile*len(df[col]))
|
||||
percentile_result = sorted_data[index]
|
||||
print(f"{percentile} Percentile is : ",percentile_result)
|
||||
|
||||
percentile_f(df,0.25)
|
||||
```
|
||||
|
||||
0.25 Percentile is : 33000
|
||||
|
||||
|
||||
We have used the method of nearest rank to calculate percentile manually.
|
||||
|
||||
Pandas uses linear interpolation of data to calculate percentiles.
|
||||
|
||||
## Correlation and Covariance
|
||||
|
||||
|
||||
```python
|
||||
df = pd.read_csv('Iris.csv')
|
||||
df.head(5)
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
| | Id | SepalLengthCm | SepalWidthCm | PetalLengthCm | PetalWidthCm | Species |
|
||||
|---|----|---------------|--------------|---------------|--------------|-------------|
|
||||
| 0 | 1 | 5.1 | 3.5 | 1.4 | 0.2 | Iris-setosa |
|
||||
| 1 | 2 | 4.9 | 3.0 | 1.4 | 0.2 | Iris-setosa |
|
||||
| 2 | 3 | 4.7 | 3.2 | 1.3 | 0.2 | Iris-setosa |
|
||||
| 3 | 4 | 4.6 | 3.1 | 1.5 | 0.2 | Iris-setosa |
|
||||
| 4 | 5 | 5.0 | 3.6 | 1.4 | 0.2 | Iris-setosa |
|
||||
|
||||
|
||||
|
||||
|
||||
```python
|
||||
df.drop(['Id','Species'],axis=1,inplace= True)
|
||||
```
|
||||
|
||||
### Covarience
|
||||
|
||||
Covariance measures the degree to which two variables change together. If the covariance between two variables is positive, it means that they tend to increase or decrease together. If the covariance is negative, it means that as one variable increases, the other tends to decrease. However, covariance does not provide a standardized measure, making it difficult to interpret the strength of the relationship between variables, especially if the variables are measured in different units.
|
||||
|
||||
$$ COV(X,Y) = \frac{\sum\limits_{i=1}^{n} (X_i - \overline{X}) (Y_i - \overline{Y})}{n - 1}$$
|
||||
|
||||
**Explanation:**
|
||||
|
||||
* $COV(X, Y)$ represents the covariance between variables X and Y.
|
||||
* $X_i$ and $Y_i$ represent the individual values for variables X and Y in the i-th observation.
|
||||
* $\overline{X}$ and $\overline{Y}$ represent the mean values for variables X and Y, respectively.
|
||||
* $n$ represents the total number of observations in the dataset.
|
||||
|
||||
|
||||
|
||||
```python
|
||||
df.cov()
|
||||
```
|
||||
|
||||
|
||||
| | SepalLengthCm | SepalWidthCm | PetalLengthCm | PetalWidthCm |
|
||||
|-------------------|-------------- |---------------|-----------------|--------------|
|
||||
| **SepalLengthCm** | 0.685694 | -0.039268 | 1.273682 | 0.516904 |
|
||||
| **SepalWidthCm** | -0.039268 | 0.188004 | -0.321713 | -0.117981 |
|
||||
| **PetalLengthCm** | 1.273682 | -0.321713 | 3.113179 | 1.296387 |
|
||||
| **PetalWidthCm** | 0.516904 | -0.117981 | 1.296387 | 0.582414 |
|
||||
|
||||
|
||||
|
||||
```python
|
||||
def cov_f(df):
|
||||
for x in df.columns:
|
||||
for y in df.columns:
|
||||
mean_x = df[x].mean()
|
||||
mean_y = df[y].mean()
|
||||
|
||||
sum = 0
|
||||
n = len(df[x])
|
||||
|
||||
for val in range(n):
|
||||
sum += (df[x].iloc[val] - mean_x)*(df[y].iloc[val] - mean_y)
|
||||
print("Covariance of {} and {} is : {}".format(x,y, sum/(n-1)))
|
||||
print()
|
||||
cov_f(df)
|
||||
```
|
||||
|
||||
Covariance of SepalLengthCm and SepalLengthCm is : 0.6856935123042504 \
|
||||
Covariance of SepalLengthCm and SepalWidthCm is : -0.03926845637583892 \
|
||||
Covariance of SepalLengthCm and PetalLengthCm is : 1.2736823266219246 \
|
||||
Covariance of SepalLengthCm and PetalWidthCm is : 0.5169038031319911
|
||||
|
||||
Covariance of SepalWidthCm and SepalLengthCm is : -0.03926845637583892 \
|
||||
Covariance of SepalWidthCm and SepalWidthCm is : 0.1880040268456377 \
|
||||
Covariance of SepalWidthCm and PetalLengthCm is : -0.32171275167785235 \
|
||||
Covariance of SepalWidthCm and PetalWidthCm is : -0.11798120805369115
|
||||
|
||||
Covariance of PetalLengthCm and SepalLengthCm is : 1.2736823266219246 \
|
||||
Covariance of PetalLengthCm and SepalWidthCm is : -0.32171275167785235 \
|
||||
Covariance of PetalLengthCm and PetalLengthCm is : 3.113179418344519 \
|
||||
Covariance of PetalLengthCm and PetalWidthCm is : 1.2963874720357946
|
||||
|
||||
Covariance of PetalWidthCm and SepalLengthCm is : 0.5169038031319911 \
|
||||
Covariance of PetalWidthCm and SepalWidthCm is : -0.11798120805369115 \
|
||||
Covariance of PetalWidthCm and PetalLengthCm is : 1.2963874720357946 \
|
||||
Covariance of PetalWidthCm and PetalWidthCm is : 0.5824143176733781
|
||||
|
||||
|
||||
|
||||
### Correlation
|
||||
|
||||
Correlation, on the other hand, standardizes the measure of relationship between two variables, making it easier to interpret. It measures both the strength and direction of the linear relationship between two variables. Correlation values range between -1 and 1, where:
|
||||
|
||||
$$r = \frac{n(\sum xy) - (\sum x)(\sum y)}{\sqrt{n(\sum x^2) - (\sum x)^2} \cdot \sqrt{n(\sum y^2) - (\sum y)^2}}$$
|
||||
|
||||
* r represents the correlation coefficient.
|
||||
* n is the number of data points.
|
||||
|
||||
|
||||
|
||||
|
||||
```python
|
||||
df.corr()
|
||||
```
|
||||
|
||||
| | SepalLengthCm | SepalWidthCm | PetalLengthCm | PetalWidthCm |
|
||||
|-------------------|---------------|--------------|---------------|--------------|
|
||||
| **SepalLengthCm** | 1.000000 | -0.109369 | 0.871754 | 0.817954 |
|
||||
| **SepalWidthCm** | -0.109369 | 1.000000 | -0.420516 | -0.356544 |
|
||||
| **PetalLengthCm** | 0.871754 | -0.420516 | 1.000000 | 0.962757 |
|
||||
| **PetalWidthCm** | 0.817954 | -0.356544 | 0.962757 | 1.000000 |
|
||||
|
||||
|
||||
```python
|
||||
import math
|
||||
def corr_f(df):
|
||||
for i in df.columns:
|
||||
for j in df.columns:
|
||||
n = len(df[i])
|
||||
|
||||
sumX = 0
|
||||
for x in df[i]:
|
||||
sumX += x
|
||||
sumY = 0
|
||||
for y in df[j]:
|
||||
sumY += y
|
||||
|
||||
sumXY = 0
|
||||
for xy in range(n):
|
||||
sumXY += (df[i].iloc[xy] * df[j].iloc[xy])
|
||||
|
||||
sumX2 = 0
|
||||
for x in df[i]:
|
||||
sumX2 += (x**2)
|
||||
sumY2 = 0
|
||||
for y in df[j]:
|
||||
sumY2 += (y**2)
|
||||
|
||||
NR = (n * sumXY) - (sumX*sumY)
|
||||
DR = math.sqrt( ( (n * sumX2) - (sumX**2))*( (n * sumY2) - (sumY ** 2) ) )
|
||||
|
||||
print("Correlation of {} and {} :{}".format(i,j,NR/DR))
|
||||
print()
|
||||
|
||||
corr_f(df)
|
||||
```
|
||||
|
||||
Correlation of SepalLengthCm and SepalLengthCm :1.0 \
|
||||
Correlation of SepalLengthCm and SepalWidthCm :-0.10936924995067286 \
|
||||
Correlation of SepalLengthCm and PetalLengthCm :0.8717541573048861 \
|
||||
Correlation of SepalLengthCm and PetalWidthCm :0.8179536333691775
|
||||
|
||||
Correlation of SepalWidthCm and SepalLengthCm :-0.10936924995067286 \
|
||||
Correlation of SepalWidthCm and SepalWidthCm :1.0 \
|
||||
Correlation of SepalWidthCm and PetalLengthCm :-0.42051609640118826 \
|
||||
Correlation of SepalWidthCm and PetalWidthCm :-0.3565440896138223
|
||||
|
||||
Correlation of PetalLengthCm and SepalLengthCm :0.8717541573048861 \
|
||||
Correlation of PetalLengthCm and SepalWidthCm :-0.42051609640118826 \
|
||||
Correlation of PetalLengthCm and PetalLengthCm :1.0 \
|
||||
Correlation of PetalLengthCm and PetalWidthCm :0.9627570970509656
|
||||
|
||||
Correlation of PetalWidthCm and SepalLengthCm :0.8179536333691775 \
|
||||
Correlation of PetalWidthCm and SepalWidthCm :-0.3565440896138223 \
|
||||
Correlation of PetalWidthCm and PetalLengthCm :0.9627570970509656 \
|
||||
Correlation of PetalWidthCm and PetalWidthCm :1.0
|
||||
|
||||
|
|
@ -1,3 +1,4 @@
|
|||
# List of sections
|
||||
|
||||
- [Pandas Series Vs NumPy ndarray](pandas_series_vs_numpy_ndarray.md)
|
||||
- [Pandas Descriptive Statistics](Descriptive_Statistics.md)
|
||||
|
|
Ładowanie…
Reference in New Issue