mirror
/
learn-python
kopia lustrzana https://github.com/animator/learn-python
1
0
Forkuj 0
Kod Zgłoszenia Packages Projekty Wydania Wiki Aktywność

Merge branch 'main' into OOPs

pull/395/head
Ankit Mahato 2024-05-23 04:57:42 +05:30 zatwierdzone przez GitHub
rodzic 07d7643c6a 30e9237bc9
commit 139f84b7d8
Nie znaleziono w bazie danych klucza dla tego podpisu
ID klucza GPG: B5690EEEBB952194
18 zmienionych plików z 1878 dodań i 3 usunięć
Pokaż statystyki Ściągnij plik aktualizacji Ściągnij plik porównania Expand all files Collapse all files

117
contrib/advanced-python/dates_and_times.md 100644
Wyświetl plik

@ -0,0 +1,117 @@
## Working with Dates and Times in Python
Handling dates and times is an essential aspect of many programming tasks.
Python provides robust modules to work with dates and times, making it easier to perform operations like formatting, parsing, and arithmetic.
This guide provides an overview of these modules and their key functionalities.
## 1. 'datetime' Module
The datetime module supplies classes for manipulating dates and times. The main classes in the datetime module are:
* date: Represents a date (year, month, day).
* time: Represents a time (hour, minute, second, microsecond).
* datetime: Combines date and time information.
* timedelta: Represents the difference between two dates or times.
* tzinfo: Provides time zone information objects.
**Key Concepts:**
* Naive vs. Aware: Naive datetime objects do not contain time zone information, while aware datetime objects do.
* Immutability: date and time objects are immutable; once created, they cannot be changed.
Example:
```bash
import datetime
# Get the current date and time
now = datetime.datetime.now()
print("Current date and time:", now)
```
## 2. Formatting Dates and Times
Formatting involves converting datetime objects into human-readable strings. This is achieved using the strftime method, which stands for "string format time."
You can specify various format codes to dictate how the output string should be structured.
**Common Format Codes:**
* %Y: Year with century (e.g., 2024)
* %m: Month as a zero-padded decimal number (e.g., 01)
* %d: Day of the month as a zero-padded decimal number (e.g., 15)
* %H: Hour (24-hour clock) as a zero-padded decimal number (e.g., 13)
* %M: Minute as a zero-padded decimal number (e.g., 45)
* %S: Second as a zero-padded decimal number (e.g., 30)
Example:
```bash
import datetime
now = datetime.datetime.now()
formatted_now = now.strftime("%Y-%m-%d %H:%M:%S")
print("Formatted current date and time:", formatted_now)
```
## 3. Parsing Dates and Times
Parsing is the process of converting strings representing dates and times into datetime objects. The strptime method, which stands for "string parse time,"
allows you to specify the format of the input string.
Example:
```bash
import datetime
date_string = "2024-05-15 13:45:30"
date_object = datetime.datetime.strptime(date_string, "%Y-%m-%d %H:%M:%S")
print("Parsed date and time:", date_object)
```
## 4. Working with Time Differences
The timedelta class is used to represent the difference between two datetime objects. This is useful for calculations involving durations, such as finding the
number of days between two dates or adding a certain period to a date.
Example:
```bash
import datetime
date1 = datetime.datetime(2024, 5, 15, 12, 0, 0)
date2 = datetime.datetime(2024, 5, 20, 14, 30, 0)
difference = date2 - date1
print("Difference:", difference)
print("Days:", difference.days)
print("Total seconds:", difference.total_seconds())
```
## 5. Time Zones
Time zone handling in Python is facilitated by the pytz library. It allows you to convert naive datetime objects into timezone-aware objects and perform
operations across different time zones.
**Key Concepts:**
* Timezone-aware: A datetime object that includes timezone information.
* Localization: The process of associating a naive datetime with a time zone.
Example:
```bash
import datetime
import pytz
# Define a timezone
tz = pytz.timezone('Asia/Kolkata')
# Get the current time in a specific timezone
now = datetime.datetime.now(tz)
print("Current time in Asia/Kolkata:", now)
```
## 6. Date Arithmetic
Date arithmetic involves performing operations like addition or subtraction on date or datetime objects using timedelta. This is useful for calculating future
or past dates based on a given date.
Example:
```bash
import datetime
today = datetime.date.today()
future_date = today + datetime.timedelta(days=10)
print("Date after 10 days:", future_date)
```
## Summary
Python’s datetime module and the pytz library provide comprehensive tools for working with dates, times, and time zones. They enable you to perform a wide range
of operations, from basic date manipulations to complex time zone conversions.

3
contrib/advanced-python/index.md
Wyświetl plik

@ -1,4 +1,7 @@
# List of sections
- [Decorators/\*args/**kwargs](decorator-kwargs-args.md)
- [Working with Dates & Times in Python](dates_and_times.md)
- [Regular Expressions in Python](regular_expressions.md)
- [JSON module](json-module.md)
- [OOPs](OOPs.md)

289
contrib/advanced-python/json-module.md 100644
Wyświetl plik

@ -0,0 +1,289 @@
# JSON Module
## What is JSON?
- [JSON]("https://www.json.org/json-en.html") (JavaScript Object Notation) is a format for structuring data.
- JSON is a lightweight, text-based data interchange format that is completely language-independent.
- Similar to XML, JSON is a format for structuring data commonly used by web applications to communicate with each other.
## Why JSON?
- Whenever we declare a variable and assign a value to it, the variable itself doesn't hold the value. Instead, the variable holds an address in memory where the value is stored. For example:
```python
age = 21
```
- When we use `age`, it gets replaced with `21`. However, *age doesn't contain 21, it contains the address of the memory location where 21 is stored*.
- While this works locally, transferring this data, such as through an API, poses a challenge. Sending your computer’s entire memory with the addresses is impractical and insecure. This is where JSON comes to the rescue.
### Example JSON
- JSON supports most widely used data types including String
, Number, Boolean, Null, Array and Object.
- Here is an example of JSON file
```json
{
"name": "John Doe",
"age": 21,
"isStudent": true,
"address": null,
"courses": ["Math", "Science", "History"],
"grades": {
"Math": 95,
"Science": 89,
"History": 76
}
}
```
# Python JSON
Python too supports JSON with a built-in package called `json`. This package provides all the necessary tools for working with JSON Objects including `parsing, serializing, deserializing, and many more`.
## 1. Python parse JSON string.
- To parse JSON string Python firstly we import the JSON module.
- JSON string is converted to a Python object using `json.loads()` method of JSON module in Python.
- Example Code:
```python
# Python program to convert JSON to Python
import json
# JSON string
students ='{"id":"01", "name": "Yatharth", "department":"Computer Science Engineering"}'
# Convert string to Python dict
students_dict = json.loads(students)
print(students_dict)
print(students_dict['name'])
```
- Ouput:
```json
{"id": "01", "name": "Yatharth", "department": "Computer Science Engineering"}
```
## 2. Python load JSON file.
- JSON data can also be directly fetch from a json file
- Example:
```python
import json
# Opening JSON file
f = open('input.json',)
# Returns JSON object as a dictionary
data = json.load(f)
# Iterating through the json file
for i in data['students']:
print(i)
# Closing file
f.close()
```
- JSON file
```json
{
"students":{
{
"id": "01",
"name": "Yatharth",
"department": "Computer Science Engineering"
},
{
"id": "02",
"name": "Raj",
"department": "Mechanical Engineering"
}
}
}
```
- Ouput
```json
{"id": "01", "name": "Yatharth", "department": "Computer Science Engineering"}
{"id": "02", "name": "Raj", "department": "Mechanical Engineering"}
```
- `json.load()`: Reads JSON data from a file object and deserializes it into a Python object.
- `json.loads()`: Deserializes JSON data from a string into a Python object.
## Addtiotnal Context
Relation between python data types and json data types is given in table below.
| Python Object | JSON Object |
|-----------------|-------------|
| Dict | object |
| list, tuple | array |
| str | string |
| int, long, float | numbers |
| True | true |
| False | false |
| None | null |
## 3. Python Dictionary to JSON String
- Parsing python dictionary to json string using `json.dumps()`.
- Example Code:
```python
import json
# Data to be written
dictionary ={
"id": "03",
"name": "Suraj",
"department": "Civil Engineering"
}
# Serializing json
json_object = json.dumps(dictionary, indent = 4)
print(json_object)
```
- Output:
``` json
{
"department": "Civil Engineering",
"id": "02",
"name": "Suraj"
}
```
## 4. Python Dictionary to JSON file.
- - Parsing python dictionary to json string using `json.dump()`.
- Example Code:
``` python
import json
# Data to be written
dictionary ={
"name" : "Satyendra",
"rollno" : 51,
"cgpa" : 8.8,
"phonenumber" : "123456789"
}
with open("sample.json", "w") as outfile:
json.dump(dictionary, outfile)
```
- Ouput: `sample.json`
``` json
{
"name" : "Satyendra",
"rollno" : 51,
"cgpa" : 8.8,
"phonenumber" : "123456789"
}
```
## 5. Append Python Dictionary to JSON String.
- Append to an already existing string using `json.update()`.
- Example :
```python
import json
# JSON data:
x = {
"id": "03",
"name": "Suraj"
}
# python object to be appended
y = { "department": "Civil Engineering"}
# parsing JSON string:
z = json.loads(x)
# appending the data
z.update(y)
# the result is a JSON string:
print(json.dumps(z))
```
- Ouput:
```json
{"id": "03", "name": "Suraj", "department": "Civil Engineering"}
```
## 6. Append Python Dictionary to JSON File.
- There is no direct function to append in file. So, we will load file in a dictionary, update dictionary then update content and convert back to json file format.
- `data.json`
``` json
{
"students":{
{
"id": "01",
"name": "Yatharth",
"department": "Computer Science Engineering"
},
{
"id": "02",
"name": "Raj",
"department": "Mechanical Engineering"
}
}
}
```
- Example Code:
``` python
import json
# function to add to JSON
def write_json(new_data, filename='data.json'):
with open(filename,'r+') as file:
# First we load existing data into a dict.
file_data = json.load(file)
# Join new_data with file_data inside students
file_data["students"].append(new_data)
# Sets file's current position at offset.
file.seek(0)
# convert back to json.
json.dump(file_data, file, indent = 4)
# python object to be appended
y = {
"id": "03",
"name": "Suraj",
"department": "Civil Engineering"
}
write_json(y)
```
- Output:
```json
{
"students":{
{
"id": "01",
"name": "Yatharth",
"department": "Computer Science Engineering"
},
{
"id": "02",
"name": "Raj",
"department": "Mechanical Engineering"
},
{
"id": "03",
"name": "Suraj",
"department": "Civil Engineering"
}
}
}
```
The Python json module simplifies the handling of JSON data, offering a bridge between Python data structures and JSON representations, vital for data exchange and storage in modern applications.

96
contrib/advanced-python/regular_expressions.md 100644
Wyświetl plik

@ -0,0 +1,96 @@
## Regular Expressions in Python
Regular expressions (regex) are a powerful tool for pattern matching and text manipulation.
Python's re module provides comprehensive support for regular expressions, enabling efficient text processing and validation.
## 1. Introduction to Regular Expressions
A regular expression is a sequence of characters defining a search pattern. Common use cases include validating input, searching within text, and extracting
specific patterns.
## 2. Basic Syntax
Literal Characters: Match exact characters (e.g., abc matches "abc").
Metacharacters: Special characters like ., *, ?, +, ^, $, [ ], and | used to build patterns.
**Common Metacharacters:**
* .: Any character except newline.
* ^: Start of the string.
* $: End of the string.
* *: 0 or more repetitions.
* +: 1 or more repetitions.
* ?: 0 or 1 repetition.
* []: Any one character inside brackets (e.g., [a-z]).
* |: Either the pattern before or after.
## 3. Using the re Module
**Key functions in the re module:**
* re.match(): Checks for a match at the beginning of the string.
* re.search(): Searches for a match anywhere in the string.
* re.findall(): Returns a list of all matches.
* re.sub(): Replaces matches with a specified string.
Examples:
```bash
import re
# Match at the beginning
print(re.match(r'\d+', '123abc').group()) # Output: 123
# Search anywhere
print(re.search(r'\d+', 'abc123').group()) # Output: 123
# Find all matches
print(re.findall(r'\d+', 'abc123def456')) # Output: ['123', '456']
# Substitute matches
print(re.sub(r'\d+', '#', 'abc123def456')) # Output: abc#def#
```
## 4. Compiling Regular Expressions
Compiling regular expressions improves performance for repeated use.
Example:
```bash
import re
pattern = re.compile(r'\d+')
print(pattern.match('123abc').group()) # Output: 123
print(pattern.search('abc123').group()) # Output: 123
print(pattern.findall('abc123def456')) # Output: ['123', '456']
```
## 5. Groups and Capturing
Parentheses () group and capture parts of the match.
Example:
```bash
import re
match = re.match(r'(\d{3})-(\d{2})-(\d{4})', '123-45-6789')
if match:
print(match.group()) # Output: 123-45-6789
print(match.group(1)) # Output: 123
print(match.group(2)) # Output: 45
print(match.group(3)) # Output: 6789
```
## 6. Special Sequences
Special sequences are shortcuts for common patterns:
* \d: Any digit.
* \D: Any non-digit.
* \w: Any alphanumeric character.
* \W: Any non-alphanumeric character.
* \s: Any whitespace character.
* \S: Any non-whitespace character.
Example:
```bash
import re
print(re.search(r'\w+@\w+\.\w+', 'Contact: support@example.com').group()) # Output: support@example.com
```
## Summary
Regular expressions are a versatile tool for text processing in Python. The re module offers powerful functions and metacharacters for pattern matching,
searching, and manipulation, making it an essential skill for handling complex text processing tasks.

153
contrib/machine-learning/ArtificialNeuralNetwork.md 100644
Wyświetl plik

@ -0,0 +1,153 @@
# Understanding the Neural Network
## Table of Contents
<details>
<summary>Click to expand</summary>
- [Introduciton](#introduction)
- [Neuron to Perceptron](#neuron-to-perceptron)
- [Key concepts](#key-concepts)
- [Layers](#layers)
- [Weights and Biases](#weights-and-biases)
- [Activation Function](#activation-functions)
- [Forward and Backward Pass](#forward-and-backward-propagation)
- [Implementation](#building-from-scratch)
</details>
## Introduction
This guide will walk you through a fundamental neural network implementation in Python. We'll build a `Neural Network` from scratch, allowing you to grasp the core concepts of how neural networks learn and make predictions.
### Let's start by Understanding the Basic Architecture of Neural Nets
## Neuron to Perceptron
| `Neuron` cells forming the humand nervous system | `Perceptron` inspired from human brain |
| :----------------------------------------------- | -------------------------------------: |
| Neurons are nerve cells that send messages all over your body to allow you to do everything from breathing to talking, eating, walking, and thinking. | The perceptron is a mathematical model of a biological neuron. Performing heavy computations to think like humans. |
| Neuron collects signals from dendrites. | The first layer is knownn as Input Layer, acting like dendritres to receive the input signal. |
| Synapses are the connections between neurons where signals are transmitted. | Weights represent synapses. |
The axon terminal releases neurotransmitters to transmit the signal to other neurons. | The output is the final result – between 1 & 0, representing classification or prediction. |
---
> Human brain has a Network of Neurons, about 86 billion neurons and more than a 100 trillion synapses connections!
## **Key Concepts**
Artificial neurons are the fundamental processing units in an ANN. They receive inputs, multiply them by weights (representing the strength of connections), sum those weighted inputs, and then apply an activation function to produce an output.
### Layers
Neurons in ANNs are organized into layers:
* **Input Layer:** Receives the raw data.
* **(n) Hidden Layers:** (Optional) Intermediate layers where complex transformations occur. They learn to detect patterns and features in the data.
* **Output Layer:** Produces the final result (prediction or classification).
### Weights and Biases
- For each input $(x_i)$, a weight $(w_i)$ is associated with it. Weights, multiplied with input units $(w_i \cdot x_i)$, determine the influence of one neuron's output on another.
- A bias $(b_i)$ is added to help influence the end product, giving the equation as $(w_i \cdot x_i + b_i)$.
- During training, the network adjusts these weights and biases to minimize errors and improve its predictions.
### Activation Functions
- An activation function is applied to the result to introduce non-linearity in the model, allowing ANNs to learn more complex relationships from the data.
- The resulting equation: $y = f(g(x))$, determines whether the neuron will "fire" or not, i.e., if its output will be used as input for the next neuron.
- Common activation functions include the sigmoid function, tanh (hyperbolic tangent), and ReLU (Rectified Linear Unit).
### Forward and Backward Propagation
- **Flow of Information:** All the above steps are part of Forward Propagation. It gives the output equation as $y = f\left(\sum_{i=1}^n w_i x_i + b_i\right)$
- **Error Correction:** Backpropagation is the algorithm used to train ANNs by calculating the gradient of error at the output layer and then propagating this error backward through the network. This allows the network to adjust its weights and biases in the direction that reduces the error.
- The chain rule of calculus is the foundational concept to compute the gradient of the error:
$
\delta_{ij}(E) = \frac{\partial E}{\partial w_{ij}} = \frac{\partial E}{\partial \hat{y}_j} \cdot \frac{\partial \hat{y}_j}{\partial \theta_j} \cdot \frac{\partial \theta_j}{\partial w_{ij}}
$
where $E$ is the error, $\hat{y}_j$ is the predicted output, $\theta_j$ is the input to the activation function of the $j^{th}$ neuron, and $w_{ij}$ is the weight from neuron $i$ to neuron $j$.
## Building From Scratch
```python
# Import required libraries
import numpy as np
import matplotlib.pyplot as plt
class SimpleNeuralNetwork:
def __init__(self, input_size, hidden_size, output_size):
self.input_size = input_size
self.hidden_size = hidden_size
self.output_size = output_size
# Initialize weights and biases
self.weights_input_hidden = np.random.randn(input_size, hidden_size)
self.bias_hidden = np.random.randn(hidden_size)
self.weights_hidden_output = np.random.randn(hidden_size, output_size)
self.bias_output = np.random.randn(output_size)
def sigmoid(self, x):
return 1 / (1 + np.exp(-x))
def sigmoid_derivative(self, x):
return x * (1 - x)
def forward(self, X):
self.hidden_layer_input = np.dot(X, self.weights_input_hidden) + self.bias_hidden
self.hidden_layer_output = self.sigmoid(self.hidden_layer_input)
self.output_layer_input = np.dot(self.hidden_layer_output, self.weights_hidden_output) + self.bias_output
self.output = self.sigmoid(self.output_layer_input)
return self.output
def backward(self, X, y, learning_rate):
output_error = y - self.output
output_delta = output_error * self.sigmoid_derivative(self.output)
hidden_error = output_delta.dot(self.weights_hidden_output.T)
hidden_delta = hidden_error * self.sigmoid_derivative(self.hidden_layer_output)
self.weights_hidden_output += self.hidden_layer_output.T.dot(output_delta) * learning_rate
self.bias_output += np.sum(output_delta, axis=0) * learning_rate
self.weights_input_hidden += X.T.dot(hidden_delta) * learning_rate
self.bias_hidden += np.sum(hidden_delta, axis=0) * learning_rate
def train(self, X, y, epochs, learning_rate):
self.losses = []
for epoch in range(epochs):
self.forward(X)
self.backward(X, y, learning_rate)
loss = np.mean(np.square(y - self.output))
self.losses.append(loss)
if epoch % 1000 == 0:
print(f"Epoch {epoch}, Loss: {loss}")
def plot_loss(self):
plt.plot(self.losses)
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.title('Training Loss Over Epochs')
plt.show()
```
### Creating the Input & Output Array
Let's create a dummy input and outpu dataset. Here, the first two columns will be useful, while the rest might be noise.
```python
X = np.array([[0,0], [0,1], [1,0], [1,1]])
y = np.array([[0], [1], [1], [1]])
```
### Defining the Neural Network
With our input and output data ready, we'll define a simple neural network with one hidden layer containing three neurons.
```python
# neural network architecture
input_size = 2
hidden_layers = 1
hidden_neurons = [2]
output_size = 1
```
### Visualizing the Training Loss
To understand how well our model is learning, let's visualize the training loss over epochs.
```python
model = NeuralNetwork(input_size, hidden_layers, hidden_neurons, output_size)
model.train(X, y, 100)
```

171
contrib/machine-learning/Regression.md 100644
Wyświetl plik

@ -0,0 +1,171 @@
# Regression
* Regression is a supervised machine learning technique which is used to predict continuous values.
> Now, Supervised learning is a category of machine learning that uses labeled datasets to train algorithms to predict outcomes and recognize patterns.
* Regression is a statistical method used to model the relationship between a dependent variable (often denoted as 'y') and one or more independent variables (often denoted as 'x'). The goal of regression analysis is to understand how the dependent variable changes as the independent variables change.
# Types Of Regression
1. Linear Regression
2. Polynomial Regression
3. Stepwise Regression
4. Decision Tree Regression
5. Random Forest Regression
6. Ridge Regression
7. Lasso Regression
8. ElasticNet Regression
9. Bayesian Linear Regression
10. Support Vector Regression
But, we'll first start with Linear Regression
# Linear Regression
* Linear regression is a fundamental statistical method used to model the relationship between a dependent variable (often denoted as
𝑌) and one or more independent variables (often denoted as
𝑋). The relationship is assumed to be linear, meaning that changes in the independent variables are associated with changes in the dependent variable in a straight-line fashion.
The basic form of linear regression for a single independent variable is:
**𝑌=𝛽0+𝛽1𝑋+𝜖**
Where:
* Y is the dependent variable.
* X is the independent variable.
* 𝛽0 is the intercept, representing the value of Y when X is zero
* 𝛽1 is the slope coefficient, representing the change in Y for a one-unit change in X
* ϵ is the error term, representing the variability in Y that is not explained by the linear relationship with X.
# Basic Code of Linear Regression
* This line imports the numpy library, which is widely used for numerical operations in Python. We use np as an alias for numpy, making it easier to reference functions and objects from the library.
```
import numpy as np
```
* This line imports the LinearRegression class from the linear_model module of the scikit-learn library.scikit-learn is a powerful library for machine learning tasks in Python, and LinearRegression is a class provided by it for linear regression.
```
from sklearn.linear_model import LinearRegression
```
* This line creates a NumPy array X containing the independent variable values. In this example, we have a simple one-dimensional array representing the independent variable. The reshape(-1, 1) method reshapes the array into a column vector, necessary for use with scikit-learn
```
X = np.array([1, 2, 3, 4, 5]).reshape(-1, 1)
```
* This line creates a NumPy array Y containing the corresponding dependent variable values. These are the observed values of the dependent variable corresponding to the independent variable values in X.
```
Y = np.array([2, 4, 5, 8, 5])
```
* This line creates an instance of the LinearRegression class, which represents the linear regression model. We'll use this object to train the model and make predictions.
```
model = LinearRegression()
```
* This line fits the linear regression model to the data. The fit() method takes two arguments: the independent variable (X) and the dependent variable (Y). This method estimates the coefficients of the linear regression equation that best fit the given data.
```
model.fit(X, Y)
```
* These lines print out the intercept (beta_0) and coefficient (beta_1) of the linear regression model. model.intercept_ gives the intercept value, and model.coef_ gives an array of coefficients, where model.coef_[0] corresponds to the coefficient of the first independent variable (in this case, there's only one).
```
print("Intercept:", model.intercept_)
print("Coefficient:", model.coef_[0])
```
* These lines demonstrate how to use the trained model to make predictions for new data.
* We create a new NumPy array new_data containing the values of the independent variable for which we want to predict the dependent variable values.
* We then use the predict() method of the model to obtain the predictions for these new data points. Finally, we print out the predicted values.
```
new_data = np.array([[6], [7]])
predictions = model.predict(new_data)
print("Predictions:", predictions)
```
# Assumptions of Linear Regression
# Linearity:
* To assess the linearity assumption, we can visually inspect a scatter plot of the observed values versus the predicted values.
* If the relationship between them appears linear, it suggests that the linearity assumption is reasonable.
```
import matplotlib.pyplot as plt
predictions = model.predict(X)
plt.scatter(predictions,Y)
plt.xlabel("Predicted Values")
plt.ylabel("Observed Values")
plt.title("Linearity Check: Observed vs Predicted")
plt.show()
```
# Homoscedasticity:
* Homoscedasticity refers to the constant variance of the residuals across all levels of the independent variable(s). We can visually inspect a plot of residuals versus predicted values to check for homoscedasticity.
```
residuals = Y - predictions
plt.scatter(predictions, residuals)
plt.xlabel("Predicted Values")
plt.ylabel("Residuals")
plt.title("Homoscedasticity Check: Residuals vs Predicted Values")
plt.axhline(y=0, color='red', linestyle='--') # Add horizontal line at y=0
plt.show()
```
# Normality of Residuals:
* To assess the normality of residuals, we can visually inspect a histogram or a Q-Q plot of the residuals.
```
import seaborn as sns
sns.histplot(residuals, kde=True)
plt.xlabel("Residuals")
plt.ylabel("Frequency")
plt.title("Normality of Residuals: Histogram")
plt.show()
import scipy.stats as stats
stats.probplot(residuals, dist="norm", plot=plt)
plt.title("Normal Q-Q Plot")
plt.show()
```
# Metrics for Regression
# Mean Absolute Error (MAE)
* MAE measures the average magnitude of the errors in a set of predictions, without considering their direction. It is the average of the absolute differences between predicted and actual values.
```
from sklearn.metrics import mean_absolute_error
mae = mean_absolute_error(Y, predictions)
print(f"Mean Absolute Error (MAE): {mae}")
```
# Mean Squared Error (MSE)
* MSE measures the average of the squares of the errors. It gives more weight to larger errors, making it sensitive to outliers.
```
from sklearn.metrics import mean_squared_error
mse = mean_squared_error(Y, predictions)
print(f"Mean Squared Error (MSE): {mse}")
```
# Root Mean Squared Error (RMSE)
* RMSE is the square root of the MSE. It provides an error metric that is in the same units as the dependent variable, making it more interpretable.
```
rmse = np.sqrt(mse)
print(f"Root Mean Squared Error (RMSE): {rmse}")
```
# R-squared (Coefficient of Determination)
* R-squared measures the proportion of the variance in the dependent variable that is predictable from the independent variables. It ranges from 0 to 1, where 1 indicates a perfect fit.
```
from sklearn.metrics import r2_score
r2 = r2_score(Y, predictions)
print(f"R-squared (R^2): {r2}")
```
> In this tutorial, The sample dataset is there for learning purpose only

70
contrib/machine-learning/confusion-matrix.md 100644
Wyświetl plik

@ -0,0 +1,70 @@
## Confusion Matrix
A confusion matrix is a fundamental performance evaluation tool used in machine learning to assess the accuracy of a classification model. It is an N x N matrix, where N represents the number of target classes.
For binary classification, it results in a 2 x 2 matrix that outlines four key parameters:
1. True Positive (TP) - The predicted value matches the actual value, or the predicted class matches the actual class.
For example - the actual value was positive, and the model predicted a positive value.
2. True Negative (TN) - The predicted value matches the actual value, or the predicted class matches the actual class.
For example - the actual value was negative, and the model predicted a negative value.
3. False Positive (FP)/Type I Error - The predicted value was falsely predicted.
For example - the actual value was negative, but the model predicted a positive value.
4. False Negative (FN)/Type II Error - The predicted value was falsely predicted.
For example - the actual value was positive, but the model predicted a negative value.
The confusion matrix enables the calculation of various metrics like accuracy, precision, recall, F1-Score and specificity.
1. Accuracy - It represents the proportion of correctly classified instances out of the total number of instances in the dataset.
2. Precision - It quantifies the accuracy of positive predictions made by the model.
3. Recall - It quantifies the ability of a model to correctly identify all positive instances in the dataset and is also known as sensitivity or true positive rate.
4. F1-Score - It is a single measure that combines precision and recall, offering a balanced evaluation of a classification model's effectiveness.
To implement the confusion matrix in Python, we can use the confusion_matrix() function from the sklearn.metrics module of the scikit-learn library.
The function returns a 2D array that represents the confusion matrix.
We can also visualize the confusion matrix using a heatmap.
```python
# Import necessary libraries
import numpy as np
from sklearn.metrics import confusion_matrix, classification_report
import seaborn as sns
import matplotlib.pyplot as plt
# Create the NumPy array for actual and predicted labels
actual = np.array(['Apple', 'Apple', 'Apple', 'Not Apple', 'Apple',
'Not Apple', 'Apple', 'Apple', 'Not Apple', 'Not Apple'])
predicted = np.array(['Apple', 'Not Apple', 'Apple', 'Not Apple', 'Apple',
'Apple', 'Apple', 'Apple', 'Not Apple', 'Not Apple'])
# Compute the confusion matrix
cm = confusion_matrix(actual,predicted)
# Plot the confusion matrix with the help of the seaborn heatmap
sns.heatmap(cm,
annot=True,
fmt='g',
xticklabels=['Apple', 'Not Apple'],
yticklabels=['Apple', 'Not Apple'])
plt.xlabel('Prediction', fontsize=13)
plt.ylabel('Actual', fontsize=13)
plt.title('Confusion Matrix', fontsize=17)
plt.show()
# Classifications Report based on Confusion Metrics
print(classification_report(actual, predicted))
```
### Results
```
1. Confusion Matrix:
[[5 1]
[1 3]]
2. Classification Report:
precision recall f1-score support
Apple 0.83 0.83 0.83 6
Not Apple 0.75 0.75 0.75 4
accuracy 0.80 10
macro avg 0.79 0.79 0.79 10
weighted avg 0.80 0.80 0.80 10
```

6
contrib/machine-learning/index.md
Wyświetl plik

@ -1,3 +1,7 @@
# List of sections
- [Section title](filename.md)
- [Regression in Machine Learning](Regression.md)
- [Confusion Matrix](confusion-matrix.md)
- [Support Vector Machine Algorithm](support-vector-machine.md)
- [Artificial Neural Network from the Ground Up](ArtificialNeuralNetwork.md)

62
contrib/machine-learning/support-vector-machine.md 100644
Wyświetl plik

@ -0,0 +1,62 @@
## Support Vector Machine
Support Vector Machine or SVM is one of the most popular Supervised Learning algorithms, which is used for Classification as well as Regression problems. However, primarily, it is used for Classification problems in Machine Learning.
SVM can be of two types -
1. Linear SVM: Linear SVM is used for linearly separable data, which means if a dataset can be classified into two classes by using a single straight line, then such data is termed as linearly separable data, and classifier is used called as Linear SVM classifier.
2. Non-linear SVM: Non-Linear SVM is used for non-linearly separated data, which means if a dataset cannot be classified by using a straight line, then such data is termed as non-linear data and classifier used is called as Non-linear SVM classifier.
Working of SVM - The goal of SVM is to find a hyperplane that separates the data points into different classes. A hyperplane is a line in 2D space, a plane in 3D space, or a higher-dimensional surface in n-dimensional space. The hyperplane is chosen in such a way that it maximizes the margin, which is the distance between the hyperplane and the closest data points of each class. The closest data points are called the support vectors.
The distance between the hyperplane and a data point "x" can be calculated using the formula −
```
distance = (w . x + b) / ||w||
```
where "w" is the weight vector, "b" is the bias term, and "||w||" is the Euclidean norm of the weight vector. The weight vector "w" is perpendicular to the hyperplane and determines its orientation, while the bias term "b" determines its position.
The optimal hyperplane is found by solving an optimization problem, which is to maximize the margin subject to the constraint that all data points are correctly classified. In other words, we want to find the hyperplane that maximizes the margin between the two classes while ensuring that no data point is misclassified. This is a convex optimization problem that can be solved using quadratic programming. If the data points are not linearly separable, we can use a technique called kernel trick to map the data points into a higher-dimensional space where they become separable. The kernel function computes the inner product between the mapped data points without computing the mapping itself. This allows us to work with the data points in the higherdimensional space without incurring the computational cost of mapping them.
1. Hyperplane:
There can be multiple lines/decision boundaries to segregate the classes in n-dimensional space, but we need to find out the best decision boundary that helps to classify the data points. This best boundary is known as the hyperplane of SVM.
The dimensions of the hyperplane depend on the features present in the dataset, which means if there are 2 features, then hyperplane will be a straight line. And if there are 3 features, then hyperplane will be a 2-dimension plane. We always create a hyperplane that has a maximum margin, which means the maximum distance between the data points.
2. Support Vectors:
The data points or vectors that are the closest to the hyperplane and which affect the position of the hyperplane are termed as Support Vector. Since these vectors support the hyperplane, hence called a Support vector.
3. Margin:
It may be defined as the gap between two lines on the closet data points of different classes. It can be calculated as the perpendicular distance from the line to the support vectors. Large margin is considered as a good margin and small margin is considered as a bad margin.
We will use the famous Iris dataset, which contains the sepal length, sepal width, petal length, and petal width of three species of iris flowers: Iris setosa, Iris versicolor, and Iris virginica. The goal is to classify the flowers into their respective species based on these four features. We load the iris dataset using load_iris and split the data into training and testing sets using train_test_split. We use a test size of 0.2, which means that 20% of the data will be used for testing and 80% for training. We set the random state to 42 to ensure reproducibility of the results.
### Implemetation of SVM in Python
```python
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score
# load the iris dataset
iris = load_iris()
# split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(iris.data,
iris.target, test_size=0.2, random_state=42)
# create an SVM classifier with a linear kernel
svm = SVC(kernel='linear')
# train the SVM classifier on the training set
svm.fit(X_train, y_train)
# make predictions on the testing set
y_pred = svm.predict(X_test)
# calculate the accuracy of the classifier
accuracy = accuracy_score(y_test, y_pred)
print("Accuracy:", accuracy)
```
#### Output
```
Accuracy: 1
```

84
contrib/mini-projects/Rock_Paper_Scissors_Game.md 100644
Wyświetl plik

@ -0,0 +1,84 @@
# Rock Paper Scissors Game
This is a simple implementation of the classic rock-paper-scissors game in Python.
## Code Explanation:
In this section, we import the required libraries (`tkinter` for GUI and `random` for generating computer choices) and define two functions:
- `determine_winner(user_choice, computer_choice)`:
- This function determines the winner of the game based on the choices made by the user and the computer.
- It returns a tuple containing the result of the game and the computer's choice.
- `play_game()`:
- This function handles the gameplay logic.
- It gets the user's choice from the radio buttons, generates a random choice for the computer, determines the winner using the `determine_winner()` function, and updates the result and computer pick labels accordingly.
### Imports and Function Definitions:
```python
import tkinter as tk
import random
def determine_winner(user_choice, computer_choice):
"""Determine the winner of the game."""
if user_choice == computer_choice:
return "It's a tie!", computer_choice
elif (user_choice == "rock" and computer_choice == "scissors") or \
(user_choice == "paper" and computer_choice == "rock") or \
(user_choice == "scissors" and computer_choice == "paper"):
return "You win!", computer_choice
else:
return "Computer wins!", computer_choice
def play_game():
"""Play the game and display the result."""
user_choice = user_var.get()
computer_choice = random.choice(["rock", "paper", "scissors"])
result, computer_pick = determine_winner(user_choice, computer_choice)
result_label.config(text=result)
computer_label.config(text=f"Computer picked: {computer_pick}")
```
### GUI Setup:
```python
# Create main window
root = tk.Tk()
root.title("Rock Paper Scissors")
# User choice options
user_var = tk.StringVar()
user_var.set("rock") # Default choice
choices = ["rock", "paper", "scissors"]
for choice in choices:
rb = tk.Radiobutton(root, text=choice, variable=user_var, value=choice)
rb.pack()
```
- Here, we create the main window for the game using `tkinter.Tk()`. We set the title to "Rock Paper Scissors".
- We define a `StringVar` to store the user's choice and set the default choice to "rock".
- We create radio buttons for the user to choose from ("rock", "paper", "scissors") and pack them into the main window.
```
```
### Play Button and Result Labels:
```python
# Play button
play_button = tk.Button(root, text="Play", command=play_game)
play_button.pack()
# Result label
result_label = tk.Label(root, text="", font=("Helvetica", 16))
result_label.pack()
# Computer pick label
computer_label = tk.Label(root, text="", font=("Helvetica", 12))
computer_label.pack()
```
- We create a "Play" button that triggers the `play_game()` function when clicked, using `tkinter.Button`.
- We create two labels to display the result of the game (`result_label`) and the computer's choice (`computer_label`). Both labels initially display no text and are packed into the main window.
```
```
### Mainloop:
```python
root.mainloop()
```
- Finally, we start the Tkinter event loop using `root.mainloop()`, which keeps the GUI window open and responsive until the user closes it.
-

36
contrib/mini-projects/dice_roller.md 100644
Wyświetl plik

@ -0,0 +1,36 @@
## Dice Roller
The aim of this project is to replicate a dice and generate a random number from the numbers 1 to 6.
For this first we will import the random library which will help make random choices.
```
import random
def dice():
dice_no = random.choice([1,2,3,4,5,6])
return "You got " + str(dice_no)
```
The above snippet of code defines a function called `dice()` which makes the random choice and returns the number that is generated.
```
def roll_dice():
print("Hey Guys, you will now roll a single dice using Python!")
while True:
start=input("Type \'k\' to roll the dice: ").lower()
if start != 'k':
print("Invalid input. Please try again.")
continue
print(dice())
roll_again = input("Do you want to reroll? (Yes/No): ").lower()
if roll_again != 'yes':
break
print("Thanks for rolling the dice.")
roll_dice()
```
The above code defines a function called `roll_dice()` which interacts with the user.
It prompts the user to give an input and if the input is `k`,the code proceeds further to generate a random number or gives the message of invalid input and asks the user to try again.
After the dice has been rolled once, the function asks the user whether they want a reroll in the form of a `yes` or `no` question. The dice is rolled again if the user gives `yes` as an answer and exits the code if the user replies with anything other than yes.

4
contrib/mini-projects/index.md
Wyświetl plik

@ -1,3 +1,5 @@
# List of sections
- [Section title](filename.md)
- [Dice Roller](dice_roller.md)
- [Rock Paper Scissors Game](Rock_Paper_Scissors_Game.md)
- [Path Finder](path-finder.md)

120
contrib/mini-projects/path-finder.md 100644
Wyświetl plik

@ -0,0 +1,120 @@
# Path Finder
This Python script uses the curses library to visualize the process of finding a path through a maze in real-time within a terminal window. The program represents the maze as a list of lists, where each list represents a row in the maze, and each string element in the lists represents a cell in the maze. The maze includes walls (#), a start point (O), and an end point (X), with empty spaces ( ) that can be traversed.
## The script includes the following main components:
- Visualization Functions: <br>
print_maze(maze, stdscr, path=[]): This function is used to display the maze in the terminal. It utilizes color pairs to distinguish between the maze walls, the path, and unexplored spaces. The current path being explored is displayed with a different color to make it stand out.
- Utility Functions: <br>
find_start(maze, start): This function searches the maze for the starting point (marked as O) and returns its position as a tuple (row, col). <br>
find_neighbors(maze, row, col): This function identifies the valid adjacent cells (up, down, left, right) that can be moved to from the current position,
ignoring any walls or out-of-bound positions.
- Pathfinding Logic: <br>
find_path(maze, stdscr): This function implements a Breadth-First Search (BFS) algorithm to find a path from the start point to the end point (X). It uses a
queue to explore each possible path sequentially. As it explores the maze, it updates the display in real-time, allowing the viewer to follow the progress
visually. Each visited position is marked and not revisited, ensuring the algorithm efficiently covers all possible paths without repetition.
Overall, the script demonstrates an effective use of the curses library to create a dynamic visual representation of the BFS algorithm solving a maze, providing both an educational tool for understanding pathfinding and an example of real-time data visualization in a terminal.
#### Below is the code of the path finder
```python
import curses
from curses import wrapper
import queue
import time
# Define the structure of the maze as a list of lists where each inner list represents a row.
maze = [
["#", "O", "#", "#", "#", "#", "#", "#", "#"],
["#", " ", " ", " ", " ", " ", " ", " ", "#"],
["#", " ", "#", "#", " ", "#", "#", " ", "#"],
["#", " ", "#", " ", " ", " ", "#", " ", "#"],
["#", " ", "#", " ", "#", " ", "#", " ", "#"],
["#", " ", "#", " ", "#", " ", "#", " ", "#"],
["#", " ", "#", " ", "#", " ", "#", "#", "#"],
["#", " ", " ", " ", " ", " ", " ", " ", "#"],
["#", "#", "#", "#", "#", "#", "#", "X", "#"]
]
# Function to print the current state of the maze in the terminal.
def print_maze(maze, stdscr, path=[]):
BLUE = curses.color_pair(1) # Color pair for walls and free paths
RED = curses.color_pair(2) # Color pair for the current path
for i, row in enumerate(maze):
for j, value in enumerate(row):
if (i, j) in path:
stdscr.addstr(i, j*2, "X", RED) # Print path character with red color
else:
stdscr.addstr(i, j*2, value, BLUE) # Print walls and free paths with blue color
# Function to locate the starting point (marked 'O') in the maze.
def find_start(maze, start):
for i, row in enumerate(maze):
for j, value in enumerate(row):
if value == start:
return i, j
return None
# Function to find a path from start ('O') to end ('X') using BFS.
def find_path(maze, stdscr):
start = "O"
end = "X"
start_pos = find_start(maze, start) # Get the start position
q = queue.Queue()
q.put((start_pos, [start_pos])) # Initialize the queue with the start position
visited = set() # Set to keep track of visited positions
while not q.empty():
current_pos, path = q.get() # Get the current position and path
row, col = current_pos
stdscr.clear() # Clear the screen
print_maze(maze, stdscr, path) # Print the current state of the maze
time.sleep(0.2) # Delay for visibility
stdscr.refresh() # Refresh the screen
if maze[row][col] == end: # Check if the current position is the end
return path # Return the path if end is reached
# Get neighbors (up, down, left, right) that are not walls
neighbors = find_neighbors(maze, row, col)
for neighbor in neighbors:
if neighbor not in visited:
r, c = neighbor
if maze[r][c] != "#":
new_path = path + [neighbor]
q.put((neighbor, new_path))
visited.add(neighbor)
# Function to find the valid neighboring cells (not walls or out of bounds).
def find_neighbors(maze, row, col):
neighbors = []
if row > 0: # UP
neighbors.append((row - 1, col))
if row + 1 < len(maze): # DOWN
neighbors.append((row + 1, col))
if col > 0: # LEFT
neighbors.append((row, col - 1))
if col + 1 < len(maze[0]): # RIGHT
neighbors.append((row, col + 1))
return neighbors
# Main function to setup curses and run the pathfinding algorithm.
def main(stdscr):
curses.init_pair(1, curses.COLOR_BLUE, curses.COLOR_BLACK) # Initialize color pair for blue
curses.init_pair(2, curses.COLOR_RED, curses.COLOR_BLACK) # Initialize color pair for red
find_path(maze, stdscr) # Find the path using BFS
stdscr.getch() # Wait for a key press before exiting
wrapper(main) # Use the wrapper to initialize and finalize curses automatically.
```

2
contrib/numpy/index.md
Wyświetl plik

@ -1,3 +1,3 @@
# List of sections
- [Section title](filename.md)
- [Introduction](introduction.md)

30
contrib/numpy/introduction.md 100644
Wyświetl plik

@ -0,0 +1,30 @@
# Introduction
## What is NumPy?
NumPy is a powerful array-processing library in Python, essential for scientific computing. It provides efficient data structures and tools for working with multidimensional arrays.
## Key Features
1. **Efficient Arrays:** NumPy offers high-performance N-dimensional array objects for swift data manipulation.
2. **Broadcasting:** Advanced broadcasting enables seamless element-wise operations on arrays of varying shapes.
3. **Interoperability:** NumPy seamlessly integrates with C, C++, and Fortran, enhancing performance and versatility.
4. **Mathematical Tools:** Comprehensive support for linear algebra, Fourier transforms, and random number generation.
## Installation
Ensure Python is installed in your system. If not you can install it from here([official Python website](https://www.python.org/)),then install NumPy via:
```bash
pip install numpy
```
## Importing NumPy
To access NumPy functions, import it with the alias `np`.
```python
import numpy as np
```
Using `np` as an alias enhances code readability and is a widely adopted convention.

573
contrib/pandas/Descriptive_Statistics.md 100644
Wyświetl plik

@ -0,0 +1,573 @@
## Descriptive Statistics
In the realm of data science, understanding the characteristics of data is fundamental. Descriptive statistics provide the tools and techniques to succinctly summarize and present the key features of a dataset. It serves as the cornerstone for exploring, visualizing, and ultimately gaining insights from data.
Descriptive statistics encompasses a range of methods designed to describe the central tendency, dispersion, and shape of a dataset. Through measures such as mean, median, mode, standard deviation, and variance, descriptive statistics offer a comprehensive snapshot of the data's distribution and variability.
Data scientists utilize descriptive statistics to uncover patterns, identify outliers, and assess the overall structure of data before delving into more advanced analyses. By summarizing large and complex datasets into manageable and interpretable summaries, descriptive statistics facilitate informed decision-making and actionable insights.
```python
import pandas as pd
import numpy as np
df = pd.read_csv("Age-Income-Dataset.csv")
df
```
| | Age | Income |
| --- | ----------- | ------ |
| 0 | Young | 25000 |
| 1 | Middle Age | 54000 |
| 2 | Old | 60000 |
| 3 | Young | 15000 |
| 4 | Young | 45000 |
| 5 | Young | 65000 |
| 6 | Young | 70000 |
| 7 | Young | 30000 |
| 8 | Middle Age | 27000 |
| 9 | Young | 23000 |
| 10 | Young | 48000 |
| 11 | Old | 52000 |
| 12 | Young | 33000 |
| 13 | Old | 80000 |
| 14 | Old | 75000 |
| 15 | Old | 35000 |
| 16 | Middle Age | 29000 |
| 17 | Middle Age | 57000 |
| 18 | Old | 43000 |
| 19 | Middle Age | 56000 |
| 20 | Old | 63000 |
| 21 | Old | 32000 |
| 22 | Old | 45000 |
| 23 | Old | 89000 |
| 24 | Middle Age | 90000 |
| 25 | Middle Age | 93000 |
| 26 | Young | 80000 |
| 27 | Young | 87000 |
| 28 | Young | 38000 |
| 29 | Young | 23000 |
| 30 | Middle Age | 38900 |
| 31 | Middle Age | 53200 |
| 32 | Old | 43800 |
| 33 | Middle Age | 25600 |
| 34 | Middle Age | 65400 |
| 35 | Old | 76800 |
| 36 | Old | 89700 |
| 37 | Old | 41800 |
| 38 | Young | 31900 |
| 39 | Old | 25600 |
| 40 | Middle Age | 45700 |
| 41 | Old | 35600 |
| 42 | Young | 54300 |
| 43 | Middle Age | 65400 |
| 44 | Old | 67800 |
| 45 | Old | 24500 |
| 46 | Middle Age | 34900 |
| 47 | Old | 45300 |
| 48 | Young | 68400 |
| 49 | Middle Age | 51700 |
```python
df.describe()
```
| | Income |
|-------|-------------|
| count | 50.000000 |
| mean | 50966.000000 |
| std | 21096.683268 |
| min | 15000.000000 |
| 25% | 33475.000000 |
| 50% | 46850.000000 |
| 75% | 65400.000000 |
| max | 93000.000000 |
### Mean
The mean, also known as the average, is a measure of central tendency in a dataset. It represents the typical value of a set of numbers. The formula to calculate the mean of a dataset is:
$$ \overline{x} = \frac{\sum\limits_{i=1}^{n} x_i}{n} $$
* $\overline{x}$ (pronounced "x bar") represents the mean value.
* $x_i$ represents the individual value in the dataset (where i goes from 1 to n).
* $\sum$ (sigma) represents the summation symbol, indicating we add up all the values from i=1 to n.
* $n$ represents the total number of values in the dataset.
```python
df['Income'].mean()
```
#### Result
```
50966.0
```
#### Without pandas
```python
def mean_f(df):
for col in df.columns:
if df[col].dtype != 'O':
temp = 0
for i in df[col]:
temp = temp +i
print("Without pandas Library -> ")
print("Average of {} is {}".format(col,(temp/len(df[col]))))
print()
print("With pandas Library -> ")
print(df[col].mean())
mean_f(df)
```
Average of Income:
- Without pandas Library -> 50966.0
- With pandas Library -> 50966.0
### Median
The median is another measure of central tendency in a dataset. Unlike the mean, which is the average value of all data points, the median represents the middle value when the dataset is ordered from smallest to largest. If the dataset has an odd number of observations, the median is the middle value. If the dataset has an even number of observations, the median is the average of the two middle values.
The median represents the "middle" value in a dataset. There are two cases to consider depending on whether the number of observations (n) is odd or even:
**Odd number of observations (n):**
In this case, the median (M) is the value located at the middle position when the data is ordered from least to greatest. We can calculate the position using the following formula:
$$ M = x_{n+1/2} $$
**Even number of observations (n):**
When we have an even number of observations, there isn't a single "middle" value. Instead, the median is the average of the two middle values after ordering the data. Here's the formula to find the median:
$$ M = \frac{x_{n/2} + x_{(n/2)+1}}{2} $$
**Explanation:**
* M represents the median value.
* n represents the total number of observations in the dataset.
* $x$ represents the individual value.
```python
df['Income'].median()
```
#### Result
```
46850.0
```
#### Without pandas
```python
def median_f(df):
for col in df.columns:
if df[col].dtype != 'O':
sorted_data = sorted(df[col])
n = len(df[col])
if n%2 == 0:
x1 =sorted_data[int((n/2))]
x2 =sorted_data[int((n/2))+1]
median=(x1+x2)/2
else:
median = sorted_data[(n+1)/2]
print("Median without library ->")
print("Median of {} is {} ".format(col,median))
print("Median with library ->")
print(df[col].median())
median_f(df)
```
Median of Income:
- Median without library -> 49850.0
- Median with library -> 46850.0
### Mode
The mode is a measure of central tendency that represents the value or values that occur most frequently in a dataset. Unlike the mean and median, which focus on the average or middle value, the mode identifies the most common value(s) in the dataset.
```python
def mode_f(df):
for col in df.columns:
if df[col].dtype == 'O':
print("Column:", col)
arr = df[col].sort_values()
prevcnt = 0
cnt = 0
ans = arr[0]
temp = arr[0]
for i in arr:
if(temp == i) :
cnt += 1
else:
prevcnt = cnt
cnt = 1
temp = i
if(cnt > prevcnt):
ans = i
print("Without pandas Library -> ")
print("Mode of {} is {}".format(col,ans))
print()
print("With pandas Library -> ")
print(df[col].mode())
mode_f(df)
```
#### Result
```
Column: Age
Without pandas Library ->
Mode of Age is Old
With pandas Library ->
0 Old
Name: Age, dtype: object
```
### Standard Deviation
Standard deviation is a measure of the dispersion or spread of a dataset. It quantifies the amount of variation or dispersion of a set of values from the mean. In other words, it indicates how much individual values in a dataset deviate from the mean.
$$s = \sqrt{\frac{\sum(x_i-\overline{x})^{2}}{n-1}}$$
* $s$ represents the standard deviation.
* $\sum$ (sigma) represents the summation symbol, indicating we add up the values for all data points.
* $x_i$ represents the individual value in the dataset.
* $\overline{x}$ (x bar) represents the mean value of the dataset.
* $n$ represents the total number of values in the dataset.
```python
df['Income'].std()
```
#### Result
```
21096.683267707253
```
#### Without pandas
```python
import math
def std_f(df):
for col in df.columns:
if len(df[col]) == 0:
print("Column is empty")
if df[col].dtype != 'O':
sum = 0
mean = df[col].mean()
for i in df[col]:
sum = sum + (i - mean)**2
std = math.sqrt(sum/len(df[col]))
print("Without pandas library ->")
print("Std : " , std)
print("With pandas library: ->")
print("Std : {}".format(np.std(df[col]))) ##ddof = 1
std_f(df)
```
Without pandas library ->
Std : 20884.6509187968 \
With pandas library: ->
Std : 20884.6509187968
### Count
```python
df['Income'].count()
```
#### Result
```
50
```
### Minimum
```python
df['Income'].min()
```
#### Result
```
15000
```
#### Without pandas
```python
def min_f(df):
for col in df.columns:
if df[col].dtype != "O":
sorted_data = sorted(df[col])
min = sorted_data[0]
print("Without pandas Library->",min)
print("With pandas Library->",df[col].min())
min_f(df)
```
Without pandas Library-> 15000 \
With pandas Library-> 15000
### Maximum
```python
df['Income'].max()
```
#### Result
```
93000
```
#### Without pandas
```python
def max_f(df):
for col in df.columns:
if df[col].dtype != "O":
sorted_data = sorted(df[col])
max = sorted_data[len(df[col])-1]
print("Without pandas Library->",max)
print("With pandas Library->",df[col].max())
max_f(df)
```
Without pandas Library-> 93000
With pandas Library-> 93000
### Percentile
```python
df['Income'].quantile(0.25)
```
#### Result
```
33475.0
```
```python
df['Income'].quantile(0.75)
```
#### Result
```
65400.0
```
#### Without pandas
```python
def percentile_f(df,percentile):
for col in df.columns:
if df[col].dtype != 'O':
sorted_data = sorted(df[col])
index = int(percentile*len(df[col]))
percentile_result = sorted_data[index]
print(f"{percentile} Percentile is : ",percentile_result)
percentile_f(df,0.25)
```
0.25 Percentile is : 33000
We have used the method of nearest rank to calculate percentile manually.
Pandas uses linear interpolation of data to calculate percentiles.
## Correlation and Covariance
```python
df = pd.read_csv('Iris.csv')
df.head(5)
```
| | Id | SepalLengthCm | SepalWidthCm | PetalLengthCm | PetalWidthCm | Species |
|---|----|---------------|--------------|---------------|--------------|-------------|
| 0 | 1 | 5.1 | 3.5 | 1.4 | 0.2 | Iris-setosa |
| 1 | 2 | 4.9 | 3.0 | 1.4 | 0.2 | Iris-setosa |
| 2 | 3 | 4.7 | 3.2 | 1.3 | 0.2 | Iris-setosa |
| 3 | 4 | 4.6 | 3.1 | 1.5 | 0.2 | Iris-setosa |
| 4 | 5 | 5.0 | 3.6 | 1.4 | 0.2 | Iris-setosa |
```python
df.drop(['Id','Species'],axis=1,inplace= True)
```
### Covarience
Covariance measures the degree to which two variables change together. If the covariance between two variables is positive, it means that they tend to increase or decrease together. If the covariance is negative, it means that as one variable increases, the other tends to decrease. However, covariance does not provide a standardized measure, making it difficult to interpret the strength of the relationship between variables, especially if the variables are measured in different units.
$$ COV(X,Y) = \frac{\sum\limits_{i=1}^{n} (X_i - \overline{X}) (Y_i - \overline{Y})}{n - 1}$$
**Explanation:**
* $COV(X, Y)$ represents the covariance between variables X and Y.
* $X_i$ and $Y_i$ represent the individual values for variables X and Y in the i-th observation.
* $\overline{X}$ and $\overline{Y}$ represent the mean values for variables X and Y, respectively.
* $n$ represents the total number of observations in the dataset.
```python
df.cov()
```
| | SepalLengthCm | SepalWidthCm | PetalLengthCm | PetalWidthCm |
|-------------------|-------------- |---------------|-----------------|--------------|
| **SepalLengthCm** | 0.685694 | -0.039268 | 1.273682 | 0.516904 |
| **SepalWidthCm** | -0.039268 | 0.188004 | -0.321713 | -0.117981 |
| **PetalLengthCm** | 1.273682 | -0.321713 | 3.113179 | 1.296387 |
| **PetalWidthCm** | 0.516904 | -0.117981 | 1.296387 | 0.582414 |
#### Without pandas
```python
def cov_f(df):
for x in df.columns:
for y in df.columns:
mean_x = df[x].mean()
mean_y = df[y].mean()
sum = 0
n = len(df[x])
for val in range(n):
sum += (df[x].iloc[val] - mean_x)*(df[y].iloc[val] - mean_y)
print("Covariance of {} and {} is : {}".format(x,y, sum/(n-1)))
print()
cov_f(df)
```
#### Result
```
Covariance of SepalLengthCm and SepalLengthCm is : 0.6856935123042504
Covariance of SepalLengthCm and SepalWidthCm is : -0.03926845637583892
Covariance of SepalLengthCm and PetalLengthCm is : 1.2736823266219246
Covariance of SepalLengthCm and PetalWidthCm is : 0.5169038031319911
Covariance of SepalWidthCm and SepalLengthCm is : -0.03926845637583892
Covariance of SepalWidthCm and SepalWidthCm is : 0.1880040268456377
Covariance of SepalWidthCm and PetalLengthCm is : -0.32171275167785235
Covariance of SepalWidthCm and PetalWidthCm is : -0.11798120805369115
Covariance of PetalLengthCm and SepalLengthCm is : 1.2736823266219246
Covariance of PetalLengthCm and SepalWidthCm is : -0.32171275167785235
Covariance of PetalLengthCm and PetalLengthCm is : 3.113179418344519
Covariance of PetalLengthCm and PetalWidthCm is : 1.2963874720357946
Covariance of PetalWidthCm and SepalLengthCm is : 0.5169038031319911
Covariance of PetalWidthCm and SepalWidthCm is : -0.11798120805369115
Covariance of PetalWidthCm and PetalLengthCm is : 1.2963874720357946
Covariance of PetalWidthCm and PetalWidthCm is : 0.5824143176733781
````
### Correlation
Correlation, on the other hand, standardizes the measure of relationship between two variables, making it easier to interpret. It measures both the strength and direction of the linear relationship between two variables. Correlation values range between -1 and 1, where:
$$r = \frac{n(\sum xy) - (\sum x)(\sum y)}{\sqrt{n(\sum x^2) - (\sum x)^2} \cdot \sqrt{n(\sum y^2) - (\sum y)^2}}$$
* r represents the correlation coefficient.
* n is the number of data points.
```python
df.corr()
```
| | SepalLengthCm | SepalWidthCm | PetalLengthCm | PetalWidthCm |
|-------------------|---------------|--------------|---------------|--------------|
| **SepalLengthCm** | 1.000000 | -0.109369 | 0.871754 | 0.817954 |
| **SepalWidthCm** | -0.109369 | 1.000000 | -0.420516 | -0.356544 |
| **PetalLengthCm** | 0.871754 | -0.420516 | 1.000000 | 0.962757 |
| **PetalWidthCm** | 0.817954 | -0.356544 | 0.962757 | 1.000000 |
#### Without using pandas
```python
import math
def corr_f(df):
for i in df.columns:
for j in df.columns:
n = len(df[i])
sumX = 0
for x in df[i]:
sumX += x
sumY = 0
for y in df[j]:
sumY += y
sumXY = 0
for xy in range(n):
sumXY += (df[i].iloc[xy] * df[j].iloc[xy])
sumX2 = 0
for x in df[i]:
sumX2 += (x**2)
sumY2 = 0
for y in df[j]:
sumY2 += (y**2)
NR = (n * sumXY) - (sumX*sumY)
DR = math.sqrt( ( (n * sumX2) - (sumX**2))*( (n * sumY2) - (sumY ** 2) ) )
print("Correlation of {} and {} :{}".format(i,j,NR/DR))
print()
corr_f(df)
```
#### Result
```
Correlation of SepalLengthCm and SepalLengthCm :1.0
Correlation of SepalLengthCm and SepalWidthCm :-0.10936924995067286
Correlation of SepalLengthCm and PetalLengthCm :0.8717541573048861
Correlation of SepalLengthCm and PetalWidthCm :0.8179536333691775
Correlation of SepalWidthCm and SepalLengthCm :-0.10936924995067286
Correlation of SepalWidthCm and SepalWidthCm :1.0
Correlation of SepalWidthCm and PetalLengthCm :-0.42051609640118826
Correlation of SepalWidthCm and PetalWidthCm :-0.3565440896138223
Correlation of PetalLengthCm and SepalLengthCm :0.8717541573048861
Correlation of PetalLengthCm and SepalWidthCm :-0.42051609640118826
Correlation of PetalLengthCm and PetalLengthCm :1.0
Correlation of PetalLengthCm and PetalWidthCm :0.9627570970509656
Correlation of PetalWidthCm and SepalLengthCm :0.8179536333691775
Correlation of PetalWidthCm and SepalWidthCm :-0.3565440896138223
Correlation of PetalWidthCm and PetalLengthCm :0.9627570970509656
Correlation of PetalWidthCm and PetalWidthCm :1.0
```

63
contrib/pandas/excel_with_pandas.md 100644
Wyświetl plik

@ -0,0 +1,63 @@
# Pandas DataFrame
The Pandas DataFrame is a two-dimensional, size-mutable, and possibly heterogeneous tabular data format with labelled axes. A data frame is a two-dimensional data structure in which the data can be organised in rows and columns. Pandas DataFrames are comprised of three main components: data, rows, and columns.
In the real world, Pandas DataFrames are formed by importing datasets from existing storage, which can be a Excel file, a SQL database or CSV file. Pandas DataFrames may be constructed from lists, dictionaries, or lists of dictionaries, etc.
Features of Pandas `DataFrame`:
- **Size mutable**: DataFrames are mutable in size, meaning that new rows and columns can be added or removed as needed.
- **Labeled axes**: DataFrames have labeled axes, which makes it easy to keep track of the data.
- **Arithmetic operations**: DataFrames support arithmetic operations on rows and columns.
- **High performance**: DataFrames are highly performant, making them ideal for working with large datasets.
### Installation of libraries
`pip install pandas` <br/>
`pip install xlrd`
- **Note**: The `xlrd` library is used for Excel operations.
Example for reading data from an Excel File:
```python
import pandas as pd
l = pd.read_excel('example.xlsx')
d = pd.DataFrame(l)
print(d)
```
Output:
```python
Name Age
0 John 12
```
Example for Inserting Data into Excel File:
```python
import pandas as pd
l = pd.read_excel('file_name.xlsx')
d = {'Name': ['Bob', 'John'], 'Age': [12, 28]}
d = pd.DataFrame(d)
L = pd.concat([l, d], ignore_index = True)
L.to_excel('file_name.xlsx', index = False)
print(L)
```
Output:
```python
Name Age
0 Bob 12
1 John 28
```
### Usage of Pandas DataFrame:
- Can be used to store and analyze financial data, such as stock prices, trading data, and economic data.
- Can be used to store and analyze sensor data, such as data from temperature sensors, motion sensors, and GPS sensors.
- Can be used to store and analyze log data, such as web server logs, application logs, and system logs

2
contrib/pandas/index.md
Wyświetl plik

@ -1,3 +1,5 @@
# List of sections
- [Pandas Series Vs NumPy ndarray](pandas_series_vs_numpy_ndarray.md)
- [Pandas Descriptive Statistics](Descriptive_Statistics.md)
- [Excel using Pandas DataFrame](excel_with_pandas.md)