Pandas - Mati & Caleb
This lesson introduces students to the Pandas library in Python for data analysis and manipulation, covering topics such as data loading, table creation, manipulation, and visualization using real-world examples.
- before we start this portion of the lesson:
- Overview:
- Learning Objectives:
- what is pandas?
- Question #2 & 3:
- but why is pandas useful?
- Question #4:
- how do i flipping use it? its so hard, my puny brain cant understand it
- example code on how to load a csv into a chart
- how to manipulate the data in pandas.
- how do i put it into a chart 😩
- Hacks
- note
- Numpy
- Data/Predictive Analysis
before we start this portion of the lesson:
check if you have pip installed since we are going to be installing some libraries today!!!!!! if you arnt sure if you have pip, check it by running this command:
pip
if your terminal says "command not found" or something else on linux, run this:
python3 -m ensurepip --default-pip
Overview:
Pandas is a powerful tool in Python that is used for data analysis and manipulation. In this lesson, we will explore how to use Pandas to work with datasets, analyze them, and visualize the results.
Learning Objectives:
By the end of this lesson, students should be able to:
- Understand what Pandas is and why it is useful for data analysis
- Load data into Pandas and create tables to store it
- Use different functions in Pandas to manipulate data, such as filtering, sorting, and grouping
- Visualize data using graphs and charts
what is pandas?
no not this
this:
- Pandas is a Python library used for data analysis and manipulation.
- it can handle different types of data, including CSV files and databases.
- it also allows you to create tables to store and work with your data.
- it has functions for filtering, sorting, and grouping data to make it easier to work with.
- it also has tools for visualizing data with graphs and charts.
- it is widely used in the industry for data analysis and is a valuable skill to learn.
- companies that use Pandas include JPMorgan Chase, Google, NASA, the New York Times, and many others.
but why is pandas useful?
- it can provides tools for handling and manipulating tabular data, which is a common format for storing and analyzing data.
- it can handle different types of data, including CSV files and databases.
- it allows you to perform tasks such as filtering, sorting, and grouping data, making it easier to analyze and work with.
- it has functions for handling missing data and can fill in or remove missing values, which is important for accurate data analysis.
- it also has tools for creating visualizations such as graphs and charts, making it easier to communicate insights from the data.
- it is fast and efficient, even for large datasets, which is important for time-critical data analysis.
- it is widely used in the industry and has a large community of users and developers, making it easy to find support and resources.
import pandas as pd
df = pd.read_csv('yourcsvfileidcjustpickoneidiot.csv')
print(df.head())
print("Average age:", df['Age'].mean())
females = df[df['Gender'] == 'Female']
print(females)
sorted_data = df.sort_values(by='Salary', ascending=False)
print(sorted_data)
uh oh!!! no pandas 😢
if see this error, enter these into your terminal:
pip install wheel
pip install pandas
on stack overflow, it said pandas is disturbed through pip as a wheel. so you need that too.
link to full forum if curious: https://stackoverflow.com/questions/33481974/importerror-no-module-named-pandas
ps: do this for this to work on ur laptop:
wget https://raw.githubusercontent.com/KKcbal/amongus/master/_notebooks/files/example.csv
import pandas as pd
# read the CSV file
df = pd.read_csv('files/example.csv')
# print the first five rows
print(df.head())
# define a function to assign each age to an age group
def assign_age_group(age):
if age < 30:
return '<30'
elif age < 40:
return '30-40'
elif age < 50:
return '40-50'
else:
return '>50'
# apply the function to the Age column to create a new column with age groups
df['Age Group'] = df['Age'].apply(assign_age_group)
# group by age group and count the number of people in each group
age_counts = df.groupby('Age Group')['Name'].count()
# print the age group counts
print(age_counts)
import pandas as pd
# load the csv file
df = pd.read_csv('files/example.csv')
# print the first five rows
print(df.head())
# filter the data to include only people aged 30 or older
df_filtered = df[df['Age'] >= 30]
# sort the data by age in descending order
df_sorted = df.sort_values('Age', ascending=False)
# group the data by gender and calculate the mean age for each group
age_by_gender = df.groupby('Gender')['Age'].mean()
# print the filtered data
print(df_filtered)
# print the sorted data
print(df_sorted)
# print the mean age by gender
print(age_by_gender)
import pandas as pd
import matplotlib.pyplot as plt
# read the CSV file
df = pd.read_csv('files/example.csv')
# create a bar chart of the number of people in each age group
age_groups = ['<30', '30-40', '40-50', '>50']
age_counts = pd.cut(df['Age'], bins=[0, 30, 40, 50, df['Age'].max()], labels=age_groups, include_lowest=True).value_counts()
plt.bar(age_counts.index, age_counts.values)
plt.title('Number of people in each age group')
plt.xlabel('Age group')
plt.ylabel('Number of people')
plt.show()
# create a pie chart of the gender distribution
gender_counts = df['Gender'].value_counts()
plt.pie(gender_counts.values, labels=gender_counts.index, autopct='%1.1f%%')
plt.title('Gender distribution')
plt.show()
# create a scatter plot of age vs. income
plt.scatter(df['Age'], df['Income'])
plt.title('Age vs. Income')
plt.xlabel('Age')
plt.ylabel('Income')
plt.show()
uh oh!!!! another error!??!!??!?! install this library:
pip install matplotlib
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
# read the CSV file
df = pd.read_csv('files/example.csv')
# define age groups
age_groups = ['<30', '30-40', '40-50', '>50']
# create a new column with the age group for each person
df['Age Group'] = pd.cut(df['Age'], bins=[0, 30, 40, 50, np.inf], labels=age_groups, include_lowest=True)
# group by age group and count the number of people in each group
age_counts = df.groupby('Age Group')['Name'].count()
# create a bar chart of the age counts
age_counts.plot(kind='bar')
# set the title and axis labels
plt.title('Number of People in Each Age Group')
plt.xlabel('Age Group')
plt.ylabel('Number of People')
# show the chart
plt.show()
magic!!!!!!
Hacks
- make your own data using your brian, google or chatgpt, should look different than mine.
- modify my code or write your own
- output your data other than a bar graph.
- write an 850+ word essay on how pandas, python or irl, affected your life. If AI score below 85%, then -1 grading point
- answer the questions below, the more explained the better.
Questions
- What are the two primary data structures in pandas and how do they differ? The two primary data structures in pandas are Series and DataFrame. A Series is a one-dimensional array-like object that can hold any data type. A DataFrame is a two-dimensional table-like data structure with rows and columns, similar to a spreadsheet.
- How do you read a CSV file into a pandas DataFrame? To read a CSV file into a pandas DataFrame, you can use the read_csv function in pandas.
- How do you select a single column from a pandas DataFrame? To select a single column from a pandas DataFrame, you can use the indexing operator [] with the column name.
- How do you filter rows in a pandas DataFrame based on a condition? To filter rows in a pandas DataFrame based on a condition, you can use boolean indexing.
- How do you group rows in a pandas DataFrame by a particular column? To group rows in a pandas DataFrame by a particular column, you can use the groupby method.
- How do you aggregate data in a pandas DataFrame using functions like sum and mean? To aggregate data in a pandas DataFrame using functions like sum and mean, you can use the agg method.
- How do you handle missing values in a pandas DataFrame? To handle missing values in a pandas DataFrame, you can use the fillna method to fill in missing values with a specific value or method, or you can use the dropna method to remove rows with missing values.
- How do you merge two pandas DataFrames together? To merge two pandas DataFrames together, you can use the merge method.
- How do you export a pandas DataFrame to a CSV file? To export a pandas DataFrame to a CSV file, you can use the to_csv method.
- What is the difference between a Series and a DataFrame in Pandas? The main difference between a Series and a DataFrame in pandas is that a Series is a one-dimensional array-like object, while a DataFrame is a two-dimensional table-like data structure. A Series can be thought of as a single column of a DataFrame, while a DataFrame can have multiple columns.
note
all hacks due saturday night, the more earlier you get them in the higher score you will get. if you miss the due date, you will get a 0. there will be no tolerance.
no questions answered
Tonight- 2.9
Friday Night- 2.8
Saturday Night - 2.7
Sunday Night - 0.0
questions answered
Tonight- 3.0
Friday Night- 2.9
Saturday Night - 2.8
Sunday Night - 0.0
wdfasdf
import pandas as pd
import matplotlib.pyplot as plt
# Read the CSV file
df = pd.read_csv('files/data.csv')
# Create a scatter plot
plt.scatter(df['Duration'], df['Calories'], color='blue')
# Set the axis labels and title
plt.xlabel('Duration')
plt.ylabel('Calories')
plt.title('Calories vs Duration')
# Display the plot
plt.show()
from skimage import io
photo = io.imread('images/waldo.jpg')
type(photo)
import matplotlib.pyplot as plt
plt.imshow(photo)
photo.shape
plt.imshow(photo[210:350, 425:500])
import numpy as np
import matplotlib.pyplot as plt
# Create a 2D array representing the complex plane
x_min, x_max = -2.5, 1.5
y_min, y_max = -2, 2
resolution = 1000
x, y = np.meshgrid(np.linspace(x_min, x_max, resolution), np.linspace(y_min, y_max, resolution))
c = x + 1j*y
# Create a 2D array representing the iterations required for each point to escape the Mandelbrot set
z = np.zeros_like(c)
for i in range(100):
z = z**2 + c
mask = np.abs(z) > 2
z[mask] = 0
if not np.any(mask):
break
# Create a grayscale image of the Mandelbrot set
plt.imshow(i - np.log2(np.log2(np.abs(z))) if np.any(mask) else 0, cmap='gray')
plt.axis('off')
plt.show()
import numpy as np
import matplotlib.pyplot as plt
# Generate some random data
data = np.random.rand(10, 10)
# Create a heatmap using imshow
plt.imshow(data, cmap='coolwarm')
# Add a colorbar for reference
plt.colorbar()
# Show the plot
plt.show()
Data/Predictive Analysis
How can Numpy and Pandas be utilized for data preprocessing in predictive analysis? Both Numpy and Pandas are tools that can be used to clean, standardize, and transform data to prepare it for predictive analysis. While Numpy is preferred for numerical data due to its ability to perform mathematical operations, Pandas offers a wider range of features to handle diverse data types.
What are some machine learning algorithms that can be used for predictive analysis, and how do they differ? There are several algorithms that can be used for predictive analysis, including linear regression, decision trees, random forests, neural networks, and support vector machines. These algorithms differ in terms of their specific applications, such as forecasting continuous outcomes or identifying optimal boundaries between different classes within a dataset.
What are some real-world applications of predictive analysis across different industries? Predictive analysis has various applications in different industries, including predicting weather forecasts, determining the likelihood of medical conditions based on scans, predicting sports game outcomes, and forecasting stock market trends.
What is the role of feature engineering in predictive analysis, and how does it impact model accuracy? Feature engineering involves selecting and manipulating variables to create a predictive analysis model. This technique can improve model accuracy by identifying crucial information that highlights patterns within the data.
How can machine learning models be deployed in real-time applications for predictive analysis? Real-time applications can use machine learning models to make predictions based on user interactions. For instance, TikTok can use machine learning models to analyze user preferences and suggest videos that align with their interests.
What are the limitations of Numpy and Pandas, and when might other data analysis tools be necessary? While Numpy is memory-efficient and preferred for numerical data, Pandas utilizes more memory but can handle diverse data types. For extremely large datasets or complex syntax, alternative tools may be necessary. Additionally, Pandas has a steep learning curve and limited documentation, and it is not compatible with 3D matrices.
How can predictive analysis improve decision-making and optimize business processes? Predictive analysis can facilitate better decision-making and optimize business processes by enabling businesses to forecast probable outcomes and tailor customer experiences to increase profits.