Numerical computing - Intro to pandas¶

Pandas is a fast, powerful, and flexible open-source data analysis library for Python. It provides two core data structures: Series (1-D) and DataFrame (2-D table).

This notebook covers:

Installation & import
Creating DataFrames
Reading DataFrames from files
Exploring a DataFrame
Accessing and iterating over columns
Iterating over columns
Selecting rows (loc, iloc, boolean, query)
Sorting
Adding and removing columns
Handling missing data
GroupBy and aggregation
Merging and concatenating
String operations
DateTime operations
Pivot tables and reshaping
Index operations
Exporting DataFrames

1. Installation¶

Pandas ships with Anaconda. To install manually:

In [1]:

!pip install pandas -q

2. Import¶

In [2]:

import pandas as pd
import numpy as np
from io import StringIO

print(f'pandas version: {pd.__version__}')

pandas version: 2.2.3

3. Creating DataFrames¶

A DataFrame can be created from a dictionary, a list of dicts, a NumPy array, or a file.

3.1 From a dictionary¶

In [3]:

data = {
    'name':   ['Alice', 'Bob', 'Carol', 'Dave', 'Eve'],
    'age':    [25, 30, 35, 28, 22],
    'score':  [88.5, 92.0, 78.3, 95.1, 84.7],
    'passed': [True, True, True, True, True],
}

df = pd.DataFrame(data)
df

Out[3]:

	name	age	score	passed
0	Alice	25	88.5	True
1	Bob	30	92.0	True
2	Carol	35	78.3	True
3	Dave	28	95.1	True
4	Eve	22	84.7	True

3.2 From a list of dicts¶

In [4]:

rows = [
    {'city': 'Pittsburgh', 'state': 'PA', 'pop': 302_971},
    {'city': 'Philadelphia', 'state': 'PA', 'pop': 1_603_797},
    {'city': 'Harrisburg',  'state': 'PA', 'pop': 50_135},
]

cities = pd.DataFrame(rows)
cities

Out[4]:

	city	state	pop
0	Pittsburgh	PA	302971
1	Philadelphia	PA	1603797
2	Harrisburg	PA	50135

3.3 From a NumPy array¶

In [5]:

arr = np.random.randint(0, 100, size=(4, 3))
df_np = pd.DataFrame(arr, columns=['x', 'y', 'z'])
df_np

Out[5]:

	x	y	z
0	46	65	70
1	39	27	5
2	59	8	41
3	49	7	28

4. Reading DataFrames from Files¶

Pandas can read from many file formats. The most common are shown below. Replace the file paths with your actual files.

4.1 CSV¶

In [6]:

# df_csv = pd.read_csv('data.csv')

# Useful parameters:
# pd.read_csv('data.csv',
#     sep=',',          # delimiter (use sep='\t' for TSV)
#     header=0,         # row number to use as column names
#     index_col=0,      # column to use as row index
#     usecols=['a','b'], # load only specific columns
#     nrows=100,        # read only first N rows
#     skiprows=2,       # skip N rows at the top
#     na_values=['NA'], # extra strings to treat as NaN
#     dtype={'col': str} # override column dtypes
# )

csv_data = """name,age,score
Alice,25,88.5
Bob,30,92.0
Carol,35,78.3
"""
df_csv = pd.read_csv(StringIO(csv_data))
df_csv

Out[6]:

	name	age	score
0	Alice	25	88.5
1	Bob	30	92.0
2	Carol	35	78.3

4.2 Excel¶

In [7]:

# df_xlsx = pd.read_excel('data.xlsx', sheet_name='Sheet1')

# Useful parameters:
# pd.read_excel('data.xlsx',
#     sheet_name=0,     # sheet name or index (0 = first sheet)
#     header=0,
#     index_col=None,
#     usecols='A:D',    # Excel column range
#     dtype={'col': float}
# )

4.3 JSON¶

In [8]:

# df_json = pd.read_json('data.json')

json_data = '[{"name":"Alice","age":25},{"name":"Bob","age":30}]'
df_json = pd.read_json(StringIO(json_data))
df_json

Out[8]:

	name	age
0	Alice	25
1	Bob	30

4.4 SQL¶

In [9]:

# import sqlite3
# conn = sqlite3.connect('database.db')
# df_sql = pd.read_sql('SELECT * FROM table_name', conn)
# conn.close()

4.5 Parquet¶

In [10]:

# df_parquet = pd.read_parquet('data.parquet')
# Requires pyarrow or fastparquet: pip install pyarrow

5. Exploring a DataFrame¶

In [11]:

df.shape          # (rows, columns)

Out[11]:

(5, 4)

In [12]:

df.dtypes         # data type of each column

Out[12]:

name       object
age         int64
score     float64
passed       bool
dtype: object

In [13]:

df.columns.tolist()  # list of column names

Out[13]:

['name', 'age', 'score', 'passed']

In [14]:

df.head()         # first 5 rows (pass n for different count)

Out[14]:

	name	age	score	passed
0	Alice	25	88.5	True
1	Bob	30	92.0	True
2	Carol	35	78.3	True
3	Dave	28	95.1	True
4	Eve	22	84.7	True

In [15]:

df.tail(3)        # last 3 rows

Out[15]:

	name	age	score	passed
2	Carol	35	78.3	True
3	Dave	28	95.1	True
4	Eve	22	84.7	True

In [16]:

df.info()         # summary: dtypes, non-null counts, memory

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5 entries, 0 to 4
Data columns (total 4 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   name    5 non-null      object 
 1   age     5 non-null      int64  
 2   score   5 non-null      float64
 3   passed  5 non-null      bool   
dtypes: bool(1), float64(1), int64(1), object(1)
memory usage: 257.0+ bytes

In [17]:

df.describe()     # statistics for numeric columns

Out[17]:

	age	score
count	5.000000	5.000000
mean	28.000000	87.720000
std	4.949747	6.543088
min	22.000000	78.300000
25%	25.000000	84.700000
50%	28.000000	88.500000
75%	30.000000	92.000000
max	35.000000	95.100000

In [18]:

df.describe(include='all')  # statistics for all columns

Out[18]:

	name	age	score	passed
count	5	5.000000	5.000000	5
unique	5	NaN	NaN	1
top	Alice	NaN	NaN	True
freq	1	NaN	NaN	5
mean	NaN	28.000000	87.720000	NaN
std	NaN	4.949747	6.543088	NaN
min	NaN	22.000000	78.300000	NaN
25%	NaN	25.000000	84.700000	NaN
50%	NaN	28.000000	88.500000	NaN
75%	NaN	30.000000	92.000000	NaN
max	NaN	35.000000	95.100000	NaN

In [19]:

df.isnull().sum()  # count of missing values per column

Out[19]:

name      0
age       0
score     0
passed    0
dtype: int64

In [20]:

df.nunique()       # number of unique values per column

Out[20]:

name      5
age       5
score     5
passed    1
dtype: int64

6. Accessing Columns¶

There are several ways to select one or more columns from a DataFrame.

6.1 Single column — returns a Series¶

In [21]:

df['name']          # bracket notation (always works)
# df.name           # attribute notation (works when name is a valid identifier)

Out[21]:

0    Alice
1      Bob
2    Carol
3     Dave
4      Eve
Name: name, dtype: object

6.2 Multiple columns — returns a DataFrame¶

In [22]:

df[['name', 'score']]

Out[22]:

	name	score
0	Alice	88.5
1	Bob	92.0
2	Carol	78.3
3	Dave	95.1
4	Eve	84.7

6.3 Column by position with iloc¶

In [23]:

df.iloc[:, 0]   # first column
df.iloc[:, -1]  # last column
df.iloc[:, 1:3] # columns 1 and 2

Out[23]:

	age	score
0	25	88.5
1	30	92.0
2	35	78.3
3	28	95.1
4	22	84.7

7. Iterating Over Columns¶

7.1 Iterate column names¶

In [24]:

for col in df.columns:
    print(col)

name
age
score
passed

7.2 Iterate (name, Series) pairs with `df.items()`¶

df.items() yields (column_name, Series) tuples — the standard way to loop over columns.

In [25]:

for col_name, col_series in df.items():
    print(f'{col_name}: dtype={col_series.dtype}, sample={col_series.iloc[0]}')

name: dtype=object, sample=Alice
age: dtype=int64, sample=25
score: dtype=float64, sample=88.5
passed: dtype=bool, sample=True

7.3 Compute a statistic for every numeric column¶

In [26]:

for col_name, col_series in df.items():
    if pd.api.types.is_numeric_dtype(col_series):
        print(f'{col_name}: mean={col_series.mean():.2f}, std={col_series.std():.2f}')

age: mean=28.00, std=4.95
score: mean=87.72, std=6.54
passed: mean=1.00, std=0.00

7.4 Build a summary table from column iteration¶

In [27]:

summary = []
for col_name, col_series in df.items():
    summary.append({
        'column':   col_name,
        'dtype':    str(col_series.dtype),
        'nulls':    col_series.isnull().sum(),
        'unique':   col_series.nunique(),
        'sample':   col_series.iloc[0],
    })

pd.DataFrame(summary)

Out[27]:

	column	dtype	unique	sample
0	name	object	5	Alice
1	age	int64	5	25
2	score	float64	5	88.5
3	passed	bool	1	True

7.5 Rename columns in a loop¶

In [28]:

# Normalize all column names: lowercase, spaces → underscores
df.columns = [c.lower().replace(' ', '_') for c in df.columns]
df.columns.tolist()

Out[28]:

['name', 'age', 'score', 'passed']

7.6 Apply a function to every column with `df.apply()`¶

apply is vectorized and faster than an explicit Python loop for large DataFrames.

In [29]:

df_numeric = df.select_dtypes(include='number')

# Apply a lambda across columns (axis=0 is the default)
col_ranges = df_numeric.apply(lambda s: s.max() - s.min())
print('Range per column:')
print(col_ranges)

Range per column:
age      13.0
score    16.8
dtype: float64

7.7 Filter rows using a column value¶

In [30]:

high_scorers = df[df['score'] > 85]
high_scorers

Out[30]:

	name	age	score	passed
0	Alice	25	88.5	True
1	Bob	30	92.0	True
3	Dave	28	95.1	True

8. Selecting Rows¶

Use .loc for label-based selection, .iloc for position-based, and .query() for expressions.

8.1 Single row and slices with `loc`¶

In [31]:

df.loc[0]           # single row as a Series

Out[31]:

name      Alice
age          25
score      88.5
passed     True
Name: 0, dtype: object

In [32]:

df.loc[1:3]         # rows with index labels 1, 2, 3 (inclusive)

Out[32]:

	name	age	score	passed
1	Bob	30	92.0	True
2	Carol	35	78.3	True
3	Dave	28	95.1	True

In [33]:

df.loc[df['score'] > 90, ['name', 'score']]  # rows where score > 90, two columns only

Out[33]:

	name	score
1	Bob	92.0
3	Dave	95.1

8.2 Position-based selection with `iloc`¶

In [34]:

df.iloc[0]          # first row

Out[34]:

name      Alice
age          25
score      88.5
passed     True
Name: 0, dtype: object

In [35]:

df.iloc[-1]         # last row

Out[35]:

name       Eve
age         22
score     84.7
passed    True
Name: 4, dtype: object

In [36]:

df.iloc[1:4, 0:2]   # rows 1-3, first two columns

Out[36]:

	name	age
1	Bob	30
2	Carol	35
3	Dave	28

8.3 Boolean indexing¶

In [37]:

# Single condition
df[df['age'] < 30]

Out[37]:

	name	age	score	passed
0	Alice	25	88.5	True
3	Dave	28	95.1	True
4	Eve	22	84.7	True

In [38]:

# Multiple conditions — use & (and) | (or), wrap each in parentheses
df[(df['age'] < 30) & (df['score'] > 85)]

Out[38]:

	name	age	score	passed
0	Alice	25	88.5	True
3	Dave	28	95.1	True

In [39]:

# isin — match a list of values
df[df['name'].isin(['Alice', 'Eve'])]

Out[39]:

	name	age	score	passed
0	Alice	25	88.5	True
4	Eve	22	84.7	True

In [40]:

# ~ inverts a boolean mask
df[~df['name'].isin(['Bob', 'Dave'])]

Out[40]:

	name	age	score	passed
0	Alice	25	88.5	True
2	Carol	35	78.3	True
4	Eve	22	84.7	True

8.4 `query()` — SQL-like filter expressions¶

In [41]:

df.query('age < 30 and score > 85')

Out[41]:

	name	age	score	passed
0	Alice	25	88.5	True
3	Dave	28	95.1	True

In [42]:

# Reference a Python variable with @
threshold = 90
df.query('score > @threshold')

Out[42]:

	name	age	score	passed
1	Bob	30	92.0	True
3	Dave	28	95.1	True

9. Sorting¶

9.1 Sort by one column¶

In [43]:

df.sort_values('age')                      # ascending (default)

Out[43]:

	name	age	score	passed
4	Eve	22	84.7	True
0	Alice	25	88.5	True
3	Dave	28	95.1	True
1	Bob	30	92.0	True
2	Carol	35	78.3	True

In [44]:

df.sort_values('score', ascending=False)   # descending

Out[44]:

	name	age	score	passed
3	Dave	28	95.1	True
1	Bob	30	92.0	True
0	Alice	25	88.5	True
4	Eve	22	84.7	True
2	Carol	35	78.3	True

9.2 Sort by multiple columns¶

In [45]:

df.sort_values(['passed', 'score'], ascending=[True, False])

Out[45]:

	name	age	score	passed
3	Dave	28	95.1	True
1	Bob	30	92.0	True
0	Alice	25	88.5	True
4	Eve	22	84.7	True
2	Carol	35	78.3	True

9.3 Top / bottom N rows¶

In [46]:

df.nlargest(3, 'score')    # top 3 scores

Out[46]:

	name	age	score	passed
3	Dave	28	95.1	True
1	Bob	30	92.0	True
0	Alice	25	88.5	True

In [47]:

df.nsmallest(2, 'age')     # 2 youngest

Out[47]:

	name	age	score	passed
4	Eve	22	84.7	True
0	Alice	25	88.5	True

9.4 Sort by index¶

In [48]:

df.sort_index(ascending=False)

Out[48]:

	name	age	score	passed
4	Eve	22	84.7	True
3	Dave	28	95.1	True
2	Carol	35	78.3	True
1	Bob	30	92.0	True
0	Alice	25	88.5	True

10. Adding and Removing Columns¶

10.1 Add a column by direct assignment¶

In [49]:

df_work = df.copy()

df_work['bonus'] = df_work['score'] * 10
df_work

Out[49]:

	name	age	score	passed	bonus
0	Alice	25	88.5	True	885.0
1	Bob	30	92.0	True	920.0
2	Carol	35	78.3	True	783.0
3	Dave	28	95.1	True	951.0
4	Eve	22	84.7	True	847.0

10.2 Add a column with `assign()` — returns a new DataFrame, does not mutate¶

In [50]:

df.assign(
    score_pct = lambda x: x['score'] / 100,
    grade     = lambda x: x['score'].apply(lambda s: 'A' if s >= 90 else 'B' if s >= 80 else 'C'),
)

Out[50]:

	name	age	score	passed	score_pct	grade
0	Alice	25	88.5	True	0.885	B
1	Bob	30	92.0	True	0.920	A
2	Carol	35	78.3	True	0.783	C
3	Dave	28	95.1	True	0.951	A
4	Eve	22	84.7	True	0.847	B

10.3 Rename columns¶

In [51]:

df.rename(columns={'name': 'full_name', 'score': 'test_score'})

Out[51]:

	full_name	age	test_score	passed
0	Alice	25	88.5	True
1	Bob	30	92.0	True
2	Carol	35	78.3	True
3	Dave	28	95.1	True
4	Eve	22	84.7	True

10.4 Drop columns¶

In [52]:

df.drop(columns=['passed'])

Out[52]:

	name	age	score
0	Alice	25	88.5
1	Bob	30	92.0
2	Carol	35	78.3
3	Dave	28	95.1
4	Eve	22	84.7

In [53]:

df.drop(columns=['age', 'passed'])

Out[53]:

	name	score
0	Alice	88.5
1	Bob	92.0
2	Carol	78.3
3	Dave	95.1
4	Eve	84.7

10.5 Drop rows¶

In [54]:

df.drop(index=[0, 4])       # drop rows by label

Out[54]:

	name	age	score	passed
1	Bob	30	92.0	True
2	Carol	35	78.3	True
3	Dave	28	95.1	True

10.6 Replace values¶

In [55]:

df.replace({'passed': {True: 'Yes', False: 'No'}})

Out[55]:

	name	age	score	passed
0	Alice	25	88.5	Yes
1	Bob	30	92.0	Yes
2	Carol	35	78.3	Yes
3	Dave	28	95.1	Yes
4	Eve	22	84.7	Yes

In [56]:

# Replace across all columns
df_work2 = df.copy()
df_work2['name'] = df_work2['name'].replace({'Alice': 'Alicia', 'Bob': 'Robert'})
df_work2

Out[56]:

	name	age	score	passed
0	Alicia	25	88.5	True
1	Robert	30	92.0	True
2	Carol	35	78.3	True
3	Dave	28	95.1	True
4	Eve	22	84.7	True

10.7 Change column data type¶

In [57]:

df_work3 = df.copy()
df_work3['age'] = df_work3['age'].astype(float)
df_work3['name'] = df_work3['name'].astype('string')   # nullable string type
df_work3.dtypes

Out[57]:

name      string[python]
age              float64
score            float64
passed              bool
dtype: object

11. Handling Missing Data¶

In [58]:

df_miss = pd.DataFrame({
    'a': [1.0, 2.0, np.nan, 4.0, 5.0],
    'b': [np.nan, 2.0, 3.0, np.nan, 5.0],
    'c': ['x', 'y', None, 'w', None],
})
df_miss

Out[58]:

	a	b	c
0	1.0	NaN	x
1	2.0	2.0	y
2	NaN	3.0	None
3	4.0	NaN	w
4	5.0	5.0	None

11.1 Detect missing values¶

In [59]:

df_miss.isna()           # True where value is NaN/None

Out[59]:

	a	b	c
0	False	True	False
1	False	False	False
2	True	False	True
3	False	True	False
4	False	False	True

In [60]:

df_miss.isna().sum()     # count per column

Out[60]:

a    1
b    2
c    2
dtype: int64

In [61]:

df_miss.isna().sum().sum()   # total missing in entire DataFrame

Out[61]:

np.int64(5)

In [62]:

df_miss.notna().all()    # True for columns with no missing values

Out[62]:

a    False
b    False
c    False
dtype: bool

11.2 Drop rows or columns with missing values¶

In [63]:

df_miss.dropna()                    # drop any row with at least one NaN

Out[63]:

	a	b	c
1	2.0	2.0	y

In [64]:

df_miss.dropna(how='all')           # drop only rows where ALL values are NaN

Out[64]:

	a	b	c
0	1.0	NaN	x
1	2.0	2.0	y
2	NaN	3.0	None
3	4.0	NaN	w
4	5.0	5.0	None

In [65]:

df_miss.dropna(subset=['a'])        # drop rows where column 'a' is NaN

Out[65]:

	a	b	c
0	1.0	NaN	x
1	2.0	2.0	y
3	4.0	NaN	w
4	5.0	5.0	None

In [66]:

df_miss.dropna(axis=1)             # drop columns that contain any NaN

Out[66]:


0
1
2
3
4

11.3 Fill missing values¶

In [67]:

df_miss.fillna(0)                   # fill all NaN with 0

Out[67]:

	a	b	c
0	1.0	0.0	x
1	2.0	2.0	y
2	0.0	3.0	0
3	4.0	0.0	w
4	5.0	5.0	0

In [68]:

# Fill each column with a different value
df_miss.fillna({'a': df_miss['a'].mean(), 'b': 0, 'c': 'unknown'})

Out[68]:

	a	b	c
0	1.0	0.0	x
1	2.0	2.0	y
2	3.0	3.0	unknown
3	4.0	0.0	w
4	5.0	5.0	unknown

In [69]:

# Forward-fill: propagate last valid value downward
df_miss[['a', 'b']].ffill()

Out[69]:

	a	b
0	1.0	NaN
1	2.0	2.0
2	2.0	3.0
3	4.0	3.0
4	5.0	5.0

In [70]:

# Backward-fill: propagate next valid value upward
df_miss[['a', 'b']].bfill()

Out[70]:

	a	b
0	1.0	2.0
1	2.0	2.0
2	4.0	3.0
3	4.0	5.0
4	5.0	5.0

11.4 Interpolate numeric columns¶

In [71]:

df_miss[['a', 'b']].interpolate()  # linear interpolation between neighbours

Out[71]:

	a	b
0	1.0	NaN
1	2.0	2.0
2	3.0	3.0
3	4.0	4.0
4	5.0	5.0

12. GroupBy and Aggregation¶

In [72]:

emp = pd.DataFrame({
    'name':   ['Alice', 'Bob', 'Carol', 'Dave', 'Eve', 'Frank', 'Grace'],
    'dept':   ['Eng', 'HR', 'Eng', 'Finance', 'HR', 'Eng', 'Finance'],
    'salary': [95000, 72000, 105000, 88000, 68000, 115000, 92000],
    'years':  [5, 3, 8, 6, 2, 10, 4],
    'rating': [4.2, 3.8, 4.5, 4.0, 3.5, 4.8, 4.1],
})
emp

Out[72]:

	name	dept	salary	years	rating
0	Alice	Eng	95000	5	4.2
1	Bob	HR	72000	3	3.8
2	Carol	Eng	105000	8	4.5
3	Dave	Finance	88000	6	4.0
4	Eve	HR	68000	2	3.5
5	Frank	Eng	115000	10	4.8
6	Grace	Finance	92000	4	4.1

12.1 Single aggregation¶

In [73]:

emp.groupby('dept')['salary'].mean()

Out[73]:

dept
Eng        105000.0
Finance     90000.0
HR          70000.0
Name: salary, dtype: float64

In [74]:

emp.groupby('dept').size()          # number of employees per dept

Out[74]:

dept
Eng        3
Finance    2
HR         2
dtype: int64

In [75]:

emp.groupby('dept')['salary'].agg(['min', 'max', 'mean', 'std'])

Out[75]:

	min	max	mean	std
dept
Eng	95000	115000	105000.0	10000.000000
Finance	88000	92000	90000.0	2828.427125
HR	68000	72000	70000.0	2828.427125

12.2 Multiple aggregations at once¶

In [76]:

emp.groupby('dept').agg(
    avg_salary = ('salary', 'mean'),
    max_salary = ('salary', 'max'),
    headcount  = ('name',   'count'),
    avg_years  = ('years',  'mean'),
)

Out[76]:

	avg_salary	max_salary	headcount	avg_years
dept
Eng	105000.0	115000	3	7.666667
Finance	90000.0	92000	2	5.000000
HR	70000.0	72000	2	2.500000

12.3 `transform` — broadcast group result back to original index¶

In [77]:

# Add a column showing each employee's salary vs. their dept average
emp['dept_avg'] = emp.groupby('dept')['salary'].transform('mean')
emp['vs_avg']   = emp['salary'] - emp['dept_avg']
emp[['name', 'dept', 'salary', 'dept_avg', 'vs_avg']]

Out[77]:

	name	dept	salary	dept_avg	vs_avg
0	Alice	Eng	95000	105000.0	-10000.0
1	Bob	HR	72000	70000.0	2000.0
2	Carol	Eng	105000	105000.0	0.0
3	Dave	Finance	88000	90000.0	-2000.0
4	Eve	HR	68000	70000.0	-2000.0
5	Frank	Eng	115000	105000.0	10000.0
6	Grace	Finance	92000	90000.0	2000.0

12.4 `filter` — keep only groups that meet a condition¶

In [78]:

# Keep only departments with more than 2 employees
emp.groupby('dept').filter(lambda g: len(g) > 2)

Out[78]:

	name	dept	salary	years	rating	dept_avg	vs_avg
0	Alice	Eng	95000	5	4.2	105000.0	-10000.0
2	Carol	Eng	105000	8	4.5	105000.0	0.0
5	Frank	Eng	115000	10	4.8	105000.0	10000.0

12.5 Aggregate the whole DataFrame¶

In [79]:

emp[['salary', 'years', 'rating']].agg(['mean', 'std', 'min', 'max'])

Out[79]:

	salary	years	rating
mean	90714.285714	5.428571	4.128571
std	16770.154896	2.819997	0.430946
min	68000.000000	2.000000	3.500000
max	115000.000000	10.000000	4.800000

13. Merging and Concatenating¶

In [80]:

dept_info = pd.DataFrame({
    'dept':            ['Eng', 'HR', 'Finance', 'Legal'],
    'location':        ['Pittsburgh', 'NYC', 'Chicago', 'Boston'],
    'headcount_limit': [20, 10, 15, 5],
})
dept_info

Out[80]:

	dept	location	headcount_limit
0	Eng	Pittsburgh	20
1	HR	NYC	10
2	Finance	Chicago	15
3	Legal	Boston	5

13.1 `merge()` — SQL-style joins¶

In [81]:

# Inner join: only rows with matching keys in both DataFrames
emp.merge(dept_info, on='dept')

Out[81]:

	name	dept	salary	years	rating	dept_avg	vs_avg	location	headcount_limit
0	Alice	Eng	95000	5	4.2	105000.0	-10000.0	Pittsburgh	20
1	Bob	HR	72000	3	3.8	70000.0	2000.0	NYC	10
2	Carol	Eng	105000	8	4.5	105000.0	0.0	Pittsburgh	20
3	Dave	Finance	88000	6	4.0	90000.0	-2000.0	Chicago	15
4	Eve	HR	68000	2	3.5	70000.0	-2000.0	NYC	10
5	Frank	Eng	115000	10	4.8	105000.0	10000.0	Pittsburgh	20
6	Grace	Finance	92000	4	4.1	90000.0	2000.0	Chicago	15

In [82]:

# Left join: keep all employees, attach dept info where available
emp.merge(dept_info, on='dept', how='left')

Out[82]:

	name	dept	salary	years	rating	dept_avg	vs_avg	location	headcount_limit
0	Alice	Eng	95000	5	4.2	105000.0	-10000.0	Pittsburgh	20
1	Bob	HR	72000	3	3.8	70000.0	2000.0	NYC	10
2	Carol	Eng	105000	8	4.5	105000.0	0.0	Pittsburgh	20
3	Dave	Finance	88000	6	4.0	90000.0	-2000.0	Chicago	15
4	Eve	HR	68000	2	3.5	70000.0	-2000.0	NYC	10
5	Frank	Eng	115000	10	4.8	105000.0	10000.0	Pittsburgh	20
6	Grace	Finance	92000	4	4.1	90000.0	2000.0	Chicago	15

In [83]:

# Right join: keep all depts, attach employees where available
emp.merge(dept_info, on='dept', how='right')

Out[83]:

	name	dept	salary	years	rating	dept_avg	vs_avg	location	headcount_limit
0	Alice	Eng	95000.0	5.0	4.2	105000.0	-10000.0	Pittsburgh	20
1	Carol	Eng	105000.0	8.0	4.5	105000.0	0.0	Pittsburgh	20
2	Frank	Eng	115000.0	10.0	4.8	105000.0	10000.0	Pittsburgh	20
3	Bob	HR	72000.0	3.0	3.8	70000.0	2000.0	NYC	10
4	Eve	HR	68000.0	2.0	3.5	70000.0	-2000.0	NYC	10
5	Dave	Finance	88000.0	6.0	4.0	90000.0	-2000.0	Chicago	15
6	Grace	Finance	92000.0	4.0	4.1	90000.0	2000.0	Chicago	15
7	NaN	Legal	NaN	NaN	NaN	NaN	NaN	Boston	5

In [84]:

# Outer join: keep everything from both sides
emp.merge(dept_info, on='dept', how='outer')

Out[84]:

	name	dept	salary	years	rating	dept_avg	vs_avg	location	headcount_limit
0	Alice	Eng	95000.0	5.0	4.2	105000.0	-10000.0	Pittsburgh	20
1	Carol	Eng	105000.0	8.0	4.5	105000.0	0.0	Pittsburgh	20
2	Frank	Eng	115000.0	10.0	4.8	105000.0	10000.0	Pittsburgh	20
3	Dave	Finance	88000.0	6.0	4.0	90000.0	-2000.0	Chicago	15
4	Grace	Finance	92000.0	4.0	4.1	90000.0	2000.0	Chicago	15
5	Bob	HR	72000.0	3.0	3.8	70000.0	2000.0	NYC	10
6	Eve	HR	68000.0	2.0	3.5	70000.0	-2000.0	NYC	10
7	NaN	Legal	NaN	NaN	NaN	NaN	NaN	Boston	5

In [85]:

# Merge on differently-named keys
df_left  = pd.DataFrame({'id': [1, 2, 3], 'value': ['a', 'b', 'c']})
df_right = pd.DataFrame({'emp_id': [2, 3, 4], 'label': ['B', 'C', 'D']})
df_left.merge(df_right, left_on='id', right_on='emp_id', how='left')

Out[85]:

	id	value	emp_id	label
0	1	a	NaN	NaN
1	2	b	2.0	B
2	3	c	3.0	C

13.2 `concat()` — stack DataFrames¶

In [86]:

# Vertical stack (append rows)
top3    = emp.head(3)
bottom3 = emp.tail(3)
pd.concat([top3, bottom3], ignore_index=True)

Out[86]:

	name	dept	salary	years	rating	dept_avg	vs_avg
0	Alice	Eng	95000	5	4.2	105000.0	-10000.0
1	Bob	HR	72000	3	3.8	70000.0	2000.0
2	Carol	Eng	105000	8	4.5	105000.0	0.0
3	Eve	HR	68000	2	3.5	70000.0	-2000.0
4	Frank	Eng	115000	10	4.8	105000.0	10000.0
5	Grace	Finance	92000	4	4.1	90000.0	2000.0

In [87]:

# Horizontal stack (add columns)
left_cols  = emp[['name', 'dept']]
right_cols = emp[['salary', 'rating']]
pd.concat([left_cols, right_cols], axis=1)

Out[87]:

	name	dept	salary	rating
0	Alice	Eng	95000	4.2
1	Bob	HR	72000	3.8
2	Carol	Eng	105000	4.5
3	Dave	Finance	88000	4.0
4	Eve	HR	68000	3.5
5	Frank	Eng	115000	4.8
6	Grace	Finance	92000	4.1

14. String Operations¶

The .str accessor vectorizes Python string methods across a Series.

In [88]:

df_str = pd.DataFrame({
    'full_name': ['  Alice Smith ', 'bob jones', 'Carol WHITE', 'dave brown  '],
    'email':     ['alice@example.com', 'bob@test.org', 'carol@example.com', 'dave@work.net'],
    'skills':    ['Python; SQL; Git', 'Excel; SQL', 'R; Python; SAS', 'Excel; Word; SQL'],
})
df_str

Out[88]:

	full_name	email	skills
0	Alice Smith	alice@example.com	Python; SQL; Git
1	bob jones	bob@test.org	Excel; SQL
2	Carol WHITE	carol@example.com	R; Python; SAS
3	dave brown	dave@work.net	Excel; Word; SQL

14.1 Case and whitespace¶

In [89]:

df_str['full_name'].str.strip()        # remove leading/trailing whitespace

Out[89]:

0    Alice Smith
1      bob jones
2    Carol WHITE
3     dave brown
Name: full_name, dtype: object

In [90]:

df_str['full_name'].str.strip().str.lower()    # strip then lowercase

Out[90]:

0    alice smith
1      bob jones
2    carol white
3     dave brown
Name: full_name, dtype: object

In [91]:

df_str['full_name'].str.strip().str.title()    # Title Case

Out[91]:

0    Alice Smith
1      Bob Jones
2    Carol White
3     Dave Brown
Name: full_name, dtype: object

14.2 Contains, startswith, endswith¶

In [92]:

df_str[df_str['email'].str.contains('example')]

Out[92]:

	full_name	email	skills
0	Alice Smith	alice@example.com	Python; SQL; Git
2	Carol WHITE	carol@example.com	R; Python; SAS

In [93]:

df_str[df_str['email'].str.endswith('.org')]

Out[93]:

	full_name	email	skills
1	bob jones	bob@test.org	Excel; SQL

14.3 Extract substrings¶

In [94]:

# Split on '@' and grab the domain part
df_str['domain'] = df_str['email'].str.split('@').str[1]
df_str[['email', 'domain']]

Out[94]:

	email	domain
0	alice@example.com	example.com
1	bob@test.org	test.org
2	carol@example.com	example.com
3	dave@work.net	work.net

In [95]:

# Count skills (number of '; ' separators + 1)
df_str['n_skills'] = df_str['skills'].str.count(';') + 1
df_str[['full_name', 'skills', 'n_skills']]

Out[95]:

	full_name	skills	n_skills
0	Alice Smith	Python; SQL; Git	3
1	bob jones	Excel; SQL	2
2	Carol WHITE	R; Python; SAS	3
3	dave brown	Excel; Word; SQL	3

14.4 Replace and split¶

In [96]:

df_str['skills'].str.replace(';', ',', regex=False)   # replace delimiter

Out[96]:

0    Python, SQL, Git
1          Excel, SQL
2      R, Python, SAS
3    Excel, Word, SQL
Name: skills, dtype: object

In [97]:

df_str['skills'].str.split('; ')              # split into a list

Out[97]:

0    [Python, SQL, Git]
1          [Excel, SQL]
2      [R, Python, SAS]
3    [Excel, Word, SQL]
Name: skills, dtype: object

In [98]:

# Explode a list column into individual rows
df_str['skills'].str.split('; ').explode().reset_index(drop=True)

Out[98]:

0     Python
1        SQL
2        Git
3      Excel
4        SQL
5          R
6     Python
7        SAS
8      Excel
9       Word
10       SQL
Name: skills, dtype: object

15. DateTime Operations¶

In [99]:

df_dt = pd.DataFrame({
    'event':    ['Launch', 'Review', 'Release', 'Maintenance'],
    'date_str': ['2024-01-15', '2024-03-22', '2024-07-04', '2024-12-31'],
    'value':    [100, 200, 150, 300],
})

df_dt['date'] = pd.to_datetime(df_dt['date_str'])
df_dt

Out[99]:

	event	date_str	value	date
0	Launch	2024-01-15	100	2024-01-15
1	Review	2024-03-22	200	2024-03-22
2	Release	2024-07-04	150	2024-07-04
3	Maintenance	2024-12-31	300	2024-12-31

15.1 Extract date parts¶

In [100]:

df_dt['year']     = df_dt['date'].dt.year
df_dt['month']    = df_dt['date'].dt.month
df_dt['day']      = df_dt['date'].dt.day
df_dt['weekday']  = df_dt['date'].dt.day_name()
df_dt['quarter']  = df_dt['date'].dt.quarter
df_dt[['event', 'date', 'year', 'month', 'day', 'weekday', 'quarter']]

Out[100]:

	event	date	year	month	day	weekday	quarter
0	Launch	2024-01-15	2024	1	15	Monday	1
1	Review	2024-03-22	2024	3	22	Friday	1
2	Release	2024-07-04	2024	7	4	Thursday	3
3	Maintenance	2024-12-31	2024	12	31	Tuesday	4

15.2 Date arithmetic¶

In [101]:

reference = pd.Timestamp('2024-01-01')
df_dt['days_since_start'] = (df_dt['date'] - reference).dt.days
df_dt[['event', 'date', 'days_since_start']]

Out[101]:

	event	date	days_since_start
0	Launch	2024-01-15	14
1	Review	2024-03-22	81
2	Release	2024-07-04	185
3	Maintenance	2024-12-31	365

In [102]:

# Add a fixed timedelta
df_dt['follow_up'] = df_dt['date'] + pd.Timedelta(days=30)
df_dt[['event', 'date', 'follow_up']]

Out[102]:

	event	date	follow_up
0	Launch	2024-01-15	2024-02-14
1	Review	2024-03-22	2024-04-21
2	Release	2024-07-04	2024-08-03
3	Maintenance	2024-12-31	2025-01-30

15.3 Generate date ranges¶

In [103]:

# Monthly date range
monthly = pd.date_range(start='2024-01-01', periods=12, freq='ME')
pd.DataFrame({'month': monthly, 'value': range(12)})

Out[103]:

	month	value
0	2024-01-31	0
1	2024-02-29	1
2	2024-03-31	2
3	2024-04-30	3
4	2024-05-31	4
5	2024-06-30	5
6	2024-07-31	6
7	2024-08-31	7
8	2024-09-30	8
9	2024-10-31	9
10	2024-11-30	10
11	2024-12-31	11

15.4 Filter by date¶

In [104]:

df_dt[df_dt['date'] >= '2024-06-01']

Out[104]:

	event	date_str	value	date	year	month	day	weekday	quarter	days_since_start	follow_up
2	Release	2024-07-04	150	2024-07-04	2024	7	4	Thursday	3	185	2024-08-03
3	Maintenance	2024-12-31	300	2024-12-31	2024	12	31	Tuesday	4	365	2025-01-30

In [105]:

df_dt[df_dt['date'].dt.month.isin([1, 7])]    # January and July only

Out[105]:

	event	date_str	value	date	year	month	day	weekday	quarter	days_since_start	follow_up
0	Launch	2024-01-15	100	2024-01-15	2024	1	15	Monday	1	14	2024-02-14
2	Release	2024-07-04	150	2024-07-04	2024	7	4	Thursday	3	185	2024-08-03

16. Pivot Tables and Reshaping¶

16.1 `pivot_table`¶

In [106]:

sales = pd.DataFrame({
    'rep':     ['Alice', 'Alice', 'Bob', 'Bob', 'Carol', 'Carol'],
    'region':  ['East', 'West', 'East', 'West', 'East', 'West'],
    'product': ['A', 'B', 'A', 'A', 'B', 'B'],
    'revenue': [120, 200, 150, 90, 180, 240],
    'units':   [3, 5, 4, 2, 6, 8],
})
sales

Out[106]:

	rep	region	product	revenue	units
0	Alice	East	A	120	3
1	Alice	West	B	200	5
2	Bob	East	A	150	4
3	Bob	West	A	90	2
4	Carol	East	B	180	6
5	Carol	West	B	240	8

In [107]:

# Average revenue per rep × region
pd.pivot_table(sales, values='revenue', index='rep', columns='region', aggfunc='sum', fill_value=0)

Out[107]:

region	East	West
rep
Alice	120	200
Bob	150	90
Carol	180	240

In [108]:

# Multiple aggregations
pd.pivot_table(sales,
               values=['revenue', 'units'],
               index='rep',
               aggfunc={'revenue': 'sum', 'units': 'mean'})

Out[108]:

	revenue	units
rep
Alice	320	4.0
Bob	240	3.0
Carol	420	7.0

16.2 `melt` — wide to long format¶

In [109]:

wide = pd.DataFrame({
    'name':  ['Alice', 'Bob', 'Carol'],
    'math':  [85, 92, 78],
    'english': [90, 76, 88],
    'science': [82, 95, 70],
})

long = wide.melt(id_vars='name', var_name='subject', value_name='score')
long

Out[109]:

	name	subject	score
0	Alice	math	85
1	Bob	math	92
2	Carol	math	78
3	Alice	english	90
4	Bob	english	76
5	Carol	english	88
6	Alice	science	82
7	Bob	science	95
8	Carol	science	70

16.3 `pivot` — long to wide format¶

In [110]:

long.pivot(index='name', columns='subject', values='score')

Out[110]:

subject	english	math	science
name
Alice	90	85	82
Bob	76	92	95
Carol	88	78	70

16.4 `stack` and `unstack`¶

In [111]:

# stack: move column labels to the row index (wide → long)
stacked = wide.set_index('name').stack()
stacked

Out[111]:

name          
Alice  math       85
       english    90
       science    82
Bob    math       92
       english    76
       science    95
Carol  math       78
       english    88
       science    70
dtype: int64

In [112]:

# unstack: move innermost row index level to columns (long → wide)
stacked.unstack()

Out[112]:

	math	english	science
name
Alice	85	90	82
Bob	92	76	95
Carol	78	88	70

17. Index Operations¶

17.1 Set and reset the index¶

In [113]:

emp_idx = emp.set_index('name')
emp_idx

Out[113]:

	dept	salary	years	rating	dept_avg	vs_avg
name
Alice	Eng	95000	5	4.2	105000.0	-10000.0
Bob	HR	72000	3	3.8	70000.0	2000.0
Carol	Eng	105000	8	4.5	105000.0	0.0
Dave	Finance	88000	6	4.0	90000.0	-2000.0
Eve	HR	68000	2	3.5	70000.0	-2000.0
Frank	Eng	115000	10	4.8	105000.0	10000.0
Grace	Finance	92000	4	4.1	90000.0	2000.0

In [114]:

emp_idx.reset_index()      # move index back to a regular column

Out[114]:

	name	dept	salary	years	rating	dept_avg	vs_avg
0	Alice	Eng	95000	5	4.2	105000.0	-10000.0
1	Bob	HR	72000	3	3.8	70000.0	2000.0
2	Carol	Eng	105000	8	4.5	105000.0	0.0
3	Dave	Finance	88000	6	4.0	90000.0	-2000.0
4	Eve	HR	68000	2	3.5	70000.0	-2000.0
5	Frank	Eng	115000	10	4.8	105000.0	10000.0
6	Grace	Finance	92000	4	4.1	90000.0	2000.0

17.2 Select rows by index label after `set_index`¶

In [115]:

emp_idx.loc['Alice']

Out[115]:

dept             Eng
salary         95000
years              5
rating           4.2
dept_avg    105000.0
vs_avg      -10000.0
Name: Alice, dtype: object

In [116]:

emp_idx.loc[['Alice', 'Carol', 'Eve']]

Out[116]:

	dept	salary	years	rating	dept_avg	vs_avg
name
Alice	Eng	95000	5	4.2	105000.0	-10000.0
Carol	Eng	105000	8	4.5	105000.0	0.0
Eve	HR	68000	2	3.5	70000.0	-2000.0

17.3 Multi-level (hierarchical) index¶

In [117]:

multi = emp.set_index(['dept', 'name'])
multi

Out[117]:

		salary	years	rating	dept_avg	vs_avg
dept	name
Eng	Alice	95000	5	4.2	105000.0	-10000.0
HR	Bob	72000	3	3.8	70000.0	2000.0
Eng	Carol	105000	8	4.5	105000.0	0.0
Finance	Dave	88000	6	4.0	90000.0	-2000.0
HR	Eve	68000	2	3.5	70000.0	-2000.0
Eng	Frank	115000	10	4.8	105000.0	10000.0
Finance	Grace	92000	4	4.1	90000.0	2000.0

In [118]:

multi.loc['Eng']            # all engineers

Out[118]:

	salary	years	rating	dept_avg	vs_avg
name
Alice	95000	5	4.2	105000.0	-10000.0
Carol	105000	8	4.5	105000.0	0.0
Frank	115000	10	4.8	105000.0	10000.0

In [119]:

multi.loc[('Eng', 'Alice')]  # single employee

Out[119]:

salary       95000.0
years            5.0
rating           4.2
dept_avg    105000.0
vs_avg      -10000.0
Name: (Eng, Alice), dtype: float64

18. Exporting DataFrames¶

In [120]:

# CSV
# df.to_csv('output.csv', index=False)

# Excel — requires openpyxl: pip install openpyxl
# df.to_excel('output.xlsx', sheet_name='Sheet1', index=False)

# JSON
# df.to_json('output.json', orient='records', indent=2)

# Parquet — requires pyarrow: pip install pyarrow
# df.to_parquet('output.parquet', index=False)

# SQL
# import sqlite3
# conn = sqlite3.connect('output.db')
# df.to_sql('table_name', conn, if_exists='replace', index=False)
# conn.close()

# Clipboard (paste into Excel / Google Sheets)
# df.to_clipboard(index=False)

19. Benchmarks¶

These benchmarks illustrate two fundamental pandas performance lessons:

Avoid Python-level row loops — iterrows / apply are far slower than vectorized operations
Prefer Parquet over CSV for repeated reads on large datasets

19.1 Setup¶

In [121]:

import time
import numpy as np
import pandas as pd

np.random.seed(0)

def bench(label, fn, repeats=3):
    times = []
    for _ in range(repeats):
        t0 = time.perf_counter()
        fn()
        times.append(time.perf_counter() - t0)
    print(f'{label:50s}  min={min(times)*1000:7.1f} ms  mean={sum(times)/len(times)*1000:7.1f} ms')

N = 500_000
df_bench = pd.DataFrame({
    'a': np.random.randn(N),
    'b': np.random.randn(N),
    'group': np.random.choice(['x', 'y', 'z'], size=N),
})
print(f'DataFrame: {N:,} rows × 3 columns')

DataFrame: 500,000 rows × 3 columns

19.2 Row iteration — `iterrows` vs `itertuples` vs vectorized¶

iterrows() deserializes each row into a Series — extremely slow. itertuples() yields lightweight named tuples — much faster, but still a Python loop. Vectorized operations run in C and are the right tool whenever possible.

In [122]:

# Use a smaller slice so iterrows doesn't take forever
df_small = df_bench.head(5_000).copy()

bench('iterrows  (5 K rows)',
      lambda: [row['a'] + row['b'] for _, row in df_small.iterrows()])

bench('itertuples (5 K rows)',
      lambda: [row.a + row.b for row in df_small.itertuples(index=False)])

bench('vectorized (5 K rows)',
      lambda: df_small['a'] + df_small['b'])

iterrows  (5 K rows)                                min=  201.2 ms  mean=  217.3 ms
itertuples (5 K rows)                               min=    2.5 ms  mean=    2.6 ms
vectorized (5 K rows)                               min=    0.1 ms  mean=    0.1 ms

In [123]:

# Scale up to show the full gap — vectorized on 500 K rows
bench('vectorized (500 K rows)',
      lambda: df_bench['a'] + df_bench['b'])

vectorized (500 K rows)                             min=    1.7 ms  mean=    2.0 ms

19.3 `apply` vs. vectorized arithmetic¶

apply with a Python lambda calls the function once per row/element. NumPy ufuncs and pandas arithmetic work on whole arrays at once.

In [ ]:

bench('apply  row-wise  (500 K rows)',
      lambda: df_bench.apply(lambda r: r['a'] * r['b'], axis=1))

bench('apply  element-wise (500 K rows)',
      lambda: df_bench['a'].apply(lambda x: x ** 2))

bench('vectorized multiply  (500 K rows)',
      lambda: df_bench['a'] * df_bench['b'])

bench('numpy square        (500 K rows)',
      lambda: df_bench['a'] ** 2)

19.4 GroupBy performance at scale¶

In [ ]:

bench('groupby mean  (500 K rows)',
      lambda: df_bench.groupby('group')['a'].mean())

bench('groupby agg   (500 K rows)',
      lambda: df_bench.groupby('group').agg({'a': ['mean', 'std'], 'b': 'sum'}))

19.5 CSV vs. Parquet read speed¶

Parquet is a columnar binary format. For repeated reads it is much faster than CSV because it skips parsing and can load only the columns you need.

In [ ]:

import os, tempfile

tmpdir = tempfile.mkdtemp()
csv_path     = os.path.join(tmpdir, 'data.csv')
parquet_path = os.path.join(tmpdir, 'data.parquet')

df_bench.to_csv(csv_path, index=False)
df_bench.to_parquet(parquet_path, index=False)

csv_mb     = os.path.getsize(csv_path)     / 1e6
parquet_mb = os.path.getsize(parquet_path) / 1e6
print(f'CSV:     {csv_mb:.1f} MB')
print(f'Parquet: {parquet_mb:.1f} MB')
print()

bench('read_csv     (500 K rows)',
      lambda: pd.read_csv(csv_path))

bench('read_parquet (500 K rows)',
      lambda: pd.read_parquet(parquet_path))

bench('read_parquet cols only (500 K rows)',
      lambda: pd.read_parquet(parquet_path, columns=['a']))

import shutil
shutil.rmtree(tmpdir)

19.6 Summary¶

Operation	Lesson
`iterrows`	Avoid — 100–1000× slower than vectorized
`itertuples`	Faster than `iterrows`, still a Python loop — avoid for large DataFrames
`apply` (row-wise)	Avoid — effectively an `iterrows` in disguise
`apply` (element-wise)	Slow — use `df['col'] ** 2` or `np.sqrt(df['col'])` instead
Vectorized / NumPy	Always prefer — operates at C speed
CSV vs. Parquet	Parquet is typically 3–10× faster to read and 2–5× smaller on disk

Licensed under the GNU General Public License v2.0 (GPL-2).
You may redistribute and/or modify this work under the terms of GPL-2.
This work is distributed WITHOUT ANY WARRANTY.
Happy computing.

icaoberg / Numerical computing - Intro to pandas

Numerical computing - Intro to pandas¶

1. Installation¶

2. Import¶

3. Creating DataFrames¶

3.1 From a dictionary¶

3.2 From a list of dicts¶

3.3 From a NumPy array¶

4. Reading DataFrames from Files¶

4.1 CSV¶

4.2 Excel¶

4.3 JSON¶

4.4 SQL¶

4.5 Parquet¶

5. Exploring a DataFrame¶

6. Accessing Columns¶

6.1 Single column — returns a Series¶

6.2 Multiple columns — returns a DataFrame¶

6.3 Column by position with iloc¶

7. Iterating Over Columns¶

7.1 Iterate column names¶

7.2 Iterate (name, Series) pairs with df.items()¶

7.3 Compute a statistic for every numeric column¶

7.4 Build a summary table from column iteration¶

7.5 Rename columns in a loop¶

7.6 Apply a function to every column with df.apply()¶

7.7 Filter rows using a column value¶

8. Selecting Rows¶

8.1 Single row and slices with loc¶

8.2 Position-based selection with iloc¶

8.3 Boolean indexing¶

8.4 query() — SQL-like filter expressions¶

9. Sorting¶

9.1 Sort by one column¶

9.2 Sort by multiple columns¶

9.3 Top / bottom N rows¶

9.4 Sort by index¶

10. Adding and Removing Columns¶

10.1 Add a column by direct assignment¶

10.2 Add a column with assign() — returns a new DataFrame, does not mutate¶

10.3 Rename columns¶

10.4 Drop columns¶

10.5 Drop rows¶

10.6 Replace values¶

10.7 Change column data type¶

11. Handling Missing Data¶

11.1 Detect missing values¶

11.2 Drop rows or columns with missing values¶

11.3 Fill missing values¶

11.4 Interpolate numeric columns¶

12. GroupBy and Aggregation¶

12.1 Single aggregation¶

12.2 Multiple aggregations at once¶

12.3 transform — broadcast group result back to original index¶

12.4 filter — keep only groups that meet a condition¶

12.5 Aggregate the whole DataFrame¶

13. Merging and Concatenating¶

13.1 merge() — SQL-style joins¶

13.2 concat() — stack DataFrames¶

14. String Operations¶

14.1 Case and whitespace¶

14.2 Contains, startswith, endswith¶

14.3 Extract substrings¶

14.4 Replace and split¶

15. DateTime Operations¶

15.1 Extract date parts¶

15.2 Date arithmetic¶

15.3 Generate date ranges¶

15.4 Filter by date¶

16. Pivot Tables and Reshaping¶

16.1 pivot_table¶

16.2 melt — wide to long format¶

16.3 pivot — long to wide format¶

16.4 stack and unstack¶

17. Index Operations¶

17.1 Set and reset the index¶

17.2 Select rows by index label after set_index¶

17.3 Multi-level (hierarchical) index¶

18. Exporting DataFrames¶

19. Benchmarks¶

7.2 Iterate (name, Series) pairs with `df.items()`¶

7.6 Apply a function to every column with `df.apply()`¶

8.1 Single row and slices with `loc`¶

8.2 Position-based selection with `iloc`¶

8.4 `query()` — SQL-like filter expressions¶

10.2 Add a column with `assign()` — returns a new DataFrame, does not mutate¶

12.3 `transform` — broadcast group result back to original index¶

12.4 `filter` — keep only groups that meet a condition¶

13.1 `merge()` — SQL-style joins¶

13.2 `concat()` — stack DataFrames¶

16.1 `pivot_table`¶

16.2 `melt` — wide to long format¶

16.3 `pivot` — long to wide format¶

16.4 `stack` and `unstack`¶

17.2 Select rows by index label after `set_index`¶

19.2 Row iteration — `iterrows` vs `itertuples` vs vectorized¶

19.3 `apply` vs. vectorized arithmetic¶