icaoberg / Dask Vs Numpy

Created Mon, 27 Apr 2026 00:00:00 +0000 Modified Mon, 27 Apr 2026 19:02:43 -0400

Jupyter Notebook python

Dask vs Numpy¶

This notebook benchmarks Dask against NumPy (CPU) for matrix multiplication and tabular data I/O. The goal is to understand when Dask's parallel, chunked execution offers a speed advantage over NumPy's eager, single-threaded operations.

In [3]:

%%capture
!pip install "dask[distributed]" -q
from dask.distributed import Client

client = Client()  # Start a local Dask client
client

INFO:distributed.scheduler:State start
INFO:distributed.scheduler:  Scheduler at:     tcp://127.0.0.1:35761
INFO:distributed.scheduler:  dashboard at:  http://127.0.0.1:37177/status
INFO:distributed.scheduler:Registering Worker plugin shuffle
INFO:distributed.nanny:        Start Nanny at: 'tcp://127.0.0.1:35791'
INFO:distributed.nanny:        Start Nanny at: 'tcp://127.0.0.1:36879'
INFO:distributed.scheduler:Register worker addr: tcp://127.0.0.1:42641 name: 1
INFO:distributed.scheduler:Starting worker compute stream, tcp://127.0.0.1:42641
INFO:distributed.core:Starting established connection to tcp://127.0.0.1:37292
INFO:distributed.scheduler:Register worker addr: tcp://127.0.0.1:40507 name: 0
INFO:distributed.scheduler:Starting worker compute stream, tcp://127.0.0.1:40507
INFO:distributed.core:Starting established connection to tcp://127.0.0.1:37296
INFO:distributed.scheduler:Receive client connection: Client-38319b6a-41f2-11f1-87db-0242ac1c000c
INFO:distributed.core:Starting established connection to tcp://127.0.0.1:37312

Dask Client and Dashboard¶

The Dask client coordinates the computations. The output above shows a link to the Dask dashboard, which provides real-time monitoring of tasks, memory usage, and CPU load across your Dask workers. This is invaluable for debugging and optimizing Dask workflows.

In [4]:

%%capture
!pip install dask tqdm seaborn faker

Setup¶

We install the core libraries needed for this notebook:

dask — parallel computing library that scales NumPy/Pandas workflows
tqdm — progress bars for loops
seaborn — statistical visualization built on Matplotlib
faker — generates realistic fake data for benchmarking

In [5]:

# import libraries
import numpy as np
import pandas as pd
from tqdm import tqdm, trange
import time
import dask.array as da

Imports¶

We import NumPy for standard array operations, Pandas for tabular data, tqdm for progress tracking, and dask.array as the Dask counterpart to NumPy arrays.

In [6]:

N = 100000
x = da.random.random((N,N),chunks=(1000, 1000))
x

Out[6]:

	Array	Chunk
Bytes	74.51 GiB	7.63 MiB
Shape	(100000, 100000)	(1000, 1000)
Dask graph	10000 chunks in 1 graph layer
Data type	float64 numpy.ndarray

Dask arrays and chunking¶

Here we create a 100,000 × 100,000 Dask array (about 80 GB if fully materialised). Dask never allocates all that memory at once — it divides the array into chunks of 1,000 × 1,000 elements and only computes each chunk when needed. The cell output shows the task graph metadata rather than the actual values.

Dask versus Numpy¶

In [7]:

SEED = 6174

def npmultiply(N=10):
    rng = np.random.default_rng(SEED)
    a = rng.random((N, N))
    b = rng.random((N, N))
    t1 = time.time()
    c = np.matmul(a, b)
    t2 = time.time()
    return t2 - t1

Benchmark functions¶

npmultiply(N) creates two N×N matrices with NumPy and times a single np.matmul call. A fixed random seed (SEED = 6174) ensures both libraries operate on equivalent data, making the timing comparison fair.

In [8]:

def daskmultiply(N=10):
    chunk = min(N, 1000)
    rng = da.random.RandomState(SEED)
    a = rng.random((N, N), chunks=(chunk, chunk))
    b = rng.random((N, N), chunks=(chunk, chunk))
    t1 = time.time()
    c = da.matmul(a, b).compute()
    t2 = time.time()
    return t2 - t1

daskmultiply(N) mirrors the NumPy version but uses dask.array. The key difference is the .compute() call at the end — Dask builds a lazy task graph first and only executes it when .compute() is invoked. The chunk size is capped at 1,000 so very small matrices don't have unnecessarily small chunks.

In [9]:

records = []
for N in tqdm(range(100, 2500, 10)):
    records.append({'N': N + 1, 'Dask': daskmultiply(N + 1), 'CPU': npmultiply(N + 1)})

df = pd.DataFrame(records)

100%|██████████| 240/240 [02:27<00:00,  1.63it/s]

Running the benchmark¶

We sweep matrix sizes from N=100 to N=2500 in steps of 10, recording wall-clock time for both daskmultiply and npmultiply at each size. Results are collected in a list of dicts and converted to a Pandas DataFrame for plotting.

In [10]:

import seaborn as sns
import matplotlib.pyplot as plt
sns.set(rc={"figure.figsize":(10, 10)}) #width=3, #height=4
sns.set_style('darkgrid')
sns.set_palette('Set2')
sns.lineplot(data=df, x='N', y='Dask', label='Dask')
sns.lineplot(data=df, x='N', y='CPU', label='CPU')
plt.title('Dask versus Numpy')
plt.xlabel('Square matrix dimension (N)')
plt.ylabel('Time (seconds)')
plt.legend(loc="upper left")
plt.show()

No description has been provided for this image

Results¶

The line plot shows elapsed time (seconds) vs. matrix dimension for Dask and NumPy. For small matrices, Dask is slower because task-graph scheduling overhead dominates. As N grows, Dask's chunked parallel execution closes the gap — and can eventually outperform NumPy when matrices no longer fit comfortably in cache or when multiple CPU cores are engaged.

Large datasets¶

In [11]:

from faker import Faker
fake = Faker()
Faker.seed(1234567890)

file = open('fake_dataset.tsv', 'w')
#from header remove address
file.write('SSN\tName\tPhone number\tCompany\tBank\tCredit Card\tCredit Card Expiration\tCredit Card Provider\n')

# not really large
for n in tqdm(range(25000)):
    ssn = fake.ssn()
    name = fake.name()
    address = fake.address() #ignore
    phone_number = fake.phone_number()
    company = fake.company()
    bank = fake.company()
    credit_card = fake.credit_card_number()
    credit_card_expiration = fake.credit_card_expire()
    credit_card_provider = fake.credit_card_provider()

    #from here remove {8} and address
    file.write('{0}\t{1}\t{2}\t{3}\t{4}\t{5}\t{6}\t{7}\n'.format(ssn, name, phone_number,company,bank,credit_card,credit_card_expiration,credit_card_provider))
file.close()

100%|██████████| 25000/25000 [00:18<00:00, 1386.30it/s]

Generating a synthetic dataset¶

We use faker to generate 25,000 rows of realistic-looking but entirely fake personal records (SSN, name, phone, company, bank, credit card). This small dataset warms up the I/O benchmark before we move on to a much larger file.

In [12]:

file = 'fake_dataset.tsv'

def load_data( file ):
    t1 = time.time()
    df = pd.read_csv( file, sep='\t' )
    t2 = time.time()
    total = t2-t1
    print(total)
    return df

df = load_data( file )

0.055648088455200195

Loading with Pandas vs Dask¶

load_data reads the TSV into a Pandas DataFrame. All rows are parsed and loaded into memory immediately, and we time the entire read operation.

load_data2 uses dask.dataframe. This function reads only the file metadata and builds a task graph; the actual data is loaded lazily only when an operation explicitly calls .compute().

In [13]:

import dask.dataframe as dd

file = 'fake_dataset.tsv'

def load_data2(file):
    t1 = time.time()
    df = dd.read_csv(file, sep='\t')
    t2 = time.time()
    print(t2 - t1)
    return df

df2 = load_data2(file)

0.028325557708740234

Large datasets¶

Now we repeat the I/O and query benchmark on a much larger file (data.tsv, ~175 MB, ~1.5 M rows) to show where Dask's lazy evaluation offers a more significant advantage. OpenOnDemand users can run this section on an HPC cluster to leverage multiple cores.

In [14]:

Faker.seed(42)
large_file = open('data.tsv', 'w')
large_file.write('SSN\tName\tPhone number\tCompany\tBank\tCredit Card\tCredit Card Expiration\tCredit Card Provider\n')
for n in tqdm(range(150000)):
    large_file.write('{0}\t{1}\t{2}\t{3}\t{4}\t{5}\t{6}\t{7}\n'.format(
        fake.ssn(), fake.name(), fake.phone_number(), fake.company(),
        fake.company(), fake.credit_card_number(), fake.credit_card_expire(), fake.credit_card_provider()
    ))
large_file.close()

100%|██████████| 150000/150000 [01:24<00:00, 1771.70it/s]

In [15]:

!du -h data.tsv

17M	data.tsv

In [16]:

!wc -l data.tsv

150001 data.tsv

In [17]:

# load dataframe using Pandas

file = 'data.tsv'
df = load_data( file )

0.3280353546142578

Pandas takes ~3 seconds to read the full 175 MB file into memory. Notice how much longer this is compared to the small 25 k-row file (~0.07 s).

In [18]:

file = 'data.tsv'
df2 = load_data2(file)

0.011517524719238281

Dask's read_csv returns almost instantly because it only reads the file header to infer schema and partition boundaries — no data is loaded yet.

In [19]:

df.columns

Out[19]:

Index(['SSN', 'Name', 'Phone number', 'Company', 'Bank', 'Credit Card',
       'Credit Card Expiration', 'Credit Card Provider'],
      dtype='object')

In [20]:

df[df['Name'].str.contains('Robert')]

Out[20]:

	SSN	Name	Phone number	Company	Bank	Credit Card	Credit Card Expiration	Credit Card Provider
10	392-05-7712	Robert Stevens	733-203-6541x4586	Wiley LLC	Carpenter LLC	502096556985	06/27	JCB 16 digit
24	045-41-7750	Roberta Hughes	+1-427-802-8951x71870	Booker, Jones and Harrington	Morgan-Schwartz	4586578091343169	02/36	VISA 16 digit
36	568-95-0673	Robert Pearson	997-982-0715x18203	Mills, Moore and Watson	Hughes-Schroeder	4515186449251922	06/35	VISA 19 digit
99	837-31-1292	Robert West	375.617.0166x0880	Hart, Pena and Bryant	Cox-Brown	4589975711308	09/29	VISA 16 digit
173	248-65-9343	Robert Hines	933.342.8013x87592	Davenport Group	Lewis-Perez	2271718275480155	04/31	Discover
...	...	...	...	...	...	...	...	...
149519	532-30-6748	Robert Gonzales	+1-477-665-3672x1445	Baker Group	Evans and Sons	4202756528368	06/28	American Express
149681	221-30-5796	Tanya Robertson	6366028680	Jennings-Perez	Johnson-Ritter	30108261221829	09/32	JCB 16 digit
149689	451-28-8165	Robert Williams	683.936.2559	Barnes, Moyer and Mosley	Hahn, Jacobson and Moon	374275007145176	10/35	JCB 16 digit
149696	287-94-0515	Robert Mcdaniel	461.940.2883x759	Galvan-Gonzalez	Stewart PLC	4012735862350032914	01/31	VISA 13 digit
149811	322-59-0909	Robert Ford	550-443-7705	Walker-Gonzalez	Smith Group	3551585152552318	04/32	JCB 15 digit

2865 rows × 8 columns

In [21]:

df2[df2['Name'].str.contains('Robert')] #map

Out[21]:

Dask DataFrame Structure:

	SSN	Name	Phone number	Company	Bank	Credit Card	Credit Card Expiration	Credit Card Provider
npartitions=1
	string	string	string	string	string	int64	string	string
	...	...	...	...	...	...	...	...

Dask Name: getitem, 5 expressions

Lazy vs. eager filtering¶

Filtering the Pandas DataFrame (df[...]) executes immediately and returns the matching rows. In contrast, filtering the Dask DataFrame (df2[...]) just appends a filter step to its internal task graph — you see the schema but no actual data is processed yet. Calling .compute() on the next line triggers the actual scan and returns the corresponding result set.

In [22]:

df2[df2['Name'].str.contains('Robert')].compute() #compute

Out[22]:

	SSN	Name	Phone number	Company	Bank	Credit Card	Credit Card Expiration	Credit Card Provider
10	392-05-7712	Robert Stevens	733-203-6541x4586	Wiley LLC	Carpenter LLC	502096556985	06/27	JCB 16 digit
24	045-41-7750	Roberta Hughes	+1-427-802-8951x71870	Booker, Jones and Harrington	Morgan-Schwartz	4586578091343169	02/36	VISA 16 digit
36	568-95-0673	Robert Pearson	997-982-0715x18203	Mills, Moore and Watson	Hughes-Schroeder	4515186449251922	06/35	VISA 19 digit
99	837-31-1292	Robert West	375.617.0166x0880	Hart, Pena and Bryant	Cox-Brown	4589975711308	09/29	VISA 16 digit
173	248-65-9343	Robert Hines	933.342.8013x87592	Davenport Group	Lewis-Perez	2271718275480155	04/31	Discover
...	...	...	...	...	...	...	...	...
149519	532-30-6748	Robert Gonzales	+1-477-665-3672x1445	Baker Group	Evans and Sons	4202756528368	06/28	American Express
149681	221-30-5796	Tanya Robertson	6366028680	Jennings-Perez	Johnson-Ritter	30108261221829	09/32	JCB 16 digit
149689	451-28-8165	Robert Williams	683.936.2559	Barnes, Moyer and Mosley	Hahn, Jacobson and Moon	374275007145176	10/35	JCB 16 digit
149696	287-94-0515	Robert Mcdaniel	461.940.2883x759	Galvan-Gonzalez	Stewart PLC	4012735862350032914	01/31	VISA 13 digit
149811	322-59-0909	Robert Ford	550-443-7705	Walker-Gonzalez	Smith Group	3551585152552318	04/32	JCB 15 digit

2865 rows × 8 columns

Conclusion¶

This notebook demonstrates the trade-offs between Dask and NumPy/Pandas for different computational tasks:

Matrix Multiplication (Dask vs NumPy)¶

Small N: For smaller matrix dimensions (N), Dask introduces overhead from task scheduling, making it slower than NumPy's eager execution.
Large N: As N increases, Dask's ability to parallelize and chunk computations allows it to close the performance gap and potentially outperform NumPy, especially when matrices no longer fit into cache or when leveraging multiple CPU cores.

Large Dataset I/O and Filtering (Dask vs Pandas)¶

Lazy Loading (Dask): Dask's read_csv returns almost instantly because it performs lazy loading, meaning it only reads metadata and builds a task graph without loading the entire dataset into memory immediately. The actual computation occurs only when a .compute() call is made.
Eager Loading (Pandas): Pandas' read_csv loads the entire dataset into memory upfront, which can be significantly slower for large files.
Lazy Filtering (Dask): Similar to loading, Dask DataFrame filtering operations also build a task graph and execute lazily. This provides a performance advantage for large datasets where only a subset of data needs to be processed, as Dask can optimize the computation plan.
Eager Filtering (Pandas): Pandas DataFrame filtering executes immediately, which can be inefficient for large datasets if the entire DataFrame needs to be scanned repeatedly.

In summary, Dask proves beneficial for larger-than-memory datasets and computationally intensive tasks by leveraging parallel processing and lazy evaluation, while NumPy and Pandas remain efficient for smaller datasets that fit comfortably in memory.