Lecture 10 - Scientific Python and Visualization

Programming for Engineers

1 Scientific Python, Data Processing and Visualization

1.1 Overview

This lecture is a showcase of practical uses of Python for scientific computing and data visualization. We will cover:

  1. Scientific Python processing: numerical arrays, tabular data, scientific routines, preprocessing, and image-processing basics.
  2. Output and visualization: terminal-oriented feedback, plotting, graph visualization, and GUI/browser-based visual tools.

It should give you an idea on how to use Python and your programming skills to solve problems in a scientific/engineering context.

Here is an overview of generally useful Python packages:

  • NumPy - useful for general numerical/array data handling
  • Pandas - working with tables (like Excel on steroids but better and can also load Excel files and many more)
  • SciPy - contains well-known numerical/scientific algorithms, tightly integrated with NumPy arrays
  • scikit-learn - basically SciPy for machine learning
  • scikit-image - image processing and analysis
  • Matplotlib / Seaborn / Plotly - plotting, graphing, and visualization

2 Part 1: Scientific Python Processing

2.1 Package Overview

2.1.1 NumPy: Numerical Data Handling

NumPy (Numerical Python) is the foundational package for numerical computing in Python. We have covered NumPy in a specific lecture, so we will not cover it in detail here. Just be aware that NumPy contains many more useful operations and data structures for computation and array manipulation.

2.1.2 SciPy: Scientific Computations

%pip install scipy

SciPy is a library used for scientific and technical computing. It builds on NumPy by adding a collection of algorithms and high-level commands for data manipulation and analysis. SciPy includes modules for optimization, integration, interpolation, eigenvalue problems, algebraic and differential equations, and others, making it a powerful tool for scientific applications.

2.1.3 Pandas: Tabular Data Processing

%pip install pandas

Pandas is a library for data manipulation and analysis. It offers data structures and operations for manipulating numerical tables and time series. Pandas introduces two new data structures to Python: Series and DataFrame, which are built on top of NumPy arrays. These structures allow for fast and efficient data manipulation.

To use Pandas with Excel files, you need to install the openpyxl package.

%pip install openpyxl

2.1.4 scikit-learn: Machine Learning Preprocessing

%pip install scikit-learn

scikit-learn is a machine learning library for the Python programming language. It features various classification, regression, and clustering algorithms, including support-vector machines, random forests, gradient boosting, k-means, DBSCAN, and many preprocessing/model-selection utilities. Designed to interoperate with the Python numerical and scientific libraries NumPy and SciPy, scikit-learn is widely used for its simplicity and efficiency in implementing machine learning models.

2.1.5 scikit-image: Image Processing

%pip install scikit-image

scikit-image is a collection of algorithms for image processing. It is designed to interoperate with NumPy and SciPy, providing a versatile toolkit for image analysis. scikit-image includes algorithms for segmentation, geometric transformations, color space manipulation, analysis, filtering, morphology, feature detection, and more. It is widely used in academic research and industry for processing and analyzing images.

2.2 NumPy: Numerical Data Handling

2.2.1 Loading Data

We have covered NPY and NPZ loading in a previous lecture. It is also possible to load structured text (CSV) files using NumPy.

import numpy as np

# Load data from a text file
data = np.loadtxt('data.csv', delimiter=',')

# Save array to a file
np.savetxt('output.csv', data, delimiter=',')

2.2.2 Practical note

NumPy can read tabular text data, but it works best when the data is strictly numeric and relatively regular. As soon as the table contains mixed types, missing values, headers, or date columns, Pandas usually becomes the better choice.

2.3 Pandas: Tabular Data Processing

Even though NumPy can load CSV data, it is best to use the library dedicated to loading and processing of tabular data: Pandas.

import pandas as pd

2.3.1 Loading Data

df_csv = pd.read_csv('sales.csv', parse_dates=['order_date'])
df_excel = pd.read_excel('sales.xlsx')

2.3.2 Creating tables

# From dictionary
data = {
    'Name': ['Alice', 'Bob', 'Charlie'],
    'Age': [25, 30, 35],
    'Score': [85, 92, 78]
}

df = pd.DataFrame(data)
print(df)
      Name  Age  Score
0    Alice   25     85
1      Bob   30     92
2  Charlie   35     78

2.3.3 Data Exploration

print("Display first few rows")
print(df_csv.head())
print("--" * 20)

print("Display last few rows")
print(df_csv.tail())
print("--" * 20)

print("Display summary statistics")
df_csv.info()
print("--" * 20)

print("Data types")
print(df_csv.dtypes)
print("--" * 20)

print("Summary statistics")
print(df_csv.describe())
print("--" * 20)

print("Unique values")
print(df_csv.nunique())
print("--" * 20)

print("Single column stats")
print("Mean sales:", df_csv['sales'].mean())
print("--" * 20)
Display first few rows
    order_id order_date     category region  sales  quantity  returned
0  ORD100083 2023-03-25        Books  South  19.06        14     False
1  ORD100366 2024-01-02     Clothing   East  30.88        11     False
2  ORD100564 2024-07-18        Books  South  10.31        11     False
3  ORD100490 2024-05-05        Books   East  20.66        11     False
4  ORD100507 2024-05-22  Electronics  South  19.22        13     False
----------------------------------------
Display last few rows
      order_id order_date  category region  sales  quantity  returned
995  ORD100387 2024-01-23     Books  North  11.48         5     False
996  ORD100324 2023-11-21     Books   East  28.33         3     False
997  ORD100861 2025-05-11     Books  South  37.67         6     False
998  ORD100708 2024-12-09  Clothing  North  13.89         6     False
999  ORD100484 2024-04-29  Clothing  North  16.61        18     False
----------------------------------------
Display summary statistics
<class 'pandas.DataFrame'>
RangeIndex: 1000 entries, 0 to 999
Data columns (total 7 columns):
 #   Column      Non-Null Count  Dtype         
---  ------      --------------  -----         
 0   order_id    1000 non-null   str           
 1   order_date  1000 non-null   datetime64[us]
 2   category    1000 non-null   str           
 3   region      1000 non-null   str           
 4   sales       1000 non-null   float64       
 5   quantity    1000 non-null   int64         
 6   returned    1000 non-null   bool          
dtypes: bool(1), datetime64[us](1), float64(1), int64(1), str(3)
memory usage: 68.0 KB
----------------------------------------
Data types
order_id                 str
order_date    datetime64[us]
category                 str
region                   str
sales                float64
quantity               int64
returned                bool
dtype: object
----------------------------------------
Summary statistics
                order_date        sales     quantity
count                 1000  1000.000000  1000.000000
mean   2024-05-14 12:00:00    21.866070     9.956000
min    2023-01-01 00:00:00     4.340000     1.000000
25%    2023-09-07 18:00:00    13.725000     5.000000
50%    2024-05-14 12:00:00    19.770000    10.000000
75%    2025-01-19 06:00:00    26.900000    15.000000
max    2025-09-26 00:00:00    86.240000    19.000000
std                    NaN    11.861339     5.346649
----------------------------------------
Unique values
order_id      1000
order_date    1000
category         4
region           4
sales          870
quantity        19
returned         2
dtype: int64
----------------------------------------
Single column stats
Mean sales: 21.86607
----------------------------------------

2.3.4 Accessing Data

print("Column:", df['Age'])                        # Column
print("Multiple columns:", df[['Name', 'Score']]) # Multiple columns
print("Row by index:", df.iloc[1])                # Row by index
print("Row by label:", df.loc[1])                 # Row by label
print("Rows by index:", df.iloc[1:3])             # Rows by index
print("Rows by label:", df.loc[1:3])              # Rows by label
print("Rows by condition:", df[df['Age'] > 30])   # Rows by condition
print("Column by index:", df.iloc[:, 1])          # Column by index
print("Column by label:", df['Age'])              # Column by label
Column: 0    25
1    30
2    35
Name: Age, dtype: int64
Multiple columns:       Name  Score
0    Alice     85
1      Bob     92
2  Charlie     78
Row by index: Name     Bob
Age       30
Score     92
Name: 1, dtype: object
Row by label: Name     Bob
Age       30
Score     92
Name: 1, dtype: object
Rows by index:       Name  Age  Score
1      Bob   30     92
2  Charlie   35     78
Rows by label:       Name  Age  Score
1      Bob   30     92
2  Charlie   35     78
Rows by condition:       Name  Age  Score
2  Charlie   35     78
Column by index: 0    25
1    30
2    35
Name: Age, dtype: int64
Column by label: 0    25
1    30
2    35
Name: Age, dtype: int64

2.3.5 Filtering and selection

# Filter by condition
print("Filter by condition:", df[df['Age'] >= 30])

print("Filter by multiple conditions:", df[(df['Age'] >= 30) & (df['Score'] < 90)])
Filter by condition:       Name  Age  Score
1      Bob   30     92
2  Charlie   35     78
Filter by multiple conditions:       Name  Age  Score
2  Charlie   35     78

2.3.6 Grouping and aggregation

df_csv.groupby('region').agg({'sales': 'sum'})
df_csv.groupby('category').agg({'sales': ['mean', 'count']})
sales
mean count
category
Books 21.376166 253
Clothing 22.035588 238
Electronics 22.467233 253
Home 21.598516 256

2.3.7 Saving Data

# Save to CSV
df.to_csv('cleaned_data.csv', index=False)

# Save to Excel
df.to_excel('cleaned_data.xlsx', index=False)

2.3.8 Notes

  • iloc is position-based indexing, while loc is label-based indexing.
  • Pandas operations often return a new object rather than modifying data in place.

2.4 SciPy: Scientific Computations

SciPy is a package containing well-known numerical methods built on top of NumPy arrays.

2.4.1 optimize: Function Minimization, Root Finding, and Curve Fitting

The scipy.optimize package offers algorithms for function minimization (scalar or multi-dimensional), root-finding, and curve-fitting.

2.4.1.1 Key Functions & Classes

  • scipy.optimize.minimize: General-purpose minimization of scalar functions of one or more variables.
  • scipy.optimize.curve_fit: Non-linear least squares fitting of a function to data.
  • scipy.optimize.root: Find roots of a function.
  • scipy.optimize.least_squares: Solve nonlinear least-squares with bounds.

2.4.1.2 Example: Minimizing a Non-Convex Function

import numpy as np
from scipy.optimize import minimize
from matplotlib import pyplot as plt


def f(x):
    return x**2 + 10*np.sin(x)

res = minimize(f, x0=0.0, method='BFGS')
print("Minimum at x =", res.x, "with value f(x) =", res.fun)

x_vals = np.linspace(res.x - 5, res.x + 5, 100)
y_vals = f(x_vals)

plt.plot(x_vals, y_vals)
plt.scatter(res.x, f(res.x), color='red')
plt.show()
Minimum at x = [-1.30644012] with value f(x) = -7.945823375615215

2.4.1.3 Example: Curve Fitting

import numpy as np
from scipy.optimize import curve_fit
import matplotlib.pyplot as plt

# Generate synthetic data
x = np.linspace(0, 10, 50)
y = 3.5 * np.sin(1.3 * x) + np.random.normal(0, 0.5, x.size)

# Define model
def model(x, a, b):
    return a * np.sin(b * x)

# Fit parameters
params, cov = curve_fit(model, x, y)
print("Fitted params:", params)

# Plot
plt.scatter(x, y, label='Data')
plt.plot(x, model(x, *params), 'r-', label='Fit')
plt.legend()
plt.show()
Fitted params: [3.49496714 1.29733859]

2.4.2 integrate: Numerical Integration and ODE Solvers

scipy.integrate provides functions to compute definite integrals, solve ordinary differential equations (ODEs), and perform multi-dimensional integration.

2.4.2.1 Key Functions

  • scipy.integrate.quad: Adaptive quadrature for single integrals.
  • scipy.integrate.solve_ivp: ODE solver interface.

2.4.2.2 Example: Definite Integral

from scipy.integrate import quad

f = lambda t: np.exp(-t**2)
result, error = quad(f, 0, np.inf)
print("Integral of exp(-t^2) from 0 to ∞ =", result)
Integral of exp(-t^2) from 0 to ∞ = 0.8862269254527579

2.4.3 interpolate: Data Interpolation

scipy.interpolate offers classes and functions for one- and multi-dimensional interpolation and smoothing splines.

2.4.3.1 Key Classes & Functions

  • scipy.interpolate.interp1d: 1-D interpolation (now deprecated but still works).
  • scipy.interpolate.CubicSpline: Interpolation.
  • scipy.interpolate.griddata: Interpolation over irregular 2-D data.
  • scipy.interpolate.BarycentricInterpolator, UnivariateSpline, RectBivariateSpline.

2.4.3.2 Deprecation note

interp1d is still widely seen in existing code, but for new code the correct approach is using CubicSpline or PchipInterpolator from scipy.interpolate.

2.4.3.3 Example: 1-D Interpolation

from scipy.interpolate import interp1d

# Original coarse data
x_raw = np.linspace(0, 10, 10)
y_raw = np.sin(x_raw)

# Create interpolator
f_linear = interp1d(x_raw, y_raw, kind='linear')
f_cubic = interp1d(x_raw, y_raw, kind='cubic')

# Evaluate at finer grid
x_fine = np.linspace(0, 10, 100)
plt.plot(x_raw, y_raw, 'o', label='Raw data points')
plt.plot(x_fine, f_linear(x_fine), '-', label='Linear interpolation')
plt.plot(x_fine, f_cubic(x_fine), '--', label='Cubic interpolation')
plt.legend()
plt.show()

“Modern” version of interpolation using scipy.interpolate.CubicSpline:

from scipy.interpolate import CubicSpline

# Original coarse data
x_raw = np.linspace(0, 10, 10)
y_raw = np.sin(x_raw)

# Create interpolator
f_cubic = CubicSpline(x_raw, y_raw)

# Evaluate at finer grid
x_fine = np.linspace(0, 10, 100)
plt.plot(x_raw, y_raw, 'o', label='Raw data points')
plt.plot(x_fine, f_cubic(x_fine), '--', label='Cubic interpolation')
plt.legend()
plt.show()

2.4.4 signal: Digital Signal Processing

scipy.signal provides signal processing tools: filtering, spectral analysis, window functions, and convolution.

2.4.4.1 Key Functions

  • scipy.signal.butter, scipy.signal.sosfilt / sosfiltfilt: Digital filter design and application.
  • scipy.signal.welch: Power spectral density estimation.
  • scipy.signal.convolve, correlate, decimate, resample.

2.4.4.2 Example: Butterworth Low-Pass Filter

from scipy.signal import butter, sosfiltfilt, convolve

# Design filter
sos = butter(4, 0.2, btype='low', output='sos')
# Create noisy signal
t = np.linspace(0, 1, 500)
sig = np.sin(2*np.pi*5*t) + 0.5*np.random.randn(500)
# Apply zero-phase filter
filtered = sosfiltfilt(sos, sig)

# Convolution with Gaussian kernel
kx = np.arange(-4, 5)
kernel = np.exp(-kx**2 / 2)
kernel /= kernel.sum()
print(kernel)
smoothed = convolve(sig, kernel, mode='same')

plt.plot(t, sig, alpha=0.5, label='Noisy')
plt.plot(t, filtered, 'r-', label='Filtered')
plt.plot(t, smoothed, 'g--', label='Smoothed')
plt.legend()
plt.show()
[1.33830625e-04 4.43186162e-03 5.39911274e-02 2.41971446e-01
 3.98943469e-01 2.41971446e-01 5.39911274e-02 4.43186162e-03
 1.33830625e-04]

2.4.5 fft: Fourier Transforms

scipy.fft contains Fast Fourier Transform routines for one- and multi-dimensional arrays.

2.4.5.1 Key Functions

  • scipy.fft.fft, ifft: Forward and inverse 1-D FFT.
  • scipy.fft.fft2, ifft2: 2-D transforms.
  • rfft, irfft: Real-input optimized transforms.

2.4.5.2 Theory note

An FFT converts a signal from the time domain (or spatial domain for, e.g., images) to the frequency domain. This is useful when the signal appears complicated in time, but is composed of a few strong frequencies. Frequency analysis using FFT is a very common technique in electrical engineering.

2.4.5.3 Example: 1-D FFT Spectral Analysis

from scipy.fft import fft, fftfreq

# Signal
t = np.linspace(0, 1, 400)
x = np.sin(2*np.pi*50*t) + 0.5*np.sin(2*np.pi*120*t) + 0.77*np.sin(2*np.pi*77*t)
x += 0.3*np.random.randn(t.size)
# Compute FFT
X = fft(x)
freqs = fftfreq(t.size, d=t[1]-t[0])
fig, [ax1, ax2] = plt.subplots(2, 1)
ax1.plot(t, x)
ax1.set_title("Time Domain")
ax2.plot(freqs[:200], np.abs(X)[:200])
ax2.set_title("Magnitude Spectrum")
ax2.set_xlabel("Frequency (Hz)")
ax2.set_ylabel("Amplitude")
plt.tight_layout()
plt.show()

2.4.6 stats: Statistical Functions and Tests

scipy.stats provides a large collection of statistical distributions, descriptive statistics, and hypothesis tests.

2.4.6.1 Key Functions & Classes

  • scipy.stats.norm, gamma, …: Continuous distributions.
  • scipy.stats.ttest_ind, ttest_rel: T-tests for independent and related samples.
  • scipy.stats.pearsonr, spearmanr: Correlation coefficients.
  • scipy.stats.kstest, chisquare: Goodness-of-fit tests.

2.4.6.2 Example: Two-Sample T-Test

from scipy.stats import ttest_ind

# Generate two samples
a = np.random.randn(100) + 0.5
b = np.random.randn(100)
stat, p = ttest_ind(a, b)
print("t-statistic =", stat, "p-value =", p)
t-statistic = 3.6559317807596217 p-value = 0.00032830207387421695

2.4.7 sparse: Sparse Matrix Tools

scipy.sparse supports sparse matrix representations for memory-efficient storage and fast arithmetics on large, sparse arrays.

2.4.7.1 Key Classes

  • lil_matrix, csr_matrix, csc_matrix, coo_matrix.
  • Methods: .dot(), .tocsc(), .toarray().

2.4.7.2 Theory note

Sparse representations matter when most matrix entries are zero. In such cases, dense storage wastes both memory and time. This appears often in graph problems (with a small edge to node ratio), text vectorization (e.g., one-hot encoding, often used with LLMs/transformers), etc.

2.4.8 spatial: KD-Tree for Nearest Neighbors

scipy.spatial.KDTree provides efficient nearest-neighbor searches in k-dimensional space.

2.4.8.1 Key Methods

  • query, query_ball_point, query_pairs.

2.4.8.2 Example: Nearest-Neighbor Query

from scipy.spatial import KDTree

points = np.random.rand(100, 2)
tree = KDTree(points)
dist, idx = tree.query([0.5, 0.5], k=5)
print("Nearest indices:", idx)

# Plot points and nearest neighbors
plt.scatter(points[:, 0], points[:, 1])
plt.scatter([0.5], [0.5], c='r')
plt.scatter(points[idx, 0], points[idx, 1], c='g')
plt.show()
Nearest indices: [28 50 41 33 49]

Speed test vs. “linear search”:

# This function computes pair-wise distances, sorts them and returns indices of k closest points
def nearest(p, P, k=5):
    return np.argsort(np.linalg.norm(np.array(p) - P, axis=1))[:k]

P = np.random.rand(1000, 2)
p = np.array([0.5, 0.5])
kdtree = KDTree(P)

%timeit kdtree = KDTree(P)
%timeit nearest(p, P, k=5)
%timeit kdtree.query(p, k=5)
189 μs ± 1.89 μs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
52.8 μs ± 3.44 μs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
25.5 μs ± 3.11 μs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

2.4.9 Further Reading

  • Official SciPy Reference: https://docs.scipy.org/doc/scipy/reference/
  • “SciPy Lecture Notes” for deeper dives: https://scipy-lectures.org/

2.5 scikit-learn: Preprocessing, Model Selection, and Basic Estimators

scikit-learn provides a unified interface to many machine-learning algorithms and data tools from preprocessing to model selection and evaluation.

2.5.1 Preprocessing sklearn.preprocessing

2.5.1.1 Scaling and Normalization

  • StandardScaler: Centers features to zero mean and unit variance. Critical for algorithms assuming similarly scaled features (e.g. many linear models, SVMs, PCA-based workflows).
  • MinMaxScaler: Scales features to a fixed range [0, 1], preserving the shape of the original distribution but remaining sensitive to outliers.
  • RobustScaler: Uses median and IQR (inter-quartile range) for centering/scaling, making it more robust to outliers.
import numpy as np
from matplotlib import pyplot as plt
from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler

N = 200
input_data = np.random.randn(N, 1)**2 * 100 - 50
sc1 = StandardScaler().fit(input_data)
data_std = sc1.transform(input_data)
sc2 = MinMaxScaler().fit(input_data)
data_mm = sc2.transform(input_data)
sc3 = RobustScaler().fit(input_data)
data_rb = sc3.transform(input_data)

# Plot
x_vals = np.arange(N)
fig, ax = plt.subplots(4, 1, figsize=(16, 8))
ax[0].scatter(x_vals, input_data)
ax[1].scatter(x_vals, data_std)
ax[2].scatter(x_vals, data_mm)
ax[3].scatter(x_vals, data_rb)
ax[0].set_title('Original Data')
ax[1].set_title('Standard Scaler')
ax[2].set_title('MinMax Scaler')
ax[3].set_title('Robust Scaler')
plt.tight_layout()
plt.show()

2.5.2 Data Splitting and Model Selection sklearn.model_selection

  • train_test_split: Quick split for train/test sets.
  • KFold, StratifiedKFold: Cross-validation iterators, with stratified variants better suited to label-preserving classification tasks.
from sklearn.model_selection import train_test_split, KFold
X = np.arange(100).reshape(10, 10)
y = np.arange(10)

X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=0)
kf = KFold(n_splits=5, shuffle=True, random_state=0)

for train_index, test_index in kf.split(X, y):
    print("TRAIN:", train_index, "TEST:", test_index)
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
TRAIN: [0 1 3 4 5 6 7 9] TEST: [2 8]
TRAIN: [0 1 2 3 5 6 7 8] TEST: [4 9]
TRAIN: [0 2 3 4 5 7 8 9] TEST: [1 6]
TRAIN: [0 1 2 4 5 6 8 9] TEST: [3 7]
TRAIN: [1 2 3 4 6 7 8 9] TEST: [0 5]

2.5.3 Estimators

2.5.3.1 Linear Models

  • LinearRegression: Ordinary Least Squares regression.
  • LogisticRegression: Regularized logistic classifier; supports l1/l2 penalties depending on the solver.
from sklearn.linear_model import LinearRegression, LogisticRegression

X_train = np.arange(100).reshape(10, 10)
y_tr_reg = np.arange(10)
y_tr_clf = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1, 1])

lr = LinearRegression().fit(X_train, y_tr_reg)
logr = LogisticRegression().fit(X_train, y_tr_clf)

print("Linear Regression intercept:", lr.intercept_)
print("Linear predictions:")
print([f"{v:.2f}" for v in lr.predict(X_train + np.random.randn(10, 10) * 0.5)])
print("Actual values:")
print(y_tr_reg)

print("Logistic Regression intercept:", logr.intercept_)
print("Logistic predictions:")
print(logr.predict(X_train + np.random.randn(10, 10) * 0.5))
print("Actual values:")
print(y_tr_clf)
Linear Regression intercept: -0.4499999999999993
Linear predictions:
['-0.04', '1.00', '2.00', '3.02', '4.00', '5.02', '5.98', '6.99', '7.99', '8.99']
Actual values:
[0 1 2 3 4 5 6 7 8 9]
Logistic Regression intercept: [-36.85911953]
Logistic predictions:
[0 0 0 0 1 1 1 1 1 1]
Actual values:
[0 0 0 0 1 1 1 1 1 1]

2.5.3.2 Clustering

  • KMeans: Centroid-based clustering.
  • DBSCAN: Density-based clustering; identifies arbitrarily shaped clusters.
  • HDBSCAN: Hierarchical density-based clustering; useful when cluster density may vary.

“Nice” data (gaussian clusters):

from sklearn.cluster import KMeans, HDBSCAN
X = np.vstack([
    (np.random.randn(100, 2)) * 20 + 50,
    (np.random.randn(200, 2)) * 20 + 100,
    (np.random.randn(200, 2)) * 20 + np.r_[50, -30]
])
km = KMeans(n_clusters=3, random_state=0).fit(X)
plt.scatter(X[:, 0], X[:, 1], c=km.labels_)
plt.show()

“Bad” data (non-gaussian clusters):

from sklearn.cluster import KMeans, HDBSCAN
X = np.vstack([
    np.random.rand(100, 2) * 1000 + 1500,
    np.random.randn(200, 2) * 1000 - 400
])
km = KMeans(n_clusters=3, random_state=0).fit(X)
db = HDBSCAN().fit(X)

# Plot
fig, ax = plt.subplots(2, 1, figsize=(8, 16))
ax[0].scatter(X[:, 0], X[:, 1], c=km.labels_)
ax[1].scatter(X[:, 0], X[:, 1], c=db.labels_)
ax[0].set_title('KMeans')
ax[1].set_title('HDBSCAN')
plt.show()
/home/syxtreme/anaconda3/envs/pge/lib/python3.12/site-packages/sklearn/cluster/_hdbscan/hdbscan.py:722: FutureWarning: The default value of `copy` will change from False to True in 1.10. Explicitly set a value for `copy` to silence this warning.
  warn(

2.5.4 Evaluation Metrics sklearn.metrics

  • Classification: accuracy_score, confusion_matrix, roc_auc_score.
  • Regression: mean_squared_error, r2_score.
  • Clustering: silhouette_score, adjusted_rand_score.

Depending on the task type, it is important to choose the appropriate metric. For example, accuracy_score belongs to classification tasks, while mean_squared_error belongs to regression tasks.

from sklearn.metrics import accuracy_score, mean_squared_error

# Classification-style example
X_train = np.arange(100).reshape(10, 10)
y_train_clf = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1, 1])
y_test_clf = np.array([0, 0, 0, 1, 1, 1, 1, 1, 0, 1])

logr = LogisticRegression().fit(X_train, y_train_clf)
y_pred_clf = logr.predict(X_train)
print("Accuracy:", accuracy_score(y_train_clf, y_pred_clf))

# Regression-style example
y_train_reg = np.arange(10).astype(float)
y_test_reg = np.arange(10, 20).astype(float)
lr = LinearRegression().fit(X_train, y_train_reg)
y_pred_reg = lr.predict(np.arange(101, 201).reshape(10, 10))
print("MSE:", mean_squared_error(y_test_reg, y_pred_reg))
Accuracy: 1.0
MSE: 0.009999999999999468

2.6 scikit-image: Image Processing

scikit-image is a library of image processing algorithms built on NumPy and SciPy.

2.6.1 I/O Plugins skimage.io

  • imread / imsave: Read/write images in multiple formats.
  • ImageCollection, MultiImage: Efficiently handle batches of images or multi-frame TIFFs.
from skimage import io
import os
from matplotlib import pyplot as plt

# base_path = os.path.dirname(__file__)
base_path = os.getcwd()
path = os.path.join(base_path, 'bridge.jpg')
img = io.imread(path)
plt.imshow(img)
io.imsave(os.path.join(base_path, 'out.png'), img)

# Load a collection of images
col = io.imread_collection(
    os.path.join(base_path, 'multi_frames', 'frames_*.png')
)
stack = io.concatenate_images(col)

2.6.2 Color Space (skimage.color)

  • rgb2gray, gray2rgb, rgb2hsv: Convert between color spaces.
from skimage.color import rgb2gray, gray2rgb
gray = rgb2gray(img)
rgb = gray2rgb(gray)
plt.imshow(gray, cmap='gray')
plt.title('Grayscale image')
plt.show()

2.6.3 Filtering (skimage.filters)

  • Edge Detectors: sobel, scharr, prewitt compute image gradients.
  • Noise Reduction: gaussian, median, threshold_otsu for binarization.
from skimage.filters import sobel, gaussian, threshold_otsu
from matplotlib import pyplot as plt

edges = sobel(gray)
blur = gaussian(img, sigma=5)
th = threshold_otsu(gray)
binary = gray > th

plt.imshow(blur)
plt.title('Gaussian Blur')
plt.show()
plt.imshow(edges)
plt.title('Sobel Edge Detection')
plt.show()
plt.imshow(binary, cmap='gray')
plt.title('Binarization')
plt.show()

2.6.4 Morphology (skimage.morphology)

Morphology operations are useful for image segmentation and feature extraction. They typically manipulate “pixel connectivity” in binary (black and white) or sometimes grayscale images.

  • Basic Ops: erosion, dilation, opening, closing.
  • Advanced: skeletonize, remove_small_objects, area_closing.
from skimage.morphology import opening, remove_small_objects
mask = gray > th
clean = remove_small_objects(mask, min_size=512)
opened = opening(clean)

plt.imshow(opened, cmap='gray')
plt.title('Removing Small Objects and Opening')
plt.show()
/tmp/ipykernel_146810/2733725184.py:3: FutureWarning: Parameter `min_size` is deprecated since version 0.26.0 and will be removed in 2.0.0 (or later). To avoid this warning, please use the parameter `max_size` instead. For more details, see the documentation of `remove_small_objects`. Note that the new threshold removes objects smaller than **or equal to** its value, while the previous parameter only removed smaller ones.
  clean = remove_small_objects(mask, min_size=512)

2.6.5 Geometric Transforms (skimage.transform)

  • resize, rotate, warp for image warping.
  • Hough Transforms: hough_line, probabilistic_hough_line for line detection.
from skimage.transform import resize, rotate
small = resize(img, (256, 256))
rotated = rotate(small, 45)

plt.imshow(small)
plt.title('Resized Image')
plt.show()
plt.imshow(rotated)
plt.title('Rotated Image')
plt.show()

3 Part 2: Output and Visualization in Python

3.1 Overview

We now move from processing to presentation. Once data is loaded, transformed, or modeled, it usually needs to be:

  • inspected during execution (debugging, validating algorithm)
  • summarized for a user (visualizing results, predictions)
  • plotted for analysis (e.g. histograms, scatterplots)
  • or displayed interactively (better for “exploratory data analysis”)

There a few ways to visualize (not only) data in Python:

  1. terminal-based output and light visual tools,
  2. scientific visualization tools,
  3. interactive and GUI/external visualization tools.

3.2 Terminal Output and Minor Visualization Tools

In data and engineering tasks, it is often useful to enhance console output for progress tracking, formatting, and simple visualization. Here, we will cover some basics, including progress bars (with tqdm), rich text/formatting (with Rich), ASCII tables (with tabulate), and basic logging techniques.

3.2.1 Key Tools

  • tqdm – wraps Python loops/iterables to display a live progress bar (percentage, ETA, etc.).
  • Rich – a toolkit for generating rich formatted console text. It prints colored/styled text, tables, and even progress bars in the terminal.
  • tabulate – formats lists or DataFrames into plain-text tables, including markdown-style output.
  • logging – built-in module for logging messages at various levels (DEBUG, INFO, WARNING, etc.).

3.3 Tqdm: Progress Bars

%pip install tqdm

The tqdm library provides progress bars for loops. Simply wrap any iterable (like range(n)) with tqdm(...) and it will show a progress bar with percent, ETA, and iterations per second. You can also use trange(n) as shorthand. Tqdm auto-detects len() of the iterable, but for unknown-length iterators you can pass total=... for accurate progress. You can also pass a description and other options to customize the progress bar.

3.3.1 Key functions/classes from tqdm

  • tqdm.tqdm(iterable, desc="desc", total=n) – wrap an iterable to display progress.
  • tqdm.trange(n) – shortcut for tqdm(range(n)).
  • tqdm.tqdm.write(msg) – safely print a message without breaking the bar.
  • tqdm.update(n) – manually advance the bar by n.
  • tqdm.set_description(desc) – change the description of the bar.
  • tqdm.contrib.tenumerate(iterable, desc="desc", total=n) – shorthand for tqdm(enumerate(iterable)).

Usage tips: Add desc="Task name" to label the bar. Use leave=False to clear the bar after completion.

from tqdm import tqdm
import time

# Example: progress bar over a loop
for i in tqdm(range(100), desc="Processing"):
    # Simulate work
    time.sleep(0.05)
Processing:   0%|          | 0/100 [00:00<?, ?it/s]Processing:   2%|▏         | 2/100 [00:00<00:04, 19.97it/s]Processing:   4%|▍         | 4/100 [00:00<00:04, 19.91it/s]Processing:   6%|▌         | 6/100 [00:00<00:04, 19.75it/s]Processing:   8%|▊         | 8/100 [00:00<00:04, 19.63it/s]Processing:  10%|█         | 10/100 [00:00<00:04, 19.61it/s]Processing:  12%|█▏        | 12/100 [00:00<00:04, 19.61it/s]Processing:  14%|█▍        | 14/100 [00:00<00:04, 19.60it/s]Processing:  16%|█▌        | 16/100 [00:00<00:04, 19.59it/s]Processing:  18%|█▊        | 18/100 [00:00<00:04, 19.60it/s]Processing:  20%|██        | 20/100 [00:01<00:04, 19.61it/s]Processing:  22%|██▏       | 22/100 [00:01<00:03, 19.60it/s]Processing:  24%|██▍       | 24/100 [00:01<00:03, 19.68it/s]Processing:  26%|██▌       | 26/100 [00:01<00:03, 19.74it/s]Processing:  28%|██▊       | 28/100 [00:01<00:03, 19.78it/s]Processing:  30%|███       | 30/100 [00:01<00:03, 19.80it/s]Processing:  32%|███▏      | 32/100 [00:01<00:03, 19.82it/s]Processing:  34%|███▍      | 34/100 [00:01<00:03, 19.80it/s]Processing:  36%|███▌      | 36/100 [00:01<00:03, 19.70it/s]Processing:  38%|███▊      | 38/100 [00:01<00:03, 19.65it/s]Processing:  40%|████      | 40/100 [00:02<00:03, 19.64it/s]Processing:  42%|████▏     | 42/100 [00:02<00:02, 19.70it/s]Processing:  44%|████▍     | 44/100 [00:02<00:02, 19.74it/s]Processing:  46%|████▌     | 46/100 [00:02<00:02, 19.77it/s]Processing:  48%|████▊     | 48/100 [00:02<00:02, 19.79it/s]Processing:  50%|█████     | 50/100 [00:02<00:02, 19.80it/s]Processing:  52%|█████▏    | 52/100 [00:02<00:02, 19.81it/s]Processing:  54%|█████▍    | 54/100 [00:02<00:02, 19.82it/s]Processing:  56%|█████▌    | 56/100 [00:02<00:02, 19.83it/s]Processing:  58%|█████▊    | 58/100 [00:02<00:02, 19.84it/s]Processing:  60%|██████    | 60/100 [00:03<00:02, 19.85it/s]Processing:  62%|██████▏   | 62/100 [00:03<00:01, 19.85it/s]Processing:  64%|██████▍   | 64/100 [00:03<00:01, 19.86it/s]Processing:  66%|██████▌   | 66/100 [00:03<00:01, 19.86it/s]Processing:  68%|██████▊   | 68/100 [00:03<00:01, 19.85it/s]Processing:  70%|███████   | 70/100 [00:03<00:01, 19.85it/s]Processing:  72%|███████▏  | 72/100 [00:03<00:01, 19.85it/s]Processing:  74%|███████▍  | 74/100 [00:03<00:01, 19.84it/s]Processing:  76%|███████▌  | 76/100 [00:03<00:01, 19.85it/s]Processing:  78%|███████▊  | 78/100 [00:03<00:01, 19.86it/s]Processing:  80%|████████  | 80/100 [00:04<00:01, 19.86it/s]Processing:  82%|████████▏ | 82/100 [00:04<00:00, 19.85it/s]Processing:  84%|████████▍ | 84/100 [00:04<00:00, 19.82it/s]Processing:  86%|████████▌ | 86/100 [00:04<00:00, 19.75it/s]Processing:  88%|████████▊ | 88/100 [00:04<00:00, 19.69it/s]Processing:  90%|█████████ | 90/100 [00:04<00:00, 19.65it/s]Processing:  92%|█████████▏| 92/100 [00:04<00:00, 19.64it/s]Processing:  94%|█████████▍| 94/100 [00:04<00:00, 19.59it/s]Processing:  96%|█████████▌| 96/100 [00:04<00:00, 19.60it/s]Processing:  98%|█████████▊| 98/100 [00:04<00:00, 19.60it/s]Processing: 100%|██████████| 100/100 [00:05<00:00, 19.60it/s]Processing: 100%|██████████| 100/100 [00:05<00:00, 19.73it/s]
from tqdm.contrib import tenumerate
import numpy as np
import time

my_list = np.random.rand(100)

for i, val in tenumerate(my_list, desc="Processing"):
    # Do something with val
    my_list[i] = val**2
    time.sleep(0.05)

3.4 Rich: Formatted Terminal Output

%pip install rich

Rich enriches terminal output with colors, styles, tables, and animations. It uses a Console object to print colored text and rich elements. Common use-cases include colorizing log output, printing status messages, and creating nice-looking tables in text form. Rich also has its own progress bar class.

3.4.1 Key classes/functions

  • Console – main class for printing. Use console = Console() and console.print() to display styled text. Supports markup such as [bold] or [underline] and named colors.
  • Markup syntax: You can wrap text in tags like [bold green]text[/].
  • Progress – Rich’s progress bar class for live updates.
  • Table – create formatted tables with columns, styles, and titles.
  • rich.logging.RichHandler – integrate with Python’s logging for colored logs.

Rich tables are easy to build: define columns with Table.add_column(...) and rows with Table.add_row(...).

from rich.console import Console
from rich.table import Table

console = Console()

table = Table(title="Employees")
table.add_column("Name", style="cyan", justify="left")
table.add_column("Department", style="magenta")
table.add_column("Salary", justify="right", style="green")

table.add_row("Alice", "Engineering", "$70k")
table.add_row("Bob", "Data Science", "$65k")
console.print(table)
            Employees            
┏━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ Name  ┃ Department   ┃ Salary ┃
┡━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━┩
│ Alice │ Engineering  │   $70k │
│ Bob   │ Data Science │   $65k │
└───────┴──────────────┴────────┘

Rich progress bars allow multi-task progress:

from rich.progress import Progress
import time

with Progress() as progress:
    task1 = progress.add_task("[cyan]Downloading file 1...", total=100)
    task2 = progress.add_task("[red]Downloading file 2...", total=100)
    for i in range(100):
        progress.update(task1, advance=1)
        if i % 10 == 0:
            progress.update(task2, advance=10)
        time.sleep(0.05)














3.5
ASCII Tables with Tabulate


The tabulate library formats tabular data into
plain-text tables. It is useful when you need to display small tables
without complex dependencies. It provides a single function
tabulate(data, headers, tablefmt) which works on lists,
dicts, NumPy arrays, or pandas DataFrames. Different
tablefmt values (like "grid" or
"pipe") control the style of borders.




3.5.1
Key functions from tabulate


    

  • tabulate(tabular_data, headers=..., tablefmt="grid") –
produce a string of the formatted table.
    • 

  • Data types supported: lists of lists, lists of
dicts, dict of lists, NumPy arrays, pandas DataFrames, etc.
    • 

  • Formatting: Common tablefmt options
include "plain", "grid",
"fancy_grid", "pipe".
    • 

  • NumPy / DataFrame: You can directly pass a
DataFrame with headers='keys' to use column names.
    • 





from tabulate import tabulate

data = [
    ["Alice", 24, "Engineer"],
    ["Bob",   30, "Data Scientist"],
    ["Eve",   28, "Analyst"]
]
print(tabulate(data, headers=["Name", "Age", "Role"], tablefmt="grid"))




+--------+-------+----------------+
| Name   |   Age | Role           |
+========+=======+================+
| Alice  |    24 | Engineer       |
+--------+-------+----------------+
| Bob    |    30 | Data Scientist |
+--------+-------+----------------+
| Eve    |    28 | Analyst        |
+--------+-------+----------------+














3.6
Logging and Timing Functions


Standard Python provides the logging module for
leveled (severity level) messages. The level can then be used to filter
out messages of different severity, e.g., showing only “ERROR”
messages.




3.6.1
Key functions from logging


    

  • logging.basicConfig(level=logging.INFO) – set global
log level.
    • 

  • logger = logging.getLogger(name) – create a
logger.
    • 

  • logger.debug/info/warning/error(msg) – log messages at
levels DEBUG, INFO, etc.
    • 

  • Customize format with
logging.basicConfig(format="%(levelname)s: %(message)s").
    • 





import logging, time
logging.basicConfig(format="%(levelname)s: %(message)s", level=logging.INFO)
logger = logging.getLogger("demo")
logger.warning("Starting processing")

start = time.time()
time.sleep(0.3)

logger.info(f"Finished in {time.time()-start:.2f} seconds")
logger.debug("Debugging info")




WARNING: Starting processing
INFO: Finished in 0.30 seconds






For real programs use logging over print(),
as it provides:


    

  • severity levels
    • 

  • better filtering
    • 

  • configurable formatting
    • 

  • easy redirection to files or monitoring tools
    • 













4
Scientific and Statistical Visualization Tools




%pip install plotly
%pip install networkx
%pip install graphviz
%pip install seaborn




    

  • Matplotlib – low-level plotting (lines, bars,
images) with fine control.
    • 

  • Seaborn – high-level statistical plots (built on
Matplotlib).
    • 

  • Plotly – interactive, web-based plots (zoom, hover,
etc.).
    • 

  • NetworkX / Graphviz – tools for creating and
drawing graph/network structures.
    • 







4.1
Matplotlib


Matplotlib’s pyplot interface is the
standard for creating static 2D plots (and more).




4.1.1
Key functions/classes


    

  • plt.figure(), plt.subplots() – create new
figure and axes.
    • 

  • plt.plot(x, y, ...) – line plots.
    • 

  • plt.scatter(x, y, ...) – scatter plots.
    • 

  • plt.bar(x, height, ...) – vertical bar chart
(plt.barh for horizontal).
    • 

  • plt.hist(data, bins=..., ...) – histogram.
    • 

  • plt.imshow(matrix, cmap=...) – display a 2D array as a
heatmap or image.
    • 

  • plt.title(), plt.xlabel(),
plt.ylabel(), plt.legend() – annotate
plots.
    • 



Usage: Usually import as
import matplotlib.pyplot as plt. After drawing, call
plt.show() (or plt.savefig(...)) to display or
save.




4.1.1.1 Line plot




import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(0, 2*np.pi, 100)
y = np.sin(x)
plt.figure()
plt.plot(x, y, label='sin(x)')
plt.xlabel("x")
plt.ylabel("sin(x)")
plt.title("Line Plot of sin(x)")
plt.legend()
plt.show()






















4.1.1.2 Scatter plot (point
data)




import numpy as np
import matplotlib.pyplot as plt

x = np.random.rand(50)
y = np.random.rand(50)
plt.figure()
plt.scatter(x, y, color='orange')
plt.title("Random Scatter Plot")
plt.xlabel("x")
plt.ylabel("y")
plt.show()






















4.1.1.3 Bar chart




categories = ["A", "B", "C"]
values = [5, 7, 3]
plt.figure()
plt.bar(categories, values, color=['red', 'green', 'blue'])
plt.title("Bar Chart Example")
plt.show()






















4.1.1.4 Heatmap




data = np.random.rand(10, 10)
plt.figure()
plt.imshow(data, cmap='viridis')
plt.colorbar()
plt.title("Random Heatmap")
plt.show()






















4.1.1.5 Subplots (multiple plots in
one figure)




fig, axs = plt.subplots(1, 2, figsize=(8, 3))
x = np.linspace(0, 2*np.pi, 100)
axs[0].plot(x, np.sin(x))
axs[0].set_title("Sine")
axs[1].plot(x, np.cos(x))
axs[1].set_title("Cosine")
plt.tight_layout()
plt.show()




























4.2
Seaborn: Statistical Plots


Seaborn builds on Matplotlib to simplify statistical
plotting. It integrates tightly with pandas DataFrames and provides easy
functions like sns.scatterplot, sns.barplot,
sns.boxplot, sns.heatmap, etc. Seaborn also
sets more analysis-friendly default styles and color palettes.




4.2.1
Key functions


    

  • sns.set_style(...) – set a seaborn theme
(e.g. "whitegrid").
    • 

  • sns.scatterplot(data=..., x=..., y=..., hue=...) –
colored scatter plot.
    • 

  • sns.lineplot(data=..., x=..., y=...) – line plot with
confidence band.
    • 

  • sns.barplot(x=..., y=..., hue=..., data=...) – bar
chart with grouping.
    • 

  • sns.boxplot(x=..., y=..., data=...) – box-and-whisker
plot.
    • 

  • sns.heatmap(data, annot=True) – annotated heatmap.
    • 

  • sns.pairplot(df) – grid of scatterplots for variable
pairs.
    • 



Usage: Usually import as
import seaborn as sns.




4.2.1.1 Scatter plot




import seaborn as sns
import numpy as np
import matplotlib.pyplot as plt

# Load a sample dataset
iris = sns.load_dataset("iris")
sns.scatterplot(data=iris, x="sepal_length", y="sepal_width", hue="species")
plt.title("Seaborn Iris Scatter Plot")
plt.show()






















4.2.1.2 Bar plot




# Bar plot example
tips = sns.load_dataset("tips")
sns.barplot(x="day", y="total_bill", hue="time", data=tips)
plt.title("Average Bill by Day and Time")
plt.show()






















4.2.1.3 Heatmap




# Heatmap example
data = np.random.rand(5, 5)
sns.heatmap(data, annot=True, cmap="coolwarm")
plt.title("Seaborn Random Heatmap")
plt.show()




























4.3
Plotly: Interactive Plots


Plotly provides interactive, web-based plotting.
Plotly charts support zooming, panning, and hover information. Its
declarative interface comes from plotly.express (a
high-level API), as well as the lower-level
plotly.graph_objects.




4.3.1
Key functions (plotly.express)


    

  • High-level API: px.line,
px.scatter, px.bar, px.histogram,
px.imshow, etc.
    • 

  • General workflow:
fig = px.some_chart(data, ...); then
fig.show() to render.
    • 

  • Figure API: Alternatively, go.Figure()
and go.* traces give more control.
    • 

  • Interactivity: Hover labels, zoom/drag tools, and
style templates are built-in.
    • 





4.3.1.1 Scatter plot




import plotly.express as px

# Scatter plot (using built-in iris dataset)
df = px.data.iris()
fig = px.scatter(
    df, x="sepal_width", y="sepal_length",
    color="species",
    title="Iris Sepal Dimensions"
)
fig.show()




                            
                        
                    










4.3.1.2 Line plot




# Line chart example
fig = px.line(x=[1, 2, 3, 4], y=[10, 15, 13, 17], title="Simple Line Chart")
fig.show()




                            
                        
                    










4.3.1.3 Bar chart




# Bar chart example
fig = px.bar(
    x=['Apples', 'Bananas', 'Cherries'],
    y=[5, 3, 7], title="Fruit Counts")
fig.show()




                            
                        
                    










4.3.1.4 Heatmap




# Heatmap example (via imshow)
matrix = [[1, 20, 30], [20, 1, 60], [30, 60, 1]]
fig = px.imshow(
    matrix,
    labels={'x':'Col','y':'Row','color':'Value'},
    title="Example Heatmap")
fig.show()




                            
                        
                    










4.3.1.5 3D scatter plot




points = np.random.rand(3, 100)
fig = px.scatter_3d(
    x=points[0], y=points[1], z=points[2],
    color=points[2], title="3D Scatter Plot")
fig.show()




                            
                        
                    
















4.4
Graphs with NetworkX and Graphviz


For network/graph data, NetworkX
allows creation and analysis of graphs. You can build a graph, add
nodes/edges (including attributes like weights), and compute graph
algorithms. For simple visualization, NetworkX can draw graphs with
nx.draw(). For more polished graph diagrams, especially for
directed graphs or hierarchical layouts, the Graphviz
package can be used via graphviz.Digraph or
Graph to generate DOT-language graphs and render them.




4.4.1
NetworkX usage


    

  • G = nx.Graph() or nx.DiGraph() – start a
new (undirected/directed) graph.
    • 

  • G.add_node(node) and
G.add_edge(u, v, weight=...).
    • 

  • Layouts: pos = nx.spring_layout(G) or other layouts to
position nodes.
    • 

  • Drawing: nx.draw(G, pos=pos, with_labels=True) or
nx.draw_networkx(G).
    • 







4.4.2
Graphviz usage


    

  • from graphviz import Digraph (for directed) or
Graph (undirected).
    • 

  • dot.node("A", "Label"),
dot.edge("A", "B", label="edge label").
    • 

  • dot.render("file") to produce an image file (e.g. PNG
or PDF).
    • 







4.4.3
Practical note


The Python graphviz package is a wrapper/interface. In
many environments, the Graphviz system binaries still need to be
installed separately for rendering to work.




import networkx as nx
import matplotlib.pyplot as plt

# Create and draw a simple NetworkX graph
G = nx.Graph()
G.add_edge("Node1", "Node2")
G.add_edge("Node2", "Node3")
pos = nx.spring_layout(G)  # force-directed layout
nx.draw(G, pos, with_labels=True, node_color='lightblue', edge_color='gray')
plt.title("NetworkX Graph")
plt.show()




















from graphviz import Digraph

# Create a simple directed graph with Graphviz
dot = Digraph(comment="Simple Graph")
dot.node("A", "Start")
dot.node("B", "End")
dot.edge("A", "B", label="goes to")






4.4.3.1 View the graph


dot.view("simple_graph")






4.4.3.2 Render (save to file) the
graph


dot.render(filename='simple_graph', format='png', cleanup=True, view=True)


The NetworkX graph is displayed inline via Matplotlib, while the
Graphviz example generates a PNG file (or PDF, depending on the
specification).














5
External and GUI Visualization Tools




%pip install open3d
%pip install opencv-python
%pip install pygame
%pip install streamlit






5.0.1
Key libraries


    

  • OpenCV (cv2) – computer vision library
for images and video (reading, displaying, drawing).
    • 

  • Pygame – 2D graphics and game development library
(window, rendering, events).
    • 

  • Open3D – 3D geometry library (point clouds, meshes,
visualization).
    • 

  • Streamlit – library to build interactive web apps
and dashboards from Python scripts.
    • 









5.1
OpenCV: Image and Video


OpenCV (imported as cv2) is widely used
for computer vision. It can load images/videos (cv2.imread,
cv2.VideoCapture), display them in windows
(cv2.imshow), and draw shapes or text on images
(cv2.rectangle, cv2.circle, etc.). Typical use
cases include processing camera feeds or image analysis.




5.1.1
Key functions


    

  • cv2.imread(path) – load an image into a NumPy array
(BGR color).
    • 

  • cv2.imshow(window_name, image) – display an image in a
window.
    • 

  • cv2.waitKey(delay) – wait for a key press or a short
GUI event-processing delay.
    • 

  • cv2.imwrite(path, image) – save an image to file.
    • 

  • cv2.VideoCapture(src) – open a video file or
camera.
    • 

  • cv2.VideoWriter() – write frames to video.
    • 

  • Drawing:
cv2.rectangle(img, pt1, pt2, color, thickness),
cv2.circle(img, center, radius, color, thickness),
cv2.line, cv2.putText, etc.
    • 



OpenCV uses BGR color format by default.






5.1.2
Practical notes


    

  • For a single static image,
cv2.waitKey(0) is usually the clearest option because it
waits until a key is pressed.
    • 

  • For live loops, cv2.waitKey(1) is
common because it updates the GUI while keeping the loop
responsive.
    • 

  • In notebook, server, container, or headless environments, GUI
windows may not work at all, even if the code is otherwise correct.
    • 





import cv2
import numpy as np

# Create a blank image
img = np.zeros((200, 300, 3), dtype=np.uint8)
# Draw a blue rectangle and a green circle
cv2.rectangle(img, (50, 50), (250, 150), (255, 0, 0), 3)
cv2.circle(img, (150, 100), 40, (0, 255, 0), 2)
cv2.putText(img, "OpenCV", (60, 45), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (255,255,255), 2)
cv2.imshow("Shapes", img)
cv2.waitKey(0)
cv2.destroyAllWindows()




For video, one could use cap = cv2.VideoCapture(0) to
capture from a webcam and
while True: ret, frame = cap.read(); cv2.imshow("Frame", frame).




import cv2

cap = cv2.VideoCapture(0)
while True:
    ret, frame = cap.read()
    if not ret:
        break
    cv2.imshow("Frame", frame)
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break
cap.release()
cv2.destroyAllWindows()












5.2
Pygame: 2D GUI Graphics and Games


Pygame is a set of Python modules for creating games
and simple GUIs. It provides a display window, drawing primitives, image
blitting, and an event loop. Use Pygame for real-time 2D graphics or
game logic (e.g. animations, keyboard input).




5.2.1
Usage


    

  • Initialization: pygame.init(), then
screen = pygame.display.set_mode((width, height)) to create
a window.
    • 

  • Event loop: Typically
for event in pygame.event.get(): to handle
QUIT or input events.
    • 

  • Drawing: screen.fill(color) to clear;
pygame.draw.rect(screen, color, rect) or
.circle, .line to draw shapes;
screen.blit(image, pos) to draw images.
    • 

  • Display update: pygame.display.flip()
or pygame.display.update().
    • 

  • Timing: clock = pygame.time.Clock()
and clock.tick(fps) to control frame rate.
    • 





import pygame

pygame.init()
screen = pygame.display.set_mode((400, 300))
pygame.display.set_caption("Pygame Example")
clock = pygame.time.Clock()
running = True

while running:
    for event in pygame.event.get():
        if event.type == pygame.QUIT:
            running = False
    screen.fill((255, 255, 255))  # white background
    # Draw a red rectangle and a blue circle
    pygame.draw.rect(screen, (255, 0, 0), pygame.Rect(150, 100, 100, 50))
    pygame.draw.circle(screen, (0, 0, 255), (100, 150), 30)
    pygame.display.flip()  # update the display
    clock.tick(30)  # limit to ~30 FPS

pygame.quit()












5.3
Open3D: 3D Graphics




%pip install open3d




Open3D is a library for 3D data (point clouds,
meshes, geometry processing). It provides visualization tools that open
interactive 3D windows. You can load 3D data (e.g. from PLY or OBJ
files) and display them. For instance,
o3d.visualization.draw_geometries() opens a viewer where
you can rotate and zoom the model.




5.3.1
Usage


    

  • o3d.geometry.PointCloud – holds 3D points (and
optionally colors, normals).
    • 

  • o3d.geometry.TriangleMesh – holds mesh vertices and
triangles.
    • 

  • o3d.io.read_point_cloud("path.ply") – load a point
cloud from file.
    • 

  • o3d.visualization.draw_geometries([geometry_list]) –
visualize the geometry list.
    • 





import open3d as o3d
import numpy as np

# Load an example point cloud (Open3D provides sample data)
pcd_data = o3d.data.PLYPointCloud()
pcd = o3d.io.read_point_cloud(pcd_data.path)
print(pcd)  # prints number of points

# draw some meshes
box = o3d.geometry.TriangleMesh.create_box()
axis = o3d.geometry.TriangleMesh.create_coordinate_frame()

# Visualize point cloud in a 3D window
o3d.visualization.draw_geometries([pcd, box, axis], window_name="Point Cloud")












5.4
Streamlit: Browser-Based Interactive Dashboards


Streamlit lets you turn Python scripts into
interactive web apps without writing HTML or JavaScript. It is useful
for quick dashboards or demos. You write code like
st.slider, st.button, and Streamlit generates
the UI. It runs a local web server, and the app is displayed in your
browser.




5.4.1
Key functions


    

  • st.title(), st.header(),
st.text() – display text and headers.
    • 

  • st.write() – versatile text/markdown display.
    • 

  • Widgets: st.button(label),
st.slider(label, min, max), st.checkbox(),
st.selectbox(), etc.
    • 

  • Charts: st.line_chart(data), st.map(df),
or any Matplotlib/Plotly figure via st.pyplot(fig) or
st.plotly_chart(fig).
    • 

  • Usage: The script is run top-to-bottom on each
interaction. To display the app, run
streamlit run script.py in a terminal.
    • 





import streamlit as st
import pandas as pd

st.title("Simple Dashboard")

df = pd.DataFrame({
    "x": [1, 2, 3, 4],
    "y": [10, 20, 15, 25]
})
st.line_chart(df.set_index("x"))  # interactive line chart

if st.checkbox("Show raw data"):
    st.write(df)








5.4.2
Usage


Streamlit requires the script to be executed by running the
streamlit run <script_name>.py command in a
terminal.


Running this script with Streamlit will open a local web interface.
It is not a visualization library in the usual sense, but
rather a framework to host your visualizations and controls in a
browser.