The Black-Box Optimization Illusion
In machine learning libraries like Scikit-learn, calling fit() hides how algorithms navigate complex search spaces. When you want to select the best subset of features from a clinical dataset, importing a pre-packaged optimization library like DEAP or PySwarms makes the process feel like magic.
For the Swarm and Evolutionary Computation course at Telkom University, third-party optimization libraries were strictly banned. Everything had to be built from scratch in pure Python and NumPy.
The objective was clear: take the Breast Cancer Wisconsin Diagnostic dataset (569 samples, 30 continuous clinical features) and find out whether metaheuristic search algorithms could prune redundant dimensions from a K-Nearest Neighbors (KNN) classifier while matching or beating its baseline diagnostic accuracy.
30 Clinical Nuclear Features
│
▼
Baseline KNN (k=5)
(All 30 Features -> Accuracy: 94.74%)
│
▼
The Continuous Search Dilemma
(Standard PSO flies through floating-point space,
but feature selection is strictly binary)
│
▼
NumPy Implementations from Scratch
├── Binary PSO (Sigmoid Velocity Binarization)
└── Genetic Algorithm (Tournament Selection & Crossover)
│
▼
Holdout Benchmark Comparison
├── GA: 17 Features -> Accuracy: 95.61% (+0.87%)
└── PSO: 14 Features -> Accuracy: 95.61% (+0.87%)
Why All 30 Features Hurt K-Nearest Neighbors
Our baseline model was a K-Nearest Neighbors classifier configured with k = 5.
KNN calculates distances in Euclidean space across all available dimensions:
Euclidean Distance:
d(p, q) = sqrt( sum( (p_i - q_i)^2 ) )
When you pass all 30 cell nuclei measurements (radius, perimeter, texture, concavity, symmetry, and fractal dimensions across mean, error, and worst values), noisy or redundant measurements add artificial distance penalties. Two cells that share critical malignant characteristics might appear distant simply because their fractal dimension measurements diverge slightly.
Evaluating all 30 features on our 114-sample holdout test set gave an accuracy of 94.74%.
Exhaustively testing every possible feature combination would require evaluating 2³⁰ - 1 subsets, totaling over 1.07 billion KNN runs. On a university laptop, an exhaustive search was computationally impossible.
How do we search through a billion possible feature masks without importing an external optimization library?
Building Binary Particle Swarm Optimization in NumPy
Particle Swarm Optimization models a flock of candidate solutions moving through feature space. Each particle tracks its personal best position (pbest) and updates its trajectory toward the global swarm best (gbest).
Standard PSO updates velocity in continuous real-valued space:
Velocity Update Equation:
v_i(t+1) = w * v_i(t) + c1 * r1 * (pbest_i - x_i) + c2 * r2 * (gbest - x_i)
However, feature selection is strictly binary: feature i is either included (1) or excluded (0).
To adapt continuous velocity updates to binary decisions without external tools, I mapped particle velocities through a Sigmoid transfer function:
import numpy as np
def sigmoid(v):
return 1.0 / (1.0 + np.exp(-np.clip(v, -10, 10)))
def update_particles(positions, velocities, pbest, gbest, w=0.7, c1=1.5, c2=1.5):
n_particles, n_features = positions.shape
r1 = np.random.rand(n_particles, n_features)
r2 = np.random.rand(n_particles, n_features)
# Velocity calculation in continuous space
new_velocities = (
w * velocities +
c1 * r1 * (pbest - positions) +
c2 * r2 * (gbest - positions)
)
# Transform continuous velocity into inclusion probability
probabilities = sigmoid(new_velocities)
# Sample binary mask
new_positions = (np.random.rand(n_particles, n_features) < probabilities).astype(int)
# Ensure at least one feature is selected per particle
for i in range(n_particles):
if np.sum(new_positions[i]) == 0:
new_positions[i, np.random.randint(0, n_features)] = 1
return new_positions, new_velocities
With 30 particles running across 50 iterations (w = 0.7, c1 = 1.5, c2 = 1.5), the swarm evaluated KNN accuracy across 1,500 feature masks in under forty seconds on CPU.
How would an evolutionary approach compare against swarm particles?
Building a Genetic Algorithm from Scratch
As a benchmark, I implemented a Genetic Algorithm across the same 30-dimensional binary chromosome space:
- Population: 30 chromosomes across 50 generations.
- Selection: Tournament selection (k = 3), picking the fittest individual among random candidate triplets.
- Crossover: Single-point crossover randomly swapping gene segments between parent chromosomes.
- Mutation: Bit-flip mutation probability (p = 0.05) per gene to introduce exploratory diversity and avoid premature convergence on local plateaus.
def crossover_and_mutate(parent1, parent2, mutation_rate=0.05):
# Single-point crossover
cut = np.random.randint(1, len(parent1))
child = np.concatenate([parent1[:cut], parent2[cut:]])
# Bit-flip mutation
mutation_mask = np.random.rand(len(child)) < mutation_rate
child = np.bitwise_xor(child, mutation_mask.astype(int))
return child
Results: Fewer Features, Better Accuracy
Evaluating both custom metaheuristic searchers against the full 30-feature baseline demonstrated that less is more in distance-based classification:
| Optimization Strategy | Features Retained | Holdout Accuracy | Improvement |
|---|---|---|---|
| Full Feature Baseline | 30 / 30 | 94.74% | Reference |
| KNN + Genetic Algorithm | 17 / 30 | 95.61% | +0.87% |
| KNN + Particle Swarm (PSO) | 14 / 30 | 95.61% | +0.87% |
Both algorithms discovered feature combinations that outperformed the 30-feature baseline. Eliminating more than half of the variables removed noise from the Euclidean metric, helping KNN establish cleaner decision boundaries.
Comparing the two metaheuristics, PSO was the more parsimonious solver: it reached the exact same 95.61% test accuracy using only 14 features compared to GA’s 17 features.
What I Learned
Writing PSO and Genetic Algorithms from scratch in NumPy demystified metaheuristic optimization for me.
When you implement velocity clamping, sigmoid thresholds, and tournament operators line by line, you realize that swarm algorithms are simply coordinated probabilistic searches through discrete feature states.
The clinical takeaway is equally clear: feeding every available sensor column into a machine learning model is often lazy engineering. Cutting 16 redundant dimensions not only speeds up inference, but actively protects distance-based classifiers from high-dimensional noise.