← Back to blog
·5 min read

Why I Built GlassBoxML?

PythonMachine LearningNumPySoftware Engineering

I didn't plan on making a machine learning library.

I was just wondering what was going on behind model.fit().

With a library like scikit-learn, being a machine learning hero can feel almost too easy:

model.fit(X, y)
predictions = model.predict(X_test)

A couple of lines and you're done.

But at some point, I started wondering:

I can understand the math in a textbook. I can also use the implementations available in libraries.

But I wanted to connect the two.

So I decided to build my own. And that became GlassBoxML.

The idea

The main language for developing GlassBoxML is Python, with NumPy at the core for numerical operations.

The aim was not to build a rival to scikit-learn. There was no reason to “reinvent the wheel” and compete with a highly mature library. The goal was much simpler:

For me to take the algorithms I was learning and implement them myself.

I wished to be able to look at the code and understand precisely what was occurring. No hidden training pipeline. No mysterious abstraction. Just the mathematics translated into code.

Hence the name: GlassBoxML. I wanted visibility, not a black box.

Starting with the fundamentals

First, I used the algorithms I knew. Linear regression was one of the first.

On paper, gradient descent is straightforward:

w ← w − η∇L(w)

In simple terms, we take the current weights, look at how the loss changes, and move the weights in the direction that reduces that loss.

Here's the problem: once you write the equation, you have to resolve all the issues a library usually tries to disguise.

The equation is only the beginning. Converting it into robust code is an entirely different story.

Growing the library

For the software that worked, I continued adding new algorithms. Eventually, GlassBoxML included implementations such as:

Linear RegressionLogistic RegressionKNNDecision TreesRandom ForestsSVMAdaBoostGradient BoostingPCALDA

I've also developed the supporting functions to support them, such as:

By this point, it wasn't entirely a “library” yet, but it wasn't just implementation either. And that was a totally different kettle of fish.

The part I didn't expect

The algorithms weren't always the hardest part. Software engineering was.

Implementing a decision tree is one thing. Another is having an API that will make you hate your own decision tree six months down the road.

I had to think about:

That was the biggest learning from the project.

Good software is not necessarily writing mathematically correct code. Now it's your job to connect the dots: you have to get the algorithm off the page and into something that someone else could actually use.

Why NumPy?

I deliberately chose NumPy as the foundation. I wanted to not be far from the mathematics.

If I used a framework that does automatic differentiation for me, with most of the machinery hidden away, I would probably end up knowing the framework instead of knowing how these algorithms really worked. The calculations themselves were more transparent when using NumPy.

Take, for example, the traditional training loop for a model:

input -> prediction -> loss -> gradient -> parameter update -> repeat

I always like to see that process come out of fairly simple array manipulation.

Testing the thing

I couldn't keep trying random examples on the growing project and hope everything was okay. I needed tests. So I packaged the models and utilities together into a test suite.

This was another useful shift in perspective. Just because an implementation works on one dataset doesn't mean it is correct. You have to think about:

This definitely gave me a much better understanding of the engineering that goes into well-developed ML libraries.

What GlassBoxML taught me

The most important thing I learned wasn't an algorithm.

A textbook can give you an equation. But turning that equation into a library means making hundreds of small decisions. You'll have to deal with data structures, shapes, numerical accuracy, APIs, validation, errors, testing, and performance.

That's where much of the interesting engineering takes place.

Why build another ML library?

This is probably the obvious question. Why build something that already exists?

The goal was never to replace scikit-learn. I created GlassBoxML to understand what libraries like it were doing.

And in a twist of irony, after I'd made my own implementations I was left with more admiration for the libraries I was attempting to learn. As you write the simplified version on your own you begin to realize everything the mature library has already solved. You start to see the abstractions rather than just the black boxes.

If you want to try it yourself, create a virtual environment and install GlassBoxML directly from PyPI:

pip install glassboxml

Give it a spin, inspect the implementations, and see what happens under the hood.

Source code & community

GlassBoxML is 100% open source under the MIT License. See the full implementation, custom vectorized routines and diagnostic benchmarks on GitHub:

Conclusion

Building GlassBoxML reaffirmed for me:

The best way to understand an abstraction is to remove it.

I didn't create GlassBoxML because the world needed another ML framework. I built it because I wanted to see through the abstraction.

I wanted to take the equations I'd been learning about, write code for them, bend them, break them, fix them, test them, and finally assemble them into something that actually worked.

At some point, a learning experiment turned into an actual piece of software. That's maybe the greatest reason I could've had for building it.

Originally published on Medium ↗