Making Deep Learning Go Brrrr From First Principles

Why do we want to be compute-bound? Because the amount of compute that we have to do in a forward path of the model is irreducible, so the fastest that we can possibly go is (FLOPs) / (# gpus * GPU compute speed)

Can minimize Python overhead by using more just-in-time techniques or cuda graphs or smth