This post discusses an older paper that shows how a clever idea (butterfly factorizations) can be learned in a typical deep-learning pipeline. It concludes with some conjecture how these fast algorithms can be applied to other slow, data-intensive algorithms: attention. Learning fast algorithms for linear transforms with butterfly factorizations. Tri Dao (author of FlashAttention), Albert…

Are Transformers all you need? pdf version of this post Timothy Hanson – November 8, 2023 Abstract This post discusses active learning and reasoning, and the strengths and limitations of using transformers for it. After setting up the problem context, we conclude that for transformers to serve as world-models to these purposes, they will need…

Hello hello — this is a blog where we’ll post research progress, thoughts, things too small to be preprints, company news, etc. Looking forward to sharing 🙂