We hand coded weights for one layer MLPs that memorises labels for input token sequences of length two. The number of facts our hand-coded models can memorise with 90% accuracy[1]scales roughly linearly with the models' parameter count[2], just like trained models for the same architecture. However, our hand-coded models' scaling prefactor still falls short of trained models' by a factor of . A hybrid solution in which we handcode the MLP input weights and learn the MLP output weights, which amounts to a linear classification problem[3], falls short by a factor of .Figure 0: Max number of facts each model can learn when requiring 90% accuracy on evaluation. The fitted lines are best fit for solving for .We pose a challenge to the community: Find better constructions than ours that can memorise more facts in the same number of weights and close the gap to the trained solution further.[4]Background: Why sequence memorization, and why an MLP?The goal of this research is to get a clearer u...