Recent theoretical understanding of neural networks has connected their training and generalization to associated kernel matrices. Due to the nonlinearity of the activation function, at random initialization, these kernel matrices are non-linear random matrices.  We consider the limiting spectral distributions of conjugate kernel and neural tangent kernel matrices for two-layer neural networks with deterministic data and random weights. When the width of the network grows faster than the size of the dataset, a deformed semicircle law appears. In this regime, we can also calculate the asymptotic testing and training errors for random feature regression. Joint work with Zhichao Wang https://arxiv.org/abs/2109.09304.