I will report on a recent result (arXiv:1911.00124, jointly with S. T. Yau). We study the Vafa-Witten invariants arising from gauge theory on noncompact 3 dimensional Calabi-Yau manifolds of particular type. The Donaldson-Thomas theory provides the means to realize these as invariants of moduli spaces of sheaves with compact support on such Calabi-Yau varieties. One important feature of Vafa-Witten invariants is that their generating series is conjecturally given by modular forms. In order to prove this conjecture, we reduce the question to studying the geometry of sheaves (or rather flags of sheaves) on surfaces, and use techniques of localization and vertex operator theory to prove that the generating function build out of these invariants is a modular form.