On a hyperelliptic curve over the rationals, there are infinitely many points defined over quadratic fields: just pull back rational points of the projective line through the degree two map. But for a positive proportion of genus g odd hyperelliptic curves, we show there can be at most 24 quadratic points not arising in this way.  The proof uses tropical geometry work of Park, as well as results of Bhargava and Gross on average ranks of hyperelliptic Jacobians.  This is joint work with Jackson Morrow.