Submitted

Deterministic Adam-Inspired Methods with Accelerated Convergence Rate

Yaxin Yu, Long Chen, and Zeyi Xu

Submitted

arXiv   Bibtex

ABSTRACT:

Adam is widely used, but its convergence theory remains
incomplete even in the deterministic full-batch setting because
momentum and adaptive preconditioning are tightly coupled. For smooth
convex objectives, we split the momentum variable through
variable-and-operator splitting, which reveals the acceleration
mechanism. We then combine a Hessian-driven correction with Adam-style
feedback based on the gradient magnitude. The resulting Adam-HNAG
(Hessian-driven Nesterov accelerated gradient with Adam-style adaptive
preconditioning) flow admits a nonnegative energy that decays
exponentially. Its discretization yields two methods, \emph{Adam-HNAG}
and the synchronous variant \emph{Adam-HNAG-s}. Under the stated
trajectory-bound and consistency conditions, both methods satisfy a
discrete Lyapunov contraction. If the exact adaptive steps are
accepted, this contraction gives an $O(k^{-2})$ objective-value
bound. Numerical experiments illustrate their behavior. These results
apply to the proposed methods, not to the original Adam
recursion.