After his father's death, Adams returned to the United States with his mother and brother. His father, a United States Army officer who fought in Korea, died while stationed in Germany when Adams was only 16.
His maternal grandfather, Thomas Lomax Hunter, was a Poet Laureate of Virginia. Adams was born in Washington, D.C., the son of Anna Campbell Stewart (née Hunter) and Robert Loughridge Adams.