Born in Cambridge, Swanson joined Arsenal at the age of 6 and signed his first professional contract with the club in April 2019.[2] In August 2020, Swanson joined Dutch Eerste Divisie side MVV Maastricht on a season-long loan deal.[3] After making just three appearances, he was recalled in December 2020.
On 4 July 2022, Swanson signed for League One side Portsmouth for an undisclosed fee on a two-year deal with the option for a further year.[4] On 29 October 2022, he scored his first senior goal in a 1–1 draw with Shrewsbury Town.[5]