Morgan Mank, 31, of Ashburn, drove down the wrong way on Franklin Street at about 3 a.m. Thursday and then reportedly reversed and struck the building and a fire hydrant.
Celebrity news site TMZ reported Mank was mumbling about Swift as he pulled at the intercom.
Police responded to a 911 call and arrested Mank. He was taken to New York Downtown Hospital for a psychological evaluation. He faces DWI charges.
It was not the first time Swift's New York City home made headlines.
In April 2021, a stalker who claimed the singer was communicating with him on social media was arrested on a trespassing charge after trying to break into the townhome.
In Sept. 2020, bullets flew outside the building after an armed robbery at a nearby sneaker store. The wild incident happened near 153 Franklin Street. Swift purchased the $18 million building in 2017. She also owns several apartments next door.