Born in Somerset, England, Withers was educated at the Sherborne School, in Dorsetshire. After a five-year apprenticeship with architect and builder Edward Mondey, he joined the London office of architect Thomas Henry Wyatt in 1849. In February 1852, Withers immigrated to the United States, beckoned by Andrew Jackson Downing to join his practice in Newburgh, New York. After Downing’s untimely death later that year, Withers formed Vaux and Withers with architect and former Downing collaborator Calvert Vaux, a partnership that lasted until 1856.
Withers enlisted in the Union Army in 1861 but resigned his commission in 1862 after falling ill. He moved his practice to New York City and by December 1863 had once again joined Vaux in a partnership that lasted until 1872. During this period, Withers designed buildings for several projects for which Olmsted, Vaux & Co. designed the landscapes. These include the Hudson River State Hospital, now a National Historic Landmark (NHL); the president's house and several faculty buildings at the Columbia Institution for the Deaf and Dumb (now Gallaudet University) in Washington, D.C.; and the chapel and a commercial block for the newly planned suburban community of Riverside, Illinois. Along with Vaux, Withers advised on the creation of the Olana estate in Hudson, New York. Perhaps his best-known project, however, is the Third Judicial District Courthouse (now the Jefferson Market Library) in New York City, also an NHL. Although he designed private residences, institutional and civic buildings, monuments, and bridges, Withers made his reputation with Gothic Revival churches. His book, Church Architecture, was published in 1873. In 1888 he formed his final partnership with architect Walter Dickson, with whom he designed several buildings on Blackwell’s Island (now Roosevelt Island). Withers retired in 1897. He died at his home in Yonkers, New York, at the age of 72 and is buried in Trinity Church Cemetery in Manhattan.