Maybe it seemed like "the Negroes disappeared." Enslaved people of African ancestry were 10% of the population of Boston in the 1750s. By 1830, the percentage of Black people in Boston was more like 3% (I took both of those numbers from this site, but I had remembered the approximate drop in percentage and absolute numbers from other sources
http://www.davidwalkermemorial.org/black-boston