Who are “they” who “have lived in idleness”?

Northern soldiers

Southern slavers

Southern enslaved people
“Now Mr Lincoln don't you think you ought to stop this thing and make them do the same by the colored men. They have lived in idleness all their lives on stolen labor, but are so furious because [colored people] are proving themselves to be men, such as have come away and got some education. It must not be so.”