Life in Texas was complicated. Since Billy refused to fight in the Civil War, he was also harassed by Confederate vigilante renegades. He knew that they wanted to force him to leave the area, and then take over his property. The family was sent to Carroll County, Arkansas while Billy remained hidden in the Texas wilderness.

He tried to think of a way to outsmart the renegades, and keep his property, yet he was under constant attack by the Apaches and Commanches. Finally the situation became too difficult and Billy left Texas to join his family in Arkansas. He started growing sugar cane in Arkansas and built a sugar mill. They had a huge house with two stories and lots of windows.