IMDB information:

Title: Nothing But a Man (1964)
Genres: Drama, Romance
Description: Born in Birmingham, Duff Anderson, the father of a male toddler, who lives with a nanny, re-locates to a small town to work on the railroad. He meets with and is attracted to Josie much to the chagrin of her preacher father. The marriage does take place nevertheless, both re-locate to live in their own house and he gets a job in a mill. He decides not to bring his son to live with them. Challenges arise when the Mill Foreman finds out that Duff is attempting to unionize the workers, forcing Duff to quit, and look for work elsewhere. Unable to reconcile himself to working on a daily wage of $2.50 picking cotton nor even as a waiter, he gets a job at a garage. He is enraged at a customer for belittling him and Josie, and is let go. Unemployed, unable to support his wife and son, he gets abusive and leaves – perhaps never to return.