Mysterious Uncle

I find it odd that the children's uncle does not want to have anything to do with them after their parents' death. The uncle's thoughts and past are not expressed in the novel, but it raises several questions to me as the reader of why he doesn't want to have any contact or have any input on decisions about the raising Flora and Miles. His situation leaves me to make my own assumptions that maybe the uncle had a bad experience in the house, is afraid of the house, or something has happened with previous servants or himself.