This was the second house built on the place by Stephen Osborn. He likely built it sometime between 1815 when he formally acquired the land and began construction of the mill at F-5-12. He and his wife Rachael were succeeded on the place by his son-in-law, Nathaniel Davis and Stephen’s oldest daughter, Mary. Stephen started as a modest farmer probably leasing the property from 1799 until his purchase in 1815. He built the mill on Bailey Brook, a modest stream with a substantial waterfall. Stephen built a mill that used the modest power available to turn handles for common hand tools.
When Stephen died in 1844, his estate inventory reflected the contents of a wood working mill and the household of a prosperous man. When Nathaniel Davis died in 1866, the inventory was that of a person of very modest means and the woodworking tools were gone. The mill had failed. Mary live on the place for a few years.
In 1875 the place was purchased by Parker and Elizabeth Hart from Hancock, Massachusetts. They seem to have used it as a summer home. It has been called the “Hart Place” ever since. In 1901 they sold it to Louis Cabot as Boston industrialist who accumulated thousands of acres in Nelson. The road to the site was closed by the town in 1922. The buildings were probably gone several years earlier.