Ezra Smith was born in Sudbury, MA in 1755. It is not known exactly when he came to Nelson. He built this house in 1781. He was a farmer (as was most everyone) and he became the pound keeper in Nelson from 1786 until his departure for New York in 1824. In 1795 the town voted to create a new pound at D-3-2 on his property as a convenience for him (the previous pound was at D-4-2). Smith sold the place to Dr. Calvin Hubbard and moved to New York State with his son. He is believed to have died in Belleville, NY, a town smaller than Nelson, on the east side of Lake Ontario, North of Syracuse. He apparently retained a Nelson connection: he is buried in the village cemetery.
Dr. Hubbard had come here from Springfield, VT. After a while he grew more interested in raising sheep than practicing medicine. Hubbard sold the place to Moors Rice in 1836 and returned to Springfield. Sometime after that the places was acquired by George Granville Hardy who with his wife Mary (Stevens) raised 5 children. At age 46 Hardy answered a call for volunteers and joined the 16th NH Regiment. He died of disease in Louisiana leaving his widow with children ages 1 through 12. Mary died seven years later.
The place was later acquired by John Dixon who was its last occupant.