Many generations of the family lived in the multi-tunneled mansion. They were deep Hobbit-holes and during the Scouring of the Shire, the Took family took refuge in them and were safe from the ruffians of Sharkey. Tooks came out of the Smials to hunt the ruffians who were on their land.
There was a room in the Smials that Gerontius Took lived in for years. It was dim and stuffy; the furniture hadn't been moved or changed for generations. Peregrin Took compared Fangorn Forest to it.