Chilworthy, described in 1633 by Thomas Gerard, topographer, as having charming views over the country and sweet fresh air, is steeped in history and was known as Chelewurthe and Chilley in earlier times and can be identified in the Saxon Charter of llrninster of 725 AD. In the manor of Hillcombe, a house was listed in the Doomesday Book. In 1242 Simon of Grydeham was in dispute with Abbot Walter of Muchelney over land at Chelewurthe. Simon of Hillcombe is shown as the owner of the property in 1243 and the Close Rolls of 1272-1337 lists Richard Miles as holding Chilley.

In the 15th century John Sydenham married a Wyndham and came into the property through the marriage. Four generations of Sydenhams resided at Chilworthy, each marrying into the great families of the district and linking together the Lovenis, Herods, Spekes, Warres and Cophams.

John Sydenham married in 1507 and held property in Combe St. Nicholas. He had five sons. In 1615 Henry Sydenham married Edith, daughter of Humphry Walrond and had five children. Legend has it that the two sons did not marry and fought in the Battle of Sedgemoor. In great secrecy the night before they left for the battle they buried the family treasure in a spot known only to them. They were both killed in the Battle and the treasure, never discovered is said to remain in the grounds to this day. The Duke of Monmouth himself is reputed to have supped at Chilworthy the very night before the unfortunate march to Sedgemoor. Henry Sydenham was living at Chilworthy in 1654. He died in 1671. He is buried with his wife at Combe St. Nicholas.

It is a great coincidence that later in the history of Chilworthy, approximately 1900, it was the residence of Gordon Halswell born 1874 who was a direct descendant of Hugh Halswell who was also killed at Sedgemoor. Two of his brothers were executed by Judge Jefferies, but the youngest brother aged 18 escaped to Holland, married Maude Van Eyk of Amsterdam and continued the Halswell line.

At the turn of the century, approximately 1910, the house was bought by Sir Herman Runge, Monica Dicken's grandfather. As a child Monica Dickens spent many holidays at Chilworthy and wrote her first two novels, Marianne and The Fancy around the house. In her autobiography, An Open Book, she recalls her childhood days at Chilworthy, and includes photographs from her family album and describes it as "a privileged paradise".

Between 1700 and 1900, there were two serious house fires at Chilworthy.  As a result of the fires parts of the house, of necessity, were rebuilt using many of the beams and mullions that were still there and, fortunately, local Chilworthy flint stone so there was little difference between the old and the rebuilt.