Through the centuries, the Château de Varennes was destroyed several times. The oldest piece, a part of the gatehouse dates from the eleventh century. The main building is mainly sixteenth century.
During the French Revolution, the owner of the castle was suspected to help King Louis XVI. The punishment for that was "beheading". King Louis himself was arrested in a town called Varennes, but that was near Paris. The castle was auctioned by the state. Since 1809, the castle owned by the family Charveriat.