To the north of the ancient market town of Haddington behind the Garleton hills sits the ruin of Garleton castle, with an intact crow stepped gabled hall-house, a row of modern cottages which incorporate earlier castle material (including 16th century cannon-loops) and the remains of a drum tower with three basement vaults set into an angled wall. L-plan in form, these vaults which suggest the distribution of weight from several levels above, are all that remain of the castle raised in the late 1500's by Sir John Seton (treasurer of the household and Lord of Session under King James VI of Scots 1567-1625) on land originally owned by the "Lyndsay" family of nearby Byres castle. Locally the Lindsays also held Luffness castle beside Aberlady and lands around Ormiston which the latter was signed over to the Cockburn family.

Garleton incorporates fabric from an earlier Lindsay tower on the site, and rubble taken from Byres castle, as it appears to have been dismantled by the English in 1548 to ensure the security of their 'Fortress of Haddingtoun' during the wars of the 'Rough Wooing' (1544-1550), where by the use of castle-sacking they hoped to force the marriage of the infant Mary Queen of Scots to the English Prince Edward.

The Seton family suffered heavily during these times with the burning of Seton Palace, Seton Collegiate church, St. Germains House, Tranent Tower, and the destruction of Winton Castle and other properties in 1544.

Garleton was acquired by George, 7th Lord Seton and passed to his 3rd son, Sir John Seton who had assisted his father in his Diplomatic post in France and was was then sent to service for the King of Spain.

The 1st Sir John Seton of Barnes, Baron of Barnes and Lord Barnes, rebuilt the Castle of Garleton as it remains now, with a completed formal courtyard, while he was waiting on his more elaborate castle at Barnes to be completed.  As he died in 1594 prior to the completion of Barnes, and Garleton Castle passed to his older brother, Robert Seton, 8th Lord Seton and later 1st Earl of Winton, who passed it to his 2nd son George Seton (later 3rd Earl of Winton), who passed it to his own younger son, the Hon. Sir John Seton of Garleton and 1st Baronet of Garleton, and hence founding the 1st of the branch known as the Seton's of Garleton.

In 1724, following their forfeiture of 1715-16, the Seton branch being somewhat impoverished, sold Garleton to the Earl of Wemyss. The presence of the modern cottages on site highlights the common practice of Victorian builders to dismantle such ancient towers as Garleton viewing them as ready made quarries failing to appreciate their historical and architectural value.