Originally the Lands of Garleton were a possession of the Lindsay family, though they were acquired by the Lords Seton during the latter part of the 16th century.  It may be that Sir David Lindsay, author of the play Ane Satyre of the Thrie Estaitis, was born in a house on this site in 1486. Garleton is an example of a courtyard castle and originally consisted of three blocks within a curtain wall.  It is one of the chain of castles stretching from the coast at Aberlady through the valley floor.

After the Lindsays, the property was briefly owned by the Towers of Inverleith, but they sold it to the Seton family. George, 7th Lord Seton acquired Garleton, as well as various other lands in the area, which were bestowed upon his 3rd son, Sir John Seton of Barnes, the diplomat at Philip II of Spain's court and Master of teh Royal Household and also of the Royal BedChamber, and later James VI's Treasurer of the Household and a Lord of Session, and who died in 1594. 

He was responsible for rebuilding the castle as we know it and which survives today, using stone from Byres Castle which was destroyed in 1548.  His descent also later owned Hailes Castle in East Lothian.  Hailes was reduced to ruins by Cromwell in 1650, and thereby remained as such.

Garleton was built as it is now, after 1550 and before Sir John Seton's death in 1594. It is now a ruined castle on the northern slopes of the Garleton Hills in East Lothian, located at East Garleton, 2 miles (3 km) north of Haddington. The courtyard castle once comprised three blocks enclosed by a curtain wall. A small section of that wall and parts of one of its round towers still survive.

The basement consists of three vaulted chambers, now below ground level, above which are thought to have been a further three storeys. At ground level are a number of gun loops. Built onto the wing is a large, circular tower measuring around 7m in diameter.  The tower house is now a substantial but precarious-looking ruin, rising to a height of around 10m, and peppered with window openings.

At the south-west corner of the site was a hall house, the basement of which was vaulted and consisted of two rooms. One was a kitchen with a wide-arched fireplace, while the other room features a canopied fireplace.  Access to the main hall above was via a round stair tower in the middle of the south wall. Gun loops provided defence for the hall house. Wemyss used stone from the site to remodel this building, although the vaulted basement, stair tower and gun loops remain. It was latterly used as a farm steading.

At the north-west corner was another building thought to be largely similar in size and design to that at the south-west corner. This corner is also the likely location of a gatehouse into the walled enclosure, with a bridge crossing the ditch.  One of the blocks was replaced by cottages in the 19th century but to the south is a complete hall-house, featuring crow-stepped gables, a round stair-tower and gun loops. Although much-altered and latterly used as a farm steading, its vaulted basement including two rooms is still extant. These rooms were a kitchen with a wide-arched range and another room featuring a canopied fireplace.

Garleton appears to have consisted of three main buildings of mixed rubble construction, a crow-stepped gabled hall-house with cannon-loops on the south side,a possible L-plan tower house and a lean-to conical roofed drum tower also with cannon-loops on the north and east corners respectively. All attached together by a roughly oblong plan courtyard wall. The present south wall linking these buildings is obviously modern, lacking cannon-loops, deviating from the logical courtyard plan and hindering the fields of fire for the tower gunners.

The west wing is completely obscured by the modern cottages but may well be the site of the courtyard gatehouse with outer bridge spanning that approach. For defensive reasons the entire site would have been surrounded by a deep fortified ditch, thus allowing the castle gunners a clear field of fire in all directions. It is unclear wither these defenses were every put to the test.  Other castles of the area: of Innerwick, Hailes, Dirleton and Tantallon; are all mentioned during Cromwell's sacking of the Lothians after the defeat of the Scots army at the battle of Dunbar in 1650. His actions may explain the disappearance of Garleton's neighbours of Kilduff tower and Athelstaneford tower.

What is misleading about the castle's location today is it wasn't a lonely tower house, but one in a chain of towers that dotted the valley floor. With Ballencrieff and Byres to the west, to the north sat Kilduff tower, to the east Athelstaneford tower and Markle castle with Barnes castle on the Garleton ridge, also built by Sir John and which never got higher than vault level before his death. 

The castle of Garleton was passed to Robert Seton, 8th Lord Seton, the older brother of Sir John Seton, and who later became the 1st Earl of Winton. He then passed the ownership to his 2nd son, George Seton, 3rd Earl of Winton (1584 - 1650), who gave Garleton Castle to his youngest son, the Hon. Sir John Seton of Garleton (1639 - 1686), 1st Baronet. Of the Seton's of Garleton, they were briefly recognized as the heirs of the Earldom of Winton in 1769, after George, 5th Earl died in exile Rome 1749.

With the Seton estates being forfeited following the involvement of the head of the House, George Seton, the the 5th Earl of Winton in the Jacobite Rising of 1715, George Seton, the 3rd Baronet of Garleton was also forfeit.  In 1724, the property was sold by George Seton, 3rd Bart. of Garleton to James Wemyss, the 5th Earl of Wemyss (1699 - 1756).

Subsequently much of the stone from the estate was reused to built the farm cottages now occupying the north side of the site.  In the 19th century, one block was replaced with new cottages built from the castle's stone, incorporating gun loops from the previous building as a decorative feature.

The presence of the modern cottages on site highlights the common practice of Victorian builders to dismantle such ancient towers as Garleton, viewing them as ready made quarries failing to appreciate their historical and architectural value.  Indeed locally the ruin of Barnes is still known as 'the vaults', and used as a store-house, and Kilduff tower and Athelstaneford tower have no traceable remains though a castle dove-cot is still around in the present village of Athelstaneford.