In 1314 King Robert the Bruce defeated King Edward II at the battle of Bannockburn near Stirling castle. Edward fled down the coast passed Edinburgh and on towards Dunbar pursued by James the 'Black' Douglas. The English garrison at North Berwick,on hearing of their King's flight and the approach of the 'Black ' Douglas,deserted their posts; "they gave up this castle and retired to the castle of Dunbar",as Lord Dunbar was still in league with the English and ships could safely dock here to evacuate Edward and his men to 'South Berwick' (Berwick on Tweed),which was still in English hands since the massacre of the entire Scots population in 1296. Because Dunbar was too strong for Douglas to besiege,he probably slighted North Berwick,as like the Bruce he had a policy of destroying castles in the Lowlands to prevent them from being used by the English as stepping stones for re-invasion.

    During the reign of King Robert II of Scots (1371-1390) the Stewart Earls of Fife held the Barony of North Berwick with it's 'Castle'. In 1373 North Berwick was granted a charter as a Royal Burgh since it was the key ferry port to and from Fife. In the late 14th century a stone tower with barmkin was raised on the site by the Lauder family,who also constructed a keep on the Bass Rock island some three miles east of North Berwick in the Firth of Forth. For some unrecorded reason the 'Castle Hill' appears to have been abandoned in favour of the more secure Bass Rock castle some time before the 1420's and may have been used as a quarry by the Lauders,who by 1426 had added 'a curtain wall and a landing stage' to their Bass castle. Also their manor house of Tyninghame was reconstructed around this time,so little remained of the 'Castle' when the site and lands around were given to the church in 1435. The Lauders were originally Constables of Tantallon castle,a great coastal fortress perched on the cliffs east of North Berwick. The principal seat of the 'Red' Douglas family. However in 1406 the Lauders came into conflict with their Douglas masters over the ownership of Tantallon,which may explain the sudden disappearance of 'Castle Hill'. The background to this conflict is complex.

    In 1388 James 2nd Earl of Douglas was killed at the battle of Otterburn. His title should then have passed to his young half brother George the 'Red' Douglas of Tantallon. However the title was seized by Archibald the 'Grim' Douglas (an illegitimate son of the 'Black' Douglas). There was even an attempt to seize Tantallon by the Stewart faction through the 2nd Earl's brother-in-law Malcom Drummond. But the Lauders with the young 'Red' Douglas's mother and her kin the Sinclairs of Herdmanston refused Drummond entry by show of force. They along with the Lyndsays of Byres also blocked any Stewart claim to Tantallon. Though the Stewarts held the nearby Barony of North Berwick the castle of Tantallon was originally built by William 1st Earl of Douglas around 1360 and was therefore a Douglas stronghold.

    In 1389, Drummond's failure to become keeper of Tantallon,resulted in him seeking English help in pursuing his own land claims to the Douglas estates. However, as he led his English army into Scotland,he was ambushed by Archibald the 'Grim' 3rd Earl of Douglas. With his forces destroyed,Drummond is thought to have fled to France as his body was never recovered from the battlefield. The 'Red' Douglas then became 1st Earl of Angus,when his mother resigned her title to give him the status required to hold Tantallon in his own right. The Stewarts then agreed to his continued occupation of Tantallon.

    These double dealings were not only confined to the nobility but extended to the Royal Stewart household. In 1402 Prince David was 'arrested' by his Uncle the Duke of Albany and his brother-in-law Archibald 4th Earl of Douglas to keep the peace. But as the Prince was 'long guarded' at Falkland castle he died of starvation. Since King Robert III of Scots (1390-1406) was a sick,weak willed man his brother Albany escaped punishment for this crime and continued to plot the King's overthrow and the murder of young Prince James (later King James I of Scots 1406-1437).

    In February 1406,Sir David Fleming with Robert Lauder and a 'strond band' of the 'leading men of Lothian' accompanied by Prince James as a symbol of Royal authority marched on Tantallon castle. It appeared King Robert or rather his lackeys were re-asserting the Stewart claim to Tantallon made and abandoned in 1388. As plots were afoot to kill the Prince,Tantallon was an ideal coastal stronghold to hide the him until he could be shipped to France. The Sinclairs of Herdmanston acting as protectors of young William the 'Red' Douglas 2nd Earl of Angus refused the Royal party entry. The violent verbal barrage between the gatekeeper and Fleming exploded into actual violence which 'shocked' the Royal army into a quick flight to Robert Lauder's North Berwick castle. Expecting an assault by the Tantallon garrison the Prince was rowed out to the even safer Bass Rock castle to await a ship to France. Fleming and the Royal escort appeared trapped. To the east was Tantallon,to the south Herdmanston castle and Byres castle both held by garrisons loyal to the 'Red' Douglas faction. To the west was Dirleton castle held by the Halyburtons whoses loyalties were doubtful. Beyond that lay Edinburgh castle held by Archibald Douglas's brother James the 'Gross' Douglas who as an ally of the Duke of Albany wanted Prince James dead so Albany could become King.

    With great haste,the Royal army raced passed Dirleton,flying the Royal banner implying the Prince was present. The army was shadowed by the Halyburtons and the Sinclairs who had already sent riders to Edinburgh castle which drew out the 'Black' Douglas army from there. After a lengthy pursuit,the Royal force was 'overtaken' at the battle of Long Hermiston Moor and after 'a terrible fight' routed with their leader David Fleming killed.

    No doubt the rebel army was sent by the Duke of Albany as Prince James was the only stumbling block to his goal of seizing the Scots throne. The rebel army marched on 'Castle Hill' to check the location of the Prince,though there is no written record of a siege of North Berwick or of the Bass castle as the rebels were unwilling,or unable,to assault the Bass,which was said to be a 'fortress'. It may have been the case that the 'Red' Douglases sacked 'Castle Hill' because the Lauders had betrayed their trust. Later Prince James boarded a ship the 'Maryenknecht',to France but was intercepted off Flamborough Head by the English and held captive for 18 years. On hearing of his son's plight,heartbroken King Robert died. His brother Albany then took control of the Kingdom as 'Governor'.

    Eventually,in 1424 Prince James,now King James I of Scots,returned to Scotland. In that same year the 4th Earl of Douglas,ally of the Stewarts of Albany,was killed in France at the battle of Verneuil. In the confused aftermath of his death King James moved quickly,sending Robert Lauder to seize Edinburgh castle from the 'Black' Douglases. Then,when the time was right,the Duke of Albany was arrested,taken from his home of Doune castle near Stirling to St Andrews castle for a mock trial then down to Caerlaverock castle in Dumfries by the new 5th Earl of Douglas,who unlike his predecessor was in subjection to his King. Albany's son Walter was taken to the Bass castle by the Lauders,his wife the Duchess of Albany was imprisoned in Tantallon castle's dungeon by the 'Red' Douglas and his father-in-law,the Earl of Lennox,was held at Edinburgh castle,also by the Lauders.

    The stage was set for James's revenge. The Duke of Albany,his son and his father-in-law were all reunited at Stirling castle,then beheaded. Bizarrely the three heads were taken first to the Bass castle,then the Lauders shipped them to Tantallon castle where the 'Red' Douglas in turn threw the heads into the dungeon beside the Duchess in an effort to drive her insane. No one knowns what pain and grief she must have experienced as she peered in the half-light to identify the head of her husband,then her son,then her father in turn. Likely the King was present to hear her despairing cries. Soon after the 'Red' Douglas took pity on the Duchess and moved her to a more comfortable chamber,where at his insistence she signed an acknowledgement to say what the King had done was just as the Stewarts of Albany were rebels. This kept the King happy as he moved on to seize Albany lands.

    Douglas continued to confine the Duchess within the bounds of his castle for her own safety,since King James was such a volatile character and,given his later treatment of the MacDonalds and the Dunbars,could turn on the Duchess at any time if she were left unprotected. Ironically the Duchess outlived the King,who was murdered by rebel Lords at Perth. She then re-asserted herself as Duchess of Albany,reclaiming most of her stolen lands.