Like all similar fabrics of its time, the Cathedral of Elgin stood due east and west, and was built in the form of a Jerusalem or Passion cross. The choir and altar faced the east, or head of the cross, with the branches, transepts, or cross wings, to the north and south, and the grand entrance through the western extremity, or foot of the cross. The grand tower rose from its centre. The west gate, flanked with two massive but elegant towers, and the chapterhouse, appended to the northern cloisters, with parts of the transepts, are all tolerably perfect; the whole displaying workmanship of the most intricate and exquisite beauty.

Surrounding the cathedral was a substantial wall. 8 feet in height, and entered by five gates. It enclosed an area 900 yards in circumference, called The College, and included the manses and gardens of the Dean, the Prebendaries, and the other dignified members of the chapter, and a paved street ran around the whole area. The only gate to the precincts now remaining is the eastern, named the Water-gate, or Pann'i port, which was formerly defended by an iron portcullis. The College was the residence of the Dean, who was Rector of Auldearn. The manse of the sub-dean still exists, but has been much enlarged and altered. The Episcopal Palace is on the south of the Cathedral.

In 1589, the Abbey of Dunfermline, with its lands and privileges, was erected into a temporal lordship, which was conferred upon Anne of Denmark, Queen of James VI. In 1593, Queen Anne appointed Alexander Seton, son of George, 7th Lord Seton, Heritable Baillie of her Lordship of Dunfermline. The house of the Bishop of Moray, in the town of Elgin, was granted to Alexander Seton, along with the Priory lands of Pluscarden and the Lordship of Urquhart.

The Elgin mansion received the name of "Dunfermline House" from the circumstance of the priories of Urquhart and Pluscarden being dependent upon the great Fifeshire Abbey. From a letter written by Lord Dunfermline to John Innes of Leuchars, in 1618, he seems to have then been engaged in embellishing the gardens of his Elgin abode. "I think," he says, "all ye have done to my yeardes wereye weill and ordourlie, and am content ye superseid the outredding of the warke, till your leisour and commoditie may permitt you to see it donne. Insteid of thankis and recompence, I am even to burdein you the forder, and to requeist you sa sone as you may in the nixt sasone, after the ground shall be redde and cleare, to cause outredde and cleare the same; for truilie I think lang to be in that countrie."

The Precentor's manse was granted to Alexander Seton simultaneously with his appointment as lay commendator of Pluscarden Priory, and he renamed the manse to Dunfermline House during the period when he was Provost of Elgin (1591–1607, and after was also Provost of Edinburgh, 1598–1608). In 1604 he was appointed by King James VI, Chancellor of Scotland, and the following year was raised to the peerage with the title of 1st Earl of Dunfermline.

In the year 1595, he had sold the barony of Pluscarden and certain other lands to Kenneth Mackenzie of Kintail, but retained the Lordship of Urquhart and the mansion-house in Elgin, where he appears to have frequently resided. In the Charter of Sale, which is now at Duff House, he describes himself "Alexander preses Collegii Justicie," and signs "A. Seton, Urquhart," along with his first wife "Lilias Drumond." 

The house being frequented by cadets of the Seton house, from the registered Testament of the " umq. Captain Patrick Seytoun, brother german to umq. John Seytoun of Lathrisk," that he died " in Elgin, in Murray, in the hous of the richt nobill and potent Lord Alex'- Lord of Fyvie, President," on the 16th of February 1600. The Will is dated two days previously, and witnessed, among others, by Lord Fyvie. The testator leaves various legacies to nephews and other relatives, — among the rest, 900 merks and his "monturs" [saddle-horses] to be as heirship to John Seytoun, his nephew and heir of line; and to Janet Duddingstoun, Lady Lathallan, 200 merks, "together with his braceletts of gold, silver saltsatt [salt-cellar], and two spoons, with a coupe."

A detailed account of Dunfermline House will be found in Chalmers's History of Dunfermline, vol. ii pp. 404 and 432. A shield of arms, surmounted by a coronet and accompanied by the date " 1688 " and the initials " I. E. D." and "I. C. D.," was formerly over a door in the north court of the building — the initials being those of the Chancellor's grandson, James, 4th and last Earl of Dunfermline, and his Countess, Jean Gordon, sister of George, 1st Duke of Gordon.

Following the forfeiture and death in France of the 4th Earl of Dunfermline, it's title and ownership passed to the Seton's of Barnes. Sir George Seton of Barnes and Hailes held claim to it, selling his share and right to the Duke of Gordon, who although preserving it for time, the house is now in ruins.

Captain A. H. Dunbar kept a sketch of an old stone at Elgin, bearing three curiously shaped escutcheons charged with the arms of Seton, Dunbar, and Falconer. The first, exhibits Seton and Buchan quarterly, between the letters "A. S." (the initials of the Chancellor), and surmounted by the words -" lESVS RENVE A RIGHT SPIRIT WITHIN GOD."