The castle, as we know it, is believed to have been built on the site of an original earth built motte and bailey. Legend has it that the stone from the castle which fell into disuse in the reign of Elizabeth I was used in the rebuilding of the town, but I doubt the truth of that.

One of the many stories that surround the death of Llewelyn says that the gatekeeper of the castle at Builth refused entry to Llewelyn, he then took refuge in a cave near Aberedw and he was then slain by English knights at Cilmery (Cefn-y-bedd), to the west of Builth. His head was severed and taken to be paraded in the London, while his body was taken to Abbey-cwm-hir, a few miles to the north, and site of a cistercian abbey. Another story has it that Llewelyn and his army camped at Rhosferreg, just across the river Irfon, and near Cilmery. On a recce, or on his way to a meeting at Llanynis, he was killed by a party of English soldiers who didn't at first realise who he was. But, there are many variations on both of these tales, and even more different stories.

At Cilmery (Cilmeri or Cefn-y-bedd), two miles to the west along the A483, a monument to the death of Llewelyn has been erected. This large monolith replaces an early, smaller, but attractive monument which is now incorporated into the walls of the steps leading to monument. At the rear of the monument site is Llewelyn's well, like many others, is said to bring good luck in return for a coin tossed into the water.The castle site, often used for grazing land, is owned by White House farm, the farmhouse itself dates back to 18th century. (The farm is the white building in the centre of this view from bailey of the castle.) The castle is under the control of CADW - the Wales department that looks after ancient monuments. The Castles of Wales web site has page upon page on the Welsh castles and the many historic figures who influenced the forming of Wales. Builth castle has an excellent article, well illustrated on the site. To the west of Builth, near Caer Beris manor, is the remains of another castle, much smaller than that of Builth. A look at the Castles of Wales website or at an Ordnance Survey map of the area reveals many more castle, or mottes scattered over the area, a fair indicator of the troubles of those early years. Little seems to have been undertaken to discover any archaeoligical evidence of the castle (apart from a dig in 1970s by Prof Spurgeon). Several underground bunkers are rumoured to have been dug during the Second World War for ammunition dumps, but were abondoned because of water and collapse, and there are certainly marks on two of the banks which back this story.