The earlier Irish castle was occupied by the Harold family, who held it as tenants of the le Bret family. It was replaced by the present building built on lands confiscated from the Eustace family of Baltinglass, to whom it had passed, because of their involvement in the Second Desmond Rebellion. The Geraldines defended the Pale from the Irish clans in the nearby Wicklow Mountains. It is believed the present castle was built around 1583 for Archbishop Adam Loftus. Originally a semi-fortified and battlemented structure it underwent extensive alterations in the 18th century.

The castle consisted of a square building four stories high with a projecting tower at each corner, the walls of which were an average of 5 feet (1.5 m) thick. On the ground level are two vaulted apartments divided by a wall nearly 10 feet (3.0 m) thick which rises to the full height of the castle. On a level with the entrance hall are the 18th century reception rooms and above this floor the former ballroom, later converted into a chapel.

Rathfarnham was described as a "waste village" when Loftus bought it. His new castle was not long built when in 1600 it had to withstand an attack by the Wicklow clans during the Nine Years' War.

The castle was sold to Delaware Properties in 1985 and it was feared that it was facing demolition. After immense public pressure to save the building, in 1987 it was purchased by the State and declared a National Monument. Currently the Office of Public Works is carrying out extensive refurbishment throughout the castle but it is still open to the public during the summer months (5 May - 12 October). The Castle is presented undergoing active conservation, where visitors can see at first hand tantalising glimpses of layers of its earlier existence uncovered during research.