Szydłowiec in the Middle Ages was the ancestral seat Odrowąż who in the fifteenth century adopted the name of its headquarters - Szydłowieccy. In the fourteenth century they built a brick mansion of wetlands on a slight elevation, which slightly increased. The first mention is from 1427 but it was only when he was owned by James and Slawek Odrowążów. In the first buildings were discovered traces of walls on the outside of the north wing. In the second half of the fifteenth century manor house has been expanded, the site of an older building established residential tower with four floors, and the western end of the island was built quadrilateral gate tower. Probably both buildings were in the outer walls.

In the years 1515-1526 Treasurer of the Crown Claus Szydlowiecki rebuilt its headquarters at the impressive Renaissance castle.

The assumption was-wing. The northern wing absorbed the remains of an earlier residential tower and the west wing joined four-sided tower with the north wing. Representative character was the eastern wing of the three floors. The southern side of the courtyard closed the curtain wall.

In the year 1550 shortly Szydłowiec part owner of the goods and the castle was Nicholas Myszkowski, and since the end of the year for nearly three centuries, the castle belonged to the Radziwill. In the early seventeenth century were carried out work on the interior. During the restoration works on the pillar supporting the large vault was discovered signature cuisine stonemasons John Poniatowski and Sebastian Młoskiego and the date 1602.

Albrecht Radziwill in the years 1619-1629 extended the residence. It was then that the external loggia on the axis of the east wing, on its corners dostawiono dependence with small terraces, and a courtyard at the junction of the north-east wing was built staircase. During this extended gate tower was raised and placed in the chapel. Castle received extensive facilities, coffered ceilings, murals and friezes.

Radzivills sold the castle Annie Sapieżynie in 1821. Then the residence was owned by the state. In the second half of the nineteenth century, the castle declined. After the destruction and subsequent of the Second World War the castle was protected in the years 1951-1952, and after 1965 years fully restored. Today it houses the Museum of Folk Musical Instruments.