Castle in Szydłów probably already existed in 1255 when he was here with his court Boleslaw the Chaste. Probably the former princely residence was located at the site of the present church of All Saints. In the fourteenth century, Kazimierz the Great surrounded by walls Szydlow, who probably already had the civic rights of the first quarter of the fourteenth century. The castle was built at the same time the city walls made ​​of them a single entity, occupying the western corner of the city; urban area ended up here escarpment descending to the valley of the river Ciekącej. Castle walls include an irregular polygon area of ​​over 7000 m2. Along the southern section of the walls stood the main residential building of the castle, which is the royal residence. This building, whose walls have been preserved to this day in full, has a plan of a rectangle with sides 13 and 35 meters, and originally was divided into two floors into two chambers. From the outside, the building had a break was probably housed the castle chapel. In the western corner stood two cylindrical towers, which are located within the walls was probably pulled out "beak", which exposes the plan Bohdan Guerquin. Slightly larger in the plan from the residence administrative building of the castle was located in the northern corner of the yard, also had two floors and the ground floor was divided into three chambers. The entrance to the castle was located in the eastern section of the walls of the town.

The castle was probably repeatedly damaged by fires ravaging the city. In the fifteenth century the two round towers were demolished and the site turned on in sequence walls erected a square tower was called Skarbczykiem, to which later was added a rectangular building with buttresses. In 1528 Skarbczyk was increased by one storey.

In 1541 years Szydlow again suffered from a fire that completely destroyed the castle buildings, among others. Soon after, it was the reconstruction of the former royal seat and vetting of 1564 years already mentioned about the castle rebuilt. During this period, however, the castle was no longer their former residence of the rulers of the function, even though the reconstruction was carried out in the Renaissance style on a grand scale.

After the fire of 1630, the damage done by the Swedish army during the flood in 1656, the castle was restored and replaced the previous entry gatehouse. Even in 1723 at the initiative of the mayor Joseph Zaluski rebuilt and modernized the castle, but soon after it was abandoned. In 1789 years was no longer able to deepening the ruins and remains of 1822 years were exposed to auction for the purpose of demolition. Plans implemented, but still dilapidated buildings.

Since the beginning of the twentieth century, the safety work. In the interwar period also were preserved city walls, and in the years 1945-1947 J.Żukowski conducted work that largely restored to their original appearance castle walls. The main building was then slightly elevated and crowned crenelage. Later still Skarbczyk rebuilt and allocated it to a small museum. Recent work started in 1999 under the direction of Leszek Kaiser.