The castle was built in this place, on the Narew, already in the fourteenth century, but the current look is completely different from the original. The first Gothic castle built bishops Płock in place of a wooden castle after burning it by the Lithuanians. In the first half of the sixteenth century it was rebuilt by Bishop S.Krzyckiego and P. Myszkowski. With this reconstruction has been assigned the Renaissance style.

In the seventeenth century and early eighteenth century was destroyed by the Swedes. The initial appearance of the lost next remodeling, in the sixteenth century, by the Bishop KH Szembek. In 1945, burned down, was rebuilt after the war.

From the castle preserved baroque chapel, bridge and underground. Currently housed in these rooms Polonia House.