The castle is first mentioned in written sources in 1307, when it was still the property of the Princes of Świdnica. In 1348, Bolko II of Świdnica sold it to the Prince-bishop Preczlaus of Pogarell (1341–1376), and since that time, the castle belonged to Breslau bishops.

During the 15th century, the castle was considerably damaged by the Hussites and therefore large-scale repairs were needed. The rebuilding of the castle took place under the rule of Bishop Jan IV Roth, at the end of the 15th century, and it was completed in 1509 by his successor – Prince-bishop John V Thurzó (1506–1520). At that time, the castle was also renamed as Johannesberg ("John's Hill"), to honor the patron of the Bishops of Breslau, John the Baptist.

The original fortified castle was later rebuilt in the Baroque style under the rule of Philipp Gotthard von Schaffgotsch (1716–1795), who made it his primary residence. During this time, Johannesberg castle and the town Javorník also became the cultural center of Upper Silesia. Among the most famous personalities living there, was August Carl Ditters von Dittersdorf, renowned Viennese composer and violinist.

Following the death of Prince-Bishop Philipp Gotthard von Schaffgotsch, the castle was once again rebuilt as a summer residence by Bishop Joseph Christian Reichsfürst von Hohenlohe-Waldenburg-Bartenstein. It remained an important centre of cultural life in the region until the beginning of the 20th century.

In 1959, the castle Jánský Vrch was loaned to the State and recovered by the Czechoslovak government in 1984, following a property agreement between the Polish and Czechoslovak Catholic archdioceses. It is now under the administration of the National Monument Institute in Olomouc and since 1 January 2002, it is on the list of Czech national cultural monuments.