However, pirate raids, the continual uprisings against the Venetian occupation by the local people and the great earthquake all led to the partial destruction of the fortifications until the Venetians themselves were forced to completely destroy them with the intention of rebuilding them. This never happened and in 1651 the town was razed and then occupied by the Turks.

During the Turkish occupation it would appear that the walls were never rebuilt but the Kazarma was restored and evidence of the Turkish extensions can be seen today, for example in the cupolas ("koubedes") on the battlements that form the watch towers. The Kazarma has since been carefully restored and is open to the public offering a panoramic view across the bay of Sitia.

Concerts, plays, lectures and art exhibitions all take place in the Kazarma during the summer months as part of the festival known as "Kornaria".