The first baronial residence was built in the thirteenth century, in the form of a simple manor with an irregular surrounding wall. Later that century, considerable fortification work was done, particularly the three towers which were added to the fortified entrance tower. A large residential building was built inside these fortifications. The fourteenth century saw considerable development of the château defences, with a moat and a new tower entrance boasting a spiked drawbridge. In the second half of this century Blandy became the model for fortified castles. The keep, 39 metres high, defended by two drawbridges, was built. The keep was altered by the addition of new towers and curtain walls.
Between the fifteenth and seventeenth centuries, the fortress gradually lost its military purpose and became a residential castle.
More domestic buildings were built. In the early eighteenth century, Maréchal Villars, the owner of Vaux-le-Vicomte, bought the lands and fortress of Blandy, which was pulled down and turned into a farm. Little by little, the fortress disappeared.