Halve and destone the plums and place in a bowl. Strip in the rosemary leaves, sprinkle with the icing sugar and leave to macerate for half an hour.
Meanwhile, make the frangipane. Cream the butter and caster sugar together, then beat in the eggs. Finely grate in the zest from the oranges and lemons and add the almond extract. Fold through the ground almonds and flour.
Preheat the oven to 180ºC/350ºF/gas 4.
Roll out the pastry on a floured surface to fit a 25cm x 35cm lined baking tray, going up the sides slightly. Bake in the middle of the oven for 15 minutes.
Remove from the oven and lower the heat to 160ºC/310ºF/gas 2½.
Spread the frangipane over the pastry and stud with the plums and flaked almonds. Bake for 30 minutes, then allow to cool slightly.
Combine the icing ingredients in a bowl and mix until smooth. Drizzle over the cooled tart, then serve.