For the last two years we have been in Kalymnos for New Year.
It is cold, down jackets, woollen hats, gloves, but can be lovely and crisp.
It could also rain and then the only thing to do is go to the cake shop in Elies! But if the sun shines you can get out to the crags. You just have to be careful of damp tufa if it has been a wet autumn (no rain so far). I think it rained for about a day when we were there last year and didn’t rain at all the year before. It’s a bit pot luck though.
Sorry I cannot give you any guarantees, Kalymnos is beautiful at all times of the year.
Ruth