Hi there, Summer temperatures in Kaly are generally around the 70s in the morning and then climb up to the high 80s in the late afternoon. Apart from the occasional day, this isn’t so much a problem for climbers as you might think: the rock is rough, there’s often a wind and so it’s OK to climb until early afternoon (or all day at north-facing places like Secret Garden). The main hold-up to harder climbing is usually humidity not temperature as this can vary from 40% to 90%! In summary, apart from maybe a handful of days, Kalymnos is one of the best summer spots for climbing in Europe.
As regards crowds, as long as you have reservations there won’t be a problem. The non-climbing tourism is fairly low key (i.e. not particularly rowdy… apart from the Greek weddings!) and the density of climbers (mainly Italian and Greek) is moderate. The main beaches, e.g. at Masouri are fairly busy (but you can always get a sunbed) but it’s not that hard to get a good beach where there’s only one or two other people. Terrible, eh!! Hope this helps you decide.