I hear this a lot at work, it can be used as an excuse not to start something new or to stop doing something, or at least to do less.
Some niggles you may get, you may put down to age but most of the time it’s just not true, allbeit the recovery process slows down, it may just be that your body is not functioning correctly.
I worked with a club runner a little while ago, it almost seemed overnight he started to significantly slow down in his running pace, so he thought it may have just been age, club mates were confirming that for him, but he came to see me and I coached him and he went back to his previous speed and then even got faster!!
So if you’re worried about an niggle, downward spiral in fitness or it being too late to start something new, keep at it, seek help if you need to, the old saying that you ‘can’t teach an old dog new tricks’, your body will adapt to anything you throw at it every day you’re alive, so go for it.