There is a baseline athletic ability needed simply because you are typically walking 10+ miles (roundtrip) and gaining 3000 vertical feet, all while breathing very thin air. However, if you prepare correctly (clothing, weather, start time, etc) and take your time it's not so bad. Altitude effects everyone differently, some people get altitude sickness while others feel no change. I grew up at a higher elevation (9000' asl) so it tends to not effect me much, but my wife can only stand about a couple of hours or so above 12,000' before she starts to get sick.