I learned to kayak in college and have been doing that along with rafting for several years now.
I only do whitewater, but most of these suggestions would apply to you.....it sounds like you are interested in flat water/recreational kayaking.
Take a class, especially if you get a sit in kayak and use a spray skirt.
I suggest never going by yourself especially if you are going to be in either moving water (river) or a large body of water (large lake/ocean). I suggest a class because there are some important safety information you need plus you can meet other like-minded folks to go along with. Most local shops that sell kayaks and gear can point you in the right direction. Try to find the specialty shop(s) in your area that deals exclusively with kayaks, canoes, and rafts rather than the mega-sports store for 1 simple reason, they have people that actually know what they are talking about can hook you up with classes/info. Typically the gear is of higher quality as well.
Know the water you are boating, most rivers and lakes people paddle on have been written about in river guides. Here in Oregon we have one called Soggy Sneakers which I constantly use. A very common problem with beginners is that they put onto a relatively calm river, but do not realize there are serious downstream hazards that veterans know about. An example are low head dams, <---------these are man killers.
Look online for used kayaks and gear on Craigslist, Boatertalk.com, mountainbuzz.com, the last two have gear swap pages as well as message boards you can post questions on.
Do not cut corners on you PFD. #1 Reason: you want it to work properly and you need it to fit so it can work properly. Also unlike some water sports you will have the thing on the entire time in the water and they the design of a kayak PFD is different than what you want of a water skiing PFD...... if it is uncomfortable it is really going to suck.