Absolutely not. If you do take ballroom lessons, I would recommend taking them consistently in a single dance for awhile. Once you learn one dance well, it will be much easier to branch out. You should also see if there are any local dance clubs in your area (clubs that play jazz music often have regular swing nights). Go online to see if there is a ballroom dance or swing dance association in your area. They may host regular events, including free lessons. Becoming an active member or a fun dancing crowd is the best way to learn.
References :

Lot's of senior citizens also want to learn ballroom dancing, and they're really learning.
References :

Lots of top pro's didn't start to your age or even a little later. And if you're doing it for fun, I know a lady who started in her late eighties and when last I saw her, she was competing at age 92, I think.
I'd also take exception to the suggestion about learning one dance at a time. I even know instructors who refuse to just teach latin or just teach standard, as each of the dances is good cross-training for the others.
References :

The great thing about ballroom is that you're never too old to learn it. There are plenty of top professional dancers that didn't start until they were your age, and some didn't start until even later! I know people that started a year ago when they were 24 and are already doing very well in competitions. I'd definitely say go for it, because it's a great skill and it is indeed heaps of fun
References :
Ballroom & Latin American dancer.