I have been living car free since October 2008. Here are some benefits I see:

1. Reduced financial burden: Direct Expenses
There is a lot of money that saved in car payments, insurane, gas, maintenance, parking etc. Even if you have a car that is paid off, you still have insurance, gas, maintenance (maybe more than on a new car), parking etc. Estimated savings are between $300 to $700 a month (that is $3600 to $8400 yearly or giving yourself a raise).
2. Reduced financial burden: Indirect Expenses
Some things that we do because we can in a car are reduced. For example, drive through fast food ordering is substituted by cooking at home or eating local foods, gyms memberships are substituted by riding a bicycle (I lost 20 lbs in 6 months without changing my diet and my work is only 2 miles away), Starbucks pit stops get reduced because riding your bicycle produces energy naturally that freshens you up etc.
3. Health benefits
Car free lifestyle make people live a more active life by bicycling or walking more. It improves health. See article here: http://commutebybike.com/2009/02/03/commuters-require-less-sick-days/

There are more benefits that I personally enjoy. There are also a lot of challenges:
1. grocery shopping: easily accomplished on an Xtracycle or a bike trailer. Inexpensive trailers can be found on Craigslist or ebay
2. long distance trips can be accomplished by car-pooling, hiring a taxi or renting a car. The amount saved in Direct Expenses easily offsets taxi costs and car rentals. Till date, I have never spent more than $100 a month on such expenses. I prefer car pooling though. It lets me spend more time with friends and they enjoy an appropriate lunch or dinner at my expense...beats traveling alone hands down.
3. social activities/bar hopping etc are better done by hiring taxi cabs. It is not safe to drive after drinking.

Car Sharing is a cheap alternative to hiring a taxi or renting a car. Alas, Jacksonville does not have any outlets for ZipCar or similar companies.

There are additional challenges to a family with children. Riding a bicycle can reduce ownership of two cars to one making a family car-lite.

Please feel free to discuss specific challenges that you may face in making the decision to go car free.