Why I Don’t Fret About Taxes

Want To Reach FI Sooner? Join more than 18,000 others and get new tips and strategies from Can I Retire Yet? every week. Subscription is free. Unsubscribe anytime:

On either side of the aisle these days, it’s not politically correct to be ambivalent about taxes. If you lean to the left, you’re supposed to feel the tax code should be more progressive, that taxes should be higher on the wealthy and on corporations. And if you lean to the right, you’re supposed to think there should be less taxes, less spending, and less government, all around.

And, almost everybody wishes the tax code were simpler. Though, if you’re in politics or finance — an advisor or accountant, perhaps — you might privately feel that our horrifically complex tax code is just great and should be left alone. After all, fear and confusion related to paying taxes is one of the primary drivers pushing customers or voters in your direction!

But I’m not in any of those camps. I have no interest in pushing a particular viewpoint. I’m just an early retiree in the middle of the road, who lives frugally, and wants to keep life simple. I’m interested in the current reality we all have to live in for retirement, whether we agree with it or not.

So, perhaps it comes as a surprise, but taxes are just not a primary concern of mine right now….

Fair Warning: I Tend to Downplay Taxes

Recently a thoughtful reader wrote me with the concern that people reading my blog and looking at their 401(k) balances would think of them as equivalent spendable dollars, without first reducing the sum by an amount for the expected tax bite. He encouraged me to write more on this piece of the puzzle.

That’s a fair concern. I don’t claim any special expertise on or interest in taxes, and I purposefully don’t write about them much here.

Most of my analysis tends to be “after-tax,” meaning I’m generally ignoring tax implications. And I need to be clear about that. (I heartily recommend reading Mike Piper over at Oblivious Investor for real tax advice.)

Fact is I loathe taxes and tax maneuvers. My philosophy is to keep my financial life simple, live frugally at low tax rates, and then pay my share without squabbling. The complexity and conniving that our tax code seems to invite, always feels like a colossal waste of time to me.

The Tax Bogeyman

Sure, I’d love to pay less in taxes. But I have no great interest here in arguing whether our current tax system is fair or unfair, or whether certain segments of society should or shouldn’t pay more in taxes, or whether the government does or doesn’t know how to spend your money better than you.

That’s all beside the point here. There is such revulsion to taxes in this country that there is a tendency to overreact to their impact, especially at lower and middle income levels.

The main thing I want to communicate is that the current tax climate actually doesn’t impact a frugal early retiree leading a modest lifestyle very much. The facts are that the effective tax rate isn’t that high for the amount of income that many of us will require to live a comfortable retirement.

But a lot of people with agendas — from politicians to bankers to financial planners — want to stir you up into believing that taxes will make or break your financial success.

One retirement site goes so far as to say that the #1 goal of retirement savings and spending should be to minimize taxes. Sorry, but I’m just not seeing that.

Another respected site says that taxes are your single biggest expense. But I’m not expecting that to be true in our experience either. Looking at our budget, I think we’ll easily spend more individually on gas, groceries, health insurance, and travel than we will on taxes.

When I was a few months from pulling the plug on my career, I went to a free consultation with a local financial planner from a respected regional firm. When they ran the projections for my retirement they used a 20% effective tax rate on all our income! That was wildly inappropriate, as I’ll shortly demonstrate. It took a marginal tax rate (which was still too high), and applied it to all our retirement income, disregarding the substantial portion which would be exempt from taxes. The end result significantly overstated our tax liability.

The New Law

With the American Taxpayer Relief Act, recently passed to avert (or postpone) the “fiscal cliff,” tax rates have apparently been decided for the near future. In short, income tax rates and capital gains and dividend tax rates will remain the same for couples making less than $450K.

The lower echelon of tax rates on those married and filing jointly for 2013 are as follows:

10% on taxable income from $0 to $17,850, plus
15% on taxable income over $17,851 to $72,500, plus
25% on taxable income over $72,501 to $146,400

Elsewhere, a mainstream blog analyzing implications of the recent fiscal cliff deal, quotes a financial expert and accountant who warns that the new legislation will impact retirees and baby boomers who “rely on interest and dividends.”

But read the law: taxes on capital gains and dividends will only increase for couples making more than $450,000/year. As I’ll show shortly, that level of income is not remotely representative of the typical retiree. I, and most of my readers, know that you can enjoy a grand retirement on a small fraction of that much income. So to imply that the rise in tax rates somehow impacts more than a very tiny fraction of real-world retirees is either misleading or foolish.

Another blog, from a favorite conservative investment advisor, reports that taxes have “gone up on 77% of Americans.” More capricious and heavy-handed government? No, it’s primarily the restoration of the 2% payroll tax cut to fund Social Security. To call that a raise in taxes in disingenuous on several levels: first of all this simply restores the status quo from before the recession, secondly it’s funding an extremely popular program from which many people will ultimately draw more in benefits than they pay in.

The way I read it, at least for now, Washington continues to protect the middle class.

The Numbers: A Frugal Retiree’s Tale

My brief Reader Survey, which many of you have taken, asks: “Assuming your home is paid for, about how much do you think you would need to live in retirement?”

With many hundreds of responses compiled now, the median answer to that question is “$4,000-$5,000/month.” So let’s say that $4,500/month, or $54,000/year is the average cost of living for prospective retirees reading this blog, the amount of income they must generate annually to live comfortably.

(I am aware that if you live in an expensive area, or have a more luxurious lifestyle, your number could be considerably higher than that, and some of what I say in this article may not apply to you. On the other hand, for the many people who require even less in retirement, then what I say here is even more applicable.)

For tax year 2013, the standard deduction for a married couple filing jointly is $12,200, and the personal exemption is $3,900 each. So that makes exactly $20,000 of income, for a couple, which is not taxed at all.

For our average retired couple that leaves, worst case, $34,000 in taxable income needed to cover the proposed $54,000 budget. Given the tax rates listed above, the first $17,850 will be taxed at 10%, and the remainder at 15%. Run those numbers and that means paying about $4,200 in taxes annually, or about $350 a month.

That is, worst case, an 8% effective tax rate on the original $54,000 income. Most retirees will pay less, some much less. Why? For starters, that calculation assumed you had no tax deductions.

But in our case there is a more significant factor: A big chunk of our savings (about half) is in after-tax accounts, where only the new gains or income will be taxed. (Many other retirees will have Roth accounts with similar tax characteristics.) So, as early retirees, much of the money we will consume in the early years of retirement will either be taxed at a lower rate, as investment income, or won’t be taxed at all!

Just because you have an expense in retirement doesn’t mean you’re going to have to pay full income tax on a commensurate amount of income. The money for paying that expense could come via withdrawals from an already-taxed account or Roth, or it could come from capital gains which are taxed at a lower rate or not at all. Any qualified dividends or long-term capital gains that fall into the 15% tax bracket, or below, will not even be taxed!

The bottom line: I expect we’ll be paying an effective income tax rate of less than 5% in coming years. And frankly, given the other variables in the retirement equation — investment returns, inflation, Social Security, health care — that small percentage in income taxes falls well down my list of concerns….

This May Not Apply to You If…

Of course, there are some readers for whom this relaxed attitude towards taxes does not work.

If you live in a high tax state, then state income taxes could add substantially to the Federal tax I’ve been discussing. (Though you’d be wise to re-evaluate your retirement location if feasible.)

Or perhaps you have the good fortune to be living on a multiple-six-digit income in retirement. If so, congratulations on your success in life. And, yes, your taxes are probably going up.

The new law enshrines higher rates for couples earning over $450K. And couples who make over $250K will also get hit with some increases because of the new health care law. People in these groups may be required to make some lifestyle changes, if they have been consuming nearly all their income each year.

The Fate of the Middle Class

So far, the politicians have largely spared the middle class from tax increases. And that’s why I can write, as I do, that taxes are not a top concern for a frugal early retiree.

CNN Money reports that only 0.7%  of all taxpayers will experience a tax increase. Most people, including a “middle class” whose definition has been expanded significantly, will not be affected by the latest from Washington.

Will this be the case going forward? Will the middle class remain safe from tax increases?

Who knows? But I don’t personally feel great cause for alarm.

Because, even if the major middle-class tax breaks are gutted, that would be unlikely to impact frugal retirees much. Another CNN Money article reports that the largest target is employer health insurance, given that workers with employer-sponsored health coverage don’t have to pay taxes on the share of premiums paid by the company. The other two prominent middle-class tax breaks are the mortgage interest deduction, and tax-deferral of retirement plan contributions. And, almost by definition, none of those items would be likely to affect the lifestyles of those at or near retirement.

Yes, the U.S. is in debt and there are bills to be paid. There is a widespread belief that taxes will somehow go up on most of us in retirement. Yet one political party is committed to no taxes, while the other party is committed to raising taxes only on the wealthy. And the middle class is largely responsible for electing them both. I don’t see how that formula leads to tax increases on middle-income retirees.

I’m more concerned about back-door mechanisms like inflation — which I do fret about.

Parting Thoughts

So taxes remain a back burner issue for me, because they haven’t been pivotal in my own retirement yet. They may not be trivial, but a 5% expense is simply not going to wreck our budget, or change our lifestyle.

I understand that for others, especially those with bigger incomes to cover bigger expenses, tax impacts might be more substantial.

And, given all that, there is one final reason I don’t fret about taxes. Because, aside from running my financial life intelligently, preparing my tax returns, and voting, there is very little I can do about them.

So how about you? Politics aside, I’m interested in your practical concerns for taxes in retirement. How will they affect you? What hard numbers can you share?

Valuable Resources

  • The Best Retirement Calculators can help you perform detailed retirement simulations including modeling withdrawal strategies, federal and state income taxes, healthcare expenses, and more. Can I Retire Yet? partners with two of the best.
  • Free Travel or Cash Back with credit card rewards and sign up bonuses.
  • Monitor Your Investment Portfolio
    • Sign up for a free Empower account to gain access to track your asset allocation, investment performance, individual account balances, net worth, cash flow, and investment expenses.
  • Our Books