When and how to part ways with your financial adviser

At first everything's great. You talk all the time, set life goals together, exchange notes. One day you notice the conversations have gotten shorter, the notes less frequent. Calls go unanswered. Maybe you two aren't such a great fit after all. The problem is, this person manages your life savings. “The good and the bad of the financial advice industry is that it's so deeply rooted in the relationships that form between advisers and their clients,” said Michael Kitces, a partner at Pinnacle Advisory Group in Columbia, Md., with a blog, Nerd's Eye View, that's widely followed in the industry. "It feels like [you're] firing a friend. And most human beings don't like confrontation with other human beings.” In one classic "bad" scenario, Mr. Kitces said, the adviser “seemed really nice and friendly, and you moved a little bit of money” into a simple financial product, only to find him becoming “increasingly aggressive” and pushing you into something “more complex, and you're nervous you don't really understand it.” Mr. Kitces's advice: Don't wait for the investment to blow up in your face. "Sometimes your intuition is trying to tell you something about the relationship," he said. "Clients even acknowledge a point in the middle when they knew something wasn't right with the dynamic but didn't do anything because it's awkward." One man who was trying to settle a family estate approached Lisa LaMarche, a partner at Milestone Wealth Advisors in Greenville, Del., and said he was "getting horrible service from his adviser's team. They wouldn't return his phone calls, wouldn't turn over the money" to the beneficiaries, Ms. LaMarche said. She took part in an exit call with the man and his adviser — whom she found polite and cooperative. However the break came about, she said, "if a client feels that their adviser is unavailable or is providing explanations that do not make sense to them, they develop a lack of trust in that person." Her view is that "when it comes to client experience, perception is reality." Nola Kulig, a Longmeadow, Mass., financial adviser, helped prep a woman to explain why she was leaving her adviser, whom she described as more of an insurance salesman. Mr. Kulig suggested the client tell him she had decided to take "a more holistic approach" to her finances and was reviewing everything in her portfolio. PIGS, LIARS AND PERFECTLY NICE GUYS If your adviser has lied to you or misrepresented fees or investments, as in this nasty cavalcade, you end it, period. If you've simply fallen out of love after a long relationship, or have a nagging feeling you could do better, it can be more like the "Seinfeld" episode in which Jerry agonizes over breaking up with his barber. But, as with Ms. LaMarche's example, it isn't always black and white. A study by the White House Council of Economic Advisers estimates that conflicted advice, short of outright lying, may cost retirement savers as much as $17 billion a year. Many of Ms. Kulig's clients come to her from big brokerage firms wondering whether they're getting the best fund or just the one that best compensates the salesperson, Ms. Kulig said. "It's hard for someone to say, 'I trust you, but I don't trust your firm or how you're compensated,'" she said. One woman on a personal finance message board said she has assumed her adviser was looking out for her best interests; she had worked with him for 16 years, and he was "really nice" and a strong supporter of his church. When she raised the "unreal fees" she was shouldering and asked him if there were cheaper products, he said she was paying for the benefit of actively managed funds and their superior performance. An astonished commenter told her to dump the guy and reassured her that "he'll survive." "If you're worried about your adviser's reaction, keep in mind that he's enjoyed 16 years of commissions, loads, 12b-1 fees, wrap fees, etc.," the commenter said. "He's done OK." Ending it can be especially hard if everything's been fine. Debora Killeen worked happily from afar for many years with an Ameriprise adviser in the Bay Area after she moved from California to France. They enjoyed each other — she was six months older, and he called her Big Sis. When she returned to the U.S., living in Sacramento, she wanted to take a more active role in her finances, with access to a local office where she could stop by with questions. He was two hours away. "The long-distance relationship wouldn't work for me," Ms. Killeen said. She began meeting with a woman at an Edward Jones office near her home, enjoying an education in money management. "Emotionally, it was like getting a divorce," she said. She composed an e-mail to her Ameriprise adviser explaining why she was moving her account and called him right after hitting send. He didn't answer immediately, but when he called back they talked for 15 minutes. "Like my friend who is a lawyer said, it's business, and he knows it's business," Ms. Killeen said. "He was very encouraging and was like, 'Take care of yourself, Big Sis.'" She was "happy that I could move on." DENIAL, ANGER, BARGAINING Advisers can be leery of clients, too. Austin Frye, an adviser in Miami, was introduced to a man at a Christmas party who was very unhappy with his adviser because his portfolio was down for the year. The guy had had a few drinks, and Mr. Frye remembers the "veins bulging out on his neck." "I'm thinking, 'Hey, the market's down, and that's why you're down," Mr. Frye said. "I don't want you as a client." Early in his career, Mr. Frye took on a client who admitted he was ditching his financial adviser because the guy had gained a lot of weight and the client feared it showed a lack of control. Two years later, Mr. Frye got a note from the client saying he was going back to his old adviser, and called to ask why. “He lost weight,” the man said. How your adviser reacts when you announce you're leaving will tell you a lot, if belatedly. "If an adviser goes into sales-pitch mode at that point, then the client should know that they have a salesman and not a money manager," said Joseph Krier, president of Krier Wealth Management in Jacksonville, Fla. "Just give me another chance. I promise it will be better," Pinnacle's Mr. Kitces recited. "I've done all this work, how can you fire me now? We've worked together for so long." So it's not surprising that many clients fire their advisers by e-mail. There's nothing wrong with that, to Mr. Kitces, and it provides documentation in case you need it down the road. Two people can leave the same conversation with two very different ideas of what they achieved in it. If you soften the blow too much when you and your adviser talk, she might not realize you fired her. You learn this when, three weeks later, she makes a bad trade on your account. Then you've got a misconduct issue to deal with. And a brand new relationship with your old flame.