Stress
Say yes to the stress. Being a literary agent is uber-competitive, and the stress doesn't let up unless you've banked a stable full of major clients who consistently take on projects for hire and crank out a major novel or blockbuster script that sells well or performs outstandingly at the box office every few years.
The odds of this happening? Pretty much zero.
Even if this did happen, agents would still have to stress about keeping their clients happy, since there's always another agent lurking around the bend ready to snatch up their clients the second a disgruntled thought pops into their head. Out in Hollywood, it's a dog-eat-dog world and even your friendly network of colleagues and industry contacts might turn against you if the price is right.
So how can agents avoid stress? They can't. But they can lessen their abandonment worries by staying organized, by being continuously present in their clients' professional and personal lives, and by having a reliable office infrastructure (i.e., having an assistant who rarely screws up).