Stress
Some days are tougher than others. You'll get emotionally involved every now and then, and that can make saying no a lot more difficult than it should be. You're not paid to feel (and believe us, those who are paid to "feel" professionally don't make a whole lot to do it). You're paid to think and come to rational conclusions—but that doesn't make it any easier to deny a loan to a prospective college student or doe-eyed entrepreneur.
The bigger stress probably won't come from clients you have to turn down because they aren't qualified, but rather from back pressure in your bank pushing you to get good candidates for whom you can actually write loans—they want those loans because customers paying back with interest (key word) is how they make any money on the loan.
Unfortunately, the good ones can be hard to find, which means you actually have to work for this.