Character Analysis
When Gandalf describes Denethor to Pippin, he uses terms like long-sighted, perceptive, and "difficult to deceive […] and dangerous to try" (5.1.93). In other words, Denethor is wily, smart, and maybe even threatening. None of these are exactly warm and fuzzy adjectives, even if Gandalf admits that, "the blood of Westernesse runs nearly true in [Denethor]" (5.1.92) (which is a good thing). This ambiguity in his character, which seems evenly divided between good (smart, perceptive) and bad (stern, cold, and dangerous) is, frankly, fascinating.
Unlike good-at-heart, humble Aragorn, Denethor is a man used to power, which he generally uses wisely. Even Gandalf admits that Denethor is a strong ruler. But his coldness, his pride, and his unfair prejudice against his own younger son, Faramir, all lead to his eventual downfall. Denethor is a truly gray character, in the sense that he is solidly balanced between Good and Evil. And even though he eventually tips more towards the Evil end of the scale, he adds some variety to a series that is otherwise a bit black-and-white, morally speaking.
One reason Denethor seems to be so cautious and suspicious is that his power is not a sure thing. He is the legal head of Gondor, which has been ruled for generations by the King's Steward. But Denethor is not king, and he will not officially take that title. He has his honor to maintain, and he won't just steal the throne outright (though he might really like to; he's not too happy about Aragorn's return).
A "steward" is a servant who takes care of the estates of an important person, usually of an aristocrat. When the kings of Gondor were out of the country on business or whatever, it used to be the Steward's job to manage things in their absence. And when the last king of Gondor (of the line of Anárion, Isildur's brother) went off to war and then didn't come back, his Steward quietly took power over Gondor. Since then, the descendants of that first Steward have ruled Gondor, supposedly in the name of the absent king. And Denethor is the current Steward of Gondor, so he only gets to keep his power as long as the king of Gondor doesn't come back. No wonder he is a bit insecure. Especially with all these rumors of a rather regal dude roaming the wilderness floating around.
Jealous, Much?
One of Denethor's worst traits is his jealousy of Gandalf. He will not listen to Gandalf's advice, because he can't handle any kind of challenge to his own authority. And in a wartime situation where only Gandalf really seems to have any kind of intel on what's going on with Sauron's strategies, Denethor's refusal to listen to him looks totally foolish.
Denethor's insane resentment of his son, Faramir, also comes out of this jealousy of Gandalf. He tells Faramir:
For Boromir was loyal to me and no wizard's pupil. He would have remembered his father's need and would not have squandered what fortune gave. He would have brought me a mighty gift. (5.4.59)
Denethor dismisses Faramir as a "wizard's pupil": a student of Gandalf. He also accuses Faramir of forgetting his duties to his own father. He imagines that Boromir would have brought Denethor the Ring (though obviously, as we know from Boromir's words at the end of The Fellowship of the Ring, his dad was the furthest thing from Boromir's mind when he was trying to grab the Ring from Frodo).
Denethor tells Faramir point-blank that he wishes it had been Faramir who had died and not Boromir. Which is messed up on many levels, especially since Faramir so obviously wants nothing more than to please Denethor and to make him proud. Denethor is definitely not a candidate to win any parenting prizes.
Oh, Now He Feels Bad
After all of these ugly words, Denethor sends Faramir into a dangerous battle in Osgiliath (which is a Gondorian fort right near the border with Mordor). Even though everyone tells Denethor that this battle is a bad idea and they should just let Osgiliath go, he won't listen. His pride demands that Gondor hold onto Osgiliath, no matter what the cost in human lives. And he is so bitter and resentful towards Faramir that he seems actively glad to send his son into a potentially fatal situation.
And of course, the inevitable happens: Faramir is horribly wounded. He faces the Lord of the Nazgûl on the battlefield, and it clearly doesn't go too well for Faramir. Once his soldiers carry Faramir back into Minas Tirith, that's when Denethor starts to feel guilty for being so mean to him. When his son is lying on his sickbed, Denethor suddenly feels some remorse for telling Faramir that he wished he were dead. We hate to say it, buddy, but that's too little, too late.
Denethor Finally Loses It
Okay, we can cut Denethor some slack. We know that he is under a lot of pressure: (a) his country borders Mordor, and Sauron is coming right for Gondor first; (b) the rightful king of Gondor is coming back to take Denethor's power away from him; (c) one of his sons has been killed, and the other one is at death's door; and (d) he has been secretly looking into Minas Tirith's palantír and watching the visions of doom that Sauron has been feeding him. But even so, none of this justifies the fact that he tries to set Faramir on fire. That's right: the man tries to burn his own son alive.
See, Sauron knows that Denethor is too honorable to be twisted into actively helping Sauron (the way he corrupted Saruman through the palantír of Orthanc). But Denethor can be pushed into despair, which can be just as damaging as actively turning evil.
So Sauron keeps sending Denethor nasty little messages about the fall of Gondor through the palantír. And Denethor gives in. When he decides to set himself and his son on fire, it's because he is totally convinced that there is no way that Gondor can possibly survive this war. They might as well give up all resistance. Denethor's worst crime is that he loses faith and stops trying, which is what sets him apart from less noble but more enduring characters like Sam or Merry or Pippin (see their "Character Analyses" for more on this contrast).
P.S. A note on fire: why does Denethor choose this particularly horrible method of attempted murder-suicide? He is so sure that he and Faramir are already dead that he decides to cremate both of them on a funeral pyre (which is a stack of wood used for burning bodies in funeral rites)—even though they are both still alive. It's an ugly way to go, but only Denethor actually winds up consumed in flames. Pippin and Gandalf rescue the injured, unconscious Faramir right at the last minute.