"Crime is for the iron nerved, who have their choice either to endure it, or, if it press too hard, to exert their fierce and savage strength for a good purpose, and fling it off at once!" (144)
Those that commit crimes have to have a certain quality about them that lets them not have a conscience, a certain characteristic that leaves them unable to care about their actions and if they might effect someone else's life other than their own. No matter how small the crime may be, one's conscience is a powerful force and has the power to destroy your mental foundation with guilt.
The minister is feeling guilty because of his dishonest actions. He has been accused of sleeping with Hester Prynne and cannot handle the consequences of his actions. Adultery is a two-way street; Hester obviously knew she had a husband, but so did the minister, therefore, he was disloyal and dishonest to the male species. Society's looks of disproval and shushed murmurings have caught up to him, creating a frenzy of denial and innocence in his mind. Now that people are starting to actually consider that he could be the father of Hester's child, he's panicking, talking himself into believing they're wrong about him.