Think of the exhaust as a system. You may have an aftermarket header and cat, and even the rear section. But you're still 'choking' your system off with the stock b-pipe. It's like sucking water from a straw, rather than a hose.
If you want to invest teh $100-$150 for performance, then I say go for it. If you're content with the sound, and performance of your car right now, then put the money into something else.