Q: I have written a novel, but it has been rejected by every publisher to whom I've submitted it. Why?
A: Because it sucks.
Q: Should I self publish my rejected novel, or put it up as an e-book on Kindle or Nook?
Self publishing a novel is almost never a good idea unless it was already published and you have reversal of rights. Non-fiction or niche market "how to" books are a different matter.
One aspect of being a successful author is dealing with the simple reality that just because you write something, that doesn't mean it is well written. Even if well written, it may not be of interest to the target demographic. It might even be the right book at the wrong time. Or perhaps it simply isn't something you really want to haunt your career, damage your career, or kill your career.
Leslie Charteris, famous author of The Saint, kept the first Saint book, 1928's MEET THE TIGER, out of print for many decades because, in his words, he was no more eager to have it paraded than "any other youthful indiscretion."
My books (in paperback) sell approx. 25,000 each for every six months that they are available. I wish they sold 250,000 or 25 million. As for Kindle or Nook, my publisher has just put two of my books up in e-format. The competition isn't less when it comes to e-books, nor is the standard any lower. Credibility counts. Books that have been rejected don't magically become better because they are an e-book. If my books were self-published, I assure you they would not sell 25,000 every six months, and I would not have a team of editors, fact checkers and lawyers vetting my non-fiction books to make sure they were grammatically sound and immune to law suits.
In short, if your novel is continually rejected, remember this maxim: "Rejection is protection." If a big house rejects it, find out why. Perhaps a small house has interest in it. If no one wants it, putting it on the market yourself may only prove an embarrassment.
If your novel isn't good enough for the market place, write another one, and another, until you write one that a publishing house is willing to back 100%. No professional author pays a cent to have a novel published. If I have a novel rejected, and I can't fix it so that it is marketable, I'm not going to damage my reputation by putting it up for sale myself. Wait. I take that back. I have put up part of a work in progress for download, and suffered the humiliation of getting slapped around for it. It was one of Amazon's clever suggestions — share part of a new work in progress with your fans. Nope. I'll wait till the work is revised, polished, and officially published. I will never again subject myself to the wrath of readers who expected a work in full, with all its rough edges fully rounded, and got less than they anticipated.