Had sales of the 3rd Edition D&D rulebook 'dried up' before they published 3.5?
Yes. Or rather they were dropping quickly. I think they could probably have had worthwhile sales for a while longer, but they weren't satisfied with the numbers they had.
I could be wrong, but my impression was that WotC came out with the corrected/updated 3.5 rulebook because they were worried that all the glitched in 3.0 might hurt the long-term survivability of the system- which is also a potential problem for DH.
No. The purpose of the .5 edition of DnD was not to fix the glitches in 3.0, but to justify the re-release of the corebooks and the introduction of 'new' supplements. The changes were pretty small, while at the same time there were enough of them to force people to buy the books again if they wanted to be compatible with the new material. The company didn't fear that the 'long-term survivability of the system' was at risk from the issues that were changed - that simply wasn't an issue or a consideration. What mattered was the number of orders they were getting from distributors, which had started to drop.
Well, DH has lower sales than D&D did during the big WotC re-launch, so a direct year-per-year comparason doesn't really work. The larger point is that I don't see how having the game system dependant on a rulebook that is widely known to be error-filled can not be having a detrimental effect on long-term sales.
Long-term sales? As has been pointed out Dark Heresy has already been around for 5 years. That is long term sales as far as an RPG is concerned.
An RPG is going to succeed or fail in its first two years. If it is still around in year 3 then it is a success, but its glory days are behind it. Sales drop through the floor at this point. But if you've had a good two years then you've probably built up enough of a fanbase that you can continue to publish supplements and expect them to turn a profit. Not a huge profit, mind you, but generally enough (what is 'enough' varies from company to company) to justify one or two supplements a year. However this level of sales isn't enough to justify much else (such as an updated '1.1' edition of the rules). It doesn't matter at this point if a system is 'error-filled' at this point because you've already secured your fanbase (if it mattered to them that much they would have stopped buying your products ages ago) and you aren't attracting much in the way of new customers anyway.
What happened with DnD 3 was that WotC decided that they couldn't be bothered doing this. They'd already been working on 3.5 since the release of 3.0 so they decided to release it a little early. Again, it wasn't really about fixing anything (they ignored far more issues than they fixed), it was just about setting the clock back to zero and releasing the core material again.
Usually after 4-6 years the number of sales will have dropped to the point that even one or two supplements a year isn't worth it (there are some companies that keep releasing material after this point, but they aren't really doing it for profit). Once you get to that point a company will usually release a new edition or else drop the game entirely.