This seems to be a fairly popular topic amongst fans.
My own opinion: No. It doesn't need to, and indeed might be counterproductive if it did.
From how I perceive the setting, the year 999.M41 ends with a huge cliffhanger whose outcome has a good chance of reshaping half the galaxy, including the Imperium. One could say that GW kind of "wrote themselves into a corner", as the only two options to continue the current timeline would be to keep the effects of the Thirteenth Black Crusade very limited (which would be a huge anti-climatic letdown after so many years of "hype"), or by going along with it and re-designing the setting, similar to how Battletech did it with the Dark Age .. and quite possibly with the same negative responses from a fandom which sees their favourite factions reshaped or entire armies ceasing to exist.
It's a risk that I just don't see GW being willing to take, especially when - as Andy Hoare once explained - 40k is "a setting designed first and foremost to house a really cool game". A setting for you to house your own games and stories in, not a story that delivers itself (though it easily would have potential to do so).
That being said, why would we have to go beyond M41, anyways? There's 10.000 years between the Horus Heresy and "now" that can be retroactively filled with new events and adventures. In fact, this is being done all the time, and I'd actually prefer if GW's studio writers would flesh out the earlier eras a bit more rather than cramming half the big events in Imperial history into the last year of its timeline.
Allow me to quote another forum's user with something that made me smile:
Let us not forget that 40K grew up out of a desire to essentially have Warhammer in Space. I tend to toss around my old battered copy of Rogue Trader to my kids friends who get into 40K and they always ask questions like "Where's the Emperor? Where's the story?"
To which I can only say, "Back then, they made it up on their own."
- from http://www.dakkadakk...91.page#5356485
Inquisitor Balthazar J. Skult said:
Often people cite that any form of love-making is instantly involving Slaanesh, or enjoying the slaughter of your enemies is on grounds of Khorne worship.
Heh, that's almost stuff for a topic on its own. In a way, those people would not be wrong. The Chaos Gods are, to a fairly large degree, fueled by human emotions, and if one were to go by Codex fluff, even love is "just another name for desire". One does not need to consciously worship a Chaos deity to indirectly serve it - the "birth" of Slaanesh has shown as much.
Of course, given that (as per its writers and creators) the franchise does not actually follow a uniform "canon truth" and that the various official sources often deliver conflicting portrayals of the world, perception of the exact degree of Grim Darkness™ is left to the individual reader and gamer. For example, the Cain novels you mentioned have no place in my interpretation of 40k, as my preferences focus on the background released by the GW studio itself.
A snippet from the rulebook to exemplify how GW material portrays the setting:
"The Imperium is home to countless billions of lost souls. The teeming masses of humanity throng the stars, but few have time to appreciate the majesty of the heavens. For the greater part of the human race, their only concern is a desperate struggle for survival.
All servants of the Imperium have a vocation that defies their existence, often alloted before they are even born. Pallid citizens toil day and night at thankless and futile tasks forced upon them by uncaring superiors, blind to the terrible truths that threaten Mankind from the void. Oblivious, they sacrifice their dreams on the altar of false hope, giving their all for the continueing survival of a decaying civilisation that cares not if they thrive or if they are ground to dust.
In the hab-complexes of the civilised worlds, the shuffle of sore-ravaged feet and the scratch of thermoquil upon vellum is punctuated by the thunder of distant war. Hunchbacked factotums and aged lickspittles slave endlessly under the unforgiving vigilance of Judicar-Prelates and Titheproctor Superians. Only a few hours' sleep is permitted each night, and even that is plagued by the grind of constant industry, for the incessant wars of the Imperium demand a heavy price. Drooling Ideosavants trade gibberish with Pendanticum, Dataslave and Stasis Clerk in a babel tongue which none truly understand. Even death is no escape; the remains of the faithful are reincarnated as servo-skulls so that they might serve the Imperium for eternity.
In the streets outside the hab-blocks and manufactorums, the Arbitrators enforce their unforgiving rule upon the desperate and the homeless. Feral children fight over the dead flesh of the fallen, their struggles lit only by flickering luminas set into crumbling masonry. Scapegoats, lepers, and pilgrims press and push in great queues that will last a lifetime, desperate in their quests for absolution they will never receive. Through this sickly gruel of flesh stride the privileged few, untouched by disease or the ravages of acidic rain. It is they who maintain the status quo for their own hidden ends, they who guide humanity itself. Some are pure of intent, some embody the corruption at the heart of the Imperium, but one thing is true for all - they care not for the fate of the common man."
In essence, this is what I want from 40k, and thus what "should" be the norm. If I wanted a brighter world, I'd settle for one of the many other franchises which would deliver. However, it is the apocalyptic dystopia I am after, the exploration of mankind's darkest side and the faint glimmer of hope which seems to shine all the more brightly when everything else is so dark. And so here I am.
This does in no way invalidate, say, Mitchell's ideas, which have found their way into your interpretation. Ultimately, neither mine nor your vision, nor anyone elses, is "more right" than the other. Games Workshop allows and actively wants us to take away what we wish from the setting and make it our own. As Gav Thorpe once put it, 40k "exists as tens of thousands of overlapping realities in the imaginations of games developers, writers, readers and gamers, and none of those interpretations is wrong."