or "endarkenment", but it depends whether this is referring to the Age of Enlightenment or to literary structures/methods

An alternative may be 'the literary device of non-enlightenment'.

I'm not familiar with Fontane's work, but I had a quick read though of 'Unterm Birnbaum' (it's at Zeno.org). I share Bernhard's reservations re. denouement because it certainly wouldn't apply to the Birnbaum story. It’s a very straightforward plot. There’s no mystery other than that there’s no concrete proof of guilt. The case is solved at the end for the villagers (the reader knows all along, although s/he cannot be 100% sure), but only by inference. Characters speak what turns out to be the truth, yet they are either ignorant of the fact that they’re right or they cannot prove their claims. Or they are certain that they know and must retrospectively concede that they had been wrong. The non-omniscient (i.e. unreliable) narrator, who keeps an ironic distance, is not very helpful either. And the question of who knows what is constantly brought up, often at significant places, like in the final sentence of a chapter. Knowledge is also juxtaposed with superstitious beliefs.

This is not detective fiction in the normal sense - - by which I mean stories like the Sherlock Holmes series. So 'Nichtaufklärung' might indeed be a reference to the Enlightenment movement and its emphasis on reason and (the possibility of) knowledge, meaning that I’d translate it as 'non-enlightenment.

Here's another definition of denouement: '(F ‘unknotting’) It may be the event or events following the major climax of the plot of a plot, or the unravelling of a plot’s complications at the end of a story or play’ The Penguin Dictionary of Literary Terms and Literary Theory, ed. by J. A. Cuddon (Penguin Books, 1999), p.215.

Cf. also:
‘The mock-Bildungsroman aspect of the novel is brought about by the ****non-enlightenment**** of the title-figure, his failure to find a meaningful love-relationship, and his inability to discover a calling that identifies both a significant direction for his individual existence and, at the same time, an appropriate accommodation to his duties to society.’
http://www.press.umich.edu/titleDetailDesc.do;jsessionid=753...

Low confidence because I really don't know much about crime fiction or its history.

--------------------------------------------------
Note added at 17 hrs (2010-10-14 12:39:56 GMT)
--------------------------------------------------

Of course, your sentence also begs the question 'Was ist anspruchsvolle Kriminalliteratur'. ;-)