Anomalous Subsurface Environment is a setting book that just happens to include a medium sized dungeon. It is a 100 page book. The dungeon part is pages 50 through 77, which is only 27% of the total page count. More than 40% is direct campaign background, 14% is new monsters, and so forth. Yet ASE1 is presented as a module (generally) and a mega-dungeon (specifically). It is my contention that ASE1 would have not been nearly as successful had it been released as The Land of One Thousand Towers campaign setting (with starter adventure included), and it would have done even worse as a cartoon science fantasy retro-clone (which it also easily could have been, with a few more classes and a chapter on combat rules).
Isle of the Unknown was sold as a hex crawl setting or location, but is also a bestiary. While I am not really interested in re-litigating whether Isle is a good setting, or has creative monsters, it seems uncontroversial, to me, to claim that it would have done better if it had focused on the lavishly illustrated monsters and wizards (say), instead of the mostly implied location-based adventure. Rather than “lack of fully realized areas,” instead the evaluation would be “each monster has a bonus encounter area detailed, all collected in the bonus hex map appendix.”
These are counterfactuals, so we can’t really know what would have happened, but examine for yourself what products have been successful. This is worth thinking about if you are making an RPG product. Consider what it is that people have already. How you position a product will guide people toward an evaluation yardstick. If you release something as a mega-dungeon, it will be compared to Barrowmaze, Rappan Athuk, and ASE. If you release something as a bestiary, it will be compared to the Fiend Folio and Teratic Tome. If you release something as a ruleset, it will be compared to Labyrinth Lord, ACKS, and Lamentations of the Flame Princess. And so forth. Think about, for one final example, how successful LotFP likely would have been had it been released as the Early Modern Weird Horror Historical Campaign Setting (which is actually just a hop, skip, and jump away from what is currently in the Rules & Magic hardcover).
A full new ruleset is probably the least effective way to present something given how saturated that particular market has become, unless you are specifically looking to appeal to the crowd that is interested in rules for rules’ sake (and in that case you really need to have new and interesting rules; an interesting setting is probably not enough). You can see this most clearly in the story games communities, where most of the successful products are innovative mechanically but often generic (or more kindly, archetypal) regarding atmosphere and setting. See, for example, Swords Without Master (wildly creative rules with an implied generic swords & sorcery setting) and Dungeon World (an unsurprising classic D&D setting variation on the elegant Apocalypse World engine).