The thief sneaks in the barracks and try to kill a guard? If they succeed on their sneak roll, that’s an automatic kill, no HP, no coup de grâce or whatever. I’d say the same goes for a successful ambush or stabbing someone unaware in a crowd : there is no “fight” to speak of. It’s pretty easy to implement in a game. As a GM, you need to be prepared to let you shiny encounter be short circuited and your big bad evil boss go down without a fight, but on a mechanical level, everything can stay the way it already is.
Oh, yeah : I would not implement this for the PCs opponents. There is no fun in dying because you failed a Spot check. That may be unrealistic, yes, but it removes a lot of their meaningful choices (I have this opinion about all save-or-die effects, but that’s for another post).
But what about 2)? When I think about situations where it does not make sense to fight, I have huge monsters and monsters made of special material (i.e. not regular flesh) in mind. The PCs are in front of a 120 feet dragon? How does swinging a sword at it makes any sense? In relative size, it’s like a cat attacking a horse. Unless PCs have some special weapon, they just can’t damage it in any meaningful sense; and unless they can touch some vital part (maybe by flying or jumping on the dragon?), they cannot hope to kill it. You hit a water elemental with your hammer? It has exactly the same effect punching a wave would have : none. That can be implemented in a quick and easy way by making those monsters special quests, where the PCs first need to acquire some special item to defeat them or to lure them in a place that cancel their advantage. As a GM, that means using them more sparingly, but apart revisiting the summoning spells and powers, there is not much to change on a mechanical level.
Those two quick fixes will encourage players to think more outside the box. The example of the thief I gave is a good one : in D&D, it’s a very risky gamble to sneak in anywhere. At the second there is an enemy that you need to eliminate silently, there is a very low chance of success : per the rules, you may deal bonus damage, but an instant kill is almost impossible unless the enemy is already very weak. Trying that means the thief is very likely to be detected (and that an alarm will be raised by the victim) and to end up in a straight up fight alone and far from the party, so a more direct approach is almost always preferable. Same goes with those special monsters : if you just can’t damage them by charging them, you have to think of some clever way to do it.
As a GM, that means you need to think on your feet more often. Let PCs foil your best-laid plans; giving a ring of “I’m never caught unaware +3” to all your big baddies is just cheap. That also means you need to be generous with information about the monsters they may face in the future (“there are rumors of a monster made of magma at the bottom of the cave”) so your PCs can prepare themselves. Tell your players that going blindly in a dungeon without having done some research is madness; tell them that always charging danger upfront is a recipe for a TPK; but also tell yourself to reward their ingenuity with at least a decent chance of success.