How exactly does a soft microprocessor (e.g Nios II) make design flexible

I am trying to figure out why people came up with idea of a soft microprocessor that is implemented on a programmable logic device.
Why do we need it and how does it make the design process flexible?
If such a thing is an IP (and thus protected by laws and license fees e.t.c) that how can one change it to meet ones needs?
If we really do need a microprocessor we can have one outside the FPGA as well right? This way all the logic elements in the FPGA can be used for other things.

Last point, how do we program such a soft processor as its called, since its architecture can keep varying and thus it can't have a single compiler right?

Some tasks require complicated sequencing. For instance, you may want to display data on a 20-charater LCD display that takes a sequence of initialization commands, then needs proper registers loaded with data. A state machine can be wired up in an FPGA. It will be a real pain to get working, and any changes will require a total rebuild of your circuit. Instead, you can instantiate a tiny microprocessor - like PicoBlaze for instance. Now making changes means you load a different program into it, and it can be done without changes to your circuit.

You could of course use a processor outside the FPGA, but you already paid for all that logic inside the FPGA, so what's the point of keeping it empty? And you will have fewer wiring problems - not to mention you can use an off-the-shelf FPGA board without an external processor.

As for your final question, sure, the architecture can be different. But if you stick with say, a Picoblaze, you can always program it using the nice assembler that comes with it. It won't change by itself.