Hi,
I'm not sure where the contradictions come from.
0.4V peak, 8 Ohms and 500mW does not match.
Thus I don't know how to help. Do you need assistance in learning formulas, physics, math .... or more in circuit design?
If the speaker is too quiet, then:
* maybe the amplifier is not suitable
* the amplifier input level is too low
* the speaker's efficiency is too low
* or your expectation is wrong
From simple mathematics (you should be able to do on your own), it's clear the 400mV can not result in 500mW.
Now we don't know where and how you measured the 400mV.
If you talk about an 8 Ohms speaker for 500mW, you also need to measure the voltage (directly) at the speaker.
Often the speaker values are specified at pure sine waveform. But human voice is not pure sine. Thus for a 0.5W average speaker power on voice I recommend that the amplifier is able to produce at least 3W (continous sine, or 6W peak) to get low distortion result.
Mind: Don't test the speaker with more than the specified 0.5W average.
Use a bandpass filter in front of the amplifier to adjust the (amplified) frequency range to the speaker.
I guess you need to give use more informations. A sketch about your circuit, and where you measured / expect the values is very very helpful.
Klaus