dielectric thickness between signal and return planes

There is no simple answer to this. There are several factors that play a role here.

The number of layers and the required finished board thickness.
The pcb material and it Dk.
Required trace widths (manufacturing) and thickness

You can play around with the numbers in Saturns "PCB Toolkit".

I don't think the thickness matters as much as you making
the other trace attributes line up with the desired z0 (if the
traces are expected to be transmission lines). Whatever
your tline form, there's PCB structure construction formulae.

Thin dielectric worsens the capacitive loading of every single-
ended trace, and if this is most of your design then go thicker.

Let's consider two PCB stackups in which signal ground and power ground are the same and the return path of the signal is on the ground layer.

Stackup A
01 Signal
02 Power
03 Power
04 Ground

Stackup B
01 Signal
02 Ground
03 Ground
04 Power

In Stackup A, the ground is on 4th layer. However, in Stackup B, the signal return will be on the 2nd layer and the power ground is on the 3rd layer. In order to maintain signal integrity, Stackup B is better then Stackup A ?

The thickness of dielectric between power and ground planes should be thin, like 3 mil or 4mil because it also provide low impedance in PDN and also slightly higher capacitance in power distribution network, which otherwise is the job of de-coupling capacitors, meaning thin layer of dielectric should be use between power and ground layer, right ?

Now back to my question. The dielectric thickness between signal and ground layer. Text book says that return current flows directly under signal but on the ground layer. What is the consequence of thick and thin dielectric layers between signal and ground layers ?

Stackup A sounds like a speculative and unrealistic example. If it describes a real design, you need to look at the details more closely, e.g. power layer copper pattern, are power nets closely bypassed to ground, how many bypass capacitors and where located?

Generally speaking, copper on the layer(s) next to the signal will always share part of the return current and affect the trace impedance. We also have designs where power planes act as transmission line ground.

Thin substrate between power and ground can act as distributed bypass capacitor, but the effect is limited to relative high frequencies. Poorly bypassed power planes can act as resonator and make the whole PCB act as an antenna.