Are you sure that your expression is correct?
Because k should appear in the numerator, you could multiply numerator and denominator with k - however, in this case we have k^2 in the denominator which is not correct.
Therefore my question above.
I think, already the first equation in your post#1 is wrong.
This is all me steps, I hope it can explain what i'm trying to do:
\[
V_x\] is:
\[
\begin{equation}
V_x\left(\dfrac{1}{R_1}+sC_2+\dfrac{1}{R_3}\right)-\dfrac{1}{R_1}* V_{in}-\dfrac{1}{R_3}* V_+-sC_2* V_{out}=0
\end{equation}
\]
\[V_+\] is:
\[
\begin{equation}
V_+\left(\dfrac{1}{R_3}+sC_4\right)-\dfrac{1}{R_3}* V_x=0
\end{equation}
\]
Isolate \[V_x\] in equation 2:
\[
\begin{align*}
V_+\left(\dfrac{1}{R_3}+sC_4\right)-\dfrac{1}{R_3}* V_x=0 \\
V_+\left(\dfrac{1}{R_3}+sC_4\right)=\dfrac{1}{R_3}* V_x \\
\dfrac{V_+\left(\dfrac{1}{R_3}+sC_4\right)}{\dfrac{1}{R_3}}=V_x
\end{align*}
\]
We know that \[V_-\] is:
\[
\begin{equation}
V_-=V_{out}* \frac{R_a}{R_a+R_b}=V_+
\end{equation}
\]
It is an ideal op amp, so that means:
\[
V_+=V_-\]
All
\[\dfrac{1}{R_{1,3}}=G_{1,3}\]
and
\[\dfrac{1}{C_{2,4}}=sC_{2,4}\]
We substitute equation 3 into equation 1.
\[ \begin{equation}
\dfrac{\left(G_3+sC_4\right)\left(G_1+sC_2+G_3\right)}{G_3}* V_+-G_1* V_{in}-G_3* V_+-sC_2=0
\end{equation}
\]
Isolating V_{in} and V_{out}:
\[\begin{equation}
\dfrac{\left(G_3+sC_4\right)\left(G_1+sC_2+G_3\right)}{G_3}* V_+-G_3* V_+=G_1* V_{in}+sC_2* V_{out}
\end{equation}
\]
Replacing \[V_+\] with \[\dfrac{R_a}{R_a+R_b}*V_{out}\]
\[\begin{equation}
\dfrac{\left(G_3+sC_4\right)\left(G_1+sC_2+G_3\right)}{G_3}* \frac{R_a}{R_a+R_b}* V_{out}-G_3* \frac{R_a}{R_a+R_b}* V_{out}=G_1* V_{in}+sC_2* V_{out}
\end{equation}
\]
Lets say that \[\dfrac{R_a}{R_a+R_b}=k\] and then we get:
\[
\begin{equation}
\dfrac{\left(G_3+sC_4\right)\left(G_1+sC_2+G_3\right)}{G_3}* k* V_{out}-G_3* k* V_{out}=G_1* V_{in}+sC_2* V_{out}
\end{equation}
\]
Then isolate all \[V_{out}\] on left side.
\[
\begin{equation}
\dfrac{\left(G_3+sC_4\right)\left(G_1+sC_2+G_3\right)}{G_3}* k* V_{out}-G_3* k* V_{out}-sC_2* V_{out}=G_1* V_{in}
\end{equation}
\]
Then can we move \[V_{out}\] out of a braces:
\[
\begin{equation}
V_{out}* \left(\dfrac{\left(G_3+sC_4\right)\left(G_1+sC_2+G_3\right)}{G_3}* k-G_3* k-sC_2\right)=G_1* V_{in}
\end{equation}
\]
Take V_{out} out, so only that is on left side:
\[
\begin{equation}
V_{out}=\dfrac{G_1* V_{in}}{\left(\dfrac{\left(G_3+sC_4\right)\left(G_1+sC_2+G_3\right)}{G_3}* k-G_3* k-sC_2\right)}
\end{equation}
\]
Again can we replace \[G_{1,3}\] with \[\dfrac{1}{R_{1,3}}\]:
\[
\begin{equation}
V_{out}=\dfrac{\dfrac{1}{R_1}* V_{in}}{\left(\dfrac{\left(\dfrac{1}{R_3}+sC_4\right)\left(\dfrac{1}{R_1}+sC_2+\dfrac{1}{R_3}\right)}{\dfrac{1}{R_3}}* k-\dfrac{1}{R_3}* k-sC_2\right)}
\end{equation}
\]