A complex number can be viewed as a 2-dimensional quantity (with real on the x-axis and imaginary on the y-axis). If we were to consider a magnitude for this 2D quantity (i.e. distance from the origin), it would be the square root of the sum of the squares of the x and y components. In other words, the square root of the sum of the real and imaginary parts... or the square root of the complex number multiplied by its conjugate.
With your matrix equation, you are already working in N dimensions (for Nx1 columns). To be clear, these are complex dimensions. You could reformulate your problem in 2N real dimensions and re-write all your proofs in 2N real dimensions. If you were to do so, you would find that (the real-space equivalents of) these conjugates were required.
Numbers are a human invention designed to help us solve problems. Complex numbers are a human invention designed to help us solve problems. You could solve all the problems of signal processing using real numbers, it just so happens that complex numbers sometimes simplify matters significantly.
For me personally, I think the same about vector and matrix notation: we could represent everything as scalars if we wanted, it would just be a pain in the... neck.