Q.In the homogeneous system $\rho(A)$ is less than the number of unknowns, then the system has :

A system $AX=B$ is homogeneous when every constant $b_i=0$, i.e. $AX=O$. Since $x_1=x_2=\cdots=x_n=0$ (the trivial solution) always satisfies it, $\rho(A)=\rho([A|O])$ automatically -- a homogeneous system is always consistent; the only real question is whether it has a non-trivial (non-zero) solution too.

Let $A$ be the $n\times n$ coefficient matrix of a homogeneous system in $n$ unknowns.

• If $\rho(A)=n$ (equivalently $|A|\ne0$, $A$ non-singular), the system has only the trivial solution.
• If $\rho(A)<n$ (equivalently $|A|=0$, $A$ singular), the system has infinitely many non-trivial solutions, forming an $(n-\rho(A))$-parameter family.

So for a square coefficient matrix, the entire question collapses to one determinant check: a non-trivial solution exists exactly when $|A|=0$. (If there are more unknowns than equations, $\rho(A)<n$ automatically, so a non-trivial solution is guaranteed without even computing a determinant.)

Worked illustration. For $x+y+z=0,\ 2x-y+z=0,\ x-2y=0$: $|A|=\begin{vmatrix}1&1&1\\2&-1&1\\1&-2&0\end{vmatrix}=1(0+2)-1(0-1)+1(-4+1)=2+1-3=0$, so a non-trivial solution exists; row-reducing $[A|O]$ recovers it as a one-parameter family.

A problem with an unknown parameter $\lambda$ in the coefficients ("find $\lambda$ so the system has a non-trivial solution") reduces to solving $|A(\lambda)|=0$ for $\lambda$ -- an ordinary polynomial equation in $\lambda$, often factored using the row/column operations that create zeros before expanding. …