Q.Which of the following statement is correct regarding homogeneous system ?

When $\rho(A)$ is less than the number of unknowns in a homogeneous system, it has the trivial solution plus infinitely many non-trivial solutions.

1. A homogeneous linear system $AX=0$ in $n$ unknowns always has at least the trivial solution $X=0$, since substituting $X=0$ satisfies every equation.
2. The system has non-trivial (non-zero) solutions if and only if $\rho(A)<n$ (rank less than the number of unknowns).
3. When $\rho(A)<n$, the solution set forms a vector subspace of dimension $n-\rho(A)\geq1$. …

For a homogeneous linear system, the trivial solution always exists, and it is the ONLY solution exactly when rank(coefficient matrix) equals the number of unknowns.

1. A homogeneous system $AX=0$ always has $X=0$ (the trivial solution), so options claiming it is 'always inconsistent' or has 'only non-trivial solutions' are false — the trivial solution always exists.
2. By the rank criterion, if $\operatorname{rank}(A) = n$ (the number of unknowns), the null space of $A$ is $\{0\}$, so $X=0$ is the unique solution.
3. If $\operatorname{rank}(A) < n$, the system has $(n - \operatorname{rank}(A))$ independent free variables, giving infinitely many non-trivial solutions in addition to $X=0$. …

The determinant condition $abc-a-b-c+2=0$ makes the numerator and denominator of the required sum identical, forcing the value $1$.

1. A homogeneous $3\times3$ linear system has a non-trivial solution iff its coefficient determinant is zero: $\begin{vmatrix}a&1&1\\1&b&1\\1&1&c\end{vmatrix}=0$
2. Expand along the first row: $a(bc-1)-1(c-1)+1(1-b)=abc-a-c+1+1-b=abc-a-b-c+2$.
3. So the condition is $abc-a-b-c+2=0$, i.e. $abc=(a+b+c)-2$. Let $s_1=a+b+c,\,s_2=ab+bc+ca,\,s_3=abc=s_1-2$.
4. Now evaluate $S=\dfrac1{1-a}+\dfrac1{1-b}+\dfrac1{1-c}$ over the common denominator $(1-a)(1-b)(1-c)$.
5. Numerator $=(1-b)(1-c)+(1-a)(1-c)+(1-a)(1-b)=3-2s_1+s_2$ (expand and collect: each product contributes $1$ to the constant, $-2$ total in each of $a,b,c$, and each pairwise product once). …

Rank less than the number of unknowns means the system is under-determined, so besides the trivial solution there are infinitely many non-trivial ones.

1. For any homogeneous system $AX=O$, $X=O$ (the trivial solution) is always a solution, since $A\cdot O=O$.
2. The question is whether non-trivial ($X\neq O$) solutions also exist, and this is governed by comparing $\rho(A)$ (the rank of the coefficient matrix) to $n$, the number of unknowns.
3. If $\rho(A)=n$, the only solution is trivial (the columns are linearly independent). …