Q.If A is a $3\times3$ non-singular matrix such that $AA^T=A^TA$ and $B=A^{-1}A^T$, then $BB^T=$

For a square matrix $A=[a_{ij}]$ of order $n$, the cofactor of $a_{ij}$ is the signed minor $A_{ij}=(-1)^{i+j}M_{ij}$ (the minor $M_{ij}$ is the determinant left after deleting row $i$ and column $j$). Replace every entry of $A$ by its cofactor to get the cofactor matrix; its transpose is the adjoint, $\operatorname{adj}A$.

Theorem (the central identity). For every square matrix $A$ of order $n$,

This follows from Laplace expansion: a row's entries dotted with their own cofactors reproduce $|A|$, while a row's entries dotted with a different row's cofactors always give $0$ -- so the product matrix is $|A|$ on the diagonal and $0$ off it.

Definition of the inverse. A square matrix $B$ with $AB=BA=I_n$ is called the inverse of $A$, written $A^{-1}$. The inverse, when it exists, is unique. $A^{-1}$ exists if and only if $A$ is non-singular ($|A|\ne0$): dividing the central identity by $|A|$ (possible exactly when $|A|\ne0$) gives the working formula

A singular matrix ($|A|=0$) has no inverse.

Worked illustration (order 2). For $A=\begin{pmatrix}a & b\\ c & d\end{pmatrix}$, the cofactors are $A_{11}=d,\ A_{12}=-c,\ A_{21}=-b,\ A_{22}=a$, so $\operatorname{adj}A=\begin{pmatrix}d & -b\\ -c & a\end{pmatrix}$ (swap the diagonal entries, negate the off-diagonal ones) and $A^{-1}=\dfrac{1}{ad-bc}\begin{pmatrix}d & -b\\ -c & a\end{pmatrix}$ whenever $ad-bc\ne0$.

Standing laws of inverses (for non-singular $A,B$ of the same order, $\lambda\ne0$ a scalar):

1. $|A^{-1}|=\dfrac{1}{|A|}$.
2. $(A^T)^{-1}=(A^{-1})^T$.
3. $(\lambda A)^{-1}=\dfrac1\lambda A^{-1}$.
4. Left/right cancellation: $AB=AC\Rightarrow B=C$; $BA=CA\Rightarrow B=C$ (pre/post-multiply by $A^{-1}$) -- this fails when $A$ is singular.
5. Reversal law: $(AB)^{-1}=B^{-1}A^{-1}$ (note the order flips, exactly as for transposes).
6. Double inverse: $(A^{-1})^{-1}=A$.

Six adjoint identities (non-singular $A$, order $n$): (i) $\operatorname{adj}(A^{-1})=(\operatorname{adj}A)^{-1}=\dfrac{A}{|A|}$; (ii) $|\operatorname{adj}A|=|A|^{n-1}$; (iii) $\operatorname{adj}(\operatorname{adj}A)=|A|^{n-2}A$; (iv) $\operatorname{adj}(\lambda A)=\lambda^{n-1}\operatorname{adj}A$; (v) $|\operatorname{adj}(\operatorname{adj}A)|=|A|^{(n-1)^2}$; (vi) $(\operatorname{adj}A)^T=\operatorname{adj}(A^T)$; and for two non-singular matrices of the same order, $\operatorname{adj}(AB)=(\operatorname{adj}B)(\operatorname{adj}A)$ (order reverses, exactly like the inverse and the transpose). …