Q.Show that $(\vec{a} \times \vec{b})^2 = a^2b^2 - (\vec{a} \cdot \vec{b})^2$.

Area from the Cross Product

The cross product $\vec{a}\times\vec{b}$ of two vectors in 3D is itself a vector, and the most useful thing about its magnitude is that it measures area.

Place the two vectors tail-to-tail. They span a parallelogram. The magnitude of their cross product is exactly the area of that parallelogram:

where $\theta$ is the angle between them.

Why sine, not cosine

The area of a parallelogram is base $\times$ height. Take $|\vec{a}|$ as the base. The height is the part of $\vec{b}$ perpendicular to $\vec{a}$, namely $|\vec{b}|\sin\theta$. Multiplying gives $|\vec{a}||\vec{b}|\sin\theta$ — precisely $|\vec{a}\times\vec{b}|$. The dot product uses $\cos\theta$ (overlap along); the cross product uses $\sin\theta$ (spread across), and "across" is what builds area.

Area of a triangle

A triangle with adjacent sides $\vec{a}$ and $\vec{b}$ is half that parallelogram:

For a triangle with vertices $A, B, C$, take $\vec{a}=\overrightarrow{AB}$ and $\vec{b}=\overrightarrow{AC}$.

A quick example

For $\vec{a}=\hat{i}+2\hat{j}$ and $\vec{b}=3\hat{i}+\hat{j}$,

$\vec{a}\times\vec{b} = \begin{vmatrix}\hat{i}&\hat{j}&\hat{k}\\1&2&0\\3&1&0\end{vmatrix} = (1\cdot 1 - 2\cdot 3)\hat{k} = -5\hat{k}.$ …