Q.Show that the area of the triangle formed by the lines $ax^2 + 2hxy + by^2 = 0$ and $lx + my + n = 0$ is $\left| \dfrac{n^2 \sqrt{h^2 - ab}}{am^2 - 2hlm + bl^2} \right|$.

Combining the two lines through $O$ with the line $lx+my+n=0$ gives area $\left|\dfrac{n^2\sqrt{h^2-ab}}{am^2-2hlm+bl^2}\right|$.

Let the pair $ax^2 + 2hxy + by^2 = 0$ represent the lines $y = m_1 x$ and $y = m_2 x$ through the origin $O$, where

$m_1 + m_2 = -\frac{2h}{b}, \qquad m_1 m_2 = \frac{a}{b}.$

These meet the line $lx + my + n = 0$ at points $P$ and $Q$. Putting $y=m_i x$ into the line:

$lx + m(m_i x) + n = 0 \Rightarrow x_i = \frac{-n}{l + m m_i}, \quad y_i = m_i x_i.$

The triangle $OPQ$ has area

$\Delta = \frac12\,|x_1 y_2 - x_2 y_1| = \frac12\,|x_1 x_2|\,|m_2 - m_1|.$

Now

$x_1 x_2 = \frac{n^2}{(l+mm_1)(l+mm_2)} = \frac{n^2}{l^2 + lm(m_1+m_2) + m^2 m_1 m_2} = \frac{n^2}{l^2 - \frac{2hlm}{b} + \frac{am^2}{b}} = \frac{n^2 b}{bl^2 - 2hlm + am^2}.$

Also …

The pair of lines through the origin has a $60^\circ$ apex angle and is symmetric about the direction $(l,m)$; the third line is perpendicular to that axis of symmetry, so the triangle is isosceles with a $60^\circ$ apex — which forces it to be equilateral. Its area then works out from the perpendicular distance of the origin to the third line.

Let $u=lx+my$ and $v=mx-ly$. Since $(l,m)\cdot(m,-l)=lm-ml=0$, the directions $u=0$ and $v=0$ are mutually perpendicular through the origin.

The pair $(lx+my)^2-3(mx-ly)^2=0$ becomes $u^2-3v^2=0$, i.e. $u=\pm\sqrt3\,v$ — two lines through the origin.

Using the true orthonormal rotated axes $U=\dfrac{u}{\sqrt{l^2+m^2}}$, $V=\dfrac{v}{\sqrt{l^2+m^2}}$ (genuine perpendicular Cartesian axes, since scaling both by the same factor preserves angles), these lines are $U=\pm\sqrt3\,V$, each making $30^\circ$ with the $U$-axis. So the angle between them (the triangle's apex angle at the origin) is $60^\circ$, and the $U$-axis bisects it.

The third line $lx+my+n=0$ is $u=-n$, i.e. $U=-\dfrac{n}{\sqrt{l^2+m^2}}$ — a line perpendicular to the $U$-axis, hence perpendicular to the bisector of the pair of lines.

…

Write the pair of lines through the origin as $y=m_1x$, $y=m_2x$ using their sum/product of slopes, find $OA$, $OB$ where each meets the transversal, then use Area $=\frac12\,OA\cdot OB\sin\theta$.

Let the pair of lines $ax^{2}+2hxy+by^{2}=0$ (through the origin $O$) be $y=m_{1}x$ and $y=m_{2}x$, where:

$m_{1}+m_{2} = -\frac{2h}{b}, \qquad m_{1}m_{2} = \frac{a}{b}$

These meet the line $lx+my+n=0$ at points $A$ and $B$.

Finding $OA$: Substitute $y=m_1x$ into $lx+my+n=0$: $x(l+mm_1)=-n$, so $x=\dfrac{-n}{l+mm_1}$, $y=m_1x$. Then:

$OA = |x|\sqrt{1+m_{1}^{2}} = \frac{|n|\sqrt{1+m_{1}^{2}}}{|l+mm_{1}|}$

Similarly, $OB = \dfrac{|n|\sqrt{1+m_{2}^{2}}}{|l+mm_{2}|}$.

Angle $\theta$ between the two lines: using direction vectors $(1,m_1)$, $(1,m_2)$:

$\sin\theta = \frac{|m_{1}-m_{2}|}{\sqrt{(1+m_{1}^{2})(1+m_{2}^{2})}}$

Area of $\triangle OAB$:

$\text{Area} = \frac{1}{2}\,OA\cdot OB\cdot\sin\theta = \frac{1}{2}\cdot\frac{n^{2}|m_{1}-m_{2}|}{|l+mm_{1}||l+mm_{2}|}$

(the $\sqrt{(1+m_1^2)(1+m_2^2)}$ factors cancel between the $OA\cdot OB$ product and $\sin\theta$'s denominator).

Simplify the denominator: …