Q.Find the equations of the pair of lines represented by $3x^2 + 7xy + 2y^2 = 0$.

Combining the two lines through $O$ with the line $lx+my+n=0$ gives area $\left|\dfrac{n^2\sqrt{h^2-ab}}{am^2-2hlm+bl^2}\right|$.

Let the pair $ax^2 + 2hxy + by^2 = 0$ represent the lines $y = m_1 x$ and $y = m_2 x$ through the origin $O$, where

$m_1 + m_2 = -\frac{2h}{b}, \qquad m_1 m_2 = \frac{a}{b}.$

These meet the line $lx + my + n = 0$ at points $P$ and $Q$. Putting $y=m_i x$ into the line:

$lx + m(m_i x) + n = 0 \Rightarrow x_i = \frac{-n}{l + m m_i}, \quad y_i = m_i x_i.$

The triangle $OPQ$ has area

$\Delta = \frac12\,|x_1 y_2 - x_2 y_1| = \frac12\,|x_1 x_2|\,|m_2 - m_1|.$

Now

$x_1 x_2 = \frac{n^2}{(l+mm_1)(l+mm_2)} = \frac{n^2}{l^2 + lm(m_1+m_2) + m^2 m_1 m_2} = \frac{n^2}{l^2 - \frac{2hlm}{b} + \frac{am^2}{b}} = \frac{n^2 b}{bl^2 - 2hlm + am^2}.$

Also

$|m_2 - m_1| = \sqrt{(m_1+m_2)^2 - 4m_1 m_2} = \sqrt{\frac{4h^2}{b^2} - \frac{4a}{b}} = \frac{2\sqrt{h^2 - ab}}{|b|}.$

Therefore

$\Delta = \frac12\cdot\frac{n^2 |b|}{|am^2 - 2hlm + bl^2|}\cdot\frac{2\sqrt{h^2-ab}}{|b|} = \left|\frac{n^2\sqrt{h^2 - ab}}{am^2 - 2hlm + bl^2}\right|.$

The pair of lines through the origin has a $60^\circ$ apex angle and is symmetric about the direction $(l,m)$; the third line is perpendicular to that axis of symmetry, so the triangle is isosceles with a $60^\circ$ apex — which forces it to be equilateral. Its area then works out from the perpendicular distance of the origin to the third line.

Let $u=lx+my$ and $v=mx-ly$. Since $(l,m)\cdot(m,-l)=lm-ml=0$, the directions $u=0$ and $v=0$ are mutually perpendicular through the origin.

The pair $(lx+my)^2-3(mx-ly)^2=0$ becomes $u^2-3v^2=0$, i.e. $u=\pm\sqrt3\,v$ — two lines through the origin.

Using the true orthonormal rotated axes $U=\dfrac{u}{\sqrt{l^2+m^2}}$, $V=\dfrac{v}{\sqrt{l^2+m^2}}$ (genuine perpendicular Cartesian axes, since scaling both by the same factor preserves angles), these lines are $U=\pm\sqrt3\,V$, each making $30^\circ$ with the $U$-axis. So the angle between them (the triangle's apex angle at the origin) is $60^\circ$, and the $U$-axis bisects it.

The third line $lx+my+n=0$ is $u=-n$, i.e. $U=-\dfrac{n}{\sqrt{l^2+m^2}}$ — a line perpendicular to the $U$-axis, hence perpendicular to the bisector of the pair of lines.

Because the cutting line is perpendicular to the axis of symmetry of a pair of lines from the origin, the resulting triangle is isosceles about that axis. An isosceles triangle with a $60^\circ$ apex angle has base angles $\dfrac{180^\circ-60^\circ}{2}=60^\circ$ each — so all three angles are $60^\circ$: the triangle is equilateral.

Height (apex to base, along the axis of symmetry) equals the perpendicular distance from the origin to $lx+my+n=0$:

$h = \frac{|n|}{\sqrt{l^2+m^2}}$

For an equilateral triangle with side $s$: $h=\dfrac{\sqrt3}{2}s \implies s=\dfrac{2h}{\sqrt3}$.

$\text{Area} = \frac{\sqrt3}{4}s^2 = \frac{\sqrt3}{4}\cdot\frac{4h^2}{3} = \frac{h^2}{\sqrt3} = \frac{n^2}{\sqrt3(l^2+m^2)}$

Write the pair of lines through the origin as $y=m_1x$, $y=m_2x$ using their sum/product of slopes, find $OA$, $OB$ where each meets the transversal, then use Area $=\frac12\,OA\cdot OB\sin\theta$.

Let the pair of lines $ax^{2}+2hxy+by^{2}=0$ (through the origin $O$) be $y=m_{1}x$ and $y=m_{2}x$, where:

$m_{1}+m_{2} = -\frac{2h}{b}, \qquad m_{1}m_{2} = \frac{a}{b}$

These meet the line $lx+my+n=0$ at points $A$ and $B$.

Finding $OA$: Substitute $y=m_1x$ into $lx+my+n=0$: $x(l+mm_1)=-n$, so $x=\dfrac{-n}{l+mm_1}$, $y=m_1x$. Then:

$OA = |x|\sqrt{1+m_{1}^{2}} = \frac{|n|\sqrt{1+m_{1}^{2}}}{|l+mm_{1}|}$

Similarly, $OB = \dfrac{|n|\sqrt{1+m_{2}^{2}}}{|l+mm_{2}|}$.

Angle $\theta$ between the two lines: using direction vectors $(1,m_1)$, $(1,m_2)$:

$\sin\theta = \frac{|m_{1}-m_{2}|}{\sqrt{(1+m_{1}^{2})(1+m_{2}^{2})}}$

Area of $\triangle OAB$:

$\text{Area} = \frac{1}{2}\,OA\cdot OB\cdot\sin\theta = \frac{1}{2}\cdot\frac{n^{2}|m_{1}-m_{2}|}{|l+mm_{1}||l+mm_{2}|}$

(the $\sqrt{(1+m_1^2)(1+m_2^2)}$ factors cancel between the $OA\cdot OB$ product and $\sin\theta$'s denominator).

Simplify the denominator:

$(l+mm_{1})(l+mm_{2}) = l^{2}+lm(m_{1}+m_{2})+m^{2}m_{1}m_{2} = l^{2}-\frac{2hlm}{b}+\frac{am^{2}}{b} = \frac{am^{2}-2hlm+bl^{2}}{b}$

Simplify $|m_1-m_2|$:

$(m_{1}-m_{2})^{2} = (m_{1}+m_{2})^{2}-4m_{1}m_{2} = \frac{4h^{2}}{b^{2}}-\frac{4a}{b} = \frac{4(h^{2}-ab)}{b^{2}} \implies |m_{1}-m_{2}| = \frac{2\sqrt{h^{2}-ab}}{|b|}$

Combine:

$\text{Area} = \frac{1}{2}\cdot n^{2}\cdot\frac{2\sqrt{h^{2}-ab}}{|b|}\cdot\frac{|b|}{|am^{2}-2hlm+bl^{2}|} = \left|\frac{n^{2}\sqrt{h^{2}-ab}}{am^{2}-2hlm+bl^{2}}\right|$

This is exactly the required formula.