Q.Find the angle between the pair of lines represented by $3x^2 + 10xy + 3y^2 = 0$.

Writing the pair of lines as $y=m_1x$ and $y=m_2x$, using the sum/product of roots from the quadratic in $m$, and the standard angle-between-lines formula gives the required identity.

The equation $ax^2+2hxy+by^2=0$ represents two lines through the origin, $y=m_1x$ and $y=m_2x$. Dividing by $x^2$: $b\left(\dfrac{y}{x}\right)^2+2h\left(\dfrac{y}{x}\right)+a=0$, i.e. $bm^2+2hm+a=0$, whose roots are $m_1,m_2$.

By Vieta's formulas:

$m_1+m_2 = -\dfrac{2h}{b}$, $\qquad m_1m_2=\dfrac{a}{b}$

The angle between the two lines satisfies:

$\tan\theta = \left|\dfrac{m_1-m_2}{1+m_1m_2}\right|$

$(m_1-m_2)^2 = (m_1+m_2)^2-4m_1m_2 = \dfrac{4h^2}{b^2}-\dfrac{4a}{b} = \dfrac{4(h^2-ab)}{b^2}$

$1+m_1m_2 = 1+\dfrac{a}{b} = \dfrac{a+b}{b}$

…

From $\tan\theta=\frac{2\sqrt{h^2-ab}}{|a+b|}$, build a right triangle with opposite $2\sqrt{h^2-ab}$ and adjacent $|a+b|$; the hypotenuse simplifies to $\sqrt{(a-b)^2+4h^2}$.

If $ax^2+2hxy+by^2=0$ represents the pair of lines $y=m_1x$ and $y=m_2x$, then $m_1+m_2=-\dfrac{2h}{b}$ and $m_1m_2=\dfrac{a}{b}$. The angle $\theta$ between them satisfies:

$\tan\theta=\left|\dfrac{m_1-m_2}{1+m_1m_2}\right|=\dfrac{\sqrt{(m_1+m_2)^2-4m_1m_2}}{|1+m_1m_2|}=\dfrac{2\sqrt{h^2-ab}}{|a+b|}$.

…

Writing the pair of lines as $b(y-m_1x)(y-m_2x)=0$ gives $m_1+m_2=-\frac{2h}{b}$, $m_1m_2=\frac{a}{b}$; substituting these into the angle-between-two-lines formula and simplifying yields the required result.

Concept

$ax^2+2hxy+by^2=0$ (a homogeneous second-degree equation) represents a pair of straight lines through the origin with slopes $m_1,m_2$ satisfying (writing $b(y-m_1x)(y-m_2x)=ax^2+2hxy+by^2$ and comparing coefficients):

$m_1+m_2 = -\dfrac{2h}{b}$, \quad $m_1m_2 = \dfrac{a}{b}$

Step 1: Angle between the two lines

$\tan\theta = \left|\dfrac{m_1-m_2}{1+m_1m_2}\right|$

Step 2: Compute $(m_1-m_2)^2$

$(m_1-m_2)^2 = (m_1+m_2)^2-4m_1m_2 = \dfrac{4h^2}{b^2}-\dfrac{4a}{b} = \dfrac{4(h^2-ab)}{b^2}$

So $|m_1-m_2| = \dfrac{2\sqrt{h^2-ab}}{|b|}$.

Step 3: Substitute

$\tan\theta = \dfrac{2\sqrt{h^2-ab}/|b|}{|1+a/b|} = \dfrac{2\sqrt{h^2-ab}}{|a+b|}$

Step 4: Convert to cosine

…