The auxiliary equation $m^2+4=0$ gives $m=\pm2i$, so the complementary function is $A\cos 2x+B\sin 2x$.

For the linear differential equation $(D^2+4)y=e^{2x}$, the complementary function (C.F.) is the general solution of the homogeneous part $(D^2+4)y=0$. Its auxiliary equation is

$m^2 + 4 = 0 \;\Rightarrow\; m^2 = -4 \;\Rightarrow\; m = \pm 2i.$

…

Put the equation in the form $\dfrac{dy}{dx}+Py=Q$ with $P=-\dfrac1x$; then $I.F.=e^{\int P\,dx}=\dfrac1x$, option (d).

Standard linear form. Divide $x\dfrac{dy}{dx}-y=x^2$ by $x$:

$\frac{dy}{dx}-\frac{1}{x}\,y=x.$

So $P=-\dfrac{1}{x}$.

Integrating factor. …

Putting $D=2$ makes the operator vanish (failure case), so the P.I. gets an extra factor $x$ and the operator is differentiated once, giving $xe^{2x}$.

The particular integral is

$P.I.=\frac{1}{3D^2+D-14}\,13e^{2x}.$

Put $D=2$ in the operator: $3(2)^2+2-14=12+2-14=0$. This is the case of failure, so we apply the rule …

For $\text{P.I.} = \frac{1}{f(D)}e^{ax}$, put $D = a = 2$. Since $f(2) = 0$ (failure case), use $\text{P.I.} = \frac{x}{f'(D)}e^{ax}$ with $f'(2) = 13$, giving $xe^{2x}$.

Step 1 — Identify $f(D)$ and $a$. Here $f(D) = 3D^2 + D - 14$ and the RHS is $13e^{2x}$, so $a = 2$.

Step 2 — Test the standard rule $\frac{1}{f(D)}e^{ax}$ at $D = 2$.

$f(2) = 3(2)^2 + 2 - 14 = 12 + 2 - 14 = 0.$

Since $f(2) = 0$, the direct substitution fails.

…