Let $I = \int x \tan^{-1} x \, dx$.
Using integration by parts,where $\tan^{-1} x$ is the first function and $x$ is the second function:
$I = \tan^{-1} x \int x \, dx - \int \left( \frac{d}{dx} \tan^{-1} x \int x \, dx \right) dx$
$I = \tan^{-1} x \left( \frac{x^2}{2} \right) - \int \left( \frac{1}{1+x^2} \cdot \frac{x^2}{2} \right) dx$
$I = \frac{x^2}{2} \tan^{-1} x - \frac{1}{2} \int \frac{x^2}{1+x^2} \, dx$
To integrate $\frac{x^2}{1+x^2}$,we rewrite the numerator as $(x^2+1-1)$:
$I = \frac{x^2}{2} \tan^{-1} x - \frac{1}{2} \int \left( \frac{x^2+1}{1+x^2} - \frac{1}{1+x^2} \right) dx$
$I = \frac{x^2}{2} \tan^{-1} x - \frac{1}{2} \int \left( 1 - \frac{1}{1+x^2} \right) dx$
$I = \frac{x^2}{2} \tan^{-1} x - \frac{1}{2} (x - \tan^{-1} x) + C$
$I = \frac{x^2}{2} \tan^{-1} x - \frac{x}{2} + \frac{1}{2} \tan^{-1} x + C$,where $C$ is the constant of integration.