The task itself is conceptually straightforward: one must compute both integrals and verify that they indeed are equal. Before starting, a glimpse into the equation that governs the region reveals a helical pattern, which begs – together with the inverse tangent – a parametrization which leverages the symmetry of the figure. Polar coordinates will thus eventually be employed, but let us begin computing the left-hand side.

We first require the curl of $\mathbf F$, which follows from a cross product with the nabla "vector": $$ \nabla \times \mathbf F = \begin{vmatrix} \ihat & \jhat & \khat\\ \partial_x & \partial_y & \partial_z\\ xz & yz & -(x^2 + y^2) \end{vmatrix} = -3y \ihat + 3x \jhat = (-3y, 3x, 0), $$ and now we parametrize the surface, letting $x = r\cos\theta$, $y = r\sin\theta$ and therefore $z = \theta$, (notice that this eliminates the multiple branches of the inverse tangent, so we can integrate $\theta$ from $0$ to $2\pi$ without any issues.) The limits of integration are $0 \leq \theta \leq 2\pi$, and $0 \leq r \leq \theta$. Under this parametrization, the direction vectors become $$ \mathbf T_r = (\cos \theta, \sin \theta, 0),\qquad \mathbf T_\theta = (-r\sin\theta, r\cos\theta, 1), $$ and consequently the normal vector is $$ \mathbf n = \mathbf T_r \times \mathbf T_\theta = \begin{vmatrix} \ihat & \jhat & \khat\\ \cos\theta & \sin\theta & 0\\ -r\sin\theta & r\cos\theta & 1 \end{vmatrix} = \sin\theta \ihat - \cos \theta \jhat + r\khat = (\sin\theta, -\cos\theta, r); $$ since the $z$-coordinate is $r$, which is always positive, we are integrating in the outer direction. This is what we want, so that the boundary is then integrated along the positive way. Finally, putting everything together we get $$ \begin{align*} \iint\limits_{\mathscr H} (\nabla \times \mathbf F) \cdot \mathbf n \diff A &= \lint_0^{2\pi} \lint_0^r (-3r\sin\theta, 3r\cos\theta, 0) \cdot (\sin\theta, -\cos\theta, r) \diff r \diff\theta\\ &= -3 \lint_0^{2\pi} \lint_0^r r (\cancel{\sin^2\theta + \cos^2\theta}) \diff r \diff \theta = -\frac{3}{2} \lint_0^{2\pi} \theta^2 \diff\theta = \color{green}\boxed{-4\pi^3.} \end{align*} $$

Now for the second part, we need to parametrize the boundary of the curve; since it must be a closed curve, its shape is shown in the figure on the left. We now look at the main curve, which must lie both in the cone and the helix: from the first condition we know that $$ x^2 + y^2 = z^2 \implies \begin{cases}x &= z\cos \theta,\\ y &= z\sin\theta, \end{cases} $$ whereas from the second we extract $$ z = \arctan\l(\frac y x\r) = \arctan\l(\frac{\cancel r \sin \theta}{\cancel r \cos \theta}\r) = \theta, $$ so overall the desired parametrization is $\sigma_1(t) = (t\cos t, t\sin t, t)$. The other two regions are trivial to deduce, so we decompose the right-hand side of our original problem into the sum $$ \oint\limits_{\partial \mathscr H} \mathbf F \cdot \diff \mathbf s = \underbrace{\lint_{S_1} \mathbf F \cdot \diff \mathbf s}_{I_1} + \underbrace{\lint_{S_2} \mathbf F \cdot \diff \mathbf s}_{I_2} + \underbrace{\lint_{S_3} \mathbf F \cdot \diff \mathbf s}_{I_3}, $$ but when $x$ and $y$ are both $0$, $\mathbf F$ also evaluates to $0$, so one can state ahead of time that $I_3 = 0$, and refrain from computing it manually. Because of the parametrization above, $$ I_1 = \lint_{S_1} \mathbf F \cdot \diff \mathbf s = \lint_0^{2\pi} \mathbf F(\sigma_1(t)) \cdot \sigma_1'(t) \diff t = \lint_0^{2\pi} (t^2 \cos t, t^2 \sin t, -t^2) \cdot (\cos t - t\sin t, \sin t + t\cos t, 1) \diff t = \lint_0^{2\pi} 0 = 0. $$ Finally, $S_2$ is parametrized by $\sigma_2(t) = (t, 0, 2\pi)$, for $t$ going from $2\pi$ to $0$, hence $$ I_2 = \lint_{2\pi}^0 \mathbf F(\sigma_2(t)) \cdot \sigma_2'(t) \diff t = -2\pi \lint_0^{2\pi} t = -4\pi^3 \implies \oint\limits_{\partial \mathscr H} \mathbf F \cdot \diff \mathbf s = \color{green}\boxed{-4\pi^3,} $$ as it was to be shown.