(b) On path AC, \(x \) and \(y \) are related by
\begin{equation*}
y = mx + b,
\end{equation*}
with
\begin{equation*}
m = \dfrac{3-2}{5-0} = 0.2,\ \ b = 2.
\end{equation*}
That is,
\begin{equation*}
y = 0.2\, x + 2.
\end{equation*}
This can be used to eliminate \(y \) and \(dy\) from equations below.
We use the unit vector for the direction from A to C to write the force vector in an analytic form. This unit vector is
\begin{equation*}
\hat u = \dfrac{ (5-0)\hat i + (3-2)\hat j}{ \sqrt{(5-0)^2 + (3-2)^2}} = \dfrac{5}{\sqrt{26}}\, \hat i + \dfrac{1}{\sqrt{26}}\, \hat j.
\end{equation*}
Since kinetic friction is pointed in the opposite direction to the displacement, we get the following analytic expression for the force vector, since the magnitude \(10\, x\, y^2\) is positive.
\begin{align*}
\vec F \amp = -10\, x\, y^2\, \hat u\\
\amp = -10\, x\, y^2 \left( \dfrac{5}{\sqrt{26}}\, \hat i + \dfrac{1}{\sqrt{26}}\, \hat j \right).
\end{align*}
The force and displacement vectors on this path are:
\begin{align*}
\amp d\vec r = \hat i\, dx + \hat j\, dy = (\hat i\, + 0.2\, \hat j)\, dx, \\
\amp \vec F = -10\, x\, (0.2x + 2)^2 \left( \dfrac{5}{\sqrt{26}}\, \hat i + \dfrac{1}{\sqrt{26}}\, \hat j \right),
\end{align*}
where I used \(dy = 0.2 dx \) on this path so that we can write the work integral as an integral over \(dx\text{.}\)
\begin{equation*}
W_{AC} = \int_{AC}\, \vec F\cdot d\vec r = \int_{0}^5\, \left( \text{ something here }\right) dx.
\end{equation*}
The dot product is ugly to work out. You can do that on a separate piece of paper. I found the following.
\begin{equation*}
W_{AC} = - \dfrac{52}{\sqrt{26}}\, \int_0^5\left(0.04\,x^3 + 0.8\, x^2 + 4\, x \right)dx = -914\text{ N.m}.
\end{equation*}
(c) Clearly, we get
\begin{equation*}
\left| W_{AC} \right| \gt \left| W_{ABC} \right|.
\end{equation*}
This says that it would cost less energy to go on path A-B-C than to go directly.