Let $X$ be a Banach space. We give characterizations of when ${\cal F}(Y,X)$ is a $u$-ideal in ${\cal W}(Y,X)$ for every Banach space $Y$ in terms of nets of finite rank operators approximating weakly compact operators. Similar characterizations are given for the cases when ${\cal F}(X,Y)$ is a $u$-ideal in ${\cal W}(X,Y)$ for every Banach space $Y$, when ${\cal F}(Y,X)$ is a $u$-ideal in ${\cal W}(Y,X^{**})$ for every Banach space $Y$, and when ${\cal F}(Y,X)$ is a $u$-ideal in ${\cal K}(Y,X^{**})$ for every Banach space $Y$.