Oliver only gets those purple/pink swirls in his eyes when he's pushed off the edge. Most of the time, he doesn't have much control over himself when those swirls appear.