Selection is driven from the surface rather than through mouse events, so stopping those isn't enough - a scrub would drag-select the text underneath it too.