Fix visual mode not respecting returnTo mode when using mouse

[TrevorCow]

When using Neovim or Vi, if in insert mode and then selecting text with the mouse, the mode is then switched to (insert) VISUAL and upon clicking to deselect, returns you to INSERT. In ideavim, no matter what mode you start in, selecting text and then clicking to deselect text returns you to NORMAL mode.

I am not super familiar with this code base, but this is this fix only changes a few lines and seems to be correct; it should respect the starting mode when switching to/from VISUAL/SELECTION mode using the mouse.

If anyone is more familiar with the codebase and has a better solution, let me know!

[AlexPl292]

Hello! Thank you for your interest!
I generally agree with the issue and the fix. Yet, I'd like to request a few changes:

• Please rebase your fixes on master to avoid unnecessary merge commits. We use a linear history approach.
• There are some tests currently failing, please check them.
• This change will also require testing to ensure it works properly.
• Can you comment on why we use the while for detecting the initialModeReturnTo and how it is supposed to work?

[TrevorCow]

Yes, the while loop is a little weird, it was added because in my testing, it was possible to get an active mode something along the lines of VISUAL(returnTo=VISUAL(returnTo=*what we actually want to return to*)) when doing something like reselecting while a selection was already made. (And this is a problem because the action of VISUAL->VISUAL is not valid) I didn't know the extent of the nesting, so thus the while loop. If we can confirm there is only ever 1 level of nesting, then we can change that, or if there is a simpler way to "collapse" the mode that could work also.

I will look into the tests, it seems to be related to it trying to switch from VISUAL->VISUAL so the while loop might need to be reworked anyway. Thanks for your comments!

[citizenmatt]

Nice change! Perhaps a question for @AlexPl292 - should this logic be part of chooseSelectionMode rather than outside it? We have a couple of callers for chooseSelectionMode - should they also get this new behaviour?

Secondly, I think it would be good to address the Visual -> Visual mode change, which does indeed sound like invalid behaviour.

[AlexPl292]

Thank you!
I don't confirm that the nesting level is only up to 1 (there are some tricky cases for i_Ctrl-o), but Visual in Visual is definitely a wrong behaviour.

Also, I agree with Matt that putting this logic right into the chooseSelectionMode is a better choice.

[AlexPl292]

See the comments above