If you look behind that panel you will see that the blank, the L/R switch and the "joystick" are all one assembly. Your only option is to get the EDM or JDM mirror switch which changes that blank to the Fold Mirror switch and then rewire it for whatever you want.
Thanks for the info - I would have never figured it out until it was too late!
Depending on the price/difficulty of the above, maybe instead I can put a switch in the center console near the e-brake, like a nice rocker switch. Are there some stock switches for that area?