How do I turn off my controller support?

[Drawing4Life]

Ok, so wording this is a bit hard, but here is the deal.
In every other Linux Distro I've been on there is no native (gaming) controller support. And I like it that way, because then I can install JoyToKey and configure it myself.
But this time I've decided to program my own software and from there configure my controller (C programming).  8)
However on LinuxLite3(64-bit) when I plug in the controller, it just works. Can move my mouse around with it... That's great!
... How do I turn that off? xD

[Drawing4Life]

Really, nothing I can do about that?  :'(

[Valtam]

No response means at this stage no one with the knowledge or experience with this specific issue thus far. Try removing the xserver-xorg-input-joystick package, then rebooting.

[tenet510h]

Wouldn't it be easier to just disable or change the config to remove the unwanted feature?