I've had idiopathic peripheral neuropathy for over 15 years. It is still progressing, starting out with my feet and lower legs and now moving up my arms from my fingers.
My experience with driving is that I also began to lose the ability to distinguish between the gas and brake pedals. I did not get into any accidents but did have a couple close calls and a considerable amount of tension due to not knowing if I could properly respond to a given situation.
I think there were two reasons that held me back from getting hand controls. First is that I was afraid that it would be very difficult to go from pedals to hand controls and might be a steep learning curve. Second reason is that every thing that you give up is another admission that your abilities have diminished which is not a comfortable thing to do.
I have been in continuous treatment for pain since 2008, have seen several neurologists, pain specialists, etc. No one forced me to get hand controls although my family was trying to convince me that I should. If I was smarter I would have listened sooner.
I found the transition to hand controls was not very difficult. I did have to get a car with hand controls before I was able to take the test and it did cost a couple of thousand dollars. I was still working at the time And there was a state program that assisted with the cost in order to extend my working life. Some automakers offer some cash on new cars for the modifications or might be worth checking into any kind of state assistance if you need it.
I'd say if this has become a serious concern for you then it is probably past time to move to hand controls. The additional cost for the controls and the admission of diminished capacity is not worth endangering the health or life of another person by driving when not safe to do so.
Also, the type of hand controls that I had installed do not impede the ability of anyone else to drive the same car. Nothing needs to be turned off, removed, or any other alteration for someone else to drive it.