I have never had this before, but I found this on the internet to reset the remote, maybe this helps (https://support.apple.com/de-de...):

If your remote does not work as expected, try restarting the remote:

Press and hold the TV/Control Centre button and the Volume Down button at the same time. Hold the buttons down for about 5 seconds until the Apple TV status indicator goes out and then lights up again.
Release the buttons. Then wait 5 to 10 seconds until the TV screen displays the message "Connection lost".
The remote control will restart. When the message "Connected" appears, you can use the remote control again.

Good luck!