RDG requires Remote Desktop Users permission

I've setup Remote Desktop Gateway to allow connections to our test environment which is on a separate domain (Domain2) connected only by a terminal 'swing' server (Domain1). I have had to add users to the "Remote Desktop Users" group on this terminal server to allow connections to to Domain2. Everything is working correctly but from what I am told, no extra permissions should be required on the server itself.
Is this true? Any thoughts are appreciated.