Is it not ~/.gem with a dot and no “s”?
It has gems installed for the user for use across projects or as shell tools. Like I run rufo to format my scripts without installing rufo in every Ruby project or mine.
Other users cannot access the gems.
While a path in /usr/... has a place for shared gems is accessible for all users to read. Like if you had multiple people using a machine or some user created to manage serving files on an a remote ubuntu machine. The problem is you need root access to write to that directory. Or you need to use root access to change permissions so that any user can edit that directory. Which is fine if you are the only person using your laptop or remote machine.