When asked about security on a multi-user Linux system, a wise man once said "everyone is root if you allow them to login as a user." There is plenty of truth in that, but embracing imminent compromise isn't always acceptable. Let's take a look at how you can limit your exposure while letting unknown and untrusted users login with a shell.
There are two groups of people who typically want to heavily restrict login users. First, the collaborators: possibly two separate organizations that have been forced to work together. Second, people who wish to allow some shady characters access to a shell but believe they may attempt to compromise security. If at all possible, the best policy is to simply not give access out, and if you do, make sure patches are applied daily.
To say that you simply shouldn't give out shells to untrustworthy users may work in a few instances, but sometimes you have to let users in. Take a fairly simple example where remote users at another site need to log in and run the same series of commands every day. Assuming their task can be easily scripted, and if it's their only purpose on the server, a shell certainly isn't necessary. OpenSSH allows a set of restrictions to be applied to an SSH key.
At the end of an SSH key entry, you can tack on these options:
This effectively restricts any SSH connections using this key to only being allowed to run the referenced script. This can even be a setuid script that restarts a Web server, for example. It's quite safe, because OpenSSH will reject any variation of the command= text. Users possessing this key will only be able to execute the command that is explicitly allowed.
Aside from that, and possibly some fancy Web-based tools or cron jobs, there aren't may options left. At times users just need to be able to login and work, and in those cases you have your work cut out for you.
It should go without saying that you need to stay up-to-date on patches. We won't focus too much on that, aside from saying: automate! Securing a machine is an entirely different topic all together, but here are a few points to consider:
Enabling SELinux (Security-Enhanced Linux) is your first line of defense against unknown attacks. SELinux can prevent buffer overflows, as opposed to simply taking the "updates" path, which requires that a publicly known hole be fixed before some tries to exploit it. SELinux provides a significantly improved access system to limit programs from accessing things they don't require to be operational. That, combined with overflow prevention makes it quite difficult to compromise a Linux system.
Further, on the topic of securing a multi-user machine, there is a much-debated precept: that users shouldn't be able to see what processes are running, unless they own them. This restriction is simple to enable in Linux and the BSD's, but does it really buy you anything? The answer is "maybe," and at the same time, "not really."
To satisfy the "maybe" camp, consider a process's arguments. When you run a command with a given set of arguments, the command as well as the arguments will show up in a 'ps' listing. If you have provided a password on the command line, it will be visible to anyone running a 'ps' while your process is still running. Many people think that allowing users to see running daemon processes on a server will allow them to know what to try attacking. This information is trivial to obtain via other means anyway, so "not really."
Every time this discussion starts, someone quickly suggests a chroot jail. "chroot" stands for "change root," which does just that.
If you run the command: 'chroot /home/charlie /bin/bash,' chroot will execute the shell /home/charlie/bin/bash, and then proceed to lock you into /home/charlie. The new root of the file system, for the lifetime of the bash shell, is /home/charlie. You now have zero access to any other part of the actual file system. Any available command, and its required libraries, needs to be copied into the chroot jail. Providing a usable environment is a ton of work. It's actually easier to give each user their own Linux Xen or Solaris Zone instance.
Finally we come to the restricted shells. The most popular, rbash, is a restricted bash shell. Setting a user's shell to rbash will provide absolutely zero security. In theory, rbash will prevent users from running anything by specifying a full path, including './' (the current directory). This implies that it's difficult for users to run commands, including scripts they write or downloaded exploits. Since $PATH is controlled globally, users can only run things in those locations. Unfortunately, /bin/ is going to need to be in their path, so all a user needs to do is run a new shell, and rbash is no longer in the picture: 'exec bash'
One method of alleviating this is to give users only one item in their path, a directory the administrator created. Within the directory, simply place symlinks to all the authorized commands. This is nearly as cumbersome as setting up chroot, but much more tolerable.
There are certainly ways to prevent users from running downloaded programs, but in the end, the multi-user security of a system will depend on security of every piece of software installed. Preventing the exploits from being successful, a la SELinux, provides the most viable method of protection. Coupled with a frequently updated system, additional restrictions such as rbash aren't generally necessary.
The real takeaway, though? Security isn't convenient, and if it is, you're doing something wrong.