A virtual private server (VPS) is a partition of a shared server that can be used for internet hosting. For those of you who didn't follow any of that sentence, we'll take a step back.
An internet (or web) hosting service gives your website a home. The files that make up your website are stored on the service's web server, a computer system that lets your site connect to the World Wide Web. A server may appear to users as a single computer, but in fact can be made up of multiple hardware units and software.
A virtual private server (VPS) is a halfway measure between these two. A regular server is partitioned into multiple "virtual" servers, each one used by a different client and appearing as a separate machine.
Each VPS (also called a virtually dedicated server or virtual machine) runs its own copy of its operating system. The client has full access to his "server". He can install nearly any software that runs on the OS; he can even reboot the VPS without affecting the others.
(Some software does not work well on a VPS, such as antivirus, firewalls, and -- not surprisingly -- software to create virtual machines.)
A virtualization solution is the software that creates VPSs on a server. Linux OpenVZ VPS servers are popular because of their ease of use. Windows VPS run on Kvm Based servers; there are also virtualizations that can run on consumer versions of Windows, thus using ordinary computers as a physical server.
Because VPSs are separated on their physical server - in effect, being separate machines - they are useful for keeping untested software from the main server environment. A physical server can have a primary "production" VPS, and a duplicate one for testing code or for running programs from untrusted sources that may present security issues.