Just for ease of copy&paste (oughta make a script out of this).

Install PostGIS (assuming Postgre is already installed); change version number, if neccessary

sudo apt-get install postgis postgresql-9.3-postgis

Setup a new database (with UTF-8 encoding):

psql -U username
create database spatial_database;

Enable PostGIS for that database:

psql -d spatial_database -f /usr/share/postgresql/9.3/contrib/postgis-2.1/postgis.sql;
psql -d spatial_database -f /usr/share/postgresql/9.3/contrib/postgis-2.1/spatial_ref_sys.sql;
psql -d spatial_database -f /usr/share/postgresql/9.3/contrib/postgis-2.1/postgis_comments.sql;

Done. :]