Modify the default runlevel in /etc/inittab if you're using init.
If you're using systemd, change the default target link in /etc/systemd/system
Normally, runlevel 1 is single user mode, 3 is full multi-user, and 5 is multi-user with GUI. So you should change 5 to 3.
Ubuntu switched to upstart recently (they used to use inittab). Anyway, their runlevels were always messed up even when they used inittab - 1 was single user mode, 2 and up were multi-user w/ X.
Anyway, this got me thinking that the easiest way to disable X would probably be to modify the kernel options passed by GRUB.
In /etc/default/grub, there's a line like:
replace it with:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash text"
and then run:
Then, at any time, you can simply run startx if you'd like to run X.