[SOLVED] Help with Ubuntu + MySQL

[SgtSpike]

I installed Ubuntu on a VPS, and then installed LAMP.  After a few issues were sorted, everything seemed to be running fine.  I restarted the server several times without any problems.

Then, I go to access my website this evening, and it seems to be having trouble.  I thought it was something wrong with the VPS, so I restarted the server.  Now MySQL won't start up, even though it is listed in the /etc/init.d/ folder.  I even tried starting it manually with "service mysql start", which just seems to lock up the SSH session (it doesn't bring me back to the prompt after typing that).

Any ideas what's going on here?  If I type "ps -e", it shows apache2 running, but not mysql.

[1714866123]

1714866123

[1714866123]

1714866123

[1714866123]

1714866123

[1714866123]

1714866123

[Xenland]

I installed Ubuntu on a VPS, and then installed LAMP.  After a few issues were sorted, everything seemed to be running fine.  I restarted the server several times without any problems.

Then, I go to access my website this evening, and it seems to be having trouble.  I thought it was something wrong with the VPS, so I restarted the server.  Now MySQL won't start up, even though it is listed in the /etc/init.d/ folder.  I even tried starting it manually with "service mysql start", which just seems to lock up the SSH session (it doesn't bring me back to the prompt after typing that).

Any ideas what's going on here?  If I type "ps -e", it shows apache2 running, but not mysql.

What is your specs? possible you don't have enough ram.

[SgtSpike]

512MB of ram... I'll have to check how much is actually being used atm.

[Xenland]

512MB of ram... I'll have to check how much is actually being used atm.

If your just running apache + mysql then you should be alright. But if you are trying to run apache + mysql + bitcoind + pushpoold... you need atleast 756MB for the memory spikes for about 1-5 workers.

[SgtSpike]

512MB of ram... I'll have to check how much is actually being used atm.

If your just running apache + mysql then you should be alright. But if you are trying to run apache + mysql + bitcoind + pushpoold... you need atleast 756MB for the memory spikes for about 1-5 workers.

Just apache + mysql + bitcoind.  I looked, and mem usage is only 14%.  I don't think that is the issue here...

[Xenland]

what happens when you....

Code:

/etc/init.d/mysqld restart

[SgtSpike]

what happens when you....

Code:

/etc/init.d/mysqld restart

Well, mysqld doesn't work, but changing that for mysql gets me this:

Code:

Rather than invoking init scripts through /etc/init.d, use the service(8)
utility, e.g. service mysql restart

Since the script you are attempting to invoke has been converted to an
Upstart job, you may also use the restart(8) utility, e.g. restart mysql

But same result - it's not bringing me back to the # prompt.  It's just kind of stuck now, and won't do anything no matter what I type until I reconnect.

[BCEmporium]

After a starting failure do:

tail /var/log/mysql.err

and check what's cooking with MySQL

[SgtSpike]

After a starting failure do:

tail /var/log/mysql.err

and check what's cooking with MySQL

After relogging into SSH, I tried that... nothing shows when I type that command.  It just pulls up another prompt.  After looking at what the command does, I pico'd into the .err file, and nothing is there.  Is there another file I should check?  These are the files in the /var/log/ directory:

Code:

apache2          dbconfig-common  kern.log        messages.1      syslog.2.gz
apparmor         debug            kern.log.1      messages.2.gz   syslog.3.gz
apt              debug.1          kern.log.2.gz   mysql           syslog.4.gz
aptitude         debug.2.gz       landscape       mysql.err       syslog.5.gz
aptitude.1.gz    dist-upgrade     lastlog         mysql.log       syslog.6.gz
auth.log         dmesg            lpr.log         mysql.log.1.gz  syslog.7.gz
auth.log.1       dmesg.0          mail.err        mysql.log.2.gz  udev
auth.log.2.gz    dmesg.1.gz       mail.info       mysql.log.3.gz  ufw.log
boot             dmesg.2.gz       mail.info.1     mysql.log.4.gz  user.log
boot.log         dmesg.3.gz       mail.info.2.gz  mysql.log.5.gz  vsftpd.log
btmp             dpkg.log         mail.log        mysql.log.6.gz  vsftpd.log.1
btmp.1           dpkg.log.1       mail.log.1      mysql.log.7.gz  vsftpd.log.2
ConsoleKit       faillog          mail.log.2.gz   news            wtmp
daemon.log       fontconfig.log   mail.warn       pycentral.log   wtmp.1
daemon.log.1     fsck             mail.warn.1     syslog
daemon.log.2.gz  installer        messages        syslog.1

[BCEmporium]

try:

tail /var/log/syslog

after a MySQL start failure.

[SgtSpike]

Edited the IP's to hide them, but each different series was a different number in the log.

Code:

Aug  2 14:56:27 111-111-111-111 postfix/smtpd[13487]: disconnect from 222-222-222-222.dynamic.hinet.net[222.222.222.222]
Aug  2 14:59:47 111-111-111-111 postfix/anvil[13491]: statistics: max connection rate 1/60s for (smtp:222.222.222.222) at Aug  2 14:56:26
Aug  2 14:59:47 111-111-111-111 postfix/anvil[13491]: statistics: max connection count 1 for (smtp:222.222.222.222) at Aug  2 14:56:26
Aug  2 14:59:47 111-111-111-111 postfix/anvil[13491]: statistics: max cache size 1 at Aug  2 14:56:26
Aug  2 15:03:33 111-111-111-111 dhclient: DHCPREQUEST of 111.111.111.111 on eth0 to 555.555.555.555 port 67
Aug  2 15:03:33 111-111-111-111 dhclient: DHCPACK of 111.111.111.111 from 444.444.444.444
Aug  2 15:03:33 111-111-111-111 dhclient: bound to 111.111.111.111 -- renewal in 16846 seconds.
Aug  2 15:09:01 111-111-111-111 CRON[15086]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -print0 | xargs -n 200 -r -0 rm)
Aug  2 15:17:01 111-111-111-111 CRON[16044]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Aug  2 15:39:01 333-333-333-333 CRON[18656]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -print0 | xargs -n 200 -r -0 rm)

This was the state of tail /var/log/syslog both before and after doing another /etc/init.d/mysql restart.

[BCEmporium]

Try to run this command directly in the console:

/usr/bin/mysqld_safe

(it's the starting command for MySQL-server)

[SgtSpike]

Try to run this command directly in the console:

/usr/bin/mysqld_safe

(it's the starting command for MySQL-server)

Not sure what you mean by "directly in the console".  This isn't a gui version of ubuntu.  All I have is SSH and VNC to the same view.  They should both be the same once logged in, right?

Anyway, I tried said command in the SSH window.  Now we're getting somewhere...

Code:

root@111-111-111-111:~# /usr/bin/mysqld_safe
110802 16:22:23 mysqld_safe Logging to syslog.
110802 16:22:23 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
110802 16:22:24 mysqld_safe mysqld from pid file /var/lib/mysql/111-111-111-111.pid ended

Similarly, the syslog now has:

Code:

Aug  2 16:22:23 111-111-111-111 mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql
Aug  2 16:22:24 111-111-111-111 mysqld_safe: mysqld from pid file /var/lib/mysql/111-111-111-111.pid ended

[BCEmporium]

SSH without GUI is "directly in the console".
So, MySQL started without issues manually. Is it running now?

[SgtSpike]

No, it is not running.

I thought

Code:

Aug  2 16:22:24 111-111-111-111 mysqld_safe: mysqld from pid file /var/lib/mysql/111-111-111-111.pid ended

meant that the process stopped running?

[Bitsky]

You could try starting it with --verbose to see if it spits out more errors.
Also, tail -f /var/log/mysql.log from another login.
Take a look into /var/log/mysql/, perhaps something interesting is in there.

If everything else fails, you can rename your database directory (/var/lib/mysql) and uninstall mysql completely. Don't forget to delete configs like /etc/my.cnf too. Then install it again and see if it's starting. If it does, stop mysql and copy your databases from your renamed directory back in. If it then fails, your database(s) are broken and you need to check/repair them.

If you run anything bad like a "control panel" that relies on mysql you may break your admin panel (which wouldn't be bad, since you only need ssh anyway).

[SgtSpike]

Ok, I tried starting mysql_safe with --verbose.  It didn't seem to change what was output.

I didn't tail mysql.log from another login, since the file has remained empty.  Seems pointless?  Maybe I'm missing something there...

I took a look inside /var/log/mysql/ and noticed an error.log file.  The lines from the most recent run attempt say this:

Code:

110802 17:41:43 [Note] Plugin 'FEDERATED' is disabled.
110802 17:41:44  InnoDB: Started; log sequence number 0 44233
110802 17:41:44 [ERROR] Can't start server: Bind on TCP/IP port: Cannot assign requested address
110802 17:41:44 [ERROR] Do you already have another mysqld server running on port: 3306 ?
110802 17:41:44 [ERROR] Aborting

110802 17:41:44  InnoDB: Starting shutdown...
110802 17:41:45  InnoDB: Shutdown completed; log sequence number 0 44233
110802 17:41:45 [Note] /usr/sbin/mysqld: Shutdown complete

Hmmm... so something is using or binding that port, yet MySQL is NOT already running.  

Gawd, this would be so simple on a Windows box too.  If only it had the security of Linux...

EDIT:  I'm going to google around for an answer to the port problem, but if anyone has more suggestions, feel free to chime in.  Thanks for the help so far guys! 

[Bitsky]

Well mysql seems to be pretty clear about the possible problem: seems like something is already using the port 3306.

Try "netstat -tunap | grep 3306" to see what program it is. Use the pid from that to do a "ps -aux | grep PID" and see more.

From there on, you can decide if you just want to kill -9 whatever process is clinging to the port. Maybe a zombie.

[SgtSpike]

Yikes!  Figured out the issue... turns out, my my.cnf file did not have 127.0.0.1 as the bind-address.  I had changed it in an effort to be able to remotely access the database with MySQL Workbench (which proved to be unsuccessful), then promptly forgot I had made the change.  On the next serve reboot, it took the change, and disallowed local access in the process.

Changed the bind-address back to the local IP, and it's working perfect now.

Community help + googling found the problem, so I thank you all for your assistance.

[BCEmporium]

Just a tip, don't bind MySQL to 0.0.0.0, it has no protection at all against brutte-forcing, rather create a XML or JSON RPC to filter and output data between servers.