Friday, November 16, 2007

Reboot Linux box after a kernel panic

If you want the server to get rebooted automatically after kernel hit by a pain error message, try adding panic=N to /etc/sysctl.conf file.

It specify kernel behavior on panic. By default, the kernel will not reboot after a panic, but this option will cause a kernel reboot after N seconds. For example following boot parameter will force to reboot Linux after 10 seconds.
Open /etc/sysctl.conf file

# vi /etc/sysctl.conf
When kernel panic’s, reboot after 10 second delay

kernel.panic = 10

Save the file.

You can do this on the fly also,by editing the GRUB
While Grub loads edit and append the following to the KERNEL line
panic=10
Alternatively, you may want to enable and use magic system request keys (SysRq).

Linux kernel includes magic system request keys. It was originally developed for kernel hackers. However, you can use this hack to reboot, shutdown or halt computer safely (remember safe reboot/shutdown == flush filesystem buffers and unmount file system and then reboot so that data loss can be avoided).

This is quite useful when Linux based system is not available after boot or after a X server crashed ( svgalib program crashes) or no display on screen. Sysrq key combo forces the kernel to respond it regardless of whatever else it is doing, unless it is completely locked up (dead).

Using further extension to iptables called ipt_sysrq (new iptables target), which allows you to do the same as the magic sysrq key on a keyboard does, but over the network. So if your network server is not responding you can still reboot it. Please note that Magic SysRq support need to be compiled in your kernel. You need to say “yes” to ‘Magic SysRq key (CONFIG_MAGIC_SYSRQ)’ when configuring the kernel. I’m assuming that you have Magic SysRq key’ support is compiled in your kernel.

Enable sysrq keys

By default it is not enabled on many Linux distributions. Add or modify following line (as soon as new Linux system installed) /etc/systctl.conf:
kernel.sysrq=1

Save and close the file and reboot system to take effect
How do I use the magic SysRq keys in emergency?

You need to use following key combination in order to reboot/halt/sync file system etc:
ALT+SysRq+COMMAND-KEY

The ‘SysRq’ key is also known as the ‘Print Screen’ key. COMMAND-KEY can be any one of the following (all keys need to hit simultaneously) :
‘b’ : Will immediately reboot the system without syncing or unmounting your disks.
‘o’ : Will shutdown your system off (if configured and supported).
’s’: Will attempt to sync all mounted filesystems.
‘u’ : Will attempt to remount all mounted filesystems read-only.
‘e’ : Send a SIGTERM to all processes, except for init.
‘h’: Show help, indeed this the one you need to remember.

So whey you need to tell your Linux computer to reboot or when your X server is crashed or you don’t see anything going across the screen then just press:

ALT+SysRQ+s : (Press and hold down ALT, then SysRQ (Print Screen) key and press ’s’) -Will try to syn all mounted system

ALT+SysRQ+r : (Press and hold down ALT, then SysRQ (Print Screen) key and press ‘r’) -Will reboot the system.

If you wish to shutdown the system instead of reboot then press following key combination:
ALT+SysRQ+o

ipt_sysrq is a new iptables target that allows you to do the same as the magic sysrq key on a keyboard does, but over the network. Sometimes a remote server hangs and only responds to icmp echo request (ping). Every administrator of such machine is very unhappy because (s)he must go there and press the reset button. It takes a long time and it’s inconvenient. So use the Network Magic SysRq and you will be able to do more than just pressing a reset button. You can remotely sync disks, remount them read-only, then do a reboot. And everything comfortably and only in a few seconds.

IP Tables network magic SysRq function 0.4



What is it?

ipt_sysrq is a new iptables target that allows you to do the same as the magic sysrq key on a keyboard does, but over the network.

Why to use the remote sysrq?

Sometimes a remote server hangs and only responds to icmp echo request (ping). Every administrator of such machine is very unhappy because (s)he must go there and press the reset button. It takes a long time and it's inconvenient. So here is a solution. Use the Network Magic SysRq and you will be able to do more than just pressing a reset button. You can remotely sync disks, remount them read-only, then do a reboot. And everything comfortably and only in a few seconds. ;-)

Is it secure?


That depends. ;-) Let me explain: You can restrict who can do this by setting the iptables firewall. But unfortunately, for simplicity, the Network Magic SysRq is based on a single packet request. This packet is encrypted and password protected, but if somebody can sniff it (s)he will be able to repeat (but not to change) the query (so-called replay attack). The query is also protected by a timestamp. When the packet is generated, it is stamped by current date and time. Then on the server side that stamp is compared with the current time of the server and if it is within the tolerance the request is accepted. Together with some other information, the timestamp is protected by SHA1 hash. This means that the potential attacker has a limited time to repeat the sniffed packet. If anybody requires a better security than this, some secure encrypted tunnel can be used. (not depending on userspace, of course! ;-))

How to install it?

Just type 'make'.

When everything is compiled type 'make install' as root and after that run 'depmod -a'. Now you can load the kernel module by the command 'modprobe ipt_SYSRQ'.

You would also like to configure the server password and the tolerance. This can be set when installing the module into a kernel, by specifying the module parameters 'passwd' for password and 'tolerance' for tolerance in seconds. The default values are passwd="" and tolerance=43200.

Example:
modprobe ipt_SYSRQ passwd="my_very_secret_password" tolerance=3600

Module options can also be specified in file /etc/modules.conf.

Example:
options ipt_SYSRQ passwd="my_very_secret_password" tolerance=3600

What to do on a server?

After the module is loaded you are able to deploy it using the iptables command.

Some examples of usage:

iptables -I INPUT -p udp --dport 9 -j SYSRQ
or
iptables -I INPUT -i eth1 -s 192.168.1.2 -p udp --dport 9 -j SYSRQ

Note that UDP port 9 is used. This is the default port for send_sysrq program, which shouldn't do any harm, as it defaults to 'discard' service.


What to do on the remote machine?


Copy the executable binary 'send_sysrq' to the remote (client) machine. Alternatively, you can compile ipt_sysrq there yourselves. After uncompressing the source package, you just need to do a 'make send_sysrq'.

Now you can use the client program 'send_sysrq' to send the sysrq request.

Usage of send_sysrq:
send_sysrq [-p ]

where
is destination UDP port (9 by default).
is DNS name or IP address of the remote host.
is a string of letters such as:
s - for sync
i - for kill all processes
u - for umount (remount read-only) all filesystems
b - for reboot
o - for power off
and other characters same as with magic sysrq key

my favorite string is "sisub"

Important note: send_sysrq sends two copies of the request for each command character separately. It does this to deal with the potential packet loss, because send_sysrq has no feedback from the remote host. Delay between the requests is one second.

Requirements

Works on Linux 2.4.x and on 2.6.x too.

To successful compile you need to have installed:
GCC
Linux kernel sources of your running kernel
Header files of your iptables command


For more information read man page of sysctl, sysctl.conf.

No comments: