Supervisord

Monitoring and Controlling Applications, Processes with Supervisord


Overview


In this article we are going to cover -

  • Install Supervisor.
  • Configure Supervisor.
  • Write a long running script which we can watch and control using Supervisor.


Prerequisites


  • One CentOS/any Linux OS installed machine.
  • Proper networking access.

In this article we are using the below machine.

  • CentOS Linux release 7.2.1511 (192.168.20.21) where we are going to install supervisord.


Brief Overview of Supervisor


Sometime we need to write a "long running script" which needs to run all the time. These scripts shouldn't fail if there's an error, or ones that should auto restart when the system reboots. To achieve this, we need something to watch and control these scripts. So, Supervisor fulfill the same need.


Supervisor is a software/process manager which is used to manage and monitor number of  long running programs or daemon process in your linux system. Supervisor will mange your process to live all time and monitored as well.



Supervisor and Python Installation


We are installing the Supervisor and python with the below command.


$ sudo yum -y install https://centos7.iuscommunity.org/ius-release.rpm
 
$ sudo yum -y install python36u
 
$ python3.6 -V
 
$ sudo yum -y install python36u-pip
 
$ pip install supervisor
 

Configuration of Supervisor


# mkdir -p /etc/supervisord/conf.d

 

# echo_supervisord_conf > /etc/supervisord/supervisord.conf    

 

The "echo_supervisord_conf "program is provided by Supervisor; it prints a sample config file to the location specified. This will create a file named "supervisord.conf  at the location specified in the above command.


# vi /etc/supervisord/supervisord.conf

[include]
files = /etc/supervisor/conf.d/*.conf

 


Setup as service: 


# wget https://gist.githubusercontent.com/mozillazg/6cbdcccbf46fe96a4edd/raw/2f5c6f5e88fc43e27b974f8a4c19088fc22b1bd5/supervisord.service -O /usr/lib/systemd/system/supervisord.service

 

Run the Supervisor


# systemctl start supervisord

 

# systemctl enable supervisord

 

# systemctl status supervisord

 

Long Running Script


Now we need to create a script which we want to run in long time. The below script is just for showing the functionality of supervisor not serving any purpose.

 

# cat /usr/local/bin/long_script.sh

#!/bin/bash

while true
do

# Print the current date and time to stdout
echo "The current date and time is: " `date`

# Sending 'error!!' to stderr
echo 'error!!' >&2
sleep 2

done

 
 

# chmod +x /usr/local/bin/long_script.sh

 

Program Configuration Files for Supervisor


For Supervisor programs the program configuration files found in the/etc/supervisor/conf.d directory, normally one file per program and a .conf extension. Now we need to do the configuration for our script which is saved at/etc/supervisor/conf.d/long_script.conf as shown below.

# cat /etc/supervisor/conf.d/long_script.conf

# Defining a program with the name "long_run_script"

[program:long_run_script 

                  
# Full path to the script

command=/usr/local/bin/long_script.sh 
autostart=true
autorestart=true

# Error log file location

stderr_logfile=/var/log/long_script_err.log

# Output log file location

stdout_logfile=/var/log/long_script_out.log

 

 
 

The autostart and autorestart define the automatic behavior of the script under certain conditions.

1) autostart - It tells to Supervisor that if it is true then this program should be started when the system boots, if it is false then will require a manual start.
 
2) autorestart - It defines how Supervisor should manage the program in the event it exits and has three options:
  • false - means not to ever restart the program after it exits.
  • unexpected - means to only restart the program if it exits with an unexpected error code (by default anything other than codes 0 or 2).
  • true - means to always restart the program after it exits.
Now our configuration file is created and saved.


New/Changed Program Configurations Acknowledged by Supervisor

Now we will tell Supervisor to look for any new or changed program configurations in the /etc/supervisor/conf.d directory with:

# supervisorctl reread

 

# supervisorctl update

 
Whenever we make a change to any program configuration file, running the two previous commands will bring the changes into effect.

Now our program should be running and we can check by looking at the output log file:

# tail -f /var/log/long_script_out.log
The current date and time is: Fri Aug 4 02:09:29 IST 2017
The current date and time is: Fri Aug 4 02:09:31 IST 2017
The current date and time is: Fri Aug 4 02:09:33 IST 2017
The current date and time is: Fri Aug 4 02:09:35 IST 2017
The current date and time is: Fri Aug 4 02:09:37 IST 2017
The current date and time is: Fri Aug 4 02:09:39 IST 2017
The current date and time is: Fri Aug 4 02:09:41 IST 2017

Managing Programs


Once our programs are running, we definitely want to stop, restart, or see their status. We can control our programs using the supervisorctl program.

To enter into the interactive mode, start supervisorctl with no arguments:

# supervisorctl
long_run_script RUNNING pid 1419, uptime 0:06:14

 

For help run the command help

supervisor> help

default commands (type help ):
=====================================
add      exit        open   reload    restart     start      tail
avail    fg          pid    remove    shutdown    status     update
clear    maintail    quit   reread    signal      stop       version

 

As we see in help we have several options like start, stop, restart, status etc.

 

supervisor> start long_run_script
long_run_script: ERROR (already started)

 

supervisor> status long_run_script
long_run_script RUNNING pid 1419, uptime 0:07:25

 

supervisor> stop long_run_script
long_run_script: stopped

 

supervisor> status long_run_script
long_run_script STOPPED Aug 04 02:15 AM

 

supervisor> restart long_run_script
long_run_script: ERROR (not running)
long_run_script: started

 

supervisor> status long_run_script
long_run_script RUNNING pid 1882, uptime 0:00:05


Output:



Web Interface


We can also configure a web interface which comes with Supervisord. This lets us see a list of all processes being monitored, as well as take action on them (restarting, stopping, clearing logs and checking output).
 
Inside of /etc/supervisord/supervisord.conf, add this:
 

# vi /etc/supervisord/supervisord.conf
 
[inet_http_server]             ; inet (TCP) server disabled by default
port=192.168.20.21:9001        ; ip_address:port specifier, *:port for all iface
 
# firewall-cmd --zone=public --permanent --add-service=http
 
# firewall-cmd --zone=public --add-port=9001/tcp --permanent
 
# firewall-cmd --reload
 
# systemctl restart supervisord
 
# systemctl status supervisord



Now we can see that Supervisor web interface is also up and running. From web interface we can perform several activity:

  • Monitor all the long running process
  • Control all the long running process
  • Restart the process
  • Stop the process
  • Tail the log file


Leave a Reply

Your email address will not be published. Required fields are marked *