Slicehost Articles - Home

Installing munin overview

2010-03-12T19:13:00Z

Anticipating problems and resource shortages on a slice can be more valuable than fixing them after they've happened. A monitoring tool like munin lets you watch your slice's resource use over time. The graphs will highlight issues before they cause downtime or bandwidth quota overages.

Why munin?

There are a number of monitoring tools out there, and each have their own strengths. Munin's focus is providing graphs over time showing system statistics like memory use, disk space, and network activity. Munin can also monitor several slices from a single master server. The setup is more complex than using a commercial monitoring service, but the benefit is that you have more control over what is monitored and how the information is presented. Well, and it's free. There's that too.

The first article in the series covers the installation of munin and some configuration options.

* What you'll need
* Installing packages
* The munin.conf file
* Email notification
* The host tree
* The munin-node.conf file
* The munin-node service

Later articles cover further configuration, testing the installation, and adding more servers to be monitored.

What you'll need

Munin's output is in html, so you will want to have a web server running to make reports available through a web browser. You can use any web server, such as apache or nginx, but for convenience the examples in this article series will assume you are running apache. It's best if you go through a tutorial series for installing apache or nginx so you understand what's being installed, but there are also barebones instructions for a default apache install in our repository if you're an experienced user and just want the web server for the purposes of accessing munin's reports.

If you want munin to send you email alerts you'll need to have a mail server running on your munin master slice that is configured to send outgoing mail messages. Slicehost has several articles that go into detail on setting up a mail server, and as with the web server, it's best if you go through a tutorial series so you get a good explanation of how to set up the mail server. If you want a quick, minimal mail server install, however, there are barebones install articles available there as well.

Munin can monitor just a single slice or it can be used to monitor several slices from one "master" slice. If you are planning on monitoring additional slices later, be sure to perform this installation on the slice you want to use as munin's master. If you are monitoring only one slice you don't have to make that decision, of course — just follow the directions in this series and don't worry about the subsequent article on installing additional nodes.

All commands assume you're running as a non-root user with sudo access.

Distribution links

To access the initial munin install article that corresponds to the Linux distribution running on your slice, click the appropriate link below:

Ubuntu: Installing munin on Ubuntu

Debian: Installing munin on Debian

CentOS: Installing munin on CentOS

Fedora: Installing munin on Fedora

Red Hat Enterprise Linux: Installing munin on RHEL

Munin configuration and testing on RHEL

2010-03-12T19:00:00Z

This article continues the installation and setup of munin on a single slice. It explains how to determine or change the URL used to access munin's reports and then check to make sure those reports are viewable and being updated.

Almost ready!

If you've been following along with the first article in this series, Installing munin, you should now have a munin master configured on your slice and have a node running and gathering data every five minutes. All that's left now is to see if you can access the reports munin generates. To do that you'll need to make sure munin is storing its generated html files in a location you can access through your web server.

The htmldir setting

If you go back to the configuration file at /etc/munin/munin.conf, toward the beginning you will find an entry that looks like:

# The next three variables specifies where the location of the RRD
# databases, the HTML output, logs and the lock/pid files.  They all
# must be writable by the user running munin-cron.  They are all
# defaulted to the values you see here.
#
# dbdir /var/lib/munin
# htmldir /var/www/html/munin
# logdir /var/log/munin
# rundir  /var/run/munin

The line you want to look at is the "htmldir" value. This is the directory where munin stores its web pages, and the default value is a subdirectory of the web server's default document root (more on that later). If you would prefer that munin store its pages in another location (as a subdirectory of an existing site, for instance) you can uncomment that line (remove the leading "#" character) and change its value to a new directory. Just make sure that you set the new directory to be writeable by the munin user or munin won't be able to generate graphs and reports there.

If you aren't sure if the munin HTML directory can be written to by the munin user, the easiest fix is to set the ownership of the directory (which is done for you in a default install). To set the ownership of the directory to the munin user, run the following command:

chown -R munin /var/www/html/munin

If you aren't using the default munin directory, make sure to substitute the new directory for "/var/www/html/munin" in the example above.

Determining the munin URL

The URL used to access munin's reports is a combination of the address you use to access your web server and where the munin HTML directory is relative to the web server's document root. Let's take a look at how you can check these details in a default apache install so you can find and change these settings when you need to.

A closer look at the default site

When apache is installed it puts it default configuration file here:

/etc/httpd/conf/httpd.conf

The default configuration sets up a basic site that can be used to serve some static files, like those munin generates. If you want to serve more than one site from apache you can refer back to our articles on apache for information that would allow you to do that with a single apache installation using a "virtual host". For now we're interested in how the default site is set up, so we'll look in that default configuration file for more details. Open it and go looking for some key lines.

The first section we'll look at sets the address and port apache will use to listen for connections:

# Listen: Allows you to bind Apache to specific IP addresses and/or
# ports, in addition to the default. See also the <VirtualHost>
# directive.
#
# Change this to Listen on specific IP addresses as shown below to 
# prevent Apache from glomming onto all bound IP addresses (0.0.0.0)
#
#Listen 12.34.56.78:80
Listen 80

By default the "Listen 80" directive tells apache to listen to all available interfaces (the IP addresses on your slice), on port 80 (the default port for unencrypted web traffic). In other words, with the default configuration you can get to the site with any address that points to your slice, be it a domain name or an IP address.

The next section we want to look at is:

#
# DocumentRoot: The directory out of which you will serve your
# documents. By default, all requests are taken from this directory, but
# symbolic links and aliases may be used to point to other locations.
#
DocumentRoot "/var/www/html"

This DocumentRoot directive tells apache where the web files are stored. That default setting means that if you visit your slice in a web browser apache will look in /var/www/html to find any pages to show you. If you have "www.example.com" pointing to your slice you can see the default index file in the web server's document root by going to this URL:

http://www.example.com

If munin's web directory is not located in the site's document root, either because you've changed apache's configuration or because you don't want the munin files to be in the document root, you have a couple options. One option is to change the munin.conf file to use a directory in a different document root (using the instructions above under the heading, "The htmldir in munin.conf"). There is another option that doesn't involve changing where munin stores its reports, however, and that is...

The Alias directive

The Alias directive in apache lets you set up an alias from a reference to a subdirectory of the document root to a location somewhere else in your slice's filesystem. For example, if your document root were "/var/www/html" but you wanted your munin HTML files to be in "/home/munin/public_html", you would set up an alias from "/munin" to "/home/munin/public_html".

The Alias directive depends on the module "mod_alias" being enabled. This mod is included and enabled in the default apache installation.

To add an alias like the example above you would want to add the alias to your site configuration in /etc/httpd/conf/httpd.conf, and would need to include a Directory entry for the directory the alias points to. The new section can be inserted at the end of the file. Your configuration might look like:

Alias /munin /home/munin/public_html
<Directory /home/munin/public_html>
    Options FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all
</Directory>

Once the Alias and Directory configuration is added to your site, you'll need to restart apache for the change to take effect.

Wasn't there something about a URL?

Now that you know where munin (or an Alias to it) is in relation to your web server's document root, finding the address of the munin reports is a matter of putting together the URL of your site and the location of munin's web pages within that site.

When you go to your base URL, like, "http://www.example.com", you're telling the web server you want to see the content in your site's document root, let's say "/var/www/html". When you go to your site with that address there's actually an implied slash at the end of the URL. Let's add that slash to make it easier to see the relationship between the URL and the document root:

http://www.example.com/

You can think of that last slash in your base URL as meaning "look in my document root". Since we're assuming /var/www for this example, that means "/" means "/var/www/html/". Now look at where the munin web directory is located in our example, "/var/www/html/munin". If you substitute a slash for the /var/www/html/ part, you get simply "/munin". Now we can put the base URL together with the relative munin location that you just worked out, and get:

http://www.example.com/munin

And there you have the address you visit in a browser to look at munin's reports. Most importantly, you have an idea of how we came up with it. You can change settings on your web server or in munin later and know how those changes could affect the address you use to see munin's web pages.

See the results

Munin collects data and generates new graphs every five minutes. If it hasn't been five minutes since you configured munin you may need to wait a little bit before there are web pages generated in the munin HTML directory. Also expect your initial graphs to be pretty sparse since there won't be much data for munin to use when drawing lines.

Point your web browser to the URL you determined above and let's see what comes up. Hopefully you'll see the host name you put into the munin.conf file, similar to:

If you get a "not found" error you may need to review this article to see if you missed a settings change or need to make the munin web directory writeable by the munin user. You might also check the web server logs in /var/log/apache2 and the munin logs in /var/log/munin for clues.

Clicking the first name in the list on the munin overview page (in the screenshot, the first "jered") will show a list of graph categories:

Once you've confirmed that the pages are being created and can be reached, wait a half hour or so to let munin accumulate enough data to draw the beginnings of a few graphs. If you click the second hostname entry on the overview page you'll get a page with all generated graphs on it. The following shots are from a munin instance that's been running a few days, to make the graphs easier to spot:

Pretty boring stuff at the top on the example server, since it's been mostly idle. You can see more active graphs if you scroll down to the system section:

From here it's just a matter of checking in on munin's reports every now and then to watch the trends and to follow up on any emails munin sends you. Keep an eye out for things like swap in/out (ideally there should be almost none), regularly high non-idle cpu use, and spikes in bandwidth use ("eth0 traffic").

For more information on the data munin reports check the documentation and FAQ at the munin project's web site.

Summary

In this article we looked at configuring the munin HTML directory and the web server's document root, and how to put that information together to get the web address for viewing the munin reports. We also viewed the munin reports to make sure they were accessible and updating.

Future articles in this series will cover installing nodes on additional slices (so you can monitor multiple slices from one munin master) and customizing munin by adding more plug-ins or improving security on the data it's generating.

If you're content with the default plug-ins and a single node, you need go no further. You're all set!

-- Jered

Installing munin on RHEL

2010-03-12T18:58:00Z

What you'll need

All commands assume you're running as a non-root user with sudo access.

Installing packages

On CentOS and Red Hat Enterprise Linux munin is not in the default repository. The easiest way to get munin is to add a repository to your installation that includes it. One such repository is the EPEL repository maintained by the Fedora Core team that contains munin and some other useful enterprise software. Once that repository is added you can install munin from there and include it in future system-wide updates.

To add the EPEL repository to your yum list, run:

sudo rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm

With the repository in place, you should be able to install munin with:

sudo yum install munin munin-node

After checking for dependencies yum will ask if you would like to import the GPG key from the EPEL repository. Say yes so yum can verify the authenticity of RPMs retrieved from that site in the future.

The munin master and node should now be installed to your slice, and your system configured to run munin. The changes made to your slice include a "munin" user to own munin's files, directories for munin's data archive and web pages, a cron.d entry to have munin generate graphs every five minutes, and an init script for munin-node.

With that, most of the basic work is done for you. Now you need to customize a couple settings and make one adjustment to improve security.

The munin.conf file

Open the file /etc/munin/munin.conf so you can change a couple important settings. This file will be covered in more depth in a later article on customizing munin.

Email notification (optional)

It's possible for munin to use the local mail command to send some basic email alerts if you have a mail server installed on the master server. The munin.conf file provides an example that sends an email whenever a plugin changes status (from OK to WARNING, for example, or when a warning situation is resolved):

# Drop somejuser@fnord.comm and anotheruser@blibb.comm an email everytime 
# something changes (OK -> WARNING, CRITICAL -> OK, etc)
#contact.someuser.command mail -s "Munin notification" somejuser@fnord.comm
#contact.anotheruser.command mail -s "Munin notification" anotheruser@blibb.comm

You can use that example as a template to add your own email notifications. For instance, you could add the following lines to the munin.conf file:

contact.john.command mail -s "Munin notification" john@example.com
contact.marcia.command mail -s "Munin notification" marcia@demoslice.com

The host tree

The "host tree" section of munin.conf describes the organization of any monitored nodes on munin's overview page. This guide only covers setting up one node on the same server as the munin master, so you can leave the default address of 127.0.0.1 alone. You might want to change the host tree name to something more descriptive, however. Find the following in the munin.conf file:

# a simple host tree
[localhost]
    address 127.0.0.1
    use_node_name yes

You can change the "localhost" entry to reflect your slice name, especially if you will be adding other nodes to your munin reports later. Don't use any spaces in the name, that confuses some of munin's graphing scripts. So you might change the host tree to look like:

[slicename]
    address 127.0.0.1
    use_node_name yes

That should do it for munin.conf.

The munin-node.conf file

The security of the default configuration can be improved by making the node (the part that actually compiles statistics) completely local so it can't accept any outside connections. There's already a directive in this file telling the munin node to reject connections from outside addresses, but it's more secure to ensure the node can't be reached by external connections to begin with. Open the /etc/munin/munin-node.conf file and look for an entry with "host *", similar to:

# Which address to bind to;
host *

To restrict the node to listen to localhost only, you should change the host entry to:

# Which address to bind to;
host 127.0.0.1

The munin-node service

The munin node wasn't started by the installer, so now that we're done binding it to localhost let's start that up:

sudo /etc/init.d/munin-node start

To make sure the munin-node service starts when your slice reboots, you'll also want to make the service active:

sudo /sbin/chkconfig munin-node on

Summary

In this article you set up munin as a master and a node on your slice and configured it to display some default reports via a web server.

The next article in this series will cover determining the URL you will use to access the munin reports and checking that the reports are being updated properly.

-- Jered

Munin configuration and testing on Fedora

2010-03-12T18:55:00Z

Almost ready!

The htmldir setting

If you go back to the configuration file at /etc/munin/munin.conf, toward the beginning you will find an entry that looks like:

# The next three variables specifies where the location of the RRD
# databases, the HTML output, and the logs, severally.  They all
# must be writable by the user running munin-cron.
dbdir   /var/lib/munin
htmldir /var/www/html/munin
logdir  /var/log/munin
rundir  /var/run/munin

The line you want to look at is the "htmldir" value. This is the directory where munin stores its web pages, and the default value is a subdirectory of the web server's default document root (more on that later). If you would prefer that munin store its pages in another location (as a subdirectory of an existing site, for instance) you can change that line to a new directory. Just make sure that you set the new directory to be writeable by the munin user or munin won't be able to generate graphs and reports there.

chown -R munin /var/www/html/munin

If you aren't using the default munin directory, make sure to substitute the new directory for "/var/www/munin" in the example above.

Determining the munin URL

A closer look at the default site

When apache is installed it puts it default configuration file here:

/etc/httpd/conf/httpd.conf

The first section we'll look at sets the address and port apache will use to listen for connections:

# Listen: Allows you to bind Apache to specific IP addresses and/or
# ports, in addition to the default. See also the <VirtualHost>
# directive.
#
# Change this to Listen on specific IP addresses as shown below to 
# prevent Apache from glomming onto all bound IP addresses (0.0.0.0)
#
#Listen 12.34.56.78:80
Listen 80

The next section we want to look at is:

#
# DocumentRoot: The directory out of which you will serve your
# documents. By default, all requests are taken from this directory, but
# symbolic links and aliases may be used to point to other locations.
#
DocumentRoot "/var/www/html"

http://www.example.com

The Alias directive

The Alias directive depends on the module "mod_alias" being enabled. This mod is included and enabled in the default apache installation.

Alias /munin /home/munin/public_html
<Directory /home/munin/public_html>
    Options FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all
</Directory>

Once the Alias and Directory configuration is added to your site, you'll need to restart apache for the change to take effect.

Wasn't there something about a URL?

http://www.example.com/

http://www.example.com/munin

See the results

Point your web browser to the URL you determined above and let's see what comes up. Hopefully you'll see the host name you put into the munin.conf file, similar to:

Clicking the first name in the list on the munin overview page (in the screenshot, the first "jered") will show a list of graph categories:

Pretty boring stuff at the top on the example server, since it's been mostly idle. You can see more active graphs if you scroll down to the system section:

For more information on the data munin reports check the documentation and FAQ at the munin project's web site.

Summary

If you're content with the default plug-ins and a single node, you need go no further. You're all set!

-- Jered

Installing munin on Fedora

2010-03-12T18:52:00Z

What you'll need

All commands assume you're running as a non-root user with sudo access.

Installing packages

To download and install the munin master package and munin node package to your master server, run the following command:

sudo yum install munin munin-node

This command installs the munin master and node to your slice, also configuring your system to run munin. The changes include creating a "munin" user to own munin's files, creating directories for munin's data archive and web pages, adding a cron.d entry to have munin generate graphs every five minutes, and an init script for munin-node.

With that, most of the basic work is done for you. Now you need to customize a couple settings and make one adjustment to improve security.

The munin.conf file

Open the file /etc/munin/munin.conf so you can change a couple important settings. This file will be covered in more depth in a later article on customizing munin.

Email notification (optional)

# Drop somejuser@fnord.comm and anotheruser@blibb.comm an email everytime 
# something changes (OK -> WARNING, CRITICAL -> OK, etc)
#contact.someuser.command mail -s "Munin notification" somejuser@fnord.comm
#contact.anotheruser.command mail -s "Munin notification" anotheruser@blibb.comm

You can use that example as a template to add your own email notifications. For instance, you could add the following lines to the munin.conf file:

contact.john.command mail -s "Munin notification" john@example.com
contact.marcia.command mail -s "Munin notification" marcia@demoslice.com

The host tree

# a simple host tree
[localhost]
    address 127.0.0.1
    use_node_name yes

[slicename]
    address 127.0.0.1
    use_node_name yes

That should do it for munin.conf.

The munin-node.conf file

# Which address to bind to;
host *

To restrict the node to listen to localhost only, you should change the host entry to:

# Which address to bind to;
host 127.0.0.1

The munin-node service

The munin node wasn't started by the installer, so now that we're done binding it to localhost let's start that up:

sudo /etc/init.d/munin-node start

To make sure the munin-node service starts when your slice reboots, you'll also want to make the service active:

sudo /sbin/chkconfig munin-node on

Summary

In this article you set up munin as a master and a node on your slice and configured it to display some default reports via a web server.

The next article in this series will cover determining the URL you will use to access the munin reports and checking that the reports are being updated properly.

-- Jered

Munin configuration and testing on Debian

2010-03-12T18:50:00Z

Almost ready!

The htmldir setting

If you go back to the configuration file at /etc/munin/munin.conf, toward the beginning you will find an entry that looks like:

# The next three variables specifies where the location of the RRD
# databases, the HTML output, and the logs, severally.  They all
# must be writable by the user running munin-cron.
dbdir   /var/lib/munin
htmldir /var/www/munin
logdir  /var/log/munin
rundir  /var/run/munin

The line you want to look at is the "htmldir" value. This is the directory where munin stores its web pages, and the default value is a subdirectory of the web server's default document root (more on that later). If you would prefer that munin store its pages in another location (as a subdirectory of an existing site, for instance) you can change that line to a new directory. Just make sure that you set the new directory to be writeable by the munin user or munin won't be able to generate graphs and reports there.

chown -R munin /var/www/munin

If you aren't using the default munin directory, make sure to substitute the new directory for "/var/www/munin" in the example above.

Determining the munin URL

A closer look at the default site

When apache is installed it sets up two directories to manage the sites that apache will serve to visitors. One directory, /etc/apache2/sites-available, contains a file for each site. By default it contains just one file, appropriately named "default". The other directory is /etc/apache2/sites-enabled, which contains links to the configuration files for sites apache should actually allow people to visit. If you haven't changed apache's configuration there will be a symlink from sites-enabled to the default file in sites-available, letting apache know to serve that site.

If you want to enable or disable another virtual host, revisit the article you used to set up apache for more details.

For now we're interested in how the site is set up, so we'll look at the default site to find those details. Open the file:

/etc/apache2/sites-available/default

Let's look at the first part of that file, which by default would be:

NameVirtualHost *
<VirtualHost *>
    ServerAdmin webmaster@localhost

    DocumentRoot /var/www/
    <Directory />
        Options FollowSymLinks
        AllowOverride None
    </Directory>
    <Directory /var/www/>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride None
        Order allow,deny
        allow from all
    </Directory>
...

The first lines, "NameVirtualHost *" and "<virtualhost>", basically tell apache that this default site configuration will handle every request that comes into the web server. If you've already set up another site configuration those lines will look different, since they're generally changed when someone wants a more specific site set-up (for a specific domain or for multiple domains). For the purposes of setting up munin, you mostly just need to know that if those lines change in the future you may need to make allowances to continue to access munin's html directory.

The next directive we want to check is a couple lines further down, "DocumentRoot /var/www/". That option tells apache where the web files are stored for this virtual host. That default setting means that if you visit this virtual host apache will look in /var/www to find any pages to show the user. If you have "www.example.com" pointing to your slice, for example, you could see the default index file in the virtual host's document root by going to this URL:

http://www.example.com

The Alias directive

The Alias directive in apache lets you set up an alias from a reference to a subdirectory of the document root to a location somewhere else in your slice's filesystem. For example, if your document root were "/var/www" but you wanted your munin HTML files to be in "/home/munin/public_html", you would set up an alias from "/munin" to "/home/munin/public_html".

The Alias directive depends on the module "mod_alias" being enabled. This mod is included and enabled in the default apache installation.

To add an alias like the example above you would want to add the alias to your site configuration in /etc/apache2/sites-available, and would need to include a Directory entry for the directory the alias points to. The new section can be inserted near the end of the file, so long as it comes before the "</virtualhost> entry. Your configuration might look like:

Alias /munin /home/munin/public_html
<Directory /home/munin/public_html>
    Options FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all
</Directory>

Once the Alias and Directory configuration is added to your site, you'll need to restart apache for the change to take effect.

Wasn't there something about a URL?

When you go to your base URL, like, "http://www.example.com", you're telling the web server you want to see the content in your site's document root, let's say "/var/www". When you go to your site with that address there's actually an implied slash at the end of the URL. Let's add that slash to make it easier to see the relationship between the URL and the document root:

http://www.example.com/

You can think of that last slash in your base URL as meaning "look in my document root". Since we're assuming /var/www for this example, that means "/" means "/var/www/". Now look at where the munin web directory is located in our example, "/var/www/munin". If you substitute a slash for the /var/www/ part, you get simply "/munin". Now we can put the base URL together with the relative munin location that you just worked out, and get:

http://www.example.com/munin

See the results

Point your web browser to the URL you determined above and let's see what comes up. Hopefully you'll see the host name you put into the munin.conf file, similar to:

Clicking the first name in the list on the munin overview page (in the screenshot, the first "jered") will show a list of graph categories:

Pretty boring stuff at the top on the example server, since it's been mostly idle. You can see more active graphs if you scroll down to the system section:

For more information on the data munin reports check the documentation and FAQ at the munin project's web site.

Summary

If you're content with the default plug-ins and a single node, you need go no further. You're all set!

-- Jered

Installing munin on Debian

2010-03-12T18:47:00Z

What you'll need

All commands assume you're running as a non-root user with sudo access.

Installing packages

To download and install the munin master package and munin node package to your master server, run the following two commands:

sudo aptitude update

sudo aptitude install munin munin-node

The first command updates the aptitude database to ensure you install the latest packages for your distribution. The second command installs the munin master and node to your slice, also configuring your system to run munin. The changes include creating a "munin" user to own munin's files, creating directories for munin's data archive and web pages, adding a cron.d entry to have munin generate graphs every five minutes, and an init script for munin-node.

With that, most of the basic work is done for you. Now you need to customize a couple settings and make one adjustment to improve security.

The munin.conf file

Open the file /etc/munin/munin.conf so you can change a couple important settings. This file will be covered in more depth in a later article on customizing munin.

Email notification (optional)

# Drop somejuser@fnord.comm and anotheruser@blibb.comm an email everytime 
# something changes (OK -> WARNING, CRITICAL -> OK, etc)
#contact.someuser.command mail -s "Munin notification" somejuser@fnord.comm
#contact.anotheruser.command mail -s "Munin notification" anotheruser@blibb.comm

You can use that example as a template to add your own email notifications. For instance, you could add the following lines to the munin.conf file:

contact.john.command mail -s "Munin notification" john@example.com
contact.marcia.command mail -s "Munin notification" marcia@demoslice.com

The host tree

# a simple host tree
[localhost.localdomain]
    address 127.0.0.1
    use_node_name yes

You can change the "localhost.localdomain" entry to reflect your slice name, especially if you will be adding other nodes to your munin reports later. Don't use any spaces in the name, that confuses some of munin's graphing scripts. You might change the host tree to look like:

[slicename]
    address 127.0.0.1
    use_node_name yes

That should do it for munin.conf.

The munin-node.conf file

# Which address to bind to;
host *

To restrict the node to listen to localhost only, you should change the host entry to:

# Which address to bind to;
host 127.0.0.1

The munin-node service

You'll need to restart the munin node to implement the change binding it to localhost:

sudo /etc/init.d/munin-node restart

Summary

In this article you set up munin as a master and a node on your slice and configured it to display some default reports via a web server.

The next article in this series will cover determining the URL you will use to access the munin reports and checking that the reports are being updated properly.

-- Jered

Munin configuration and testing on CentOS

2010-03-12T18:42:00Z

Almost ready!

The htmldir setting

If you go back to the configuration file at /etc/munin/munin.conf, toward the beginning you will find an entry that looks like:

# The next three variables specifies where the location of the RRD
# databases, the HTML output, logs and the lock/pid files.  They all
# must be writable by the user running munin-cron.  They are all
# defaulted to the values you see here.
#
# dbdir /var/lib/munin
# htmldir /var/www/html/munin
# logdir /var/log/munin
# rundir  /var/run/munin

chown -R munin /var/www/html/munin

If you aren't using the default munin directory, make sure to substitute the new directory for "/var/www/html/munin" in the example above.

Determining the munin URL

A closer look at the default site

When apache is installed it puts it default configuration file here:

/etc/httpd/conf/httpd.conf

The first section we'll look at sets the address and port apache will use to listen for connections:

# Listen: Allows you to bind Apache to specific IP addresses and/or
# ports, in addition to the default. See also the <VirtualHost>
# directive.
#
# Change this to Listen on specific IP addresses as shown below to 
# prevent Apache from glomming onto all bound IP addresses (0.0.0.0)
#
#Listen 12.34.56.78:80
Listen 80

The next section we want to look at is:

#
# DocumentRoot: The directory out of which you will serve your
# documents. By default, all requests are taken from this directory, but
# symbolic links and aliases may be used to point to other locations.
#
DocumentRoot "/var/www/html"

http://www.example.com

The Alias directive

The Alias directive depends on the module "mod_alias" being enabled. This mod is included and enabled in the default apache installation.

Alias /munin /home/munin/public_html
<Directory /home/munin/public_html>
    Options FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all
</Directory>

Once the Alias and Directory configuration is added to your site, you'll need to restart apache for the change to take effect.

Wasn't there something about a URL?

http://www.example.com/

http://www.example.com/munin

See the results

Point your web browser to the URL you determined above and let's see what comes up. Hopefully you'll see the host name you put into the munin.conf file, similar to:

Clicking the first name in the list on the munin overview page (in the screenshot, the first "jered") will show a list of graph categories:

Pretty boring stuff at the top on the example server, since it's been mostly idle. You can see more active graphs if you scroll down to the system section:

For more information on the data munin reports check the documentation and FAQ at the munin project's web site.

Summary

If you're content with the default plug-ins and a single node, you need go no further. You're all set!

-- Jered

Installing munin on CentOS

2010-03-12T18:39:00Z

What you'll need

All commands assume you're running as a non-root user with sudo access.

Installing packages

To add the EPEL repository to your yum list, run:

sudo rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm

With the repository in place, you should be able to install munin with:

sudo yum install munin munin-node

With that, most of the basic work is done for you. Now you need to customize a couple settings and make one adjustment to improve security.

The munin.conf file

Open the file /etc/munin/munin.conf so you can change a couple important settings. This file will be covered in more depth in a later article on customizing munin.

Email notification (optional)

# Drop somejuser@fnord.comm and anotheruser@blibb.comm an email everytime 
# something changes (OK -> WARNING, CRITICAL -> OK, etc)
#contact.someuser.command mail -s "Munin notification" somejuser@fnord.comm
#contact.anotheruser.command mail -s "Munin notification" anotheruser@blibb.comm

You can use that example as a template to add your own email notifications. For instance, you could add the following lines to the munin.conf file:

contact.john.command mail -s "Munin notification" john@example.com
contact.marcia.command mail -s "Munin notification" marcia@demoslice.com

The host tree

# a simple host tree
[localhost]
    address 127.0.0.1
    use_node_name yes

[slicename]
    address 127.0.0.1
    use_node_name yes

That should do it for munin.conf.

The munin-node.conf file

# Which address to bind to;
host *

To restrict the node to listen to localhost only, you should change the host entry to:

# Which address to bind to;
host 127.0.0.1

The munin-node service

The munin node wasn't started by the installer, so now that we're done binding it to localhost let's start that up:

sudo /etc/init.d/munin-node start

To make sure the munin-node service starts when your slice reboots, you'll also want to make the service active:

sudo /sbin/chkconfig munin-node on

Summary

In this article you set up munin as a master and a node on your slice and configured it to display some default reports via a web server.

The next article in this series will cover determining the URL you will use to access the munin reports and checking that the reports are being updated properly.

-- Jered

Munin configuration and testing on Ubuntu

2010-03-12T18:32:00Z

Almost ready!

The htmldir setting

If you go back to the configuration file at /etc/munin/munin.conf, toward the beginning you will find an entry that looks like:

# The next three variables specifies where the location of the RRD
# databases, the HTML output, and the logs, severally.  They all
# must be writable by the user running munin-cron.
dbdir   /var/lib/munin
htmldir /var/www/munin
logdir  /var/log/munin
rundir  /var/run/munin

The line you want to look at is the "htmldir" value. This is the directory where munin stores its web pages, and the default value is a subdirectory of the web server's default document root (more on that later). If you would prefer that munin store its pages in another location (as a subdirectory of an existing site, for instance) you can change that line to a new directory. Just make sure that you set the new directory to be writeable by the munin user or munin won't be able to generate graphs and reports there.

chown -R munin /var/www/munin

If you aren't using the default munin directory, make sure to substitute the new directory for "/var/www/munin" in the example above.

Determining the munin URL

A closer look at the default site

If you want to enable or disable another virtual host, revisit the article you used to set up apache for more details.

For now we're interested in how the site is set up, so we'll look at the default site to find those details. Open the file:

/etc/apache2/sites-available/default

Let's look at the first part of that file, which by default would be:

NameVirtualHost *
<VirtualHost *>
    ServerAdmin webmaster@localhost

    DocumentRoot /var/www/
    <Directory />
        Options FollowSymLinks
        AllowOverride None
    </Directory>
    <Directory /var/www/>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride None
        Order allow,deny
        allow from all
    </Directory>
...

http://www.example.com

The Alias directive

The Alias directive in apache lets you set up an alias from a reference to a subdirectory of the document root to a location somewhere else in your slice's filesystem. For example, if your document root were "/var/www" but you wanted your munin HTML files to be in "/home/munin/public_html", you would set up an alias from "/munin" to "/home/munin/public_html".

The Alias directive depends on the module "mod_alias" being enabled. This mod is included and enabled in the default apache installation.

Alias /munin /home/munin/public_html
<Directory /home/munin/public_html>
    Options FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all
</Directory>

Once the Alias and Directory configuration is added to your site, you'll need to restart apache for the change to take effect.

Wasn't there something about a URL?

http://www.example.com/

http://www.example.com/munin

See the results

Point your web browser to the URL you determined above and let's see what comes up. Hopefully you'll see the host name you put into the munin.conf file, similar to:

Clicking the first name in the list on the munin overview page (in the screenshot, the first "jered") will show a list of graph categories:

Pretty boring stuff at the top on the example server, since it's been mostly idle. You can see more active graphs if you scroll down to the system section:

For more information on the data munin reports check the documentation and FAQ at the munin project's web site.

Summary

If you're content with the default plug-ins and a single node, you need go no further. You're all set!

-- Jered

Installing munin on Ubuntu

2010-03-12T18:09:00Z

What you'll need

All commands assume you're running as a non-root user with sudo access.

Installing packages

To download and install the munin master package and munin node package to your master server, run the following two commands:

sudo aptitude update

sudo aptitude install munin munin-node

With that, most of the basic work is done for you. Now you need to customize a couple settings and make one adjustment to improve security.

The munin.conf file

Open the file /etc/munin/munin.conf so you can change a couple important settings. This file will be covered in more depth in a later article on customizing munin.

Email notification (optional)

# Drop somejuser@fnord.comm and anotheruser@blibb.comm an email everytime 
# something changes (OK -> WARNING, CRITICAL -> OK, etc)
#contact.someuser.command mail -s "Munin notification" somejuser@fnord.comm
#contact.anotheruser.command mail -s "Munin notification" anotheruser@blibb.comm

You can use that example as a template to add your own email notifications. For instance, you could add the following lines to the munin.conf file:

contact.john.command mail -s "Munin notification" john@example.com
contact.marcia.command mail -s "Munin notification" marcia@demoslice.com

The host tree

# a simple host tree
[localhost.localdomain]
    address 127.0.0.1
    use_node_name yes

[slicename]
    address 127.0.0.1
    use_node_name yes

That should do it for munin.conf.

The munin-node.conf file

# Which address to bind to;
host *

To restrict the node to listen to localhost only, you should change the host entry to:

# Which address to bind to;
host 127.0.0.1

The munin-node service

You'll need to restart the munin node to implement the change binding it to localhost:

sudo /etc/init.d/munin-node restart

Summary

In this article you set up munin as a master and a node on your slice and configured it to display some default reports via a web server.

The next article in this series will cover determining the URL you will use to access the munin reports and checking that the reports are being updated properly.

-- Jered

Speed up resizes - Part 2

2010-03-09T18:23:00Z

In the first part of this how-to we looked at how resize time is extended when a slice hosts many small files or many files that are being updated during the resize.

In this second part we look at the effect on resize time of large, constantly-updated files and how to mitigate it.

Slices with huge, constantly-updated files

Very large files that are being updated during a resize pose similar problems for resize downtime as smaller, constantly-updated files — but on steroids. If your slice hosts files that are frequently written to and which are larger than 10Gb, this section is for you.

These large, constantly updating files will need to be completely re-copied during the reboot phase to capture any updates made since the resize began. This will extend 'resize downtime' considerably due to the size of that second copy, which must complete before the resized slice can be brought back online.

MySQL database servers that use the InnoDB data-file format write data to a single file that can — and will — grow very large indeed. Similarly, MySQL in InnoDB mode usually logs to a single very large log file.

An update to a large InnoDB MySQL data or log file (/var/lib/mysql/ibdata1, etc) forces the resize process to re-copy the entire file in the final copy phase. If these files are large then the re-copy can take some time, which keeps the database offline.

Another source of large files are application logs, especially logs produced by mail servers and some web servers. Apache can easily produce log files that are 16Gb or larger so it's not safe to assume that Apache's default logging will help you avoid this large-file issue.

MySQL transaction logs can also grow large if you have turned on transaction logging. It's rare that people do, and it's even rarer that they turn on transaction logging without running out of disk space shortly afterwards! Still, it's wise to keep an eye out for this possibility.

How to mitigate

As described above, pruning databases of cruft may help reduce the total copy time. Archive and delete old or obsolete databases before resizing.

Again, turning off database writes, if possible, will reduce 'resize downtime'. Turning off logging may also help in the case of InnoDB databases.

If you have turned on MySQL transaction logging and the transaction log file is large, it's worth turning it off and archiving then deleting it on the slice before starting a resize.

On mail servers, check the size of /var/log/mail.log or /var/log/maillog before resizing. Consider turning the mail server off before resizing (you have a secondary 'backup' mail server to pick up the load, right?).

Similarly, check how Apache is logging. If it is logging all requests to one file, check the size of that file and consider archiving and deleting it or turning Apache off prior to starting the resize. The same advice applies to any other application that you know is writing to a large log file.

For the above applications and any others, review your logrotate policy (if you have one) to check that it is keeping a check on your log file sizes. This will save you downtime during resizes and and make life with any Linux server easier.

Pack your toolbag

Of course, it is difficult to track what files have been created after a slice has been set up. That is especially true for applications that create session files. It pays to find — and cull — these large collections of little files.

You can identify the ten largest directories and files quite easily by issuing this command as root:

du -a / | sort -n -r | head -n 10

Change that final '10' to any other number to alter how many files and directories the search returns. This command is a good middle-ground tool for identifying large directories of small files and large files.

If you only want to look for large files, try this large file finder (as root):

find ~/ -mount -type f -ls|sort -rnk7 |head -30|awk '{printf "%10d MB\t%s\n",($7/1024)/1024,$NF}'

And if you want to find large directories only, try this large directory finder (as root):

du -x --max-depth=4 ~/|sort -rn|head -n30|awk '{printf "%10d MB\t%s\n",$1/1024,$2}'

Technical details

If your slice does not match any of the common types we have examined above, you may be able to assess how its resize time might look if you consider your applications with an understanding of how the resize process works.

The first stage of a resize is a live copy of the slice's entire file system. All applications are left running during this stage.

Here's where predicting resize time runs into its first uncertainty. Without detailed knowledge of your usage of your slice's file-system, the resize function cannot accurately predict how long the 'file copying' stage of a resize will take to complete.

This unpredictability is particularly true for the final directory on Linux file systems: the /var/ directory. It's called 'var' because the size of the data it holds is 'variable' in size and likely to change while the slice's applications are running. The resize process copies this /var/ directory last, which is why the resize progress counter sometimes sits at '98%' or '99% complete' for what may feel like an uncomfortably long time.

The second uncertainty is that the final phase of a resize includes a reboot (downtime) component during which any files updated since the resize preparation began are copied again. The length of the downtime depends on the size and number of updated files. Again, the resize process cannot tell in advance how many updates applications like MySQL are writing to data files so it cannot predict how long this final 'update' copy will take.

Summary

If you know how your applications are using diskspace and writing to files you may be able to judge when a resize will take longer than you would like and make preparations accordingly. At the very least, you should be able to use your new-found resize knowledge to better schedule resizes to fit your timing requirements.

Speed up resizes - Part 1

2010-03-09T18:21:00Z

This guide will help you shorten slice resize times, slice moves, and slice backup times. If your slice's applications match any of the characteristics highlighted in this how-to, then read on — this article will probably make resize times shorter and more predictable for you. We'll show you how cut resize times by taking preventative action before resizing, whether you are resizing up or resizing down.

And, as we like to give value for money, these tips will also speed things up for you if we ever have to move your slice or migrate it from failing hardware. As yet another benefit, backup times will be reduced.

Resize, backup and clone times are generally determined by the volume of used disk space on a slice. That is, the more data to copy, the longer the job takes. Typically an empty OS resize or clone takes 10-15 minutes to complete. However, certain slice 'personalities' dramatically extend resize time. They are:

1. Slices containing lots of small files, such as Ruby session files, cache files, mail server message files, or web server image thumbnails. These resizes take longer to complete than you might expect, but the reboot - and associated downtime - that completes the resize will usually be quite short.

2. Slices containing many files that are being updated during resize. Typically these are MySQL MyISAM table files or web servers hosting multiple domains, each configured to log to separate files. The need to update these actively-written files after the resize's first-pass copy tends to extend the reboot downtime that completes the resize process.

3. Slices containing one or more large files that are updated during resize. Examples are MySQL databases using InnoDB format, mail servers with large mail logs and webservers that log to single, large files. The need to update these actively-written files after the resize's first-pass copy greatly extends the reboot downtime that completes the resize process.

So, the secret to cutting resize and clone times is to manage data on your slice and to identify any applications that are writing to disk during the resize. Let's look at these factors more closely to see how we can mitigate each.

Beware! Small files overhead...

Although they don't take up much individual disk space, slices hosting many small files force the resize process to carry out many 'file-open, file-copy, file-check' processes. These occur during the first — or 'preparation' — stage of a resize, while the slice is still running. They don't affect the operation of the slice but they do extend the time required before the final 'reboot' phase of the resize, making it less predictable for you.

As an example, a one-gigabyte data file only requires one file-open, file-copy, file-check process. Contrast that with a one-gigabyte chunk of data spread across 10,000 individual files that requires 10,000 individual file-open, file-copy, and file-checks. That's a lot more system and network overhead.

Check if your applications are among those that are more likely to create many small files and see longer resize preparation times. They are:

* Web servers serving many small thumbnails or image files

* Caching servers that cache on disk with small files

* Email servers with large 'archives' of undeleted email

* Ruby/Rails servers - which tend to create lots of small session files and not delete them

* git repositories

* Custom application servers that create - but do not delete - session files for each visitor

While it may take some time to copy these small files, once they are copied the 'resize down-time' during reboot will usually be quite short.

To summarise, the 'small files problem' makes resize preparation time longer and therefore less predictable. And that makes the time of the final reboot phase harder to predict. However, the final reboot time - the 'resize downtime' - will usually remain quite short in these 'small files' cases.

How to mitigate

Check your slice applications against our list above. If your applications are on the list, then do what you can to prune unwanted small files before flicking the resize switch.

If you are running Ruby/Rails, assume the worst and search community forums for typical session and cache file locations, as well as how to identify which ones you can safely delete. Look into storing session date in MySQL with the command:

rake db:sessions:create

and truncating log files in log/ to zero bytes with the command:

rake log:clear

If you are running a caching server that caches to disk (as opposed to RAM), identify its file-storage directory and prune with vigour.

Check your filesystem for small session and cache files created by custom applications. Again, prune with vigour.

If resizing an email server that has a Mail Delivery Agent (MDA) such as Dovecot installed, have your email users clean their email archives of old cruft first.

Constantly-changing files

Files that change between the start of a resize and the final reboot stage of a resize have to be copied again from scratch during the reboot that completes a slice resize.

This certainly extends 'resize downtime' and therefore extends 'resize completion time'.

Database servers are the most frequent culprits that change large files of data between the start and end of a resize. These changes force the system to copy the entire database file again in the second 'update' part of the copy process that occurs in resizes. The slice is down during the second copy so time spent re-copying updated database files is time spent with the server offline.

Some combinations of database structure and type tend to exacerbate this kind of problem. For example, if you have a MySQL multi-table MyISAM database with many table files that are all updated within single SQL transactions, then many or all of the table files may need to be copied again during the second copy.

Given that database files can be many gigabytes in size, the implications of these updates for 'resize downtime' become clear. It also illustrates how difficult it is for the resize system to predict accurately how long a resize will take - after all, how can it predict what and how many SQL updates will occur between resize start and finish?

How to mitigate

If your database contains a lot of obsolete data it may pay to archive that data and then prune it from the live database before resizing. MySQL, for example, allows you to archive data using the mysqldump script, after which you can delete obsolete data in the live database. The large mysqldump output file containing the obsolete data will not extend 'resize downtime' because it is not being written to during resize.

Another option if you have applications writing to many files during resize is to set the application into read-only mode immediately before the resize. Databases can usually be set into temporary read-only mode. With other applications your mileage may vary.

You can also prevent writes to multiple files by turning applications off, but setting applications to read-only mode is usually the preferred option.

In the second part of this how-to we will look at the impact of slices with large constantly-updated files and how to mitigate their impact on resize times.

How to use Rescue Mode

2010-03-09T18:19:00Z

Rescue Mode grants you full access to a non-bootable slice's filesystem. You can use it to modify problem configuration files or to use scp to copy data from the slice to a remote location.

Rescue Mode gives you full access to a non-bootable slice's fileystem so that you can repair it. Typically, you will use it to remove startup scripts in /etc/init.d/, or to modify problem configuration files, or - at worst - to use scp to copy off data from the slice to a remote location.

Get the slice into Rescue Mode

To get a slice into Rescue Mode, log into Slice Manager at: https://manage.slicehost.com

Then click on the slice name (if you have more than one slice).

Then click on the 'Rescue' link which is at the top right.

This should display a page of text with two options. Read and understand the text and click on the top 'Rescue' button.

Watch the page to see the temporary root password that will be displayed at the top against a green background. Make a note of that password. You will be able to ssh in to the slice as the 'root' user using this temporary root password. The Rescue Mode IP will be the usual primary IP for the slice.

How to work in Rescue Mode

You can identify a slice that is in rescue mode by issuing the mount command. If /dev/sdb1 is mounted, it's in Rescue Mode.

You will be able to 'mount' the slice's normal '/' filesystem with the following command:

mount /dev/sda1 /mnt

Your slice files will now be visible in /mnt/.

Now you can carry out repairs. For example, if your slice has been booting straight into swapping mode and thus denying access via the usual ssh and Console methods, you might use Rescue Mode access to temporarily remove the MySQL start up script - bearing in mind that the startup script's location is now:

/mnt/etc/init.d/

Similarly, any configuration files you want to modify are in their normal location but with the /mnt/ directory prepended to the directory path.

You will also now be able to use 'scp' and 'sftp' to securely copy files on to and off of the slice.

You will be able to transfer files over the slice's normal public network interface, or over its normal private interface to another slice (if you have another slice).

Note: You can also use Slice Manager's Console mode to access the slice in Rescue Mode. If you do, you will also be able to see the slice booting into Rescue mode. This may help you understand the process a slice goes through to get into Rescue Mode.

How to get out of Rescue Mode

Once you are done, unmount the '/mnt' filesystem with:

umount /mnt

To exit Rescue Mode go to the Slice Manager and, once again, click on the 'Rescue' link at the top right. This should display a page of text with two options. Read and understand the text and click on the top 'Exit Rescue Mode' button.

And your slice should now be out of Rescue Mode!

Summary

Slicehost's Rescue Mode is quite easy to use once you understand what it does and how to access your root filesystem.

Further Information:

The original Slicehost announcement concerning the addition of Rescue Mode to the Slice Manager can make for good background.

For more on Rescue Mode, read page 11 of our Slice Administration Guide.

Using ServerDensity to monitor a slice

2010-03-04T20:03:00Z

You have a number of options for monitoring your slice. Commercial services like ServerDensity can be easier to set up and maintain than free monitoring applications.

Why ServerDensity?

Fixing problems with a slice can be good, but catching problems before they happen is even better. If you can monitor your slice's resource use over time you can identify some potential problems and compensate for them before they cause downtime.

A commercial monitoring service like ServerDensity can be easier to set up and maintain than a free monitoring package like munin or mrtg. While we encourage you to check all your options, we chose ServerDensity as our subject for an introduction to commercial monitoring. They provide free monitoring for a single slice (useful for many of our customers), and their paid options include access to some spiffy extras like an iPhone app for checking your graphs.

A site like ServerDensity is great for tracking performance over time, but if you simply want to set up alerts to track slice or program uptime it's better to use a service designed for that purpose (like Pingdom).

Prerequisites

To make sure your setup is supported by ServerDensity, check their system requirements page. The agent installation is the only part of this guide that varies according to the distribution running on your slice. The instructions here encompass the distributions we provide images for that are supported by ServerDensity: Ubuntu, Debian, CentOS, Red Hat Enterprise, Fedora, and Gentoo.

You will need to have a Python installation on your slice to run the ServerDensity agent. All of Slicehost's images for the ServerDensity-supported distributions have Python installed by default.

Getting an account

The first step is to get an account with ServerDensity. Direct your web browser to their sign-up page to begin the process.

Create an account name you will use to sign on to their site, and enter your email address. The initial password for your account will be emailed to that address. You will also need to create a host name that will be used to access your reports. If you enter "slicename" for the host, the address you would visit to view your statistics and make changes would be:

https://slicename.serverdensity.com

We recommend making the host address different from your account name for security purposes (no need to make it easy for someone to find out what your account name is, eh?). The host name you enter does not have to match your slice name of course, we're just using "slicename" as a convenient example.

Once the account is created, check your email for the initial password. When you log in you'll be able to add the name of the first slice you want to track, and then you'll get a page with instructions for setting up the agent on your slice. Don't close that page just yet — you'll need some of the information listed there.

Installing the agent

In order to gather data from your slice you need to install ServerDensity's "agent" program. This is a small Python program that runs in the background periodically to send data about the slice's resource use to ServerDensity. The agent consumes very few resources by itself, and sends information about your system and resource use only (it doesn't gather the contents of any of your data files, for instance).

Downoad the agent

To download the agent package from ServerDensity run the following command:

wget http://www.serverdensity.com/downloads/sd-agent.tar.gz

Install the agent files

Once you have the agent package downloaded, run the following commands to unpack the files and then install them to /usr/bin:

tar xvzf sd-agent.tar.gz
sudo mv sd-agent /usr/bin/sd-agent

Editing the configuration

Now that the files are in place you'll need to add information specific to your slice to the agent's configuration file. This is where those instructions come in handy.

If you closed that page despite me telling you not to, you can get back to it via the ServerDensity servers page. For example, if you used "slicename" as your host name, you can go to:

https://slicename.serverdensity.com/servers/

On that page you would click the "Agent" link next to your slice's entry to bring the instructions back up.

With the instructions page in front of you, scroll down until you find the entry for the "agent key". You'll want to copy that.

Next edit the agent's configuration file at:

/usr/bin/sd-agent/config.cfg

By default the contents of that file should look like:

[Main]
sd_url: http://example.serverdensity.com
agent_key: 

apache_status_url: http://www.example.com/server-status/?auto

mysql_server:
mysql_user:
mysql_pass:

nginx_status_url: http://www.example.com/nginx_status

report_anon_stats: yes

The two entries you will definitely want to change are "sd_url" and "agent_key".

For the sd_url entry, change "example" to the host name you entered for your slice. Note that if you prefer that your reported data be encrypted in transit, you can change the "http" at the beginning of the sd_url entry to "https". Using "slicename" as the host name, then, you could change that entry to:

sd_url: https://slicename.serverdensity.com

For the agent_key entry you want to paste in that agent key you found on the instructions page. Once you paste the key in, your entry would look something like:

agent_key: 5514ab444a2b2d61638eaf0904406cdb

The rest of the settings are pretty self-explanatory if you can use them, and unnecessary if you can't. Leave them alone for now (so we can test the agent with the most basic settings first). After a successful test you can check the ServerDensity documentation to make sure each service you run supports status reporting to ServerDensity.

The last line of the configuration file, the "report_anon_stats" setting, is not used yet by the ServerDensity agent. It will control whether or not the agent reports anonymous statistics like your Linux distribution and python version for use in further ServerDensity development. If you do not want the agent to report that information, set this entry to "no".

Adding iptables rules (optional)

If your slice was set up using one of Slicehost's setup articles you won't have outgoing traffic filtered by iptables by default, so you won't need to make any changes to iptables to allow the agent to communicate with ServerDensity.

If you have iptables filtering outgoing traffic you will need to open either port 80 or 443 for the agent's communication, depending on whether you set the sd_url entry to use "http" or "https".

Starting at boot time

To start the ServerDensity agent when your slice reboots you will need to first copy the init script for the agent to /etc/init.d on your system:

sudo cp /usr/bin/sd-agent/sd-agent.init /etc/init.d/sd-agent
sudo chmod 0755 /etc/init.d/sd-agent

The steps you take to enable the init script will depend on the Linux distribution running on your slice.

Ubuntu and Debian

On Ubuntu and Debian systems, run the command:

sudo /usr/sbin/update-rc.d sd-agent defaults

CentOS, Fedora, and RHEL

On RPM-based systems (CentOS, Fedora, and Red Hat Enterprise Linux), run the following commands to enable the init script:

sudo /sbin/chkconfig --add sd-agent
sudo /sbin/chkconfig sd-agent on

Gentoo

For Gentoo systems you enable the sd-agent init script with:

sudo /sbin/rc-update add sd-agent default

Starting the agent

To start the ServerDensity agent, run the command:

sudo /etc/init.d/sd-agent start

Once the agent has started, a process ID file should be created at:

/var/run/sd-agent.pid

If that file doesn't exist the agent did not start properly, and you may need to double-check the permissions on the init script or the settings in your configuration file.

Viewing the reports

Once the agent has started you'll need to wait about five minutes to make sure the agent collects some data and reports to ServerDensity. I know, waiting's not fun, but the agent only collects data periodically, and you'll want at least one collection in order to initialize the reporting screen. Use the time to plot a practical joke on your co-workers or to start a load of laundry, depending on whether you're setting this up from an office or from home. Or if you're in a friend's home, offer to do their laundry, it would be a nice way to pay them back for letting you use their Internet connection.

Now that a little time has passed, go to your dashboard using the address you set up when you created your account. If you used the hostname "slicename", that would be:

https://slicename.serverdensity.com/

You should see your slice on the list of servers, and be able to click it to see more information. That link would be something like:

https://slicename.serverdensity.com/servers/view/1/

There won't be much of a graph there yet, since it takes a while for the agent to report back enough data to make much of a line. If you don't see any reports at all you may need to recheck your agent configuration and make sure the agent is running.

Further information

Your account should be collecting data from the agent running on your slice now, and you'll eventually be able to see how your slice uses system resources like memory and CPU over time. Good work! Where do you go from here?

For starters, to get more details on using the ServerDensity service, as well as information on setting up status reports for apache, nginx, and mySQL, visit the ServerDensity documentation page. But some useful bits to keep in mind...

Trial account

It's important to bear in mind that your ServerDensity account starts as a trial of their paid service. If you want to continue using the service for a single slice, you'll need to downgrade your trial account to a free account before the trial period expires. If you decide you want to continue on the paid service (to get reporting for multiple slices, or to use their iPhone app, for example), you will need to upgrade your account before the end of the trial period.

Adding more slices

If you want to set up reporting for more than one slice you'll need to set up a copy of the ServerDensity agent on each slice. This means going through the installation and configuration in this guide for each slice - there isn't really a practical way to get around that.

Fortunately if you have several slices you can at least make the process of setting the slices up on your dashboard a little easier by using the "Add Cloud" option on ServerDensity's "Add Server" interface. You will need to add your Slicehost API key to your ServerDensity account (you can find it in the Slice Manager's account settings, after enabling API access), and then you'll be able to select from a list of your slices which ones you'll be adding to the monitoring service. Once they are created in the dashboard you can get agent keys for each from their "Agent" pages.

Updating the agent

You can check for agent updates and install them with the following commands:

cd /usr/bin/sd-agent
sudo python agent.py update
sudo python agent.py restart

Summary

Your ServerDensity account should be set up and recording data for your slice. Keep an eye on the graphs it generates and watch for trends like increased traffic or low memory conditions, then adjust your application configurations or slice size accordingly. You can also poke around in the ServerDensity configuration screens to set up email alerts and what data is reported in your dashboard.

-- Jered

Slicehost Articles - Home

Installing munin overview

Why munin?

Contents

What you'll need

Distribution links

Further reading

Munin configuration and testing on RHEL

Almost ready!

The htmldir setting

Determining the munin URL

A closer look at the default site

The Alias directive

Wasn't there something about a URL?

See the results

Summary

Installing munin on RHEL

What you'll need

Installing packages

The munin.conf file

Email notification (optional)

The host tree

The munin-node.conf file

The munin-node service

Summary

Munin configuration and testing on Fedora

Almost ready!

The htmldir setting

Determining the munin URL

A closer look at the default site

The Alias directive

Wasn't there something about a URL?

See the results

Summary

Installing munin on Fedora

What you'll need

Installing packages

The munin.conf file

Email notification (optional)

The host tree

The munin-node.conf file

The munin-node service

Summary

Munin configuration and testing on Debian

Almost ready!

The htmldir setting

Determining the munin URL

A closer look at the default site

The Alias directive

Wasn't there something about a URL?

See the results

Summary

Installing munin on Debian

What you'll need

Installing packages

The munin.conf file

Email notification (optional)

The host tree

The munin-node.conf file

The munin-node service

Summary

Munin configuration and testing on CentOS

Almost ready!

The htmldir setting

Determining the munin URL

A closer look at the default site

The Alias directive

Wasn't there something about a URL?

See the results

Summary

Installing munin on CentOS

What you'll need

Installing packages

The munin.conf file

Email notification (optional)

The host tree

The munin-node.conf file

The munin-node service

Summary

Munin configuration and testing on Ubuntu