Extending Zabbix monitoring using the Rust programming language

Since Zabbix release 2.2.0, users and developers of Zabbix can now extend Zabbix monitoring features by using loadable modules.

Up until that release the only way to extend Zabbix was by using scripts in the form of user parameters, external scripts and system.run[] items.

However the problem with using scripts for your metrics is that each time you need to get that metric Zabbix needs to fork(2) a new process and that could be a serious performance penalty, especially if you have lots of metrics.

On the other hand using the loadable modules in recent versions of Zabbix you could now create new Zabbix items easily without the overhead and performance issues that external scripts have.

A loadable module in Zabbix is essentially a shared library, which is loaded by the Zabbix Agent, Server or Proxy daemons during startup. In order for a shared library to be recognized as a valid Zabbix module your plugin needs to provide certain symbols in it's symbols table.

Loadable modules for Zabbix are usually written in C and you can already find a good starting point for creating a new Zabbix plugin at the Zabbix loadable modules page.

Another way that we could extend Zabbix monitoring features by using loadable modules is to use the Rust programming language, and this is what we will focus on in this post.

Rust is a systems programming language with focus on safety, speed and concurrency. Rust can easily hook into existing C libraries using the Rust FFI interface, but you could also write Rust code that compiles to a shared library and can be called from C (or other languages, e.g. Python, Ruby, etc.) as well.

In this post we will see how to create a low-level Rust library by translating the current Zabbix C API to Rust and once we are ready with the low-level stuff we will see how to use that Rust crate in order to create a Zabbix loadable module written in Rust.

Throughout this post we have used:

  • CentOS release 7.1.1503 (3.10.0-229.el7.x86_64)
  • Rust 1.1.0 (35ceea399 2015-06-19)
  • Zabbix 2.4.5-1.el7

You can also find the code used throughtout this post in the rust-zbx repository.

With that said, lets get started!

Translating the Zabbix C API to Rust

As a first step we would do is to get the existing Zabbix C API translated to Rust, so that we can later build our loadable module on top of it.

Lets start first by creating a new Rust project.

$ cargo new rust-zbx

Our Cargo.toml file looks like this:

[package]
name = "zbx"
version = "0.1.0"
description = "Crate for creating Zabbix loadable modules"
authors = ["Marin Atanasov Nikolov <dnaeon@gmail.com>"]

[dependencies]
libc = "*"

The Zabbix C API is located within the include/module.h header file of the Zabbix source tree and this is the file that we will now translate to Rust. In this header you will find various C macros and structs used by the Zabbix modules.

First, lets translate the C macros to Rust now.

/* include/module.h */

#define ZBX_MODULE_OK   0
#define ZBX_MODULE_FAIL -1

#define ZBX_MODULE_API_VERSION_ONE      1

/* flags for command */
#define CF_HAVEPARAMS           0x01    /* item accepts either optional or mandatory parameters */
#define CF_MODULE               0x02    /* item is defined in a loadable module */
#define CF_USERPARAMETER        0x04    /* item is defined as user parameter */

/* agent result types */
#define AR_UINT64       0x01
#define AR_DOUBLE       0x02
#define AR_STRING       0x04
#define AR_TEXT         0x08
#define AR_LOG          0x10
#define AR_MESSAGE      0x20

#define SYSINFO_RET_OK          0
#define SYSINFO_RET_FAIL        1

At the top of our src/lib.rs file we will first add the crates that we'll use throughout our library.

extern crate libc;

use std::{ffi, mem};
use libc::{c_char, c_int, c_uint, uint64_t,  c_double, malloc, strncpy};

And now translating the above C macros to Rust would look like this:

// Return codes used by module during (un)initialization
pub const ZBX_MODULE_OK: c_int = 0;
pub const ZBX_MODULE_FAIL: c_int = -1;

// Module API versions
pub const ZBX_MODULE_API_VERSION_ONE: c_int = 1;

// Flags for commands
// Item does not accept parameters
pub const CF_NOPARAMS: c_uint = 0;

// Item accepts either optional or mandatory parameters
pub const CF_HAVEPARAMS: c_uint = 1;

// Item is defined in a loadable module
pub const CF_MODULE: c_uint = 2;

// Item is defined as user parameter
pub const CF_USERPARAMETER: c_uint = 4;

// Agent result types
pub const AR_UINT64: c_int = 1;
pub const AR_DOUBLE: c_int = 2;
pub const AR_STRING: c_int = 4;
pub const AR_TEXT: c_int = 8;
pub const AR_LOG: c_int = 16;
pub const AR_MESSAGE: c_int = 32;

// Return codes used by item callbacks
pub const SYSINFO_RET_OK: c_int = 0;
pub const SYSINFO_RET_FAIL: c_int = 1;

Pretty straightforwad, now lets translate the C structs as well into Rust add implement common methods that we will use when working with these types in Rust.

typedef struct
{
        char            *key;
        unsigned        flags;
        int             (*function)();
        char            *test_param;    /* item test parameters; user parameter items keep command here */
}
ZBX_METRIC;

/* agent request structure */
typedef struct
{
        char            *key;
        int             nparam;
        char            **params;
        zbx_uint64_t    lastlogsize;
        int             mtime;
}
AGENT_REQUEST;

typedef struct
{
        char            *value;
        char            *source;
        zbx_uint64_t    lastlogsize;
        int             timestamp;
        int             severity;
        int             logeventid;
        int             mtime;
}
zbx_log_t;

/* agent return structure */
typedef struct
{
        int             type;
        zbx_uint64_t    ui64;
        double          dbl;
        char            *str;
        char            *text;
        char            *msg;

        /* null-terminated list of pointers */
        zbx_log_t       **logs;
}
AGENT_RESULT;

The ZBX_METRIC struct is used for creating new Zabbix items and contains information about an item such as the name, whether the item accepts parameters or not, a callback function used for processing and calculating the item and optional test parameters.

The AGENT_REQUEST struct is used whenever a new request is sent to Zabbix for processing. An instance of AGENT_REQUEST struct is usually passed by Zabbix to our callback function which contains all details about the actual request - name of the item, the number of parameters passed to the item, the actual parameters and others.

The AGENT_RESULT struct is used for storing the result from our callback. Zabbix passes a pointer to an actual instance of AGENT_RESULT and our callback is expected to update this instance with some result. The result can optionally contain a message with it, usually used for indicating an error condition.

Now, lets translate these structs to Rust.

#[repr(C)]
pub struct ZBX_METRIC {
    pub key: *const c_char,
    pub flags: c_uint,
    pub function: extern "C" fn(*mut AGENT_REQUEST, *mut AGENT_RESULT) -> c_int,
    pub test_param: *const c_char,
}

#[repr(C)]
pub struct AGENT_REQUEST {
    key: *const c_char,
    nparam: c_int,
    params: *const *const c_char,
    lastlogsize: uint64_t,
    mtime: c_int,
}

#[repr(C)]
pub struct zbx_log_t {
    value: *const c_char,
    source: *const c_char,
    lastlogsize: uint64_t,
    timestamp: c_int,
    severity: c_int,
    logeventid: c_int,
    mtime: c_int,
}

#[repr(C)]
pub struct AGENT_RESULT {
    _type: c_int,
    ui64: uint64_t,
    dbl: c_double,
    _str: *const c_char,
    text: *const c_char,
    msg: *const c_char,
    logs: *const *const zbx_log_t,
}

Again, this is a pretty straightforward task. Since these types need to be passed between Rust and C we need to make sure that our Rust structs are compatible with the C representation, therefore we apply the #[repr(C)] attribute to our Rust structs.

In order to create new Zabbix items easily from Rust we will also introduce a new Rust type. This will be a high-level type that we'll use for creating new Zabbix items for our loadable modules.

// Type used for creating new Zabbix item keys
pub struct Metric {
    pub key: ffi::CString,
    pub flags: c_uint,
    pub function: extern "C" fn(*mut AGENT_REQUEST, *mut AGENT_RESULT) -> c_int,
    pub test_param: ffi::CString,
}

impl Metric {
    pub fn new(key: &str, flags: u32, function: extern "C" fn(*mut AGENT_REQUEST, *mut AGENT_RESULT) -> c_int, test_param: &str) -> Metric {
        Metric {
            key: ffi::CString::new(key).unwrap(),
            flags: flags as c_uint,
            function: function,
            test_param: ffi::CString::new(test_param).unwrap(),
        }
    }

    pub fn to_zabbix_item(&self) -> ZBX_METRIC {
        ZBX_METRIC {
            key: self.key.as_ptr(),
            flags: self.flags as c_uint,
            function: self.function,
            test_param: self.test_param.as_ptr(),
        }
    }
}

The Zabbix C API also provides a C macro used for retrieving the parameters passed to an item. This is how the C macro looks like.

#define get_rparam(request, num)        (request->nparam > num ? request->params[num] : NULL)

We will translate this C macro to a Rust associated function for the AGENT_REQUEST type. The result of the AGENT_REQUEST::get_params associated function is a Vector containing the parameters of this request as passed by Zabbix.

impl AGENT_REQUEST {
    pub fn get_params<'a>(request: *mut AGENT_REQUEST) -> Vec<&'a[u8]> {
        unsafe {
            let len = (*request).nparam;
            let mut v = Vec::new();

            for i in 0..len {
                let ptr = (*request).params.offset(i as isize);
                let param = ffi::CStr::from_ptr(*ptr).to_bytes();
                v.push(param);
            }

            v
        }
    }
}

Our Rust callbacks also need to set result, which will be returned to Zabbix. To do that we will also need to translate the C API macros which deal with setting result. These are the C API Zabbix macros used for setting result from an agent request.

#define SET_UI64_RESULT(res, val)               \
(                                               \
        (res)->type |= AR_UINT64,               \
        (res)->ui64 = (zbx_uint64_t)(val)       \
)

#define SET_DBL_RESULT(res, val)                \
(                                               \
        (res)->type |= AR_DOUBLE,               \
        (res)->dbl = (double)(val)              \
)

/* NOTE: always allocate new memory for val! DON'T USE STATIC OR STACK MEMORY!!! */
#define SET_STR_RESULT(res, val)                \
(                                               \
        (res)->type |= AR_STRING,               \
        (res)->str = (char *)(val)              \
)

/* NOTE: always allocate new memory for val! DON'T USE STATIC OR STACK MEMORY!!! */
#define SET_TEXT_RESULT(res, val)               \
(                                               \
        (res)->type |= AR_TEXT,                 \
        (res)->text = (char *)(val)             \
)

/* NOTE: always allocate new memory for val! DON'T USE STATIC OR STACK MEMORY!!! */
#define SET_LOG_RESULT(res, val)                \
(                                               \
        (res)->type |= AR_LOG,                  \
        (res)->logs = (zbx_log_t **)(val)       \
)

/* NOTE: always allocate new memory for val! DON'T USE STATIC OR STACK MEMORY!!! */
#define SET_MSG_RESULT(res, val)                \
(                                               \
        (res)->type |= AR_MESSAGE,              \
        (res)->msg = (char *)(val)              \
)

You should note that when the result of an item is a string (text, message and log) Zabbix expects to receive a raw pointer to a previously allocated memory of the resulting string and once done with the result Zabbix will free(3) the memory.

If we tried to return a reference to a string from Rust to C we would end up in some ugly situations such as double-freeing the result, as Zabbix would try to free(3) the memory, when it was already deallocated by the Rust borrow checker once our string goes out of scope.

In order to deal with this we need to allocate some memory from Rust for the resulting string and actually leak that memory to Zabbix, which will be deallocated by Zabbix once done with the result. For this purpose we will introduce a helper function that we will use when working with string results in Zabbix.

// When the result of a Zabbix item is text (string, text and message)
// Zabbix expects to receive a pre-allocated pointer with the result
// string, which is free(3)'d by Zabbix once done with the result.
unsafe fn string_to_malloc_ptr(src: &str) -> *mut c_char {
    let c_src = ffi::CString::new(src).unwrap();
    let len = c_src.to_bytes_with_nul().len() as u64;

    let dst = malloc(len) as *mut c_char;
    strncpy(dst, c_src.as_ptr(), len);

    dst
}

Now, lets implement the functions for setting results as well. We will implement them as associated functions for the AGENT_RESULT type.

impl AGENT_RESULT {
    pub fn set_uint64_result(result: *mut AGENT_RESULT, value: u64) {
        unsafe {
            (*result)._type |= AR_UINT64;
            (*result).ui64 = value as uint64_t;
        }
    }

    pub fn set_f64_result(result: *mut AGENT_RESULT, value: f64) {
        unsafe {
            (*result)._type |= AR_DOUBLE;
            (*result).dbl = value as c_double;
        }
    }

    pub fn set_str_result(result: *mut AGENT_RESULT, value: &str) {
        unsafe {
            (*result)._type |= AR_STRING;
            (*result)._str = string_to_malloc_ptr(value);
        }
    }

    pub fn set_text_result(result: *mut AGENT_RESULT, value: &str) {
        unsafe {
            (*result)._type |= AR_TEXT;
            (*result).text = string_to_malloc_ptr(value);
        }
    }

    pub fn set_msg_result(result: *mut AGENT_RESULT, value: &str) {
        unsafe {
            (*result)._type |= AR_MESSAGE;
            (*result).msg = string_to_malloc_ptr(value);
        }
    }

    // TODO: Implement set_log_result(...)
}

Dereferencing a raw pointer is considered unsafe operation in Rust, therefore we are using the unsafe keyword in our functions above.

As a last thing we will create another helper function that we can later use in our loadable modules for easy item creation.

pub fn create_items(metrics: &Vec<Box<Metric>>) -> *const ZBX_METRIC {
    let items = metrics
        .iter()
        .map(|metric| metric.to_zabbix_item())
        .collect::<Vec<_>>();

    // XXX: leak items into the void
    let ptr = items.as_ptr();
    mem::forget(items);

    ptr
}

So far we have built a Rust low-level interface of the Zabbix loadable modules API. In the next section of this post we will see how to create a fully functional loadable module for Zabbix in Rust.

Building a Zabbix module in Rust

Now that we have the low-level stuff sorted out, lets create a simple Zabbix loadable module in Rust using the library that we've created in the previous section.

First, lets create a new Rust project:

$ cargo new dummy

Our Cargo.toml file for this Zabbix dummy module like this:

[package]
name = "dummy"
version = "0.1.0"
description = "Example Zabbix loadable module written in Rust"
authors = ["Marin Atanasov Nikolov <dnaeon@gmail.com>"]

[lib]
name = "rust_dummy"
crate-type = ["dylib"]

[dependencies.rand]
version = "*"

[dependencies.zbx]
git = "https://github.com/dnaeon/rust-zbx.git"
version = "*"

As mentioned in the beginning of this post the Zabbix loadable modules are simply shared libraries, and that is why our Rust crate needs to be of dylib type.

This setting simply instructs the Rust compiler to build the resulting Rust library as a shared library, which can then be used from other languages as well.

A Zabbix loadable module should provide certain symbols in it's symbols table in order be to recognized as valid Zabbix plugin and our Rust library should provide these symbols as well.

These are the zbx_module_api_version, zbx_module_init, zbx_module_uninit and zbx_module_item_list symbols.

This is how the C prototypes for the above functions look like:

int         zbx_module_api_version(void);
int         zbx_module_init(void);
int         zbx_module_uninit(void);
ZBX_METRIC  *zbx_module_item_list(void);

The zbx_module_api_version callback should return the module API version, and currently the only supported version for it is ZBX_MODULE_API_VERSION_ONE.

The zbx_module_init callback is used by modules for performing any initialiazation that the module needs to perform before it can be used. This function is called during module loading and should return either a success or a failure code indicating the result of the initialization procedure.

Similar to the zbx_module_init the zbx_module_uninit callback is used as the shutdown procedure for a Zabbix module, e.g. de-allocating any resources, closing sockets, etc.

The zbx_module_item_list callback is the one that returns an array of items that Zabbix will be able to use for processing any metrics requests.

Lets first add the Rust crates that we'll use throughout our Rust module at the top of our src/lib.rs file.

// Example Zabbix loadable module written in Rust

extern crate zbx;
extern crate rand;

use std::{boxed, mem, str};
use rand::Rng;

Now, lets implement the zbx_module_api_version function now.

#[no_mangle]
pub extern fn zbx_module_api_version() -> i32 {
    zbx::ZBX_MODULE_API_VERSION_ONE
}

Our initialization and shutdown module functions are pretty easy as well.

#[no_mangle]
pub extern fn zbx_module_init() -> i32 {
    zbx::ZBX_MODULE_OK
}

#[no_mangle]
pub extern fn zbx_module_uninit() -> i32 {
    zbx::ZBX_MODULE_OK
}

Since our example Zabbix module does not require any special initialization or shutdown procedures we are simply returning a success code back to Zabbix. In case your module requires any initialization and shutdown procedures this is the place where you would implement them.

And now we need to implement the zbx_module_item_list function, which should return an array of items, that can later be used by Zabbix for item processing.

#[no_mangle]
pub extern fn zbx_module_item_list() -> *const zbx::ZBX_METRIC {
    let metrics = vec![
        boxed::Box::new(zbx::Metric::new("rust.echo", zbx::CF_HAVEPARAMS, rust_echo, "")),
        boxed::Box::new(zbx::Metric::new("rust.random", zbx::CF_NOPARAMS, rust_random, "")),
    ];

    // XXX: Leak items into the void
    let items = zbx::create_items(&metrics);
    mem::forget(metrics);

    items
}

In the above example function we have created two new Zabbix items using the zbx::Metric::new associated function.

The rust.echo item accepts parameters from Zabbix, while rust.random does not and both of these items do not have any test parameters.

Finally we are returning the items back to Zabbix by using the zbx::create_items helper function.

Now we need to implement the rust_echo and rust_random callbacks as well, which are the ones that perform the actual item processing when a new request is sent to Zabbix.

Lets implement the rust_echo function now.

#[no_mangle]
pub extern fn rust_echo(request: *mut zbx::AGENT_REQUEST, result: *mut zbx::AGENT_RESULT) -> i32 {
    let params = zbx::AGENT_REQUEST::get_params(request);

    if params.len() != 1 {
        zbx::AGENT_RESULT::set_msg_result(result, "Invalid number of parameters");
        return zbx::SYSINFO_RET_FAIL;
    }

    let param = match str::from_utf8(params[0]) {
        Ok(p)  => p,
        Err(e) => panic!("Invalid UTF-8 sequence: {}", e),
    };

    zbx::AGENT_RESULT::set_str_result(result, param);

    zbx::SYSINFO_RET_OK
}

The rust_echo function expects to receive exactly one parameter from Zabbix and echoes it back.

And this is our rust_random function that will generate a random number each time it gets called and returns it to Zabbix.

#[no_mangle]
#[allow(unused_variables)]
pub extern fn rust_random(request: *mut zbx::AGENT_REQUEST, result: *mut zbx::AGENT_RESULT) -> i32 {
    let mut rng = rand::thread_rng();
    let num = rng.gen::<u64>();

    zbx::AGENT_RESULT::set_uint64_result(result, num);

    zbx::SYSINFO_RET_OK
}

If you have followed this post by far you should now have a fully working Zabbix module written in Rust!

Time to test things out and see our Rust module in action!

First, lets build our Rust library using Cargo.

$ cargo build --release

Once you successfully build your project you should find the Rust library within your target/release directory.

$ ls -l target/release/librust_dummy.so
-rwxr-xr-x 1 mnikolov mnikolov 3095064 Jul 27 14:41 target/release/librust_dummy.so

In order to load this library in your Zabbix Agent, Server or Proxy you should update their respective configuration files and update the LoadModule and LoadModulePath configuration settings.

This is how my Zabbix Agent settings look like:

####### LOADABLE MODULES #######

### Option: LoadModulePath
#       Full path to location of agent modules.
#       Default depends on compilation options.
#
# Mandatory: no
# Default:
LoadModulePath=/usr/lib/zabbix/modules

### Option: LoadModule
#       Module to load at agent startup. Modules are used to extend functionality of the agent.
#       Format: LoadModule=<module.so>
#       The modules must be located in directory specified by LoadModulePath.
#       It is allowed to include multiple LoadModule parameters.
#
# Mandatory: no
# Default:
LoadModule=librust_dummy.so

Finally, install the librust_dummy.so library in the path to which LoadModulePath setting points to, so that the Zabbix services can find and load the library.

$ sudo install target/release/librust_dummy.so /usr/lib/zabbix/modules

We can now actually test our Rust module for Zabbix by using the zabbix_get(8) tool.

Lets get some random numbers by using the rust.random Zabbix item.

$ zabbix_get -s 127.0.0.1 -p 10050 -k rust.random
13559620181874123117

Every time we call rust.random item we should get a random number generated from Rust.

And now lets test the rust.echo Zabbix item as well.

$ zabbix_get -s 127.0.0.1 -p 10050 -k rust.echo['Rust is awesome!']
Rust is awesome!

Conclusion

Throughout this post we have seen how Rust can be used in places where you would usually use C and that is probably one of the features that makes Rust a language with great potential. Of course Rust is more than just that - Rust has a really nice and vibrant community behind it and a constantly growing ecosystem of libraries that you could pick from.

I'm still making my baby steps with Rust and learning the language, but so far I really enjoy coding in it and I'm definitely looking at my next project to develop in Rust!

Written on July 27, 2015