{"id":922,"date":"2019-12-11T22:13:03","date_gmt":"2019-12-11T20:13:03","guid":{"rendered":"http:\/\/www.mythryll.com\/?p=922"},"modified":"2020-01-19T21:26:43","modified_gmt":"2020-01-19T19:26:43","slug":"migrating-from-mrtg-rrd-cgi-to-grafana-influxdb-telegraf-on-docker-containers-for-enterprise-network-performance-monitoring-with-snmp-part-1","status":"publish","type":"post","link":"https:\/\/www.mythryll.com\/?p=922","title":{"rendered":"Migrating from MRTG\/RRD\/CGI to Grafana\/InfluxDB\/Telegraf on Docker containers for Enterprise Network Performance Monitoring with SNMP – Part 1"},"content":{"rendered":"\n

(if you are looking for part2, it’s here<\/a>)<\/p>\n\n\n\n

Introduction <\/h2>\n\n\n\n

I like reading guides, not writing them (it takes too long for me). So this isn’t a guide, it’s more like an adventure tale, it’s a “I got out one day for a stroll and look what happened!”. I had no knowledge of Grafana, InfluxDB, Telegraf or Docker before doing this, apart from a simple definition for each term. The previous monitoring infrastructure was old, difficult to maintain and limited in capabilities, its components slowly dying and out of support. Furthermore, new monitoring needs and novel tech such as Streaming Telemetry and Network Automation were almost impossible to integrate. A big change was due, and it needed to be done soon.<\/p>\n\n\n\n

Everything that follows was implemented in a production environment, although test VMs were used for the first tests on installing and using the software.<\/p>\n\n\n\n

For those of you that don’t know me, I am a network engineer, I mostly work with Cisco Network Equipment and Software, I have a strong background with Network Performance and Fault management and Monitoring and I try to have an open mind, combining tech from different areas to solve operational challenges and deploy new services. I also have a keen interest in Network Automation. You can find me on Twitter under the @mythryll handle.
<\/p>\n\n\n\n

A little history<\/h2>\n\n\n\n

A little over a month ago (maybe two) I took a big plunge. I had been working at my current employer for more than 15 years and almost since the beginning, bringing in a lot of experience with SNMP, I had deployed SNMP network throughput monitoring on MRTG<\/a>, the well-known network traffic grapher created by Tobie Oeticker, probably the best friend of most network engineers for a long long time. Deploying MRTG was also the start of my involvement with Linux, as I used OpenSuSE linux for it, putting everything (MRTG+Apache) in one server box, and began a parallel life as a semi-serious Linux system admin, taking care of my systems. <\/p>\n\n\n\n

\"\"
A familiar image for most network engineers<\/figcaption><\/figure>\n\n\n\n

That monitoring infrastructure became larger little by little in order to provide metrics for every port in the network (a few thousands) for every network device (a few hundreds). I added more servers, spread out, using physical at start and then migrated them to virtual machines as Vmware took over the Datacenter (I had a hand in that). CPU load, Memory, Temperatures, PS Status, pretty much everything important that had an available SNMP gauge parameter was displayed, even custom metrics produced by running Perl scripts on the core switches (measuring MLS QoS queues on Cat 6500s), or composite metrics (products of several SNMP gathered metrics). <\/p>\n\n\n\n

During the migration to the virtual world, the servers were trimmed down as much as possible and RRD\/CGI was added to the mix, in order to take the load off from creating PNG files constantly whenever transforming SNMP metrics into graphs. Instead, by using the MRTG-CGI script created by Jan “Yenya” Kasprzak<\/a><\/em> and using RRD to store the data, monitored object web pages and graphs were only created on demand. I had created a decent hierarchical site structure, calling the MRTG-CGI <\/a>script whenever every link was used, with just the right parameters for every object in order to dynamically create the page for it. I finally added a little Javascript in order to create a light structure, added AD authentication and migrated to Https for securing the access for the right users and groups. <\/p>\n\n\n\n

One central web server (Apache) served as the website skeleton and three measuring servers were running MRTG\/RRD\/CGI for gathering the SNMP metrics and serving them through Apache by responding to requests from links used on the main web server.<\/p>\n\n\n\n

The old dog<\/h3>\n\n\n\n
\"\"
Can you teach an old dog new tricks?<\/figcaption><\/figure><\/div>\n\n\n\n

I had reached the limit for that infrastructure. There were constant problems, RRD files becoming corrupted and having to be deleted, file rights often overwritten, system files updates would cause conflicts with perl snmp libraries used by MRTG and needed manual patching, etc. I had little notification of such issues, only connectivity problems would generate emails for the mrtg local user. Further than that, provisioning for new items to monitor was a pain, since every OID required one specific entry for every network node in the MRTG config (if you ever did that, you know how it is) and once that was done, the WEB UI part was also another task to complete, that usually stayed behind and out of date. Btw, I never liked other solutions like Cacti or Routers2 (sorry Steve), as they didn’t allow enough freedom with the UI. <\/p>\n\n\n\n

Upgrading the OS became a nightmare, as RRD files don’t behave well once the system has changed and so history could be lost. The time to perform the upgrade, set everything back as it was, copy over config and data and start the engine again, was too long. There would be gaps in the monitoring data and I had to look for such “free” periods in my schedule, that rarely came up. If the OS upgrade procedure failed (it often did) I had to revert to a snapshot or create everything again from scratch.. <\/p>\n\n\n\n

Updating perl modules with CPAN was also an issue, as the Security Division’s tools in my Organization would most often cut part of the access to the CPAN mirrors, resulting in file corruption (Security will get you, every time..).<\/p>\n\n\n\n

On top of that I had to learn a lot about Apache as well, as significant changes in the software made it necessary to learn the new access rights model and how it would affect the tool, and the dynamic part, based on CGI, was not getting any younger. The CGI script used for dynamic page and graph creation, mrtg-cgi<\/a>, dated its latest changes back to 2003 and since 2014 there were even security issues involved, you can read about that here<\/a>.<\/p>\n\n\n\n

After I started playing with automation (python, Ansible<\/a>, netmiko<\/a>, nornir<\/a>, netbox<\/a>, PyATS\/Genie<\/a>, etc) I developed big plans to use Ansible to automatically manage the configuration part. I had already created python scripts to produce the web pages for each network node with every interface on it, began studying Django and DBs, which in fact could come in handy later, and generically sensed that such an old tech could not go any further. RRD is great but not so flexible and not querible. The look was also not so great but I told myself that I am not a web developer so that was ok. <\/p>\n\n\n\n

Meeting with Grafana<\/h3>\n\n\n\n

I had seen Grafana before. I believe it was Cisco’s Jason Davis<\/a>, that published pics from a Cisco Live Noc at some point (probably Jan 2018) and I began taking a look at it. Then Cisco Devnet’s Stuart Clark<\/a> mentioned it during a NetDevOps Live Session and I became aware of the fact that there are finally software solutions for network engineers who monitor the network and seek simpler and information rich tools to visualize performance metrics.<\/p>\n\n\n\n

At first I thought of going for a small step further, still using MRTG for the collection of data, but replacing RRD with a similar component, like this<\/a>. Little after starting my search, I realized there was a solution that provided everything in one package, had enough simplicity and robustness to fit in with my goals:
Telegraf<\/a>, InfluxDB <\/a>and Grafana<\/a>, the so-called TIG Stack.<\/p>\n\n\n\n

I also came across a post <\/a>by Jeremy Cohoe in the Cisco Devnet developer blog, about setting up Streaming Telemetry using the TIG stack to consume and visualize the data. As I wanted to deploy Streaming Telemetry for our network as well (I will not develop this here and now), I thought that stepping towards the TIG stack with SNMP, would probably ease that transition as well, later.<\/p>\n\n\n\n

So I decided to try that, and began looking for more information. The official documentation for each tool was rich but had no reference on how to combine them in one solution. <\/p>\n\n\n\n

Ok, I am convinced.. so where do I start?<\/h3>\n\n\n\n

I found a few posts about how it worked and how to set it up. Some used it to monitor Host performance metrics on isolated systems, others for home network monitoring (setup on Rasberrypi), others for limited SNMP metrics, and others for Virtual environments, like Vmware\/VCenter<\/a>. I didn’t find any posts or comments about it being used for a large network infrastructure with SNMP. Nothing came up on Twitter either. The Devops world has produced a lot of monitoring tools lately but it’s probably still early for network engineers that traditionally used simpler tools to adopt and use them, so I guess it’s normal. <\/p>\n\n\n\n

At first I started with a test server in a VM, Ubuntu 18.04 LTS. I decided to try the procedure suggested in this post<\/a>, explaining how to setup most things for simple SNMP monitoring and only a few agents (nodes), and combined it with this post <\/a>in order to better prepare InfluxDB, as the first one does not create a separate Database. Looking into more posts, I started to realize how InfluxDB is very close to a regular database, providing a lot more features but at the same time needing some administration and housekeeping. Besides doing regular queries and deleting test or erroneous data, I found that a very important issue is data retention policy. But more on that later. <\/p>\n\n\n\n

What were the main installation steps?<\/h3>\n\n\n\n