cv2snmp: CPE-VIEW to SNMP proxy

Introduction and Description

This project began life as an SNMP proxy for StarACS (an Alvarion/Telrad-rebranded version of Friendly Tech ACS), and this version is a rewrite that hooks into BreezeVIEW/NCS instead of StarACS. It allows you to use the SNMP protocol to collect current KPIs for all CPEs on your Telrad network managed by CPE-VIEW, so that you can avoid having to implement a BreezeVIEW/NCS API client of your own, or scrape the KPI values off of the BV web interface. Since CPE-VIEW currently lacks retention of historical KPI data for CPEs, this SNMP interface provides you with a useful means of polling these KPIs by way of some other network monitoring tool that natively supports SNMP (e.g., Cacti, Zabbix, and others), which can in turn retain the historical values for you.

This service is written in Perl and depends on the Net-SNMP project's SNMP agent Perl classes (which are just wrappers around the Net-SNMP C libraries).

(jump to downloads and installation instuctions)

Current Known Limitations and Quirks

I know that there are many, many, many areas where error handling either could be much better, or even simply implemented, period...especially when it comes to its interactions with outside services such as PostgreSQL, Net-SNMP, or BreezeVIEW itself. I'm also fully aware that my Perl conventions are probably non-kosher, that this thing probably needs a great deal of refactoring, that my coding may be atrocious, and that my [lack of] commenting certainly is.

I am happy to consider patches and contributions from anyone brave enough to submit any.

Additional Implementation Details and Features

Rather than come up with a MIB from scratch, I decided to re-use an existing one: while all Telrad CPE models support TR-69, and some only support TR-69, there are some models of Telrad CPE (specifically, all of the ones manufactured by Gemtek, which includes the 7000, 9000, and 12000 but not the 12000U) that do natively support SNMP. So cv2snmp takes the Gemtek SNMP MIB, and allows you to use the same Gemtek private OIDs to collect KPIs across all Telrad CPE models managed by your CPE-VIEW instance, regardless of whether it's a model that natively supports SNMP or not. So you don't need to know which model of CPE it is or who made it before you poll its KPIs via SNMP: cv2snmp abstracts the CPE make and model details away from you.

The following OIDs from the Gemtek MIB are supported:

The Gemtek MIB is being slightly...erm, "abused", though: since it was intended to be used to query a CPE directly at that CPE's own IP address, all of the OIDs we are interested in are defined as single-instance scalars. But cv2snmp treats them as table columns instead, with each row being a particular CPE. So each CPE managed by CPE-VIEW is assigned an index # starting at 1 for each KPI column. There are also additional columns in the table that allow you to identify which CPE holds that row/index# (by either CPE serial # or by SIM IMSI, though it is easy to extend if you want to add others).

The practical advantage to all of this is that it makes historical graph creation easier to automate: even assuming that 100% of your deployed CPE were SNMP-capable, you would have to add each CPE to your network monitoring/graphing tool of choice individually, with each as a separate monitored device with its own IP address. However, if your SNMP management tool only has to know about one host to probe, then generating graphs for each CPE becomes easier since most of these tools support checking to see if the index/row count for the table has changed since the last time it checked (usually via an SNMP walk), and then auto-generating new graphs for the new indexes.

The Gemtek MIB has the odd distinction of defining every OID to be of datatype string, even ones that you would expect to only contain numeric data or be some other obvious type (like a counter). cv2snmp honors this. On the one hand, this might possibly give some SNMP management software fits, but many do support converting numeric data contained in a string to a number and treating it as such. So on the other hand, if for some bizarre reason you're forced to stick to SNMP v1 for your queries, you shouldn't have 32-bit counter overrun problems, because those counter values will be returned as an ASCII number representation inside of a string...

Besides hooking into CPE-VIEW, this rewrite has a couple of key improvements over the original StarACS version:

  1. Performance is better, as some operations are now being parallelized. Every SNMP GET request that it processes for a given KPI causes cv2snmp to make a BreezeVIEW API call to obtain the most-recently-collected value of that KPI for that CPE. This is by far the most obvious performance bottleneck, as making an API call to BV is expensive (as was the equivalent action for StarACS). Net-SNMP's agent implements its own event loop, and by default any request sent to the agent is a blocking one, so while one GET request is in the middle of being handled, others will be queued up and not be handled at the same time. This didn't scale well, and in a production environment, the more CPEs that were being monitored, the worse things got, leading to more and more dropped samples in the historical KPIs.

    I initially tried to deal with this by implementing threads, but I discovered in the course of attempting this that Net-SNMP's agent library is not thread-safe, so I was forced to pivot to something else. In the end, I took one of the suggestions in the Net-SNMP FAQ and ran with it, which was to fork off multiple subagent processes for different OID "branches"; thus, each individual KPI "column" in the table is being handled by a different process, within which each request received (per-CPE) is still processed serially. Still, it ended up making a significant difference. The main process still spins off some threads that I decided to leave in place, such as one that periodically refreshes the list of CPEs from CPE-VIEW, since they don't make any calls into Net-SNMP library functions.

    The so-called right answer to this problem is probably to either implement the Net-SNMP FAQ's second suggestion (use setDelegated()), or to simply retrieve from BV and cache the latest KPIs in memory at some interval. But the latter has the risk of making the data returned via SNMP even more stale than it already otherwise would be, while the former would have required a considerable re-re-write of the main subagent hander. For now, the multi-process solution turned out to be the easiest and most pragmatic to implement, but I'm open to revisiting this down the road.

  2. One problem that we ran into on occasion with the StarACS implementation was what would happen to historical KPIs for a customer that had to have their CPE replaced. StarACS actually internally assigned each managed CPE a particular index number (which it turns out was being generated by a database auto-increment ID column), and to keep things simple, I just re-used the StarACS-generated index as the SNMP table row index. But StarACS was mapping CPE serial numbers to these indexes, which meant that even if we were changing out the CPE for an existing customer, and stuck their same SIM card into the new CPE, our SNMP management tool would track the KPIs for that as if it were a new customer, since it saw a new index number for that CPE. Even worse, if we re-used a CPE for an install that had been previously servicing a different customer in the past, because index numbers were assigned based on unique hardware serial numbers, StarACS would see that existing CPE come back on-line, and new KPIs for this install would get added to the old KPIs from the previous customer who had formerly been using that CPE.

    CPE-VIEW forced my hand into dealing with this issue, since unlike StarACS, it has no internal ID for each managed CPE that I could rely on, other than the CPE hardware serial number itself. And the API returns a list of managed CPEs in the order of lowest serial to highest serial, which means that if we were to base our SNMP index numbers on the order in which CPE-VIEW returned them, every time a new CPE got installed with a serial number lower than the highest serial number already in the system, the index numbers for existing CPEs would change, which is no good. So cv2snmp keeps track of a unique identifier of your choice (by default the hardware serial number, just like StarACS did, but I recommend you change that to the SIM IMSI, which is also an option!) and fixes a given index number so that it's always pointing at the CPE linked to that unique identifier. Thus, if you choose to, say, have SNMP indexes track IMSIs, then if a SIM card moves from one CPE to another, whichever CPE has that SIM in it is the one whose KPIs you will see reported back on the SNMP table row identified by that index.

Downloads and Installation Instructions

Assuming all CPAN package requirements are met, cv2snmp will likely run just fine under any relatively modern build of Perl, provided it was configured and built with threads support. You will also need to install and run snmpd from a recent-ish version of Net-SNMP, as well as supply a running PostgreSQL instance (though if you feel like making modifications to the script, it should be easy to substitute in another database, provided a Perl DBI driver exists for it).

Although not strictly required, the service was written assuming that it would be running on the same server that BreezeVIEW is installed and running on, and also that Net-SNMP will be sharing the same server as well, so the following instructions will walk you through doing just that: installing all of the necessary components on your BreezeVIEW server. It's absolutely possible to run cv2snmp on a separate server, and even to have your snmpd daemon running on yet a third, but you'll have to work out for yourself how to do that if it's a requirement of yours (among other things, you'll have to use a TCP network socket instead of a local one between cv2snmp and snmpd if they are on separate servers, which may require minor modifications to cv2snmp and your NetSNMP::Agent Perl module).

You will need to download the following bits:

Before beginning, make sure that all of your CPEs are running the latest public firmware releases; at the time of this writing, these would be:

(For future firmware updates, you will need to make sure that any firmware-version-specific TR-69/CWMP attribute notification definitions for that particular CPE model are re-added for the new firmware; see the cpeview-update-params.txt file for examples on how to do this via BreezeVIEW CLI.)

On the BreezeVIEW server, as root/superuser, do the following:

  1. Install Net-SNMP from CentOS repository:

    yum -y install net-snmp net-snmp-utils

  2. Stop snmpd:

    service snmpd stop

  3. Upload and copy the Gemtek MIB file to /usr/share/snmp/mibs

  4. Upload and replace the stock conf files in /etc/snmp with the supplied ones

  5. Edit /etc/snmp/snmpd.conf to your liking (I recommend a different community string other than the default 'public' one)

  6. Start snmpd back up:

    service snmpd start

  7. Install a whole host of Perl modules from CentOS repository:

    yum -y install perl-IPC-ShareLite perl-DateTime perl-DateTime-Format-ISO8601 perl-JSON-PP perl-libwww-perl perl-DBI perl-DBD-Pg perl-Digest-CRC perl-Data-Alias perl-Proc-Daemon net-snmp-perl

  8. Upload and import the Postgres script to create the necessary table for CPE index tracking:

    cat cv2snmp.sql | psql -U postgres

  9. Upload and import the CPE-VIEW default-parameter-attributes update script into BreezeVIEW:

    cat cpeview-update-params.txt | ncs_cli -u admin

  10. Open up cv2snmp.pl in a text editor, and make any necessary configuration changes indicated at the top

    (at minimum, you'll need to supply valid BreezeVIEW credentials -- ones with read and write permissions -- and maybe consider changing $index_key to 'imsi' instead of the default 'serial')

At this point, you should be able to run cv2snmp.pl as root (cv2snmp will daemonize itself), give it a few seconds to retrieve your list of CPEs from CPE-VIEW, and then verify that it is working by walking the CPE KPI OIDs with something like snmpwalk -v 1 -c public localhost pmpDevCpe.

If you wish to run cv2snmp as a user other than root, that can be done; you'll just need to make sure that whatever user you run it as has full permissions/access to the snmpd AgentX local socket at /var/agentx/master. Either that, or choose a different type of communications socket for cv2snmp to communicate to snmpd with.

Software License

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

nathana@fsr.com / Last updated 11 May 2020