Company Accelerates Application Performance Troubleshooting Customer Challenge

CUSTOMER CHALLENGE

Headquartered in Herndon, Virginia, Boxwood Technology provides job boards and other online career center services for associations. The company is endorsed by the American Society of Association Executives and is a charter member of the International Association of Employment Websites.

Boxwood employees in two locations depend on business applications housed in a collocation facility, including a ticketing system, customer databases, Microsoft® SharePoint, accounting software, and an IT logging tool. Previously, if users reported slow application performance, Boxwood’s systems administrators used the “show IP NBAR” command to investigate. “But the command returned limited information about only a few protocols, and interpreting the data was time-consuming,” says William Bordeau, systems administrator for Boxwood Technology. “To accelerate troubleshooting and provide a good user experience, we wanted deeper insight into end-to-end application performance, and an easy-to-use graphical user interface.”

Complicating the challenge, Boxwood was preparing to replace a traditional phone system with an all-IP voice system. “We knew we needed the ability to visualize voice traffic from end-to-end, because voice quality affects customer satisfaction when they call for sales or support,” Bordeau says. The existing, third-party NetFlow traffic analyzer would not work for voice traffic monitoring, because it relied on traceroute, which only shows traffic between routers, not between switches. This meant that a portion of the traffic flow for interoffice calls, which goes through the Multiprotocol Label Switching (MPLS) cloud, would be hidden. To see the end-to-end flow, Boxwood needed a router that supported Performance Monitoring technology that discovers switches as well as routers. Cisco® Performance Monitoring technology follows flows hop-by-hop, collecting statistics across the flow path. It leverages another Performance Monitoring technology, which collects diagnostic data including packet loss, jitter, hop-by-hop latency, and response time.

Ease-of-use was critical. “Our staff is busy and needs a simple tool that displays actionable information without requiring them to jump through hoops,” Bordeau says. “We wanted an easy-to-use visualization tool that would require less detective work.”

SOLUTION

Boxwood found a comprehensive solution to visualizing application flows by using services on the Cisco Integrated Services Router Generation 2 (ISR G2) in conjunction with LiveAction (now known as LiveNX*).

• Cisco Application Visibility and Control (AVC) is a suite of services in Cisco network devices that provides application-level classification, monitoring and traffic control. It uses deep-packet inspection to identify more than 1000 applications, collecting performance statistics such as bandwidth use, latency, and response time. On a recent day, Cisco AVC reported that the top 10 types of traffic by volume included HTTP, Common Internet File System (CIFS), Exchange, Active Directory, Cisco WebEx®, Simple Network Management Protocol (SNMP), and YouTube.
• Performance Monitoring technology supported in Cisco ISR G2 routers, monitors voice flows from point to point, across routers as well as switches. “With Performance Monitoring, nothing is blocked from view, including voice traffic traversing our MPLS cloud,” Bordeau says.
• LiveAction provides an intuitive GUI for filtering the information from Cisco AVC and Performance Monitoring and presenting it in an easy-to-understand, visual format. If an employee reports slow network performance, Bordeau can visualize current or historical network activity on the LiveAction interface, which color-codes traffic links by volume and protocol. This capability makes it easy to see the protocol responsible for the saturation, such as CIFS for file transfers, and the originating endpoint. “LiveAction is a single interface we can use with Cisco AVC and Performance Monitoring to see congested areas, visualize flows from end-to-end, and apply changes that LiveAction pushes to our routers,” Bordeau says. “The visual representation helps us pinpoint the location of a network incident that’s affecting the user experience in just minutes, compared to hours or even days before we had the tools.”

CUSTOMER RESULTS

Successes to-date include quickly discovering the sources of network congestion, detecting and mitigating an attack against the company’s web server farm, and identifying a misconfigured ACL before it caused voice quality problems.

Accelerated Troubleshooting of Network Performance Issues

Boxwood uses Cisco AVC and LiveAction primarily for diagnostics, to identify the cause of slow performance for TCP-based applications. “We use LiveAction on-demand anytime we need to troubleshoot or diagnose network performance issues,” Bordeau says. “If a user reports slow application performance, I can apply a filter in LiveAction to look at metrics from that server. Being able to visualize network activity by application, not just the port, is very valuable for us.” If LiveAction shows zero retransmissions for a large file transfer, Bordeau can narrow down the problem to the server, not the network. The combination of LiveAction and Cisco AVC also makes it easy to see if slow application performance is a result of client network delay or server network delay.

“LiveAction is a single interface we can use with Cisco AVC and Performance Monitoring to detect congested areas, visualize flows from end-to-end, and apply changes that LiveAction pushes to our routers.”

– WILLIAM BORDEAU, SYSTEMS ADMINISTRATOR, BOXWOOD TECHNOLOGY

Boxwood experienced the value of Cisco AVC and LiveAction soon after implementation, when email traffic slowed abruptly. Cisco AVC reported congestion in an uplink to the MPLS cloud, clearly visible on the LiveAction interface. From within LiveAction, Bordeau built a filter to look for file transfers with a few clicks, quickly identifying the source. Further investigation revealed that an employee in the sales and marketing office was backing up large, offline mailboxes.

Blocked Network Attack

LiveAction and Cisco AVC also helped Boxwood’s IT team detect a Structured Query Language (SQL) injection attack on company web servers. The firewall at the facility reports the source country of all connection attempts. When Bordeau used the LiveAction to view historical data, he noticed multiple connections in a country where the company does not do business, always from 2 a.m. to 3 a.m. With this information, he was able to construct a filter that blocked the traffic.

Averted Voice Quality Problem by Discovering Misconfigured ACL

When preparing for the migration to unified communications, Bordeau connected an IP phone to the network for testing purposes. Although the phone worked, LiveAction showed that the connection was going across a different interface than expected. With further investigation, Bordeau discovered that an access control list (ACL) was helping to enable calls to travel over the public Internet instead of just the MPLS VPN. “Without Cisco Performance Monitoring and LiveAction, we might never have discovered the error, allowing some calls to travel on a path without QoS,” he says. “Discovering the problem helped us protect voice quality for our employees and customers.”

Next Steps

Today, Boxwood uses LiveAction and Cisco AVC primarily for diagnostics. Later, the company might use the alerting feature in LiveAction to gain early awareness of issues before they affect the user experience.

Bordeau concludes, “We’re excited to have the tools to collect detailed information about all application flows and view it from one easy-to-use interface. And visualizing the network is helping us provide a great quality of experience by identifying and resolving issues more quickly.”

Read more or download PDF