Citrix Engineering has a dedicated Globalization Services team that helps ensure our products meet the needs of our non-English-speaking customers. Part of that involves making sure things like the display of dates and times — and even colors and fonts — appear in the format customers prefer. Supporting these “globalization requirement” requests helps us to deliver a higher-quality user experience.

To date, one of our challenges has been identifying which requests to address more broadly (and how quickly to address them). Since early 2018, we’ve worked to apply machine learning to analyze these requests, better understand the voice of the customer, measure our team’s effectiveness, and find opportunities for product improvements.

Let’s turn back the clock to 2016.

Back then, the Globalization Services team was analyzing customer voice by going through support tickets manually. With more than 100,000 a year, it gave us plenty of opportunity to learn about our customers and their needs, but it was a time-consuming process. And we found that more than half our support tickets were related to globalization requirements.

Thanks to artificial intelligence and natural language processing, we’ve been able to analyze customer voice and dig deeper into globalization issues. We leveraged the data from our manual analysis of data from 2016 and 2017 to build a decision engine powered by machine learning.

Our algorithm is shown in the flow chart above (click the image for a larger view). We train our model with historical customer data and use it to classify customer data and determine the relevance to globalization requirements.

How do we train it? First, we clean the text samples by removing items that won’t be helpful in future analyses. Then, we extract features that help to distinguish the data samples from each other. Data samples are then aligned with these features, which results in numeric vectors. The model learns the rules from those samples, saving them for future use, and we end up with a trained random forest model.

The chart below shows the prediction process (click the image for a larger view). Customer data gets processed and, after feature extraction, is represented with numeric vectors. The vectors go to our random forest model, which enables us to identify whether the customer data is related to a globalization issue.

The tuning of the classification algorithm is an ongoing process, but we’re analyzing tickets with increasing accuracy and created a dashboard (see the image below) to track the data. We’re hoping to exceed 85 percent accuracy by year-end across all product lines. The algorithm also enables rapid analysis — more than 27,000 tickets in three hours—and it’s helping to shape our products. For example, we created a time zone health check tool, so customers can identify issues related to time zone settings. We’re expecting it to lead to an 8 percent decrease in globalization-related customer support issues.

You have to listen to the customer to deliver a high-quality user experience. By using machine learning, we’re able to dig deeper into our customers’ challenges faster and ensure our products meet their globalization requirements.