Case Study: US Mobile Closes NoSQL Analytics Gap
SlamData Provides Missing Platform for NoSQL Data Insight
US Mobile launched in 2015 with the goal of giving mobile phone customers the freedom to “choose what they need and never pay for what they don’t.” Letting customers create their own talk, text and data plans proved popular, but the resulting growth presented its own challenges.
In 2016, US Mobile rebuilt its core business applications (which previously ran on a relational database) on MongoDB, a NoSQL data platform. The move gave US Mobile flexibility, scalability and better support for microservices, but it also meant existing queries, reports, data visualizations and analyses had to be rebuilt. What’s more, support for SQL, the query language native to relational databases, is not natively supported in MongoDB.
Many organizations struggle to fill the gap left by the lack of SQL support when using NoSQL stores — whether MongoDB or other options, like Couchbase or MarkLogic. US Mobile considered several approaches to close the gap. US Mobile chose SlamData for its combination of SQL querying against NoSQL sources coupled with interactive data-analysis, visualization and dashboarding capabilities.
SlamData was deployed on a synced replica of US Mobile’s production MongoDB database within one day. The company was able to start creating reports for sales, marketing, finance and other areas within hours. Dozens of sophisticated dashboards and interactive analyses have since been made available to US Mobile employees for self-service to real- time insight.
The Company: US Mobile
US Mobile is a nationwide wireless carrier that offers 4G LTE network coverage that reaches more than 300 million Americans. Like other mobile virtual network operators (MVNOs), such as Boost Mobile, Cricket Wireless and TracFone, US Mobile leases wireless telephone and data services from the largest national networks. Competitive pricing is usually the key appeal for MVNOs, but US Mobile launched in late 2015 with a strategy to also differentiate itself with flexible, a la carte service plans and responsive customer service.
Technology is a key differentiator for US Mobile. The company operates entirely online, with no physical retail locations. The online customer experience is crucial. Customers can flexibly and instantly change their plans through a web-based dashboard, adding talk, text and data capacity as needed. In addition to supporting smartphones at an average cost of $15 per month, US Mobile offers SIM cards for other devices, including alarms, GPS trackers, cars, tablets and smartwatches. Support is handled via phone (within 20 seconds), chat (within 10 seconds) and email (within 10 minutes) 24 hours a day, seven days per week.
US Mobile’s strategy has succeeded. The company surpassed 40,000 customers in 2017. Growing from five (mostly engineering- focused) employees at launch, the company has grown to more than 40 employees across sales, marketing, product management, design, engineering and executive management. The increasingly diverse team has been eager to gain greater visibility and deeper insight into customer behavior, company operations, performance and profitability.
Michael Melmed, vice president of operations and strategy, joined in mid-2016 in part to enhance the company’s analytical capabilities. At the time, US Mobile faced challenges tied to rapid growth. US Mobile was struggling to scale up its business applications, which ran on MariaDB, an open source relational database. US Mobile wanted to move to a scalable, microservices-based approach, so it decided to rewrite its core applications on MongoDB, an open source NoSQL database. MongoDB was seen as the best fit by US Mobile developers, who wanted to be able to quickly introduce new features and services. The NoSQL platform gave the developers flexibility to quickly add new dimensions of data and to scale out to support rapid subscriber growth.
About US Mobile
- US Mobile leases wireless telephone and data services from the largest national networks.
- Technology is a key differentiator for US Mobile. The company operates entirely online, with no physical retail locations.
- US Mobile wanted to move to a scalable, microservices-based approach, so it decided to rewrite its core applications on MongoDB, an open source NoSQL database.
The Challenges: RDBMS to NoSQL (MongoDB)
Rewriting US Mobile’s business applications to run on MongoDB was just the beginning. Upon completion of that effort, in November 2016, the second, analytical phase of the project began. Before Melmed was hired, US Mobile engineers were the only employees who had access to data. They built out essential, customer-coded sales, marketing, and financial reports and they made these reports accessible to managers through an administrative portal. In his first few months with US Mobile, Melmed supplemented these administrative reports with dozens of ad hoc analyses requested by US Mobile business users. He used Data Grip, an SQL querying tool, against MariaDB and built sharable analyses in Google Spreadsheets or Microsoft Excel. The more popular of these analyses were scheduled and rerun as needed, but Melmed spent a lot of time on manual preparation and updates because users lacked self-service options.
Once the MongoDB migration was completed, it was up to Melmed to rebuild US Mobile’s analytical capabilities from scratch. The trouble was that ANSI (American National Standards Institute) SQL, the venerable query language used with relational databases, is not generally supported in MongoDB or other NoSQL databases. Instead, there are alternative query languages and capabilities that lack the breadth, depth and power of SQL. MongoDB, for example, offers an aggregation framework for summarizing large groups of data, a query language, a MapReduce API and a Find API. It also offers a Spark Connector and a BI Connector for tools like Tableau and Business Objects. US Mobile didn’t use Spark or have an incumbent BI tool, so these options didn’t meet Melmed’s needs.
“The MongoDB Query Language wasn’t too difficult to pick up when looking at one collection, but it didn’t seem to have all that much in common with SQL,” says Melmed. “The complexity [of querying] was amplified when I started to look at MongoDB’s MapReduce and aggregate functions.”
As Melmed went deeper into MongoDB-native analytical options, the lack of multi-set join capabilities, commonly used in the SQL world, made it particularly difficult to analyze customer data, which US Mobile stores in one MongoDB collection, together with mobile device and SIM card data, which are handled in separate collections within MongoDB. “I needed to bring our subscriber data together with the devices that they’re using and the plan purchases that they’re making, but the trouble just with these few joins was a big hurdle,” Melmed explains.
Migrating to MongoDB Was Initially Good for Developers. Not for BI.
- Once the MongoDB migration was completed, it was up to Melmed to rebuild US Mobile’s analytical capabilities from scratch.
- As Melmed went deeper into MongoDB-native analytical options, the lack of multi-set join capabilities, commonly used in the SQL world, made it particularly difficult to analyze customer data.
The Solution: Native MongoDB Anlalytics vis SlamData
Melmed considered a number of options for getting data out of MongoDB to drive sophisticated analytics. The first (and most common) path considered was extracting, transforming and flattening data from MongoDB into a relational model and then loading it into a relational database. This is a time- and labor-intensive path and many practitioners then struggle with poorly transformed data that is either too complicated and repetitive or oversimplified and missing detail. Melmed also considered popular, developer-oriented third-party tools including Studio 3T (a tool he still uses for browsing and filtering MongoDB data sets), but none of them met his analytical needs.
The next option explored was SlamData. At the heart of SlamData is a compiler that translates ANSI-standard SQL into query languages and API calls native to NoSQL platforms, including MongoDB, Couchbase, MarkLogic and HDFS, through Apache Spark. SlamData has also extended SQL with additional operators to query and drill down into the nested arrays and subdocuments common in NoSQL platforms and that are key to their data flexibility. Finally, the software also includes the kind of reporting, charting, dashboarding and data visualization capabilities typically found in relational analytics and BI products.
Melmed downloaded a trial version of SlamData and installed and started using it over a weekend. By the next week, he was negotiating with SlamData to set up a commercially supported production deployment.
“It felt like a light switch was turned on when I saw that I could keep using my SQL skills but with direct access to the [NoSQL] database,” Melmed says.
Melmed was able to set up the pilot deployment singlehandedly, using credentials to access US Mobile’s MongoDB database. The production deployment required the support of engineers, who set up a replica of the production database to eliminate the possibility of SlamData queries affecting mission-critical database performance. The engineers also synced the SlamData software with US Mobile’s Google single-sign-on services to ensure authorized access. Despite these extra steps, the SlamData production deployment was completed in a single day.
Melmed immediately began work on new reports and analyses. He does this work singlehandedly, with no need for help from engineers to connect or prepare data. As for the users of these reports, a key appeal of SlamData turned out to be the ability to create interactive, self-service analyses that users can filter and run at will against the latest data in MongoDB.
“Before SlamData, my approach was to answer the questions of the moment from different users and departments,” Melmed explains. “SlamData’s ability to let users refresh reports on demand and use interactive features to filter the data has changed my way of thinking. Now when I build a report, I try to think ahead and build something that will be useful for ongoing analysis.”
Today, when US Mobile employees ask questions that are similar to questions that have been asked before, Melmed can just point to a report that already exists or he can easily tweak that report to add new dimensions of data or filters. One report that has seen long- term use was created for the product team to track adoption of US Mobile’s AutoPay feature (see Figure 1). The analysis was requested in the wake of a user-interface upgrade, but Melmed thought long term and built a report to track activations over time, with user- adjustable date ranges and selectable tabs for activations, activations with AutoPay and a key performance metric for total activations including the AutoPay feature.
Behind the scenes, Melmed used SQL statements to query the relevant variables and applied SlamData charting to format bar charts and key performance indicators (KPI). The AutoPay analysis is simple, but Melmed says it would be easy to go deeper to analyze what types of customers are or aren’t choosing AutoPay by adding variables on the talk, text and data plans that customers are choosing.
Melmed routinely adds a SlamData data-export feature to reports so users can drill down and extract specific subsets of data. Marketing, for example, can target specific customer segments with campaigns about relevant new features and upgrades. This is another example of interactive functionality that anticipates the self-service needs of business users.
“When I first saw SlamData, the key selling point was being able to use SQL,” Melmed says, “but the interactivity I’ve been able to build into reports has turned out to be extremely important.”
0 to 60 In 10 Minutes
- Melmed considered a number of options for getting data out of MongoDB to drive sophisticated analytics.
- Melmed downloaded a trial version of SlamData and installed and started using it over a weekend. By the next week, he was negotiating with SlamData to set up a commercially supported production deployment.
The Impact of Implementing SlamData
Nine months after US Mobile started using SlamData, Melmed has created hundreds of direct analyses of the data in MongoDB. The payoff on the SlamData subscription began with Melmed’s ability to use SQL, a language with which he and millions of other data professionals are familiar. In the case of MongoDB, SlamData compiles SQL queries and translates them into the supported query capabilities expressed in the database’s Aggregation, MapReduce and Find APIs. Some of Melmed’s most sophisticated analyses include multi-way joins that would have to be handled with multiple lookup stages or MapReduce in MongoDB. In SlamData, Melmed doesn’t have to get into API calls or unfamiliar MapReduce procedures. He does it all in SQL without help from engineers.
Beyond the productivity of working in SQL, Melmed says he has built SlamData’s self- service refresh and interactive filtering capabilities into more than 50 reports. This has slashed the number of overlapping and repetitive requests for analyses. Most of these reports are seeing long-term, day-to-day use, with US Mobile employees filtering down to the insights of interest, refreshing at will to see up-to-the-minute results and, in many cases, downloading subsets of data for marketing campaigns, customer-service initiatives and custom spreadsheet analysis.
“The ROI has been in saving my time building, refreshing and tweaking reports and saving the time of engineers, who would otherwise have to build custom reports for our administrative portal,” says Melmed.
Each report that gets repeated, long-term use saves Melmed five minutes to 30 minutes per week that would otherwise be spent responding to requests, he says. And for each analysis he can create on his own, Melmed estimates he’s saving a full day of an engineer’s time that would otherwise be spent on custom report development.
“I think it is safe to say that our annual cost for SlamData was paid for in savings within three months,” he concludes.
Additional author subscriptions are $1,500 per year, but there are no extra charges for data or interactive report consumers. The majority of SlamData customers, including US Mobile, run on a single server instance. SlamData says its largest customers and those running lots of sophisticated, interactive analyses opt for a second server instance to ensure fast query performance.
ROI In 3 Months
- Nine months after US Mobile started using SlamData, Melmed has created hundreds of direct analyses of the data in MongoDB.
- Annual license pays for itself in 3 months.
Here’s a recap of US Mobile’s journey along with best-practice advice and forward-looking insight on next steps for US Mobile and would- be SlamData customers:
Many organizations that embrace NoSQL databases for their agility and scalability assume that they have to convert that information back into flat, rigidly modeled and structured data in order to develop reports and analyses. What many discover when flattening NoSQL data is that nuances of nested detail are either rendered in cumbersome, repetitive structures that are hard to transform or the data is oversimplified and details are lost.
US Mobile learned that it’s possible, with SlamData, to use standard SQL to query a NoSQL database without having to learn a new query language, developer-oriented API calls, MapReduce or other data-scientist-level techniques or languages.
A second lesson learned at US Mobile is that it’s best to build reports for broad, long-term use, looking beyond questions of the moment to the questions that specific types of users and departments will need to ask every day. The approach reduces redundant and overlapping query requests and lets report authors focus on crucial new data-driven insights that might drive data-monetization opportunities and business innovation.
For every report that US Mobile can build without the help of engineers, the company is saving at least a day of work from a highly paid developer. And for every report that it can design for long-term use, exploiting interactive filtering, data selection, refresh and data- export functionalities, US Mobile is saving five minutes to 30 minutes of report author or power-user time per week. Melmed says that these savings returned US Mobile’s annual subscription cost for SlamData within three months. US Mobile did not detail its expenditures, but standard subscription costs for SlamData are $15,000 per year for a single server instance with one author. Additional author subscriptions are $1,500 per year, but there are no extra charges based on data capacity, data downloads or numbers of consumers of interactive reports.
The majority of US Mobile’s SlamData analyses are returned within a few seconds or less, according to Melmed. More sophisticated analyses, such as a dashboard with upwards of a dozen visualizations with multi-way joins behind the scenes, initially took as long as a minute to render. Melmed says he applied SlamData’s caching capabilities and suggested best practices on join sequencing to improve the performance of more sophisticated reports. In the SQL world, it’s common for SQL queries to get quite long and complex, with embedded subqueries. The best practice with SlamData is to take a data-pipeline approach, breaking down the elements of a sophisticated query into a sequence of simpler queries. SlamData offers documentation and support resources on this topic.
What To Avoid
Don’t assume that relational data must remain in the relational world or that relational and NoSQL data can’t mix. Constellation spoke to Ocasta, a SlamData customer that built a question-and-answer application for a retailer in which data on registered user names and their roles and locations were stored in a relational database. The question-and-answer aspects of the app were handled in MongoDB, which could flexibly handle changes, additions and variations in questions. Ocasta’s customer wanted to track the Q & A performance of specific users over time, so the app developer made a point of capturing and adding these relational dimensions of data to the MongoDB record. With this approach, organizations can make the most of NoSQL flexibility, capturing all data required for an analysis inside the NoSQL platform and using SlamData for the analysis.
The New To-Do List
- Organizations can use SlamData and avoid having to learn a new query language, developer-oriented API calls, MapReduce or other data-scientist-level techniques or languages.
- BI architecture should focus on scale — not one-off or ad hoc query writing and reports.
- Don’t assume that relational data must remain in the relational world or that relational and NoSQL data can’t mix.
Those adopting NoSQL databases should fully embrace the flexible data management, agile development and scalability of these platforms. Constellation finds that application- development, data-management and analytics professionals who have no experience with NoSQL products greatly benefit from adequate training and experimentation before venturing into major new projects. It takes time and a bit of experience to wrap one’s mind around what is a very different approach to working with data in the NoSQL world. Fortunately, NoSQL databases are very flexible, adaptable and forgiving, so developers won’t face the same schema-change and broken-functionality nightmares they would face when making rookie mistakes with a relational database.
To make the most of a NoSQL platform, Constellation also recommends that developers and data and analytics professionals learn about the native capabilities of the database. SlamData converts SQL queries to operate against database- native query languages and APIs, so it’s helpful to know what’s possible and what’s simple versus what’s complex for the underlying database.
US Mobile found that its more complex queries
— ones that would have required MapReduce techniques in the underlying database – might suffer from slow query performance.
Performance improved when US Mobile applied caching and best-practice query sequencing techniques. Melmed might have implemented these techniques in advance had he known what to expect, but there’s no substitute for experience when it comes to learning how to make the most of any technology.
Inevitably, organizations that are heavily invested in relational BI and analytics tools will want to analyze combinations of NoSQL and relational data. For now, SlamData is geared to developing analytics on NoSQL platforms including MongoDB, Couchbase, MarkLogic and HDFS through Apache Spark. The vendor’s ambition is to provide a platform that’s suitable for broad analytical capabilities against all data. With this in mind, it has on its roadmap connectors to relational sources, starting with PostgreSQL, MySQL and popular commercial databases, as soon as early 2018. Also on the roadmap are machine learning capabilities, with an eye toward predictive analyses. Finally, it also plans to add a columnar store to support analyses not supported in database-native APIs and query languages.
Constellation sees SlamData as an SQL-friendly option for developing descriptive and diagnostic analytics against NoSQL sources. Those looking to blend NoSQL and relational data without the drawbacks of moving and flattening data should keep a close eye on this vendor’s coming product releases and talk to reference customers about their success.
What and What Not To Do
- Data analytics professionals should adopt native NoSQL analytics solutions.
- Organizations that are heavily invested in relational BI and analytics tools will want to analyze combinations of NoSQL and relational data.
Doug Henschen is Vice President and Principal Analyst at Constellation Research, Inc., focusing on data- driven decision making. His Data-to-Decisions research examines how organizations employ data analysis to reimagine their business models and gain a deeper understanding of their customers. Data insights also figure into tech optimization and innovation in human-to-machine and machine-to-machine business processes in manufacturing, retailing and services industries.
Henschen’s research acknowledges the fact that innovative applications of data analysis require a multi- disciplinary approach, starting with information and orchestration technologies, continuing through business intelligence, data visualization, and analytics, and moving into NoSQL and Big Data analysis, third-party data enrichment, and decision management technologies. Insight-driven business models and innovations are of interest to the entire C-suite.
Previously, Henschen led analytics, Big Data, business intelligence, optimization, and smart applications research and news coverage at InformationWeek. His experiences include leadership in analytics, business intelligence, database, data warehousing, and decision-support research and analysis for Intelligent Enterprise. Further, Henschen led business process management and enterprise content management research and analysis at Transform magazine. At DM News, he led the coverage of database marketing and digital marketing trends and news.
About Constellation Research
About Constellation Research
Constellation Research is an award-winning, Silicon Valley-based research and advisory firm that helps organizations navigate the challenges of digital disruption through business models transformation and the judicious application of disruptive technologies. Unlike the legacy analyst firms, Constellation Research is disrupting how research is accessed, what topics are covered and how clients can partner with a research firm to achieve success. Over 350 clients have joined from an ecosystem of buyers, partners, solution providers, C-suite, boards of directors and vendor clients. Our mission is to identify, validate and share insights with our clients.