Case study
Digital Retail Platform with a Data Lake, Telemetry Visualization, & Analytics
We’ve built and optimized a consumer engagement platform for data analytics in retail to ensure accurate statistics and data-driven decision-making
Business challenge of a Fortune 500 enterprise
Our client, a Fortune 500 multinational retailer with a presence in over 180 countries, was seeking to orchestrate their next-generation consumer engagement ecosystem for selling revolutionary new products. The platform’s microservice architecture and complex infrastructure encompassed numerous independent components built on diverse technologies and developed by different remote teams — and even vendors — across the globe.
Our client’s Technical Management Department was missing a clear overview of all interactions between separate systems within the complex network of retail data acquisition services. To achieve that degree of visibility, they needed a holistic system that would monitor performance, applications, and infrastructure. They also needed to build a centralized data lake management software for logging and telemetry data as well as develop a set of clearly defined rules for writing and managing logs from microservices.
BitsByteSoft caught the eye of our client as a strong and trustworthy engineering partner. They started by assigning us a pilot project for collecting, storing, and visualizing telemetry data.
Key features
Collect and store huge data sets
Visualize telemetry & customer data
Detect anomalies and ensure security
Industry:Retail
Market: Global
Team size: 20 engineers
Cooperation: July 2017 – January 2020
Technologies:
AWS / Elasticsearch / Grafana / HashiCorp / Java / JavaScript / Kafka / Kibana / Kubernetes / Node.JS / Python / Salesforce / Terraform
Retail data solution delivered
The BitsByteSoft team of developers, DevOps, and QA engineers soon got a handle on our client’s business and technical needs and came up with a detailed roadmap for project development. After this roadmap was approved by our client’s Principal Architect, our team set to work, improving, building on, and testing the platform.
Data lake
We started by building and managing our client’s data lake for collecting, accumulating, and storing telemetry data and logs from the platform’s applications and microservices. The types of metrics gathered ranged from business and sales data to technical information.
Logging guidelines
Another of our team’s goals was to establish firm practices that would address the lack of a unified approach to logging events from microservices in the data storage. We created a centralized Elasticsearch system for writing logs, defined logging rules, and developed practical guides to be used by all other teams.
The introduction of guidelines for logging practices became a real game changer for our client’s organization, bringing all teams onto the same page and ensuring well-coordinated platform management.
Data visualization
We were also responsible for creating and visualizing interconnections between the platform’s systems to provide our client’s Technical Department with a view of how the platform operates across regions, countries, and teams. Our team empowered our client’s S&M Department with retail sales analytics to keep track of the whole sales cycle, from order to delivery. Using Elasticsearch, we formalized the approach to tracing transactions and implemented data visualization using Kibana.
In effect, we built a powerful monitoring system with custom dashboards for visualizing collected telemetry data on business and technical processes. Data visualized includes the number of registered users, the number of products ordered and purchased, the customer journey flow, absolute sales records, website attendance, time spent on the site, domain redirects, technical incidents and errors, resolved urgent requests, information on servers, etc.
Anomaly detection and an alert & notification system
We set up a comprehensive monitoring tool for analytics in retail industry that enables anomaly detection and proactive incident resolution. Incidents are classified according to the seriousness of impact on the system’s performance. In case an anomaly or malfunction is detected, support teams receive a notification and can quickly act on it.
Thanks to an efficient alerting system that our team implemented, the speed of response to critical incidents decreased from several hours to a few minutes.
DevOps
Our DevOps engineers oversaw the whole infrastructure management process, performing deployment activities, implementing a new version of the system, dealing with a number of integrations, and supporting custom retail analytics solutions within our client’s platform.
Business outcome
Our team provided stability, security, and failover capabilities that our client’s platform was lacking. This allowed our partner to extend the geography of their global presence and accelerate the time to market of their innovative products. We fine-tuned communications between the platform’s scattered systems, built a scalable data lake architecture, and provided efficient visualization tools for monitoring and troubleshooting.
One of our main contributions to our client’s business success was in acting as a software vendor inside the organization, delivering multiple out-of-the-box solutions to our client’s internal teams around the world.
This initial joint effort between BitsByteSoft and our client has evolved into a long-lasting win-win collaboration. The successful outcome of this partnership has resulted in BitsByteSoft helping our client with two other large projects.
Let’s work together on your
next project
Drop us a line about your project at contact@bitsbytesoft.com or via the contact form below with attachments, and we will contact you within a business day.