Interview Question: Tell us about a recent challenge that you faced in your current role, and how you overcame it.
Problem: To develop a microservice-based high-performance pricing execution engine (In the cloud with concurrent processing, scalability, and observability) to solve complex configure price quote (CPQ), use cases by processing the data of the Salesforce platform (Current eco-system falls short in terms of performance).
Use case: Large global enterprise companies often deal with significant amounts of complexity when creating configurations and pricing within their quoting process. They have to work with hundreds of thousands of pricing rules and configuration rules each time they create or update a quote.
Brainstorming: We had brainstormed with the team and break down the problem statement into smaller pieces/modules, and everyone has taken end-to-end ownership/responsibility of the one module. here I have taken ownership of observability.
On my Plate: I have to find out the most suitable solution for observability that support involves measuring, collecting, and analyzing various diagnostics signals from a distributed system. These signals may include metrics, traces, logs, events, profiles.
My Challenges: Develop an end-to-end diligent plan/roadmap for Observability under high uncertainty, and demonstrate how to implement observability throughout the organization from scratch. I visualized there was a steep learning curve.
Team Immediate thought: As we are developers who have experience in the Microsoft technology stack. In front of us, we have an easy/time-tested/proven approach to use Azure Application insight and buy the Phobos NuGet for Enterprise Application Monitoring using Akka.NET for Actor monitoring, Concurrent processing, Distributed Tracing.
Action/Overcome: I was made a decision to evaluate/explore the best option for observability.
I had made the to-do list as per attention-to-details and a sense of urgency, it helps me to navigate, stay prioritized, and be on track.
- Made the holistic roadmap with a deadline timeline.
- Technology Evaluation: Find out the Observability technologies Landscape which solves the use case. In the journey, I have learned OpenTracing, Akka.NET Phobos, Microsoft Application Insights, StatsD, Graphite time-series database, Grafan, Prometheus, Jaeger, Zipkin, Elasticsearch, Kubernetes, deployment of the container, Akka.NET framework, and persistence for observability.
- Proof of Concept (POC): Write the code for multiple POC for Observability how to support trace, logs, metrics for application performance measures, and trace distributed calls.
- Documentation: Made the documentation for How to integrate the Observability Nuget in other microservices.
- Demo to a Larger audience in Organization: Based on the most suitable technology of the observability use case, I had given the demo on OpenTracing.io standard. I am happier to answered to team’s questions on Observability.
- Draw the roadmap - Observability journey has started to implement its own reusable observability Nuget (Which contains metric and tracing telemetry) in an application, demonstrate the application performance gain in observability tool using metrics show in Grafana, and made the support team more efficient to find out issue by analyzing the distributed trace in Jeager.
Achievement: Developed solution designs in collaboration with software architects that improved product support team efficiency by 100% and made it easy for a development team to debug applications.
