Elasticsearch - Nodes, clusters, and shards

Elastic Stack Video - Load your gun in short time.

 
Beginner's Crash Course to Elastic Stack Playlist
Nodes, clusters, and shards in Elasticsearch - S1E3:Mini Beginner's Crash Course

What is Ingress

Ingress.

Enabling traffic.

• Openning the cluster to receive external traffic.

Traffic routing

• Defining the traffic routes to backend services.

Traffic Reliability

• Ensure Reliable, Secure communication.

Orchestration Tools

Orchestration Tools.

Orchestration - automation that supports processes and workflows, such as provisioning resources.

• Scale up and scale down applications on request
• Auto scale applications based on usage
• Create self-healing systems by spinning down unhealthy node and replacing them with new ones

Orchestration Tools Examples

• Docker Swarm
• kubernetes
• Zookeeper
• Terraform

Configuration Management Tools

Configuration Management Tools.

What is configuration Management?

• Resolves configuration drift
• Maintaining desired state
• The process of systematically handling chnages to a system
• Maintain integrity of a system over time
• Automation plays an key role
  • Puppet
  • Ansibel
  • Chef
  • Salt

Examples

Ansible

• Open Source
• Declarative configuration
• YAML configuration files
• No control server needed - but ansible tower available
• No agents needed, just python and ssh

Puppet

• Declarative configuration
• Manage state through a UI
• Custom modules use puppet DSL
• Pushes chnages to clients using control server and agents installed on clients.

Chef

• Procedural configuration
• Agent/Server
• Uses chef DSL

Salt

• Declarative configuration
• Agent (minions) /Server (Master) - but can support agentless
• Uses YAML
• Support for event-driven automation

Note:
Idempotent Behavior - is an operation that can be applied multiple times without changing the result beyound the initial application. E.g. 1 x 1 = 1
- Configuration management tools will avoid reapeating tasks.
- The desired state is maintained even if you run it multiple times.

Why use Configuration Management?

• Quick provisioning of New servers
• Quick Recovery from Critical Events
• No more snowflake
• Version controlfor the server Envirnment
• Replaced Environment

Build Automation Tools

Build Automation Tools.

Examples

• Java: ant, maven, gradle
• JavaScript: npm, grunt, gulp
• Packer: build machine images and containers

DevOps Tools Periodic Table

Search the world's most comprehensive index of full-text books.

The Periodic Table of DevOps Tools is the industry's go-to resource for identifying best-of-breed tools across the software delivery lifecycle

DevOps Periodic table - https://digital.ai/learn/devops-periodic-table/

Recent challenge that you faced

Interview Question: Tell us about a recent challenge that you faced in your current role, and how you overcame it.

Problem: To develop a microservice-based high-performance pricing execution engine (In the cloud with concurrent processing, scalability, and observability) to solve complex configure price quote (CPQ), use cases by processing the data of the Salesforce platform (Current eco-system falls short in terms of performance).

Use case: Large global enterprise companies often deal with significant amounts of complexity when creating configurations and pricing within their quoting process. They have to work with hundreds of thousands of pricing rules and configuration rules each time they create or update a quote.

Brainstorming: We had brainstormed with the team and break down the problem statement into smaller pieces/modules, and everyone has taken end-to-end ownership/responsibility of the one module. here I have taken ownership of observability.

On my Plate: I have to find out the most suitable solution for observability that support involves measuring, collecting, and analyzing various diagnostics signals from a distributed system. These signals may include metrics, traces, logs, events, profiles.

My Challenges: Develop an end-to-end diligent plan/roadmap for Observability under high uncertainty, and demonstrate how to implement observability throughout the organization from scratch. I visualized there was a steep learning curve.

Team Immediate thought: As we are developers who have experience in the Microsoft technology stack. In front of us, we have an easy/time-tested/proven approach to use Azure Application insight and buy the Phobos NuGet for Enterprise Application Monitoring using Akka.NET for Actor monitoring, Concurrent processing, Distributed Tracing.

Action/Overcome: I was made a decision to evaluate/explore the best option for observability.

I had made the to-do list as per attention-to-details and a sense of urgency, it helps me to navigate, stay prioritized, and be on track.

1. Made the holistic roadmap with a deadline timeline.
2. Technology Evaluation: Find out the Observability technologies Landscape which solves the use case. In the journey, I have learned OpenTracing, Akka.NET Phobos, Microsoft Application Insights, StatsD, Graphite time-series database, Grafan, Prometheus, Jaeger, Zipkin, Elasticsearch, Kubernetes, deployment of the container, Akka.NET framework, and persistence for observability.
3. Proof of Concept (POC): Write the code for multiple POC for Observability how to support trace, logs, metrics for application performance measures, and trace distributed calls.
4. Documentation: Made the documentation for How to integrate the Observability Nuget in other microservices.
5. Demo to a Larger audience in Organization: Based on the most suitable technology of the observability use case, I had given the demo on OpenTracing.io standard. I am happier to answered to team’s questions on Observability.
6. Draw the roadmap - Observability journey has started to implement its own reusable observability Nuget (Which contains metric and tracing telemetry) in an application, demonstrate the application performance gain in observability tool using metrics show in Grafana, and made the support team more efficient to find out issue by analyzing the distributed trace in Jeager.

Achievement: Developed solution designs in collaboration with software architects that improved product support team efficiency by 100% and made it easy for a development team to debug applications.