Orchestration Tools

Orchestration Tools.

Orchestration - automation that supports processes and workflows, such as provisioning resources.

• Scale up and scale down applications on request
• Auto scale applications based on usage
• Create self-healing systems by spinning down unhealthy node and replacing them with new ones

Orchestration Tools Examples

• Docker Swarm
• kubernetes
• Zookeeper
• Terraform

Configuration Management Tools

Configuration Management Tools.

What is configuration Management?

• Resolves configuration drift
• Maintaining desired state
• The process of systematically handling chnages to a system
• Maintain integrity of a system over time
• Automation plays an key role
  • Puppet
  • Ansibel
  • Chef
  • Salt

Examples

Ansible

• Open Source
• Declarative configuration
• YAML configuration files
• No control server needed - but ansible tower available
• No agents needed, just python and ssh

Puppet

• Declarative configuration
• Manage state through a UI
• Custom modules use puppet DSL
• Pushes chnages to clients using control server and agents installed on clients.

Chef

• Procedural configuration
• Agent/Server
• Uses chef DSL

Salt

• Declarative configuration
• Agent (minions) /Server (Master) - but can support agentless
• Uses YAML
• Support for event-driven automation

Note:
Idempotent Behavior - is an operation that can be applied multiple times without changing the result beyound the initial application. E.g. 1 x 1 = 1
- Configuration management tools will avoid reapeating tasks.
- The desired state is maintained even if you run it multiple times.

Why use Configuration Management?

• Quick provisioning of New servers
• Quick Recovery from Critical Events
• No more snowflake
• Version controlfor the server Envirnment
• Replaced Environment

Build Automation Tools

Build Automation Tools.

Examples

• Java: ant, maven, gradle
• JavaScript: npm, grunt, gulp
• Packer: build machine images and containers

DevOps Tools Periodic Table

Search the world's most comprehensive index of full-text books.

The Periodic Table of DevOps Tools is the industry's go-to resource for identifying best-of-breed tools across the software delivery lifecycle

DevOps Periodic table - https://digital.ai/learn/devops-periodic-table/

Recent challenge that you faced

Interview Question: Tell us about a recent challenge that you faced in your current role, and how you overcame it.

Problem: To develop a microservice-based high-performance pricing execution engine (In the cloud with concurrent processing, scalability, and observability) to solve complex configure price quote (CPQ), use cases by processing the data of the Salesforce platform (Current eco-system falls short in terms of performance).

Use case: Large global enterprise companies often deal with significant amounts of complexity when creating configurations and pricing within their quoting process. They have to work with hundreds of thousands of pricing rules and configuration rules each time they create or update a quote.

Brainstorming: We had brainstormed with the team and break down the problem statement into smaller pieces/modules, and everyone has taken end-to-end ownership/responsibility of the one module. here I have taken ownership of observability.

On my Plate: I have to find out the most suitable solution for observability that support involves measuring, collecting, and analyzing various diagnostics signals from a distributed system. These signals may include metrics, traces, logs, events, profiles.

My Challenges: Develop an end-to-end diligent plan/roadmap for Observability under high uncertainty, and demonstrate how to implement observability throughout the organization from scratch. I visualized there was a steep learning curve.

Team Immediate thought: As we are developers who have experience in the Microsoft technology stack. In front of us, we have an easy/time-tested/proven approach to use Azure Application insight and buy the Phobos NuGet for Enterprise Application Monitoring using Akka.NET for Actor monitoring, Concurrent processing, Distributed Tracing.

Action/Overcome: I was made a decision to evaluate/explore the best option for observability.

I had made the to-do list as per attention-to-details and a sense of urgency, it helps me to navigate, stay prioritized, and be on track.

1. Made the holistic roadmap with a deadline timeline.
2. Technology Evaluation: Find out the Observability technologies Landscape which solves the use case. In the journey, I have learned OpenTracing, Akka.NET Phobos, Microsoft Application Insights, StatsD, Graphite time-series database, Grafan, Prometheus, Jaeger, Zipkin, Elasticsearch, Kubernetes, deployment of the container, Akka.NET framework, and persistence for observability.
3. Proof of Concept (POC): Write the code for multiple POC for Observability how to support trace, logs, metrics for application performance measures, and trace distributed calls.
4. Documentation: Made the documentation for How to integrate the Observability Nuget in other microservices.
5. Demo to a Larger audience in Organization: Based on the most suitable technology of the observability use case, I had given the demo on OpenTracing.io standard. I am happier to answered to team’s questions on Observability.
6. Draw the roadmap - Observability journey has started to implement its own reusable observability Nuget (Which contains metric and tracing telemetry) in an application, demonstrate the application performance gain in observability tool using metrics show in Grafana, and made the support team more efficient to find out issue by analyzing the distributed trace in Jeager.

Achievement: Developed solution designs in collaboration with software architects that improved product support team efficiency by 100% and made it easy for a development team to debug applications.

Machine Learning

Machine Learning.

A category of artificial intelligence that uses algorithms to analyze data, learn from the data, and then make inferences about the data.

Use cases

• Mark email as spam or not spam
• Categorize articles as technology, sports or politics
• Determine if test is negative or positive
• Facial recognition
• Fraud detection/prevention
• Revenue predictions
• Success rate measurements
• Credit scoring

Data Analytics VS Data Science

Data Analytics	Data Science
Definition Process and analyze data sets to model and communicate answer to specific questions with actionable data.	A multi-disciplinary field that specializes in exposing insights from large data sets.
Skills Statistics Computer Science AI (machine learning, deep learning, etc.)	Scientific method Advanced mathematics Computer Science AI (machine learning, deep learning, etc.) Data Analysis
Goals Provide solution to known unknowns Answer specific questions	Understand phenomena Expose unknown unknowns To identify the right questions to ask Find better ways to analyze data

Programming eBooks - read online

Search the world's most comprehensive index of full-text books.

Programming eBooks - https://books.google.co.in/books

Elasticsearch _msearch

When using the msearch (multi-search) API in Elasticsearch, you can search across multiple indices in a single request. Each search query in the msearch request can be targeted to a specific index..

Create Elasticsearch Index

PUT product-index
{
    "settings": {
        "analysis": {
            "normalizer": {
                "lowercaseNormalizer": {
                    "type": "custom",
                    "filter": [
                        "lowercase",
                        "asciifolding"
                    ]
                }
            }
        }
    },
    "mappings": {
        "properties": {
            "Product": {
                "type": "object",
                "properties": {
                    "ProductName": {
                        "type": "keyword",
                        "normalizer": "lowercaseNormalizer"
                    },
                    "ProductCode": {
                        "type": "keyword",
                        "normalizer": "lowercaseNormalizer"
                    },
                    "ProductDescription": {
                        "type": "text"
                    },
                    "ProductFamily": {
                        "type": "keyword",
                        "normalizer": "lowercaseNormalizer"
                    },
                    "Color": {
                        "type": "keyword"
                    }
                }
            }
        }
    }
}     
       

Adding sample data/records

        
#Add first Document in elasticsearch 
PUT /product-index/_doc/1
{
    "Product": {
        "ProductName": "CBC Anti Virus",
        "ProductCode": "Network",
        "ProductDescription": "APAC",
        "ProductFamily": "Sales Cloud",
        "Color": "Silver"
    }
}


PUT /product-index/_doc/2
{
    "Product": {
        "ProductName": "CBC Esoteric",
        "ProductDescription": "CBC Panel - 1",
        "ProductFamily": "Hardware",
        "Color": "BLUE"
    }
}

PUT /product-index/_doc/3
{
    "Product": {
        "ProductName": "CBC Oncology",
        "ProductCode": "CBC Health",
        "ProductFamily": "Software"
    }
}

PUT /product-index/_doc/4
{
    "Product": {
        "ProductName": "CBC Panel",
        "ProductCode": "Secure Wifi",
        "ProductDescription": "Only One Charge Type"
    }
}

PUT /product-index/_doc/5
{
    "Product": {
        "ProductName": "CBC Panel (Invitro)",
        "ProductCode": "Secure Wifi",
        "ProductDescription": "CBC Panel",
        "ProductFamily": "Hardware",
        "Color": "GREEN"
    }
}


PUT /product-index/_doc/6
{
    "Product": {
        "ProductName": "CBC Panel - Beijing",
        "ProductCode": "CBC Panel",
        "ProductDescription": "Only One Charge Type",
        "ProductFamily": "Software",
        "Color": "BLUE"
    }
}


PUT /product-index/_doc/7
{
    "Product": {
        "ProductName": "Esoteric",
        "ProductCode": "CBC",
        "ProductFamily": "Marketing Cloud",
        "Color": "WHITE"
    }
}


PUT /product-index/_doc/8
{
    "Product": {
        "ProductName": "Oncology",
        "ProductCode": "Network",
        "ProductDescription": "CBC",
        "ProductFamily": "Sales Cloud",
        "Color": "WHITE"
    }
}

        

Create Elasticsearch Index

PUT order-index
{
    "settings": {
        "analysis": {
            "normalizer": {
                "lowercaseNormalizer": {
                    "type": "custom",
                    "filter": [
                        "lowercase",
                        "asciifolding"
                    ]
                }
            }
        }
    },
    "mappings": {
        "properties": {
            "Order": {
                "type": "object",
                "properties": {
                    "OrderCode": {
                        "type": "keyword",
                        "normalizer": "lowercaseNormalizer"
                    },
                    "Description": {
                        "type": "text"
                    },
                    "Family": {
                        "type": "keyword",
                        "normalizer": "lowercaseNormalizer"
                    },
                    "ProductName": {
                        "type": "keyword",
                        "normalizer": "lowercaseNormalizer"
                    },
                    "ProductId": {
                        "type": "keyword"
                    }
                }
            }
        }
    }
}    
       

Adding sample data/records

        
#Add first Document in elasticsearch 
PUT /order-index/_doc/5c57754b-86a6-41f1-a4af-de65b531fe8f
{
    "Order": {
        "OrderCode": "Network",
        "Description": "APAC",
        "Family": "Sales Cloud",
        "ProductName": "CBC Anti Virus",
        "ProductId": 1
    }
}


PUT /order-index/_doc/218c4907-89c2-482f-90cc-61dd7e27f999
{
    "Order": {
        "OrderCode": "Network",
        "Description": "CBC Panel - 1",
        "Family": "Hardware",
        "ProductName": "CBC Esoteric",
        "ProductId": 2
    }
}

 

        

View data/regcords

GET /product-index/_search
GET /order-index/_search

Search case-insensitive records

In the request body, provide the search queries in a newline-separated JSON format. Each search query should have two parts: the header and the search body. The header contains information about the index and search type, while the search body contains the actual search query DSL.

#Try 1
POST http://localhost:9200/_msearch
Content-type:application/json
Payload as below

{"index" : "product-index"}
{"query": {"match": {"Product.OrderCode": "Network"}}, "from" : 0, "size" : 10}
{"index" : "order-index"}
{"query": {"match": {"Order.OrderCode": "Network"}}}

       

Note

• Text fields are analyzed.
• For keyword fields we need to add custom elasticsearch setting for analysis.

Reference

• run elasticsearch locally run elasticsearch locally
• Documentation: Elasticsearch - _msearch

Multiple Index Query

The following request searches the product-index and order-index indices.

Create Elasticsearch Index

PUT product-index
{
  "settings": {
    "analysis": {
      "normalizer": {
        "lowercaseNormalizer": {
          "type": "custom",
          "filter": ["lowercase", "asciifolding"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "ProductName": {
        "type": "keyword",
        "normalizer": "lowercaseNormalizer"
      },
     "ProductCode": {
        "type": "keyword",
        "normalizer": "lowercaseNormalizer"
      },
      "ProductDescription": {
        "type": "text"
      },
      "ProductFamily": {
        "type": "keyword",
        "normalizer": "lowercaseNormalizer"
      },
      "Color": {
        "type": "keyword"
      }
    }
  }
}       
       

Adding sample data/records

        
#Add first Document in elasticsearch 
PUT /product-index/_doc/1
{
  "ProductName": "CBC Anti Virus",
  "ProductCode": "Network",
  "ProductDescription": "APAC",
  "ProductFamily": "Sales Cloud",
  "Color": "Silver"
}


PUT /product-index/_doc/2
{
  "ProductName": "CBC Esoteric",
  "ProductDescription": "CBC Panel - 1",
  "ProductFamily": "Hardware",
  "Color": "BLUE"
}

PUT /product-index/_doc/3
{
  "ProductName": "CBC Oncology",
  "ProductCode": "CBC Health",
  "ProductFamily": "Software"
}

PUT /product-index/_doc/4
{
  "ProductName": "CBC Panel",
  "ProductCode": "Secure Wifi",
  "ProductDescription": "Only One Charge Type"
}

PUT /product-index/_doc/5
{
  "ProductName": "CBC Panel (Invitro)",
  "ProductCode": "Secure Wifi",
  "ProductDescription": "CBC Panel",
  "ProductFamily": "Hardware",
  "Color": "GREEN"
}


PUT /product-index/_doc/6
{
  "ProductName": "CBC Panel - Beijing",
  "ProductCode": "CBC Panel",
  "ProductDescription": "Only One Charge Type",
  "ProductFamily": "Software",
  "Color": "BLUE"
}


PUT /product-index/_doc/7
{
  "ProductName": "Esoteric",
  "ProductCode": "CBC",
  "ProductFamily": "Marketing Cloud",
  "Color": "WHITE"
}


PUT /product-index/_doc/8
{
  "ProductName": "Oncology",
  "ProductCode": "Network",
  "ProductDescription": "CBC",
  "ProductFamily": "Sales Cloud",
  "Color": "WHITE"
}

        

Create Elasticsearch Index

PUT order-index
{
    "settings": {
        "analysis": {
            "normalizer": {
                "lowercaseNormalizer": {
                    "type": "custom",
                    "filter": [
                        "lowercase",
                        "asciifolding"
                    ]
                }
            }
        }
    },
    "mappings": {
        "properties": {
            "OrderCode": {
                "type": "keyword",
                "normalizer": "lowercaseNormalizer"
            },
            "Description": {
                "type": "text"
            },
            "Family": {
                "type": "keyword",
                "normalizer": "lowercaseNormalizer"
            },
            "ProductName": {
                "type": "keyword",
                "normalizer": "lowercaseNormalizer"
            },
            "ProductId": {
                "type": "keyword"
            }
        }
    }
}      
       

Adding sample data/records

        
#Add first Document in elasticsearch 
PUT /order-index/_doc/5c57754b-86a6-41f1-a4af-de65b531fe8f
{
    "OrderCode": "Network",
    "Description": "APAC",
    "Family": "Sales Cloud",
    "ProductName": "CBC Anti Virus",
    "ProductId": 1
}


PUT /order-index/_doc/218c4907-89c2-482f-90cc-61dd7e27f999
{
    "OrderCode": "Network",
    "Description": "CBC Panel - 1",
    "Family": "Hardware",
    "ProductName": "CBC Esoteric",
    "ProductId": 2
}

 

        

View data/regcords

GET /product-index/_search
GET /order-index/_search

Search case-insensitive records

#Try 1
GET /order-index,product-index/_search
{
    "_source": {
        "include": [
            "Id",
            "OrderCode",
            "ProductId",
            "ProductCode"
        ],
        "exclude": []
    },
    "from": 0,
    "size": 100,
    "track_total_hits": true,
    "query": {
        "bool": {
            "filter": [
                {
                    "bool": {
                        "should": [
                            {
                                "query_string": {
                                    "query": "*Network*",
                                    "fields": [
                                        "ProductCode",
                                        "OrderCode"
                                    ],
                                    "boost": 7
                                }
                            }
                        ]
                    }
                }
            ]
        }
    }
}
       

Note

• Text fields are analyzed.
• For keyword fields we need to add custom elasticsearch setting for analysis.

Reference

• run elasticsearch locally run elasticsearch locally
• Documentation: Elasticsearch - Search multiple data stream and indices