Azure Blaze

Tuesday, 22 June 2021

SLA summary for Azure services

Service-level agreements (SLAs) describe Microsoft’s commitments for uptime and connectivity. The SLA for individual Azure services are listed below.

Reference

Azure SLA: https://azure.microsoft.com/en-us/support/legal/sla/summary/
Azure Support plan: https://azure.microsoft.com/en-in/support/plans/
Azure how to: https://www.youtube.com/playlist?list=PLLasX02E8BPBKgXP4oflOL29TtqTzwhxR
Azure app service plan to: https://azure.microsoft.com/en-in/pricing/details/app-service/windows/

Sunday, 20 June 2021

Azure Cli

The Azure command-line interface (Azure CLI) is a set of commands used to create and manage Azure resources. The Azure CLI is available across Azure services and is designed to get you working quickly with Azure, with an emphasis on automation.

Get list of commands in csv file

Get-Command | Export-Csv -Path .\AzCommands.csv

clear
      
#List of resource group
az group list
      
#Create VM using CLI
PS /home/cloud> az vm create `
>> --name LabVM1 `
>> --resource-group 72-ed951e1f-accessing-and-using-the-azure-cloud-sh `
>> --image UbuntuLTS `
>> --admin-username azureuser `
>> --generate-ssh-keys


#Vm list
az vm list

#Get Resource Group List
Get-AzResource

#Get Storage account
Get-AzStorageAccount

#Get VM
Get-AzVM

#Get Azure resource by  resource type
Get-AzResource | ft

#remove Azure VM (Provide Azure VM Name and ResourceGroup)
Remove-AzVM

Reference

Azure CLI Overview: https://docs.microsoft.com/en-us/cli/azure/?view=a
Azure CLI Reference: https://docs.microsoft.com/en-us/cli/azure/reference-index?view=azure-cli-latest
Azure PowerShell Overview: https://docs.microsoft.com/en-us/powershell/azure/overview?view=azps-1.6.0
Quickstart for Bash in Azure Cloud Shell: https://docs.microsoft.com/en-us/azure/cloud-shell/quickstart
Quickstart for PowerShell in Azure Cloud Shell: https://docs.microsoft.com/en-us/azure/cloud-shell/quickstart-powershell

Monday, 8 March 2021

Elasticsearch for Stocks

Store the BSE/NSE daily stock feeds in Elasticsearch for research.

Create Elasticsearch Index

PUT stock-index
{
  "settings": {
    "index": {
      "number_of_shards": 1,
      "number_of_replicas": 2,
      "analysis": {
        "normalizer": {
          "lowercaseNormalizer": {
            "type": "custom",
            "filter": [
              "lowercase",
              "asciifolding"
            ]
          }
        }
      }
    }
  },
  "mappings": {
    "dynamic": "false",
    "properties": {
      "Symbol": {
        "type": "keyword",
        "normalizer": "lowercaseNormalizer"
      },
      "CreatedDate": {
        "type": "date",
        "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
      },
      "Open": {
        "type": "double"
      },
      "High": {
        "type": "double"
      },
      "Low": {
        "type": "double"
      },
      "Close": {
        "type": "double"
      },
      "Volume": {
        "type": "double"
      },
      "FeedProvider": {
        "type": "text"
      }
    }
  }
}

Adding sample data/records

        
#Add first Stock Document in elasticsearch 

PUT /stock-index/_doc/1
{
  "Symbol": "SBI",
  "CreatedDate": "2021-03-08",
    "Close": 389.6,
    "Open": 388.45,
    "High": 393.4,
    "Low": 386.05,
  "FeedProvider": "zerodha"
}

PUT /stock-index/_doc/2
{
  "Symbol": "SBI",
  "CreatedDate": "2021-03-09",
    "Close": 387.6,
    "Open": 393.45,
    "High": 394.9,
    "Low": 382.25,
  "FeedProvider": "zerodha"
}

PUT /stock-index/_doc/3
{
  "Symbol": "SBI",
  "CreatedDate": "2021-03-10",
    "Close": 387.7,
    "Open": 391.1,
    "High": 393,
    "Low": 385.7,
  "FeedProvider": "zerodha"
}

PUT /stock-index/_doc/4
{
  "Symbol": "SBI",
 "CreatedDate": "2021-03-12",
    "Close": 381.1,
    "Open": 391.7,
    "High": 393.2,
    "Low": 377.5,
  "FeedProvider": "zerodha"
}


PUT /stock-index/_doc/5
{
  "Symbol": "SBI",
  "CreatedDate": "2021-03-15",
    "Close": 383.9,
    "Open": 382.5,
    "High": 386,
    "Low": 369.35,
  "FeedProvider": "zerodha"
}


PUT /stock-index/_doc/6
{
  "Symbol": "SBI",
  "CreatedDate": "2021-03-16",
    "Close": 378.55,
    "Open": 385.6,
    "High": 387.85,
    "Low": 376.1,
  "FeedProvider": "zerodha"
}

PUT /stock-index/_doc/7
{
  "Symbol": "SBI",
   "CreatedDate": "2021-03-17",
    "Close": 368.15,
    "Open": 378.2,
    "High": 379.65,
    "Low": 366.05,
  "FeedProvider": "zerodha"
}


PUT /stock-index/_doc/8
{
  "Symbol": "SBI",
   "CreatedDate": "2021-03-18",
    "Close": 367.1,
    "Open": 371.5,
    "High": 375,
    "Low": 361.1,
  "FeedProvider": "zerodha"
}

PUT /stock-index/_doc/9
{
  "Symbol": "SBI",
 "CreatedDate": "2021-03-19",
    "Close": 371.15,
    "Open": 363.95,
    "High": 372.9,
    "Low": 357.25,
  "FeedProvider": "zerodha"
}

PUT /stock-index/_doc/10
{
  "Symbol": "SBI",
   "CreatedDate": "2021-03-22",
    "Close": 367,
    "Open": 372,
    "High": 372.8,
    "Low": 363.5,
  "FeedProvider": "zerodha"
}

PUT /stock-index/_doc/11
{
  "Symbol": "SBI",
   "CreatedDate": "2021-03-23",
    "Close": 372.7,
    "Open": 368.7,
    "High": 377.95,
    "Low": 367.05,
  "FeedProvider": "zerodha"
}

PUT /stock-index/_doc/12
{
  "Symbol": "SBI",
  "CreatedDate": "2021-03-24",
    "Close": 359.85,
    "Open": 368.5,
    "High": 369.05,
    "Low": 358.65,
  "FeedProvider": "zerodha"
}

PUT /stock-index/_doc/13
{
  "Symbol": "SBI",
   "CreatedDate": "2021-03-25",
    "Close": 355.2,
    "Open": 360,
    "High": 360.85,
    "Low": 345.2,
  "FeedProvider": "zerodha"
}

PUT /stock-index/_doc/14
{
  "Symbol": "SBI",
  "CreatedDate": "2021-03-26",
    "Close": 356.4,
    "Open": 360,
    "High": 361.8,
    "Low": 354.95,
  "FeedProvider": "zerodha"
}


GET /stock-index/_search
{
  "query": {
    "match": {
      "Symbol": "SBI"
    }
  }
}

View data/regcords

GET /stock-index/_search

Searching .

Search a particular stock query or field as belows

#Try 1
GET /stock-index/_search
{
  "query": {
    "match": {
      "Symbol": "SBI"
    }
  }
}

#Try
GET /stock-index/_search
{
   "query": {
     "range": {
       "Close": {
         "gte": 100,
         "lte": 389
       }
     }
   }
}

#Try
GET /stock-index/_search
{
   "query": {
     "range": {
       "CreatedDate": {
         "gte": "2021-03-08",
         "lte": "2021-03-09",
          "format": "yyyy-MM-dd"
       }
     }
   }
}

#Try
GET /stock-index/_search
{
  "size": 10,
  "query": {
    "range": {
      "CreatedDate": {
        "gte": "now-90d/d",
        "lte": "now/d",
        "format": "yyyy-MM-dd"
      }
    }
  }
}


#Simple moving avg
GET /stock-index/_search
{
  "size": 0,
  "aggs": {
    "hourly_data": {
      "date_histogram": {
        "field": "CreatedDate",
        "interval": "day"
      },
      "aggs": {
        "stock_value": {
          "sum": {
            "field": "Close"
          }
        },
        "mva_demo": {
          "moving_avg": {
            "buckets_path": "stock_value",
            "window": 5,
            "model": "simple"
          }
        }
      }
    }
  }
}
 


#Linear Moving Average
GET /stock-index/_search
{
  "size": 0,
  "aggs": {
    "hourly_data": {
      "date_histogram": {
        "field": "CreatedDate",
        "interval": "day"
      },
      "aggs": {
        "stock_value": {
          "sum": {
            "field": "Close"
          }
        },
        "mva_simple": {
          "moving_avg": {
            "buckets_path": "stock_value",
            "window": 10,
            "model": "simple"
          }
        },
        "mva_linear": {
          "moving_avg": {
            "buckets_path": "stock_value",
            "window": 10,
            "model": "linear"
          }
        }
      }
    }
  }
}


#Exponentially Weighted Moving Average
GET /stock-index/_search
{
  "size": 0,
  "aggs": {
    "hourly_data": {
      "date_histogram": {
        "field": "CreatedDate",
        "interval": "day"
      },
      "aggs": {
        "stock_value": {
          "sum": {
            "field": "Close"
          }
        },
        "mva_simple": {
          "moving_avg": {
            "buckets_path": "stock_value",
            "window": 10,
            "model": "simple"
          }
        },
        "mva_linear": {
          "moving_avg": {
            "buckets_path": "stock_value",
            "window": 10,
            "model": "linear"
          }
        },
        "mva_ewma": {
          "moving_avg": {
            "buckets_path": "stock_value",
            "window": 10,
            "model": "ewma",
            "settings": {
              "alpha": 0.3
            }
          }
        }
      }
    }
  }
}

 #minimum of the price monthly
 GET /stock-index/_search
 {
  "size": 0,
  "aggs": {
    "price_per_month": {
      "date_histogram": {
        "field": "CreatedDate",
        "calendar_interval": "month"
      },
      "aggs": {
        "price": {
          "min": {
            "field": "Close"
          }
        }
      }
    },
    "min_monthly_price": {
      "min_bucket": {
        "buckets_path": "price_per_month>price" 
      }
    }
  }
}


#Moving Min 
GET /stock-index/_search
{
  "size": 0,
  "aggs": {
    "day_data": {
      "date_histogram": {
        "field": "CreatedDate",
        "interval": "day"
      },
      "aggs": {
        "stock_value": {
          "sum": {
            "field": "Close"
          }
        },
         "the_movfn": {
          "moving_fn": {
            "buckets_path": "stock_value",  
            "window": 10,
            "script": "MovingFunctions.min(values)"
          }
        }
      }
    }
  }
}

Reference

Documentation: BSE
Aggs: ES search-aggregations

Sunday, 10 January 2021

Elasticsearch - Boost, aggs

Boosting is the process by which you can modify the relevance of a document. You can boost a document while you are indexing it or when you query for the document.we definitely recommend you use the query-time boosting because it’s the most flexible and allows you to change your mind about what fields or terms are important.

Create Elasticsearch Index

PUT product-index
{
  "settings": {
    "analysis": {
      "normalizer": {
        "lowercaseNormalizer": {
          "type": "custom",
          "filter": ["lowercase", "asciifolding"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "ProductName": {
        "type": "keyword",
        "normalizer": "lowercaseNormalizer"
      },
     "ProductCode": {
        "type": "keyword",
        "normalizer": "lowercaseNormalizer"
      },
      "ProductDescription": {
        "type": "text"
      },
      "ProductFamily": {
        "type": "keyword",
        "normalizer": "lowercaseNormalizer"
      },
      "Color": {
        "type": "keyword"
      }
    }
  }
}

Adding sample data/records

        
#Add first Document in elasticsearch 
PUT /product-index/_doc/1
{
  "ProductName": "CBC Anti Virus",
  "ProductCode": "Network",
  "ProductDescription": "APAC",
  "ProductFamily": "Sales Cloud",
  "Color": "Silver"
}


PUT /product-index/_doc/2
{
  "ProductName": "CBC Esoteric",
  "ProductDescription": "CBC Panel - 1",
  "ProductFamily": "Hardware",
  "Color": "BLUE"
}

PUT /product-index/_doc/3
{
  "ProductName": "CBC Oncology",
  "ProductCode": "CBC Health",
  "ProductFamily": "Software"
}

PUT /product-index/_doc/4
{
  "ProductName": "CBC Panel",
  "ProductCode": "Secure Wifi",
  "ProductDescription": "Only One Charge Type"
}

PUT /product-index/_doc/5
{
  "ProductName": "CBC Panel (Invitro)",
  "ProductCode": "Secure Wifi",
  "ProductDescription": "CBC Panel",
  "ProductFamily": "Hardware",
  "Color": "GREEN"
}


PUT /product-index/_doc/6
{
  "ProductName": "CBC Panel - Beijing",
  "ProductCode": "CBC Panel",
  "ProductDescription": "Only One Charge Type",
  "ProductFamily": "Software",
  "Color": "BLUE"
}


PUT /product-index/_doc/7
{
  "ProductName": "Esoteric",
  "ProductCode": "CBC",
  "ProductFamily": "Marketing Cloud",
  "Color": "WHITE"
}


PUT /product-index/_doc/8
{
  "ProductName": "Oncology",
  "ProductCode": "Network",
  "ProductDescription": "CBC",
  "ProductFamily": "Sales Cloud",
  "Color": "WHITE"
}

View data/regcords

GET /product-index/_search

Searching with relevancy

Boosting the score of a particular query or field as belows

#Try 1
POST /product-index/_search
{
  "query": {
    "match": {
      "ProductFamily": {
        "query": "hArdware",
        "boost": 2
      }
    }
  }
}


#Try 2
POST /product-index/_search
{
  "query": {
    "bool": {
      "should": [
        {
          "query_string": {
            "query": "*Wifi*",
            "default_field": "ProductCode",
            "boost": 3
          }
        },
        {
          "match": {
            "ProductFamily": {
              "query": "hArdware",
              "boost": 4
            }
          }
        }
      ]
    }
  }
}

#Try 3
GET /product-index/_search
{
  "query": {
    "query_string": {
      "query": "*Secure* OR *panel*",
      "default_field": "ProductCode",
      "boost": 1
    }
  }
}

#Try 4
POST /product-index/_search
{
  "query": {
    "bool": {
      "should": [
        {
          "query_string": {
            "query": "*CBC*",
            "default_field": "ProductName",
            "boost": 100
          }
        },
         {
          "query_string": {
            "query": "*hard*",
            "default_field": "ProductFamily",
            "boost": 91
          }
        },
        {
          "query_string": {
            "query": "*one*",
            "default_field": "ProductDescription",
            "boost": 92
          }
        },
        {
          "query_string": {
            "query": "*Blue*",
            "default_field": "Color",
            "boost": 90
          }
        },
        {
          "query_string": {
            "query": "*Health*",
            "default_field": "ProductCode",
            "boost": 93
          }
        }
      ]
    }
  }
}


#Try 5
POST /product-index/_search
{
  "query": {
    "bool": {
      "should": [
        {
          "query_string": {
            "query": "*cbc*",
            "default_field": "ProductName",
            "boost": 49
          }
        },
        {
          "query_string": {
            "query": "*cbc*",
            "default_field": "ProductCode",
            "boost": 7
          }
        },
        {
          "query_string": {
            "query": "*cbc*",
            "fields": [ "ProductDescription", "ProductFamily"]
          }
        }
      ]
    }
  }
}

#Try 6
GET /product-index/_search
{
  "query" : {
    "query_string" : {
      "query" : "*cbc*",
      "fields"  : ["*"]
    }
  }
}


#Try 7
POST /product-index/_search
{
  "query": {
    "bool": {
      "should": [
        {
          "query_string": {
            "query": "*cbc*",
            "default_field": "ProductName",
            "boost": 49
          }
        },
        {
          "query_string": {
            "query": "*cbc*",
            "default_field": "ProductCode",
            "boost": 7
          }
        },
        {
          "query_string": {
            "query": "*cbc*",
            "fields": [ "*"]
          }
        }
      ]
    }
  }
}



#Try Demonstration for aggregation
GET /product-index/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "match": { "ProductFamily" : "hArdware"}
        }
      ]
    }
  },
  "aggs": {
    "Family": {
      "terms": {
        "field": "ProductFamily"
      }
    }
  }
}

#Try Another example of aggregation
GET /product-index/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "match": {
            "ProductFamily": "hArdware"
          }
        },
        {
          "bool": {
            "should": [
              {
                "query_string": {
                  "query": "*cbc* OR *network*",
                  "fields": [
                    "ProductName",
                    "ProductCode",
                    "ProductDescription",
                    "ProductFamily",
                    "Color"
                  ]
                }
              }
            ]
          }
        }
      ]
    }
  },
  "aggs": {
    "Family": {
      "terms": {
        "field": "ProductFamily"
      }
    },
    "Color": {
      "terms": {
        "field": "Color"
      }
    }
  },
  "_source": [
    "ProductName",
    "ProductCode",
    "ProductDescription",
    "ProductFamily",
    "Color"
    ]
}

Reference

Documentation: Elasticsearch Boost elasticsearch-in-action

Tuesday, 5 January 2021

Elasticsearch case-insensitive searches

A simple normalizer called lowercase ships with elasticsearch and can be used.

Create Elasticsearch Index

PUT product-index
{
  "settings": {
    "analysis": {
      "normalizer": {
        "lowercaseNormalizer": {
          "type": "custom",
          "filter": ["lowercase", "asciifolding"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "ProductName": {
        "type": "keyword",
        "normalizer": "lowercaseNormalizer"
      },
     "ProductCode": {
        "type": "keyword",
        "normalizer": "lowercaseNormalizer"
      },
      "ProductDescription": {
        "type": "text"
      },
      "ProductFamily": {
        "type": "keyword",
        "normalizer": "lowercaseNormalizer"
      },
      "Color": {
        "type": "keyword"
      }
    }
  }
}

Adding sample data/records

        
#Add first Document in elasticsearch 
PUT /product-index/_doc/1
{
  "ProductName": "CBC Anti Virus",
  "ProductCode": "Network",
  "ProductDescription": "APAC",
  "ProductFamily": "Sales Cloud",
  "Color": "Silver"
}


PUT /product-index/_doc/2
{
  "ProductName": "CBC Esoteric",
  "ProductDescription": "CBC Panel - 1",
  "ProductFamily": "Hardware",
  "Color": "BLUE"
}

PUT /product-index/_doc/3
{
  "ProductName": "CBC Oncology",
  "ProductCode": "CBC Health",
  "ProductFamily": "Software"
}

PUT /product-index/_doc/4
{
  "ProductName": "CBC Panel",
  "ProductCode": "Secure Wifi",
  "ProductDescription": "Only One Charge Type"
}

PUT /product-index/_doc/5
{
  "ProductName": "CBC Panel (Invitro)",
  "ProductCode": "Secure Wifi",
  "ProductDescription": "CBC Panel",
  "ProductFamily": "Hardware",
  "Color": "GREEN"
}


PUT /product-index/_doc/6
{
  "ProductName": "CBC Panel - Beijing",
  "ProductCode": "CBC Panel",
  "ProductDescription": "Only One Charge Type",
  "ProductFamily": "Software",
  "Color": "BLUE"
}


PUT /product-index/_doc/7
{
  "ProductName": "Esoteric",
  "ProductCode": "CBC",
  "ProductFamily": "Marketing Cloud",
  "Color": "WHITE"
}


PUT /product-index/_doc/8
{
  "ProductName": "Oncology",
  "ProductCode": "Network",
  "ProductDescription": "CBC",
  "ProductFamily": "Sales Cloud",
  "Color": "WHITE"
}

View data/regcords

GET /product-index/_search

Search case-insensitive records

#Try 1
GET /product-index/_search
{
  "query": {
    "match": { "ProductFamily" : "hArdware"}
  }
}

#Try 2
GET /product-index/_search
{
  "query" : {
    "query_string" : {
      "query" : "*bei*",
      "fields"  : ["ProductName"]
    }
  }
}

#Try 3
GET /product-index/_search
{
  "query": {
    "query_string": {
      "query": "*Secure* OR *panel*",
      "default_field": "ProductCode"
    }
  }
}

#Try 4
GET /product-index/_search
{
  "query" : {
    "query_string" : {
      "query" : "*cHarge*",
      "fields"  : ["ProductDescription"]
    }
  }
}

Note

Text fields are analyzed.
For keyword fields we need to add custom elasticsearch setting for analysis.

Reference

run elasticsearch locally run elasticsearch locally
Documentation: Elasticsearch normalizer

Sunday, 27 December 2020

Term Vs Match queries - Elasticsearch

Term	Match Queries
Looks for the exact term in the inverted index.	Queries are analyzed before looking up the inverted index.
Does not know of the presence of the analyzer.	Understands how the fields have ben analyzed.
Great for keywords, numbers, dates where exact matches are important.	Useful when searching full text fields with a large body of text.
Less likely to match irrelevant documents.	More likely to match irrelevant documents.
---	Does not go through a query parsing process, does not support wildcards, prefixes etc.

Wednesday, 29 July 2020

Metric types

Counter : A counter is a cumulative metric that represents a single monotonically increasing counter whose value can only increase or be reset to zero on restart.
Gauge: A gauge is a metric that represents a single numerical value that can arbitrarily go up and down.
Histogram: A histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. It also provides a sum of all observed values.
Summary: Similar to a histogram, a summary samples observations (usually things like request durations and response sizes).

when to use which metric type?

Counters are values that can only increase that keep track of the number of events. For example, we would use counters to start a number of requests, number of errors or the number of Finnish transactions.
Gauge is another type of metric that represents numerical value on like counters. Gauges can go up and down to represent the current state of the system. We can use gauges for information that can both increase and decrease over time. For example, memory usage, CPU usage temperature but also a number of conquering requests.
Histogram sample observations and count them in configurable buckets. This way we can check how many requests finish within a given time or how many responses fit within a given size, Histogram exposed several metrics for us to query.
Summaries calculate configurable. lt's over a sliding time window. In general, many use cases of summaries can be covered with Histogram.

Reference

Documentation: Metric Types

Azure Blaze

Search This Blog

Tuesday, 22 June 2021

SLA summary for Azure services

Reference

Sunday, 20 June 2021

Azure Cli

Get list of commands in csv file

Reference

Monday, 8 March 2021

Elasticsearch for Stocks

Create Elasticsearch Index

Adding sample data/records

View data/regcords

Searching .

Reference

Sunday, 10 January 2021

Elasticsearch - Boost, aggs

Create Elasticsearch Index

Adding sample data/records

View data/regcords

Searching with relevancy

Reference

Tuesday, 5 January 2021

Elasticsearch case-insensitive searches

Create Elasticsearch Index

Adding sample data/records

View data/regcords

Search case-insensitive records

Note

Reference

Sunday, 27 December 2020

Term Vs Match queries - Elasticsearch

Wednesday, 29 July 2020

Metric types

when to use which metric type?

Reference

Creating a NuGet Package Feed to Host Artifacts

Recent Post