Azure Blaze

Tuesday, 1 August 2023

Elasticsearch _msearch

When using the msearch (multi-search) API in Elasticsearch, you can search across multiple indices in a single request. Each search query in the msearch request can be targeted to a specific index..

Create Elasticsearch Index

PUT product-index
{
    "settings": {
        "analysis": {
            "normalizer": {
                "lowercaseNormalizer": {
                    "type": "custom",
                    "filter": [
                        "lowercase",
                        "asciifolding"
                    ]
                }
            }
        }
    },
    "mappings": {
        "properties": {
            "Product": {
                "type": "object",
                "properties": {
                    "ProductName": {
                        "type": "keyword",
                        "normalizer": "lowercaseNormalizer"
                    },
                    "ProductCode": {
                        "type": "keyword",
                        "normalizer": "lowercaseNormalizer"
                    },
                    "ProductDescription": {
                        "type": "text"
                    },
                    "ProductFamily": {
                        "type": "keyword",
                        "normalizer": "lowercaseNormalizer"
                    },
                    "Color": {
                        "type": "keyword"
                    }
                }
            }
        }
    }
}

Adding sample data/records

        
#Add first Document in elasticsearch 
PUT /product-index/_doc/1
{
    "Product": {
        "ProductName": "CBC Anti Virus",
        "ProductCode": "Network",
        "ProductDescription": "APAC",
        "ProductFamily": "Sales Cloud",
        "Color": "Silver"
    }
}


PUT /product-index/_doc/2
{
    "Product": {
        "ProductName": "CBC Esoteric",
        "ProductDescription": "CBC Panel - 1",
        "ProductFamily": "Hardware",
        "Color": "BLUE"
    }
}

PUT /product-index/_doc/3
{
    "Product": {
        "ProductName": "CBC Oncology",
        "ProductCode": "CBC Health",
        "ProductFamily": "Software"
    }
}

PUT /product-index/_doc/4
{
    "Product": {
        "ProductName": "CBC Panel",
        "ProductCode": "Secure Wifi",
        "ProductDescription": "Only One Charge Type"
    }
}

PUT /product-index/_doc/5
{
    "Product": {
        "ProductName": "CBC Panel (Invitro)",
        "ProductCode": "Secure Wifi",
        "ProductDescription": "CBC Panel",
        "ProductFamily": "Hardware",
        "Color": "GREEN"
    }
}


PUT /product-index/_doc/6
{
    "Product": {
        "ProductName": "CBC Panel - Beijing",
        "ProductCode": "CBC Panel",
        "ProductDescription": "Only One Charge Type",
        "ProductFamily": "Software",
        "Color": "BLUE"
    }
}


PUT /product-index/_doc/7
{
    "Product": {
        "ProductName": "Esoteric",
        "ProductCode": "CBC",
        "ProductFamily": "Marketing Cloud",
        "Color": "WHITE"
    }
}


PUT /product-index/_doc/8
{
    "Product": {
        "ProductName": "Oncology",
        "ProductCode": "Network",
        "ProductDescription": "CBC",
        "ProductFamily": "Sales Cloud",
        "Color": "WHITE"
    }
}

Create Elasticsearch Index

PUT order-index
{
    "settings": {
        "analysis": {
            "normalizer": {
                "lowercaseNormalizer": {
                    "type": "custom",
                    "filter": [
                        "lowercase",
                        "asciifolding"
                    ]
                }
            }
        }
    },
    "mappings": {
        "properties": {
            "Order": {
                "type": "object",
                "properties": {
                    "OrderCode": {
                        "type": "keyword",
                        "normalizer": "lowercaseNormalizer"
                    },
                    "Description": {
                        "type": "text"
                    },
                    "Family": {
                        "type": "keyword",
                        "normalizer": "lowercaseNormalizer"
                    },
                    "ProductName": {
                        "type": "keyword",
                        "normalizer": "lowercaseNormalizer"
                    },
                    "ProductId": {
                        "type": "keyword"
                    }
                }
            }
        }
    }
}

Adding sample data/records

        
#Add first Document in elasticsearch 
PUT /order-index/_doc/5c57754b-86a6-41f1-a4af-de65b531fe8f
{
    "Order": {
        "OrderCode": "Network",
        "Description": "APAC",
        "Family": "Sales Cloud",
        "ProductName": "CBC Anti Virus",
        "ProductId": 1
    }
}


PUT /order-index/_doc/218c4907-89c2-482f-90cc-61dd7e27f999
{
    "Order": {
        "OrderCode": "Network",
        "Description": "CBC Panel - 1",
        "Family": "Hardware",
        "ProductName": "CBC Esoteric",
        "ProductId": 2
    }
}

View data/regcords

GET /product-index/_search
GET /order-index/_search

Search case-insensitive records

In the request body, provide the search queries in a newline-separated JSON format. Each search query should have two parts: the header and the search body. The header contains information about the index and search type, while the search body contains the actual search query DSL.

#Try 1
POST http://localhost:9200/_msearch
Content-type:application/json
Payload as below


{"index" : "product-index"}
{"query": {"match": {"Product.OrderCode": "Network"}}, "from" : 0, "size" : 10}
{"index" : "order-index"}
{"query": {"match": {"Order.OrderCode": "Network"}}}

Note

Text fields are analyzed.
For keyword fields we need to add custom elasticsearch setting for analysis.

Reference

run elasticsearch locally run elasticsearch locally
Documentation: Elasticsearch - _msearch

Tuesday, 4 July 2023

Multiple Index Query

The following request searches the product-index and order-index indices.

Create Elasticsearch Index

PUT product-index
{
  "settings": {
    "analysis": {
      "normalizer": {
        "lowercaseNormalizer": {
          "type": "custom",
          "filter": ["lowercase", "asciifolding"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "ProductName": {
        "type": "keyword",
        "normalizer": "lowercaseNormalizer"
      },
     "ProductCode": {
        "type": "keyword",
        "normalizer": "lowercaseNormalizer"
      },
      "ProductDescription": {
        "type": "text"
      },
      "ProductFamily": {
        "type": "keyword",
        "normalizer": "lowercaseNormalizer"
      },
      "Color": {
        "type": "keyword"
      }
    }
  }
}

Adding sample data/records

        
#Add first Document in elasticsearch 
PUT /product-index/_doc/1
{
  "ProductName": "CBC Anti Virus",
  "ProductCode": "Network",
  "ProductDescription": "APAC",
  "ProductFamily": "Sales Cloud",
  "Color": "Silver"
}


PUT /product-index/_doc/2
{
  "ProductName": "CBC Esoteric",
  "ProductDescription": "CBC Panel - 1",
  "ProductFamily": "Hardware",
  "Color": "BLUE"
}

PUT /product-index/_doc/3
{
  "ProductName": "CBC Oncology",
  "ProductCode": "CBC Health",
  "ProductFamily": "Software"
}

PUT /product-index/_doc/4
{
  "ProductName": "CBC Panel",
  "ProductCode": "Secure Wifi",
  "ProductDescription": "Only One Charge Type"
}

PUT /product-index/_doc/5
{
  "ProductName": "CBC Panel (Invitro)",
  "ProductCode": "Secure Wifi",
  "ProductDescription": "CBC Panel",
  "ProductFamily": "Hardware",
  "Color": "GREEN"
}


PUT /product-index/_doc/6
{
  "ProductName": "CBC Panel - Beijing",
  "ProductCode": "CBC Panel",
  "ProductDescription": "Only One Charge Type",
  "ProductFamily": "Software",
  "Color": "BLUE"
}


PUT /product-index/_doc/7
{
  "ProductName": "Esoteric",
  "ProductCode": "CBC",
  "ProductFamily": "Marketing Cloud",
  "Color": "WHITE"
}


PUT /product-index/_doc/8
{
  "ProductName": "Oncology",
  "ProductCode": "Network",
  "ProductDescription": "CBC",
  "ProductFamily": "Sales Cloud",
  "Color": "WHITE"
}

Create Elasticsearch Index

PUT order-index
{
    "settings": {
        "analysis": {
            "normalizer": {
                "lowercaseNormalizer": {
                    "type": "custom",
                    "filter": [
                        "lowercase",
                        "asciifolding"
                    ]
                }
            }
        }
    },
    "mappings": {
        "properties": {
            "OrderCode": {
                "type": "keyword",
                "normalizer": "lowercaseNormalizer"
            },
            "Description": {
                "type": "text"
            },
            "Family": {
                "type": "keyword",
                "normalizer": "lowercaseNormalizer"
            },
            "ProductName": {
                "type": "keyword",
                "normalizer": "lowercaseNormalizer"
            },
            "ProductId": {
                "type": "keyword"
            }
        }
    }
}

Adding sample data/records

        
#Add first Document in elasticsearch 
PUT /order-index/_doc/5c57754b-86a6-41f1-a4af-de65b531fe8f
{
    "OrderCode": "Network",
    "Description": "APAC",
    "Family": "Sales Cloud",
    "ProductName": "CBC Anti Virus",
    "ProductId": 1
}


PUT /order-index/_doc/218c4907-89c2-482f-90cc-61dd7e27f999
{
    "OrderCode": "Network",
    "Description": "CBC Panel - 1",
    "Family": "Hardware",
    "ProductName": "CBC Esoteric",
    "ProductId": 2
}

View data/regcords

GET /product-index/_search
GET /order-index/_search

Search case-insensitive records

#Try 1
GET /order-index,product-index/_search
{
    "_source": {
        "include": [
            "Id",
            "OrderCode",
            "ProductId",
            "ProductCode"
        ],
        "exclude": []
    },
    "from": 0,
    "size": 100,
    "track_total_hits": true,
    "query": {
        "bool": {
            "filter": [
                {
                    "bool": {
                        "should": [
                            {
                                "query_string": {
                                    "query": "*Network*",
                                    "fields": [
                                        "ProductCode",
                                        "OrderCode"
                                    ],
                                    "boost": 7
                                }
                            }
                        ]
                    }
                }
            ]
        }
    }
}

Note

Text fields are analyzed.
For keyword fields we need to add custom elasticsearch setting for analysis.

Reference

run elasticsearch locally run elasticsearch locally
Documentation: Elasticsearch - Search multiple data stream and indices

Saturday, 5 November 2022

Elasticsearch Reindex / migration

ElasticSearch: Zero downtime reindexing
ElasticSearch has a solution to the problem, index aliases. The alias is like a symbolic link which can point to one or more indices. It gives us the flexibility to create a new index in the background and making the changes in a way that is almost unnoticeable to the application.

Reference

Elasticsearch reindex APIhttps://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html
Elasticsearch reindex 1 https://nasa.github.io/cumulus/docs/troubleshooting/reindex-elasticsearch
reindex status https://nasa.github.io/cumulus-api/#reindex-status
Elasticsearch reindex explained https://youtu.be/nME4ZjfjwiA
Elasticsearch reindex needs https://dzone.com/articles/elasticsearch-fault-tolerance-reindexing-need-and-1
Elasticsearch reindex steps https://www.thirdrocktechkno.com/blog/6-steps-to-reindex-elasticsearch-data/

Friday, 4 November 2022

Elasticsearch

Elasticsearch is the distributed search and analytics engine at the heart of the Elastic Stack.

Reference

What is Elasticsearch? https://youtu.be/ZP0NmfyfsoM

Thursday, 3 November 2022

AWS OpenSearch Deployment Best Practices

3 Dedicated Master Nodes

Offload cluster managment task like health checks and maintain routing and cluster state.

Data Nodes deployed in multiples of 3

Store the data in shards and perform searches, query request, and CRUD operations.

Deploy across 3AZs

Destribute the data nodes equally across 3 AZs for the hightest availability.

Reference

ES: https://www.elastic.co/guide/en/elasticsearch/reference/8.5/elasticsearch-intro.html

Saturday, 22 October 2022

Validate Subsequence

List array = new List() { 5, 1, 22, 25, 6, -1, 8, 10 };
List sequence = new List { 1, 6, -1, 10 };
bool isValid = IsValidSubsequence(array, sequence);
Console.WriteLine(isValid);
//Expected Output: true

Solution 1.

public static bool IsValidSubsequence0(List array, List sequence)
{
    int seqIdx = 0;
    for (int arrIdx = 0; arrIdx < array.Count && seqIdx < sequence.Count; arrIdx++)
    {
        if (array[arrIdx] == sequence[seqIdx])
        {
            seqIdx++;
        }
    }
    return seqIdx == sequence.Count;
}

Test case

Test Case 1
{
  "array": [5, 1, 22, 25, 6, -1, 8, 10],
  "sequence": [1, 6, -1, 10]
}
Test Case 2
{
  "array": [5, 1, 22, 25, 6, -1, 8, 10],
  "sequence": [5, 1, 22, 25, 6, -1, 8, 10]
}
Test Case 3
{
  "array": [5, 1, 22, 25, 6, -1, 8, 10],
  "sequence": [5, 1, 22, 6, -1, 8, 10]
}
Test Case 4
{
  "array": [5, 1, 22, 25, 6, -1, 8, 10],
  "sequence": [22, 25, 6]
}
Test Case 5
{
  "array": [5, 1, 22, 25, 6, -1, 8, 10],
  "sequence": [1, 6, 10]
}
Test Case 6
{
  "array": [5, 1, 22, 25, 6, -1, 8, 10],
  "sequence": [5, 1, 22, 10]
}
Test Case 7
{
  "array": [5, 1, 22, 25, 6, -1, 8, 10],
  "sequence": [5, -1, 8, 10]
}
Test Case 8
{
  "array": [5, 1, 22, 25, 6, -1, 8, 10],
  "sequence": [25]
}
Test Case 9
{
  "array": [1, 1, 1, 1, 1],
  "sequence": [1, 1, 1]
}
Test Case 10
{
  "array": [5, 1, 22, 25, 6, -1, 8, 10],
  "sequence": [5, 1, 22, 25, 6, -1, 8, 10, 12]
}
Test Case 11
{
  "array": [5, 1, 22, 25, 6, -1, 8, 10],
  "sequence": [4, 5, 1, 22, 25, 6, -1, 8, 10]
}
Test Case 12
{
  "array": [5, 1, 22, 25, 6, -1, 8, 10],
  "sequence": [5, 1, 22, 23, 6, -1, 8, 10]
}
Test Case 13
{
  "array": [5, 1, 22, 25, 6, -1, 8, 10],
  "sequence": [5, 1, 22, 22, 25, 6, -1, 8, 10]
}
Test Case 14
{
  "array": [5, 1, 22, 25, 6, -1, 8, 10],
  "sequence": [5, 1, 22, 22, 6, -1, 8, 10]
}
Test Case 15
{
  "array": [5, 1, 22, 25, 6, -1, 8, 10],
  "sequence": [1, 6, -1, -1]
}
Test Case 16
{
  "array": [5, 1, 22, 25, 6, -1, 8, 10],
  "sequence": [1, 6, -1, -1, 10]
}
Test Case 17
{
  "array": [5, 1, 22, 25, 6, -1, 8, 10],
  "sequence": [1, 6, -1, -2]
}
Test Case 18
{
  "array": [5, 1, 22, 25, 6, -1, 8, 10],
  "sequence": [26]
}
Test Case 19
{
  "array": [5, 1, 22, 25, 6, -1, 8, 10],
  "sequence": [5, 1, 25, 22, 6, -1, 8, 10]
}
Test Case 20
{
  "array": [5, 1, 22, 25, 6, -1, 8, 10],
  "sequence": [5, 26, 22, 8]
}
Test Case 21
{
  "array": [1, 1, 6, 1],
  "sequence": [1, 1, 1, 6]
}
Test Case 22
{
  "array": [5, 1, 22, 25, 6, -1, 8, 10],
  "sequence": [1, 6, -1, 10, 11, 11, 11, 11]
}
Test Case 23
{
  "array": [5, 1, 22, 25, 6, -1, 8, 10],
  "sequence": [5, 1, 22, 25, 6, -1, 8, 10, 10]
}
Test Case 24
{
  "array": [5, 1, 22, 25, 6, -1, 8, 10],
  "sequence": [1, 6, -1, 5]
}

Multi String Search

string bigstring = "this is a big string";
string[] smallstrings = { "this", "yo", "is", "a", "bigger", "string", "kappa" };

List result = MultistringSearch(bigstring, smallstrings);
foreach (bool b in result)
{
    Console.WriteLine(b);
}

//Expected output: [true, false, true, true, false, true, false]

Solution 1.

public static List MultistringSearch(string bigstring, string[] smallstrings)
{
    List result = new List();
    foreach (string smalString in smallstrings)
    {
        result.Add(isContain(smalString, bigstring));
    }
    return result;
}


public static bool isContain(string smallString, string bigString)
{
    for (int i = 0; i <= bigString.Length - smallString.Length; i++)
    {
        int j;
        for (j = 0; j < smallString.Length; j++)
        {
            if (bigString[i + j] != smallString[j])
            {
                break;
            }
        }
        if (j == smallString.Length)
        {
            return true;
        }
    }
    return false;
}

Solution 2:

public static bool IsContain(string smallString, string bigString)
{
    for (int i = 0; i < bigString.Length; i++)
    {
        if (i + smallString.Length > bigString.Length)
        {
            break;
        }

        if(IsContain(smallString, bigString, i))
        {
            return true;
        }
    }
    return false;
}

private static bool IsContain(string smallString, string bigString, int startIdx)
{
    int leftBigidx = startIdx;
    int rightBigIdx = startIdx + smallString.Length - 1;
    int leftSmallIdx = 0;
    int rightSmallIdx = smallString.Length - 1;
    while (leftBigidx <= rightBigIdx)
    {
        if (bigString[leftBigidx] != smallString[leftSmallIdx] ||
                bigString[rightBigIdx] != smallString[rightSmallIdx])
        {
            return false;
        }
        leftBigidx++;
        rightBigIdx--;
        leftSmallIdx++;
        rightSmallIdx--;
    }
    return true;
}

Test case

Test Case 1
{
  "bigString": "this is a big string",
  "smallStrings": ["this", "yo", "is", "a", "bigger", "string", "kappa"]
}

Test Case 2
{
  "bigString": "abcdefghijklmnopqrstuvwxyz",
  "smallStrings": ["abc", "mnopqr", "wyz", "no", "e", "tuuv"]
}

Test Case 3
{
  "bigString": "abcdefghijklmnopqrstuvwxyz",
  "smallStrings": ["abcdefghijklmnopqrstuvwxyz", "abc", "j", "mnopqr", "pqrstuvwxyz", "xyzz", "defh"]
}

Test Case 4
{
  "bigString": "hj!)!%Hj1jh8f1985n!)51",
  "smallStrings": ["%Hj7", "8f198", "!)5", "!)!", "!!", "jh81", "j181hf"]
}

Test Case 5
{
  "bigString": "Mary goes to the shopping center every week.",
  "smallStrings": ["to", "Mary", "centers", "shop", "shopping", "string", "kappa"]
}

Test Case 6
{
  "bigString": "adcb akfkw afnmc fkadn vkaca jdaf dacb cdba cbda",
  "smallStrings": ["abcd", "acbd", "adbc", "dabc", "cbda", "cabd", "cdab"]
}

Test Case 7
{
  "bigString": "test testing testings tests testers test-takers",
  "smallStrings": ["tests", "testatk", "testiing", "trsatii", "test-taker", "test"]
}

Test Case 8
{
  "bigString": "ndbajwhfawkjljkfaopwdlaawjk dawkj awjkawkfjhkawk ahjwkjad jadfljawd",
  "smallStrings": ["abc", "akwbc", "awbc", "abafac", "ajjfbc", "abac", "jadfl"]
}

Test Case 9
{
  "bigString": "Is this particular test going to pass or is it going to fail? That is the question.",
  "smallStrings": ["that", "the", "questions", "goes", "mountain", "passes", "passed", "going", "is"]
}

Test Case 10
{
  "bigString": "Everything in this test should fail.",
  "smallStrings": ["everything", "inn", "that", "testers", "shall", "failure"]
}

Test Case 11
{
  "bigString": "this ain't a big string",
  "smallStrings": ["this", "is", "yo", "a", "bigger"]
}

Test Case 12
{
  "bigString": "bbbabb",
  "smallStrings": ["bbabb"]
}

Azure Blaze

Search This Blog

Tuesday, 1 August 2023

Elasticsearch _msearch

Create Elasticsearch Index

Adding sample data/records

Create Elasticsearch Index

Adding sample data/records

View data/regcords

Search case-insensitive records

Note

Reference

Tuesday, 4 July 2023

Multiple Index Query

Create Elasticsearch Index

Adding sample data/records

Create Elasticsearch Index

Adding sample data/records

View data/regcords

Search case-insensitive records

Note

Reference

Saturday, 5 November 2022

Elasticsearch Reindex / migration

Reference

Friday, 4 November 2022

Elasticsearch

Reference

Thursday, 3 November 2022

AWS OpenSearch Deployment Best Practices

Reference

Saturday, 22 October 2022

Validate Subsequence

Test case

Multi String Search

Test case

API Authentication

Recent Post