Azure Blaze: Elasticsearch

Showing posts with label Elasticsearch. Show all posts

Monday, 29 April 2024

Elasticsearch - Nodes, clusters, and shards

Elastic Stack Video - Load your gun in short time.

Beginner's Crash Course to Elastic Stack Playlist
Nodes, clusters, and shards in Elasticsearch - S1E3:Mini Beginner's Crash Course

Tuesday, 1 August 2023

Elasticsearch _msearch

When using the msearch (multi-search) API in Elasticsearch, you can search across multiple indices in a single request. Each search query in the msearch request can be targeted to a specific index..

Create Elasticsearch Index

PUT product-index
{
    "settings": {
        "analysis": {
            "normalizer": {
                "lowercaseNormalizer": {
                    "type": "custom",
                    "filter": [
                        "lowercase",
                        "asciifolding"
                    ]
                }
            }
        }
    },
    "mappings": {
        "properties": {
            "Product": {
                "type": "object",
                "properties": {
                    "ProductName": {
                        "type": "keyword",
                        "normalizer": "lowercaseNormalizer"
                    },
                    "ProductCode": {
                        "type": "keyword",
                        "normalizer": "lowercaseNormalizer"
                    },
                    "ProductDescription": {
                        "type": "text"
                    },
                    "ProductFamily": {
                        "type": "keyword",
                        "normalizer": "lowercaseNormalizer"
                    },
                    "Color": {
                        "type": "keyword"
                    }
                }
            }
        }
    }
}

Adding sample data/records

        
#Add first Document in elasticsearch 
PUT /product-index/_doc/1
{
    "Product": {
        "ProductName": "CBC Anti Virus",
        "ProductCode": "Network",
        "ProductDescription": "APAC",
        "ProductFamily": "Sales Cloud",
        "Color": "Silver"
    }
}


PUT /product-index/_doc/2
{
    "Product": {
        "ProductName": "CBC Esoteric",
        "ProductDescription": "CBC Panel - 1",
        "ProductFamily": "Hardware",
        "Color": "BLUE"
    }
}

PUT /product-index/_doc/3
{
    "Product": {
        "ProductName": "CBC Oncology",
        "ProductCode": "CBC Health",
        "ProductFamily": "Software"
    }
}

PUT /product-index/_doc/4
{
    "Product": {
        "ProductName": "CBC Panel",
        "ProductCode": "Secure Wifi",
        "ProductDescription": "Only One Charge Type"
    }
}

PUT /product-index/_doc/5
{
    "Product": {
        "ProductName": "CBC Panel (Invitro)",
        "ProductCode": "Secure Wifi",
        "ProductDescription": "CBC Panel",
        "ProductFamily": "Hardware",
        "Color": "GREEN"
    }
}


PUT /product-index/_doc/6
{
    "Product": {
        "ProductName": "CBC Panel - Beijing",
        "ProductCode": "CBC Panel",
        "ProductDescription": "Only One Charge Type",
        "ProductFamily": "Software",
        "Color": "BLUE"
    }
}


PUT /product-index/_doc/7
{
    "Product": {
        "ProductName": "Esoteric",
        "ProductCode": "CBC",
        "ProductFamily": "Marketing Cloud",
        "Color": "WHITE"
    }
}


PUT /product-index/_doc/8
{
    "Product": {
        "ProductName": "Oncology",
        "ProductCode": "Network",
        "ProductDescription": "CBC",
        "ProductFamily": "Sales Cloud",
        "Color": "WHITE"
    }
}

Create Elasticsearch Index

PUT order-index
{
    "settings": {
        "analysis": {
            "normalizer": {
                "lowercaseNormalizer": {
                    "type": "custom",
                    "filter": [
                        "lowercase",
                        "asciifolding"
                    ]
                }
            }
        }
    },
    "mappings": {
        "properties": {
            "Order": {
                "type": "object",
                "properties": {
                    "OrderCode": {
                        "type": "keyword",
                        "normalizer": "lowercaseNormalizer"
                    },
                    "Description": {
                        "type": "text"
                    },
                    "Family": {
                        "type": "keyword",
                        "normalizer": "lowercaseNormalizer"
                    },
                    "ProductName": {
                        "type": "keyword",
                        "normalizer": "lowercaseNormalizer"
                    },
                    "ProductId": {
                        "type": "keyword"
                    }
                }
            }
        }
    }
}

Adding sample data/records

        
#Add first Document in elasticsearch 
PUT /order-index/_doc/5c57754b-86a6-41f1-a4af-de65b531fe8f
{
    "Order": {
        "OrderCode": "Network",
        "Description": "APAC",
        "Family": "Sales Cloud",
        "ProductName": "CBC Anti Virus",
        "ProductId": 1
    }
}


PUT /order-index/_doc/218c4907-89c2-482f-90cc-61dd7e27f999
{
    "Order": {
        "OrderCode": "Network",
        "Description": "CBC Panel - 1",
        "Family": "Hardware",
        "ProductName": "CBC Esoteric",
        "ProductId": 2
    }
}

View data/regcords

GET /product-index/_search
GET /order-index/_search

Search case-insensitive records

In the request body, provide the search queries in a newline-separated JSON format. Each search query should have two parts: the header and the search body. The header contains information about the index and search type, while the search body contains the actual search query DSL.

#Try 1
POST http://localhost:9200/_msearch
Content-type:application/json
Payload as below


{"index" : "product-index"}
{"query": {"match": {"Product.OrderCode": "Network"}}, "from" : 0, "size" : 10}
{"index" : "order-index"}
{"query": {"match": {"Order.OrderCode": "Network"}}}

Note

Text fields are analyzed.
For keyword fields we need to add custom elasticsearch setting for analysis.

Reference

run elasticsearch locally run elasticsearch locally
Documentation: Elasticsearch - _msearch

Tuesday, 4 July 2023

Multiple Index Query

The following request searches the product-index and order-index indices.

Create Elasticsearch Index

PUT product-index
{
  "settings": {
    "analysis": {
      "normalizer": {
        "lowercaseNormalizer": {
          "type": "custom",
          "filter": ["lowercase", "asciifolding"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "ProductName": {
        "type": "keyword",
        "normalizer": "lowercaseNormalizer"
      },
     "ProductCode": {
        "type": "keyword",
        "normalizer": "lowercaseNormalizer"
      },
      "ProductDescription": {
        "type": "text"
      },
      "ProductFamily": {
        "type": "keyword",
        "normalizer": "lowercaseNormalizer"
      },
      "Color": {
        "type": "keyword"
      }
    }
  }
}

Adding sample data/records

        
#Add first Document in elasticsearch 
PUT /product-index/_doc/1
{
  "ProductName": "CBC Anti Virus",
  "ProductCode": "Network",
  "ProductDescription": "APAC",
  "ProductFamily": "Sales Cloud",
  "Color": "Silver"
}


PUT /product-index/_doc/2
{
  "ProductName": "CBC Esoteric",
  "ProductDescription": "CBC Panel - 1",
  "ProductFamily": "Hardware",
  "Color": "BLUE"
}

PUT /product-index/_doc/3
{
  "ProductName": "CBC Oncology",
  "ProductCode": "CBC Health",
  "ProductFamily": "Software"
}

PUT /product-index/_doc/4
{
  "ProductName": "CBC Panel",
  "ProductCode": "Secure Wifi",
  "ProductDescription": "Only One Charge Type"
}

PUT /product-index/_doc/5
{
  "ProductName": "CBC Panel (Invitro)",
  "ProductCode": "Secure Wifi",
  "ProductDescription": "CBC Panel",
  "ProductFamily": "Hardware",
  "Color": "GREEN"
}


PUT /product-index/_doc/6
{
  "ProductName": "CBC Panel - Beijing",
  "ProductCode": "CBC Panel",
  "ProductDescription": "Only One Charge Type",
  "ProductFamily": "Software",
  "Color": "BLUE"
}


PUT /product-index/_doc/7
{
  "ProductName": "Esoteric",
  "ProductCode": "CBC",
  "ProductFamily": "Marketing Cloud",
  "Color": "WHITE"
}


PUT /product-index/_doc/8
{
  "ProductName": "Oncology",
  "ProductCode": "Network",
  "ProductDescription": "CBC",
  "ProductFamily": "Sales Cloud",
  "Color": "WHITE"
}

Create Elasticsearch Index

PUT order-index
{
    "settings": {
        "analysis": {
            "normalizer": {
                "lowercaseNormalizer": {
                    "type": "custom",
                    "filter": [
                        "lowercase",
                        "asciifolding"
                    ]
                }
            }
        }
    },
    "mappings": {
        "properties": {
            "OrderCode": {
                "type": "keyword",
                "normalizer": "lowercaseNormalizer"
            },
            "Description": {
                "type": "text"
            },
            "Family": {
                "type": "keyword",
                "normalizer": "lowercaseNormalizer"
            },
            "ProductName": {
                "type": "keyword",
                "normalizer": "lowercaseNormalizer"
            },
            "ProductId": {
                "type": "keyword"
            }
        }
    }
}

Adding sample data/records

        
#Add first Document in elasticsearch 
PUT /order-index/_doc/5c57754b-86a6-41f1-a4af-de65b531fe8f
{
    "OrderCode": "Network",
    "Description": "APAC",
    "Family": "Sales Cloud",
    "ProductName": "CBC Anti Virus",
    "ProductId": 1
}


PUT /order-index/_doc/218c4907-89c2-482f-90cc-61dd7e27f999
{
    "OrderCode": "Network",
    "Description": "CBC Panel - 1",
    "Family": "Hardware",
    "ProductName": "CBC Esoteric",
    "ProductId": 2
}

View data/regcords

GET /product-index/_search
GET /order-index/_search

Search case-insensitive records

#Try 1
GET /order-index,product-index/_search
{
    "_source": {
        "include": [
            "Id",
            "OrderCode",
            "ProductId",
            "ProductCode"
        ],
        "exclude": []
    },
    "from": 0,
    "size": 100,
    "track_total_hits": true,
    "query": {
        "bool": {
            "filter": [
                {
                    "bool": {
                        "should": [
                            {
                                "query_string": {
                                    "query": "*Network*",
                                    "fields": [
                                        "ProductCode",
                                        "OrderCode"
                                    ],
                                    "boost": 7
                                }
                            }
                        ]
                    }
                }
            ]
        }
    }
}

Note

Text fields are analyzed.
For keyword fields we need to add custom elasticsearch setting for analysis.

Reference

run elasticsearch locally run elasticsearch locally
Documentation: Elasticsearch - Search multiple data stream and indices

Saturday, 5 November 2022

Elasticsearch Reindex / migration

ElasticSearch: Zero downtime reindexing
ElasticSearch has a solution to the problem, index aliases. The alias is like a symbolic link which can point to one or more indices. It gives us the flexibility to create a new index in the background and making the changes in a way that is almost unnoticeable to the application.

Reference

Elasticsearch reindex APIhttps://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html
Elasticsearch reindex 1 https://nasa.github.io/cumulus/docs/troubleshooting/reindex-elasticsearch
reindex status https://nasa.github.io/cumulus-api/#reindex-status
Elasticsearch reindex explained https://youtu.be/nME4ZjfjwiA
Elasticsearch reindex needs https://dzone.com/articles/elasticsearch-fault-tolerance-reindexing-need-and-1
Elasticsearch reindex steps https://www.thirdrocktechkno.com/blog/6-steps-to-reindex-elasticsearch-data/

Friday, 4 November 2022

Elasticsearch

Elasticsearch is the distributed search and analytics engine at the heart of the Elastic Stack.

Reference

What is Elasticsearch? https://youtu.be/ZP0NmfyfsoM

Thursday, 3 November 2022

AWS OpenSearch Deployment Best Practices

3 Dedicated Master Nodes

Offload cluster managment task like health checks and maintain routing and cluster state.

Data Nodes deployed in multiples of 3

Store the data in shards and perform searches, query request, and CRUD operations.

Deploy across 3AZs

Destribute the data nodes equally across 3 AZs for the hightest availability.

Reference

ES: https://www.elastic.co/guide/en/elasticsearch/reference/8.5/elasticsearch-intro.html

Monday, 8 March 2021

Elasticsearch for Stocks

Store the BSE/NSE daily stock feeds in Elasticsearch for research.

Create Elasticsearch Index

PUT stock-index
{
  "settings": {
    "index": {
      "number_of_shards": 1,
      "number_of_replicas": 2,
      "analysis": {
        "normalizer": {
          "lowercaseNormalizer": {
            "type": "custom",
            "filter": [
              "lowercase",
              "asciifolding"
            ]
          }
        }
      }
    }
  },
  "mappings": {
    "dynamic": "false",
    "properties": {
      "Symbol": {
        "type": "keyword",
        "normalizer": "lowercaseNormalizer"
      },
      "CreatedDate": {
        "type": "date",
        "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
      },
      "Open": {
        "type": "double"
      },
      "High": {
        "type": "double"
      },
      "Low": {
        "type": "double"
      },
      "Close": {
        "type": "double"
      },
      "Volume": {
        "type": "double"
      },
      "FeedProvider": {
        "type": "text"
      }
    }
  }
}

Adding sample data/records

        
#Add first Stock Document in elasticsearch 

PUT /stock-index/_doc/1
{
  "Symbol": "SBI",
  "CreatedDate": "2021-03-08",
    "Close": 389.6,
    "Open": 388.45,
    "High": 393.4,
    "Low": 386.05,
  "FeedProvider": "zerodha"
}

PUT /stock-index/_doc/2
{
  "Symbol": "SBI",
  "CreatedDate": "2021-03-09",
    "Close": 387.6,
    "Open": 393.45,
    "High": 394.9,
    "Low": 382.25,
  "FeedProvider": "zerodha"
}

PUT /stock-index/_doc/3
{
  "Symbol": "SBI",
  "CreatedDate": "2021-03-10",
    "Close": 387.7,
    "Open": 391.1,
    "High": 393,
    "Low": 385.7,
  "FeedProvider": "zerodha"
}

PUT /stock-index/_doc/4
{
  "Symbol": "SBI",
 "CreatedDate": "2021-03-12",
    "Close": 381.1,
    "Open": 391.7,
    "High": 393.2,
    "Low": 377.5,
  "FeedProvider": "zerodha"
}


PUT /stock-index/_doc/5
{
  "Symbol": "SBI",
  "CreatedDate": "2021-03-15",
    "Close": 383.9,
    "Open": 382.5,
    "High": 386,
    "Low": 369.35,
  "FeedProvider": "zerodha"
}


PUT /stock-index/_doc/6
{
  "Symbol": "SBI",
  "CreatedDate": "2021-03-16",
    "Close": 378.55,
    "Open": 385.6,
    "High": 387.85,
    "Low": 376.1,
  "FeedProvider": "zerodha"
}

PUT /stock-index/_doc/7
{
  "Symbol": "SBI",
   "CreatedDate": "2021-03-17",
    "Close": 368.15,
    "Open": 378.2,
    "High": 379.65,
    "Low": 366.05,
  "FeedProvider": "zerodha"
}


PUT /stock-index/_doc/8
{
  "Symbol": "SBI",
   "CreatedDate": "2021-03-18",
    "Close": 367.1,
    "Open": 371.5,
    "High": 375,
    "Low": 361.1,
  "FeedProvider": "zerodha"
}

PUT /stock-index/_doc/9
{
  "Symbol": "SBI",
 "CreatedDate": "2021-03-19",
    "Close": 371.15,
    "Open": 363.95,
    "High": 372.9,
    "Low": 357.25,
  "FeedProvider": "zerodha"
}

PUT /stock-index/_doc/10
{
  "Symbol": "SBI",
   "CreatedDate": "2021-03-22",
    "Close": 367,
    "Open": 372,
    "High": 372.8,
    "Low": 363.5,
  "FeedProvider": "zerodha"
}

PUT /stock-index/_doc/11
{
  "Symbol": "SBI",
   "CreatedDate": "2021-03-23",
    "Close": 372.7,
    "Open": 368.7,
    "High": 377.95,
    "Low": 367.05,
  "FeedProvider": "zerodha"
}

PUT /stock-index/_doc/12
{
  "Symbol": "SBI",
  "CreatedDate": "2021-03-24",
    "Close": 359.85,
    "Open": 368.5,
    "High": 369.05,
    "Low": 358.65,
  "FeedProvider": "zerodha"
}

PUT /stock-index/_doc/13
{
  "Symbol": "SBI",
   "CreatedDate": "2021-03-25",
    "Close": 355.2,
    "Open": 360,
    "High": 360.85,
    "Low": 345.2,
  "FeedProvider": "zerodha"
}

PUT /stock-index/_doc/14
{
  "Symbol": "SBI",
  "CreatedDate": "2021-03-26",
    "Close": 356.4,
    "Open": 360,
    "High": 361.8,
    "Low": 354.95,
  "FeedProvider": "zerodha"
}


GET /stock-index/_search
{
  "query": {
    "match": {
      "Symbol": "SBI"
    }
  }
}

View data/regcords

GET /stock-index/_search

Searching .

Search a particular stock query or field as belows

#Try 1
GET /stock-index/_search
{
  "query": {
    "match": {
      "Symbol": "SBI"
    }
  }
}

#Try
GET /stock-index/_search
{
   "query": {
     "range": {
       "Close": {
         "gte": 100,
         "lte": 389
       }
     }
   }
}

#Try
GET /stock-index/_search
{
   "query": {
     "range": {
       "CreatedDate": {
         "gte": "2021-03-08",
         "lte": "2021-03-09",
          "format": "yyyy-MM-dd"
       }
     }
   }
}

#Try
GET /stock-index/_search
{
  "size": 10,
  "query": {
    "range": {
      "CreatedDate": {
        "gte": "now-90d/d",
        "lte": "now/d",
        "format": "yyyy-MM-dd"
      }
    }
  }
}


#Simple moving avg
GET /stock-index/_search
{
  "size": 0,
  "aggs": {
    "hourly_data": {
      "date_histogram": {
        "field": "CreatedDate",
        "interval": "day"
      },
      "aggs": {
        "stock_value": {
          "sum": {
            "field": "Close"
          }
        },
        "mva_demo": {
          "moving_avg": {
            "buckets_path": "stock_value",
            "window": 5,
            "model": "simple"
          }
        }
      }
    }
  }
}
 


#Linear Moving Average
GET /stock-index/_search
{
  "size": 0,
  "aggs": {
    "hourly_data": {
      "date_histogram": {
        "field": "CreatedDate",
        "interval": "day"
      },
      "aggs": {
        "stock_value": {
          "sum": {
            "field": "Close"
          }
        },
        "mva_simple": {
          "moving_avg": {
            "buckets_path": "stock_value",
            "window": 10,
            "model": "simple"
          }
        },
        "mva_linear": {
          "moving_avg": {
            "buckets_path": "stock_value",
            "window": 10,
            "model": "linear"
          }
        }
      }
    }
  }
}


#Exponentially Weighted Moving Average
GET /stock-index/_search
{
  "size": 0,
  "aggs": {
    "hourly_data": {
      "date_histogram": {
        "field": "CreatedDate",
        "interval": "day"
      },
      "aggs": {
        "stock_value": {
          "sum": {
            "field": "Close"
          }
        },
        "mva_simple": {
          "moving_avg": {
            "buckets_path": "stock_value",
            "window": 10,
            "model": "simple"
          }
        },
        "mva_linear": {
          "moving_avg": {
            "buckets_path": "stock_value",
            "window": 10,
            "model": "linear"
          }
        },
        "mva_ewma": {
          "moving_avg": {
            "buckets_path": "stock_value",
            "window": 10,
            "model": "ewma",
            "settings": {
              "alpha": 0.3
            }
          }
        }
      }
    }
  }
}

 #minimum of the price monthly
 GET /stock-index/_search
 {
  "size": 0,
  "aggs": {
    "price_per_month": {
      "date_histogram": {
        "field": "CreatedDate",
        "calendar_interval": "month"
      },
      "aggs": {
        "price": {
          "min": {
            "field": "Close"
          }
        }
      }
    },
    "min_monthly_price": {
      "min_bucket": {
        "buckets_path": "price_per_month>price" 
      }
    }
  }
}


#Moving Min 
GET /stock-index/_search
{
  "size": 0,
  "aggs": {
    "day_data": {
      "date_histogram": {
        "field": "CreatedDate",
        "interval": "day"
      },
      "aggs": {
        "stock_value": {
          "sum": {
            "field": "Close"
          }
        },
         "the_movfn": {
          "moving_fn": {
            "buckets_path": "stock_value",  
            "window": 10,
            "script": "MovingFunctions.min(values)"
          }
        }
      }
    }
  }
}

Reference

Documentation: BSE
Aggs: ES search-aggregations

Sunday, 10 January 2021

Elasticsearch - Boost, aggs

Boosting is the process by which you can modify the relevance of a document. You can boost a document while you are indexing it or when you query for the document.we definitely recommend you use the query-time boosting because it’s the most flexible and allows you to change your mind about what fields or terms are important.

Create Elasticsearch Index

PUT product-index
{
  "settings": {
    "analysis": {
      "normalizer": {
        "lowercaseNormalizer": {
          "type": "custom",
          "filter": ["lowercase", "asciifolding"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "ProductName": {
        "type": "keyword",
        "normalizer": "lowercaseNormalizer"
      },
     "ProductCode": {
        "type": "keyword",
        "normalizer": "lowercaseNormalizer"
      },
      "ProductDescription": {
        "type": "text"
      },
      "ProductFamily": {
        "type": "keyword",
        "normalizer": "lowercaseNormalizer"
      },
      "Color": {
        "type": "keyword"
      }
    }
  }
}

Adding sample data/records

        
#Add first Document in elasticsearch 
PUT /product-index/_doc/1
{
  "ProductName": "CBC Anti Virus",
  "ProductCode": "Network",
  "ProductDescription": "APAC",
  "ProductFamily": "Sales Cloud",
  "Color": "Silver"
}


PUT /product-index/_doc/2
{
  "ProductName": "CBC Esoteric",
  "ProductDescription": "CBC Panel - 1",
  "ProductFamily": "Hardware",
  "Color": "BLUE"
}

PUT /product-index/_doc/3
{
  "ProductName": "CBC Oncology",
  "ProductCode": "CBC Health",
  "ProductFamily": "Software"
}

PUT /product-index/_doc/4
{
  "ProductName": "CBC Panel",
  "ProductCode": "Secure Wifi",
  "ProductDescription": "Only One Charge Type"
}

PUT /product-index/_doc/5
{
  "ProductName": "CBC Panel (Invitro)",
  "ProductCode": "Secure Wifi",
  "ProductDescription": "CBC Panel",
  "ProductFamily": "Hardware",
  "Color": "GREEN"
}


PUT /product-index/_doc/6
{
  "ProductName": "CBC Panel - Beijing",
  "ProductCode": "CBC Panel",
  "ProductDescription": "Only One Charge Type",
  "ProductFamily": "Software",
  "Color": "BLUE"
}


PUT /product-index/_doc/7
{
  "ProductName": "Esoteric",
  "ProductCode": "CBC",
  "ProductFamily": "Marketing Cloud",
  "Color": "WHITE"
}


PUT /product-index/_doc/8
{
  "ProductName": "Oncology",
  "ProductCode": "Network",
  "ProductDescription": "CBC",
  "ProductFamily": "Sales Cloud",
  "Color": "WHITE"
}

View data/regcords

GET /product-index/_search

Searching with relevancy

Boosting the score of a particular query or field as belows

#Try 1
POST /product-index/_search
{
  "query": {
    "match": {
      "ProductFamily": {
        "query": "hArdware",
        "boost": 2
      }
    }
  }
}


#Try 2
POST /product-index/_search
{
  "query": {
    "bool": {
      "should": [
        {
          "query_string": {
            "query": "*Wifi*",
            "default_field": "ProductCode",
            "boost": 3
          }
        },
        {
          "match": {
            "ProductFamily": {
              "query": "hArdware",
              "boost": 4
            }
          }
        }
      ]
    }
  }
}

#Try 3
GET /product-index/_search
{
  "query": {
    "query_string": {
      "query": "*Secure* OR *panel*",
      "default_field": "ProductCode",
      "boost": 1
    }
  }
}

#Try 4
POST /product-index/_search
{
  "query": {
    "bool": {
      "should": [
        {
          "query_string": {
            "query": "*CBC*",
            "default_field": "ProductName",
            "boost": 100
          }
        },
         {
          "query_string": {
            "query": "*hard*",
            "default_field": "ProductFamily",
            "boost": 91
          }
        },
        {
          "query_string": {
            "query": "*one*",
            "default_field": "ProductDescription",
            "boost": 92
          }
        },
        {
          "query_string": {
            "query": "*Blue*",
            "default_field": "Color",
            "boost": 90
          }
        },
        {
          "query_string": {
            "query": "*Health*",
            "default_field": "ProductCode",
            "boost": 93
          }
        }
      ]
    }
  }
}


#Try 5
POST /product-index/_search
{
  "query": {
    "bool": {
      "should": [
        {
          "query_string": {
            "query": "*cbc*",
            "default_field": "ProductName",
            "boost": 49
          }
        },
        {
          "query_string": {
            "query": "*cbc*",
            "default_field": "ProductCode",
            "boost": 7
          }
        },
        {
          "query_string": {
            "query": "*cbc*",
            "fields": [ "ProductDescription", "ProductFamily"]
          }
        }
      ]
    }
  }
}

#Try 6
GET /product-index/_search
{
  "query" : {
    "query_string" : {
      "query" : "*cbc*",
      "fields"  : ["*"]
    }
  }
}


#Try 7
POST /product-index/_search
{
  "query": {
    "bool": {
      "should": [
        {
          "query_string": {
            "query": "*cbc*",
            "default_field": "ProductName",
            "boost": 49
          }
        },
        {
          "query_string": {
            "query": "*cbc*",
            "default_field": "ProductCode",
            "boost": 7
          }
        },
        {
          "query_string": {
            "query": "*cbc*",
            "fields": [ "*"]
          }
        }
      ]
    }
  }
}



#Try Demonstration for aggregation
GET /product-index/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "match": { "ProductFamily" : "hArdware"}
        }
      ]
    }
  },
  "aggs": {
    "Family": {
      "terms": {
        "field": "ProductFamily"
      }
    }
  }
}

#Try Another example of aggregation
GET /product-index/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "match": {
            "ProductFamily": "hArdware"
          }
        },
        {
          "bool": {
            "should": [
              {
                "query_string": {
                  "query": "*cbc* OR *network*",
                  "fields": [
                    "ProductName",
                    "ProductCode",
                    "ProductDescription",
                    "ProductFamily",
                    "Color"
                  ]
                }
              }
            ]
          }
        }
      ]
    }
  },
  "aggs": {
    "Family": {
      "terms": {
        "field": "ProductFamily"
      }
    },
    "Color": {
      "terms": {
        "field": "Color"
      }
    }
  },
  "_source": [
    "ProductName",
    "ProductCode",
    "ProductDescription",
    "ProductFamily",
    "Color"
    ]
}

Reference

Documentation: Elasticsearch Boost elasticsearch-in-action

Tuesday, 5 January 2021

Elasticsearch case-insensitive searches

A simple normalizer called lowercase ships with elasticsearch and can be used.

Create Elasticsearch Index

PUT product-index
{
  "settings": {
    "analysis": {
      "normalizer": {
        "lowercaseNormalizer": {
          "type": "custom",
          "filter": ["lowercase", "asciifolding"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "ProductName": {
        "type": "keyword",
        "normalizer": "lowercaseNormalizer"
      },
     "ProductCode": {
        "type": "keyword",
        "normalizer": "lowercaseNormalizer"
      },
      "ProductDescription": {
        "type": "text"
      },
      "ProductFamily": {
        "type": "keyword",
        "normalizer": "lowercaseNormalizer"
      },
      "Color": {
        "type": "keyword"
      }
    }
  }
}

Adding sample data/records

        
#Add first Document in elasticsearch 
PUT /product-index/_doc/1
{
  "ProductName": "CBC Anti Virus",
  "ProductCode": "Network",
  "ProductDescription": "APAC",
  "ProductFamily": "Sales Cloud",
  "Color": "Silver"
}


PUT /product-index/_doc/2
{
  "ProductName": "CBC Esoteric",
  "ProductDescription": "CBC Panel - 1",
  "ProductFamily": "Hardware",
  "Color": "BLUE"
}

PUT /product-index/_doc/3
{
  "ProductName": "CBC Oncology",
  "ProductCode": "CBC Health",
  "ProductFamily": "Software"
}

PUT /product-index/_doc/4
{
  "ProductName": "CBC Panel",
  "ProductCode": "Secure Wifi",
  "ProductDescription": "Only One Charge Type"
}

PUT /product-index/_doc/5
{
  "ProductName": "CBC Panel (Invitro)",
  "ProductCode": "Secure Wifi",
  "ProductDescription": "CBC Panel",
  "ProductFamily": "Hardware",
  "Color": "GREEN"
}


PUT /product-index/_doc/6
{
  "ProductName": "CBC Panel - Beijing",
  "ProductCode": "CBC Panel",
  "ProductDescription": "Only One Charge Type",
  "ProductFamily": "Software",
  "Color": "BLUE"
}


PUT /product-index/_doc/7
{
  "ProductName": "Esoteric",
  "ProductCode": "CBC",
  "ProductFamily": "Marketing Cloud",
  "Color": "WHITE"
}


PUT /product-index/_doc/8
{
  "ProductName": "Oncology",
  "ProductCode": "Network",
  "ProductDescription": "CBC",
  "ProductFamily": "Sales Cloud",
  "Color": "WHITE"
}

View data/regcords

GET /product-index/_search

Search case-insensitive records

#Try 1
GET /product-index/_search
{
  "query": {
    "match": { "ProductFamily" : "hArdware"}
  }
}

#Try 2
GET /product-index/_search
{
  "query" : {
    "query_string" : {
      "query" : "*bei*",
      "fields"  : ["ProductName"]
    }
  }
}

#Try 3
GET /product-index/_search
{
  "query": {
    "query_string": {
      "query": "*Secure* OR *panel*",
      "default_field": "ProductCode"
    }
  }
}

#Try 4
GET /product-index/_search
{
  "query" : {
    "query_string" : {
      "query" : "*cHarge*",
      "fields"  : ["ProductDescription"]
    }
  }
}

Note

Text fields are analyzed.
For keyword fields we need to add custom elasticsearch setting for analysis.

Reference

run elasticsearch locally run elasticsearch locally
Documentation: Elasticsearch normalizer

Sunday, 27 December 2020

Term Vs Match queries - Elasticsearch

Term	Match Queries
Looks for the exact term in the inverted index.	Queries are analyzed before looking up the inverted index.
Does not know of the presence of the analyzer.	Understands how the fields have ben analyzed.
Great for keywords, numbers, dates where exact matches are important.	Useful when searching full text fields with a large body of text.
Less likely to match irrelevant documents.	More likely to match irrelevant documents.
---	Does not go through a query parsing process, does not support wildcards, prefixes etc.

Search This Blog

Monday, 29 April 2024

Elastic Stack Video - Load your gun in short time.

Tuesday, 1 August 2023

Create Elasticsearch Index

Adding sample data/records

Create Elasticsearch Index

Adding sample data/records

View data/regcords

Search case-insensitive records

Note

Reference

Tuesday, 4 July 2023

Create Elasticsearch Index

Adding sample data/records

Create Elasticsearch Index

Adding sample data/records

View data/regcords

Search case-insensitive records

Note

Reference

Saturday, 5 November 2022

Reference

Friday, 4 November 2022

Reference

Thursday, 3 November 2022

Reference

Monday, 8 March 2021

Create Elasticsearch Index

Adding sample data/records

View data/regcords

Searching .

Reference

Sunday, 10 January 2021

Create Elasticsearch Index

Adding sample data/records

View data/regcords

Searching with relevancy

Reference

Tuesday, 5 January 2021

Create Elasticsearch Index

Adding sample data/records

View data/regcords

Search case-insensitive records

Note

Reference

Sunday, 27 December 2020

Recent Post