| Data Analytics | Data Science |
|---|---|
| Definition Process and analyze data sets to model and communicate answer to specific questions with actionable data. |
A multi-disciplinary field that specializes in exposing insights from large data sets. |
Skills
|
|
Goals
|
|
Friday, 18 August 2023
Data Analytics VS Data Science
Tuesday, 1 August 2023
Programming eBooks - read online
Search the world's most comprehensive index of full-text books.
Programming eBooks - https://books.google.co.in/books
Elasticsearch _msearch
When using the msearch (multi-search) API in Elasticsearch, you can search across multiple indices in a single request. Each search query in the msearch request can be targeted to a specific index..
Create Elasticsearch Index
PUT product-index
{
"settings": {
"analysis": {
"normalizer": {
"lowercaseNormalizer": {
"type": "custom",
"filter": [
"lowercase",
"asciifolding"
]
}
}
}
},
"mappings": {
"properties": {
"Product": {
"type": "object",
"properties": {
"ProductName": {
"type": "keyword",
"normalizer": "lowercaseNormalizer"
},
"ProductCode": {
"type": "keyword",
"normalizer": "lowercaseNormalizer"
},
"ProductDescription": {
"type": "text"
},
"ProductFamily": {
"type": "keyword",
"normalizer": "lowercaseNormalizer"
},
"Color": {
"type": "keyword"
}
}
}
}
}
}
Adding sample data/records
#Add first Document in elasticsearch
PUT /product-index/_doc/1
{
"Product": {
"ProductName": "CBC Anti Virus",
"ProductCode": "Network",
"ProductDescription": "APAC",
"ProductFamily": "Sales Cloud",
"Color": "Silver"
}
}
PUT /product-index/_doc/2
{
"Product": {
"ProductName": "CBC Esoteric",
"ProductDescription": "CBC Panel - 1",
"ProductFamily": "Hardware",
"Color": "BLUE"
}
}
PUT /product-index/_doc/3
{
"Product": {
"ProductName": "CBC Oncology",
"ProductCode": "CBC Health",
"ProductFamily": "Software"
}
}
PUT /product-index/_doc/4
{
"Product": {
"ProductName": "CBC Panel",
"ProductCode": "Secure Wifi",
"ProductDescription": "Only One Charge Type"
}
}
PUT /product-index/_doc/5
{
"Product": {
"ProductName": "CBC Panel (Invitro)",
"ProductCode": "Secure Wifi",
"ProductDescription": "CBC Panel",
"ProductFamily": "Hardware",
"Color": "GREEN"
}
}
PUT /product-index/_doc/6
{
"Product": {
"ProductName": "CBC Panel - Beijing",
"ProductCode": "CBC Panel",
"ProductDescription": "Only One Charge Type",
"ProductFamily": "Software",
"Color": "BLUE"
}
}
PUT /product-index/_doc/7
{
"Product": {
"ProductName": "Esoteric",
"ProductCode": "CBC",
"ProductFamily": "Marketing Cloud",
"Color": "WHITE"
}
}
PUT /product-index/_doc/8
{
"Product": {
"ProductName": "Oncology",
"ProductCode": "Network",
"ProductDescription": "CBC",
"ProductFamily": "Sales Cloud",
"Color": "WHITE"
}
}
Create Elasticsearch Index
PUT order-index
{
"settings": {
"analysis": {
"normalizer": {
"lowercaseNormalizer": {
"type": "custom",
"filter": [
"lowercase",
"asciifolding"
]
}
}
}
},
"mappings": {
"properties": {
"Order": {
"type": "object",
"properties": {
"OrderCode": {
"type": "keyword",
"normalizer": "lowercaseNormalizer"
},
"Description": {
"type": "text"
},
"Family": {
"type": "keyword",
"normalizer": "lowercaseNormalizer"
},
"ProductName": {
"type": "keyword",
"normalizer": "lowercaseNormalizer"
},
"ProductId": {
"type": "keyword"
}
}
}
}
}
}
Adding sample data/records
#Add first Document in elasticsearch
PUT /order-index/_doc/5c57754b-86a6-41f1-a4af-de65b531fe8f
{
"Order": {
"OrderCode": "Network",
"Description": "APAC",
"Family": "Sales Cloud",
"ProductName": "CBC Anti Virus",
"ProductId": 1
}
}
PUT /order-index/_doc/218c4907-89c2-482f-90cc-61dd7e27f999
{
"Order": {
"OrderCode": "Network",
"Description": "CBC Panel - 1",
"Family": "Hardware",
"ProductName": "CBC Esoteric",
"ProductId": 2
}
}
View data/regcords
GET /product-index/_search
GET /order-index/_search
Search case-insensitive records
In the request body, provide the search queries in a newline-separated JSON format. Each search query should have two parts: the header and the search body. The header contains information about the index and search type, while the search body contains the actual search query DSL.
#Try 1
POST http://localhost:9200/_msearch
Content-type:application/json
Payload as below
{"index" : "product-index"}
{"query": {"match": {"Product.OrderCode": "Network"}}, "from" : 0, "size" : 10}
{"index" : "order-index"}
{"query": {"match": {"Order.OrderCode": "Network"}}}
Note
- Text fields are analyzed.
- For keyword fields we need to add custom elasticsearch setting for analysis.
Reference
- run elasticsearch locally run elasticsearch locally
- Documentation: Elasticsearch - _msearch
Tuesday, 4 July 2023
Multiple Index Query
The following request searches the product-index and order-index indices.
Create Elasticsearch Index
PUT product-index
{
"settings": {
"analysis": {
"normalizer": {
"lowercaseNormalizer": {
"type": "custom",
"filter": ["lowercase", "asciifolding"]
}
}
}
},
"mappings": {
"properties": {
"ProductName": {
"type": "keyword",
"normalizer": "lowercaseNormalizer"
},
"ProductCode": {
"type": "keyword",
"normalizer": "lowercaseNormalizer"
},
"ProductDescription": {
"type": "text"
},
"ProductFamily": {
"type": "keyword",
"normalizer": "lowercaseNormalizer"
},
"Color": {
"type": "keyword"
}
}
}
}
Adding sample data/records
#Add first Document in elasticsearch
PUT /product-index/_doc/1
{
"ProductName": "CBC Anti Virus",
"ProductCode": "Network",
"ProductDescription": "APAC",
"ProductFamily": "Sales Cloud",
"Color": "Silver"
}
PUT /product-index/_doc/2
{
"ProductName": "CBC Esoteric",
"ProductDescription": "CBC Panel - 1",
"ProductFamily": "Hardware",
"Color": "BLUE"
}
PUT /product-index/_doc/3
{
"ProductName": "CBC Oncology",
"ProductCode": "CBC Health",
"ProductFamily": "Software"
}
PUT /product-index/_doc/4
{
"ProductName": "CBC Panel",
"ProductCode": "Secure Wifi",
"ProductDescription": "Only One Charge Type"
}
PUT /product-index/_doc/5
{
"ProductName": "CBC Panel (Invitro)",
"ProductCode": "Secure Wifi",
"ProductDescription": "CBC Panel",
"ProductFamily": "Hardware",
"Color": "GREEN"
}
PUT /product-index/_doc/6
{
"ProductName": "CBC Panel - Beijing",
"ProductCode": "CBC Panel",
"ProductDescription": "Only One Charge Type",
"ProductFamily": "Software",
"Color": "BLUE"
}
PUT /product-index/_doc/7
{
"ProductName": "Esoteric",
"ProductCode": "CBC",
"ProductFamily": "Marketing Cloud",
"Color": "WHITE"
}
PUT /product-index/_doc/8
{
"ProductName": "Oncology",
"ProductCode": "Network",
"ProductDescription": "CBC",
"ProductFamily": "Sales Cloud",
"Color": "WHITE"
}
Create Elasticsearch Index
PUT order-index
{
"settings": {
"analysis": {
"normalizer": {
"lowercaseNormalizer": {
"type": "custom",
"filter": [
"lowercase",
"asciifolding"
]
}
}
}
},
"mappings": {
"properties": {
"OrderCode": {
"type": "keyword",
"normalizer": "lowercaseNormalizer"
},
"Description": {
"type": "text"
},
"Family": {
"type": "keyword",
"normalizer": "lowercaseNormalizer"
},
"ProductName": {
"type": "keyword",
"normalizer": "lowercaseNormalizer"
},
"ProductId": {
"type": "keyword"
}
}
}
}
Adding sample data/records
#Add first Document in elasticsearch
PUT /order-index/_doc/5c57754b-86a6-41f1-a4af-de65b531fe8f
{
"OrderCode": "Network",
"Description": "APAC",
"Family": "Sales Cloud",
"ProductName": "CBC Anti Virus",
"ProductId": 1
}
PUT /order-index/_doc/218c4907-89c2-482f-90cc-61dd7e27f999
{
"OrderCode": "Network",
"Description": "CBC Panel - 1",
"Family": "Hardware",
"ProductName": "CBC Esoteric",
"ProductId": 2
}
View data/regcords
GET /product-index/_search
GET /order-index/_search
Search case-insensitive records
#Try 1
GET /order-index,product-index/_search
{
"_source": {
"include": [
"Id",
"OrderCode",
"ProductId",
"ProductCode"
],
"exclude": []
},
"from": 0,
"size": 100,
"track_total_hits": true,
"query": {
"bool": {
"filter": [
{
"bool": {
"should": [
{
"query_string": {
"query": "*Network*",
"fields": [
"ProductCode",
"OrderCode"
],
"boost": 7
}
}
]
}
}
]
}
}
}
Note
- Text fields are analyzed.
- For keyword fields we need to add custom elasticsearch setting for analysis.
Reference
- run elasticsearch locally run elasticsearch locally
- Documentation: Elasticsearch - Search multiple data stream and indices
Saturday, 5 November 2022
Elasticsearch Reindex / migration
ElasticSearch: Zero downtime reindexing
ElasticSearch has a solution to the problem, index aliases. The alias is like a symbolic link which can point to one or more indices. It gives us the flexibility to create a new index in the background and making the changes in a way that is almost unnoticeable to the application.
Reference
- Elasticsearch reindex APIhttps://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html
- Elasticsearch reindex 1 https://nasa.github.io/cumulus/docs/troubleshooting/reindex-elasticsearch
- reindex status https://nasa.github.io/cumulus-api/#reindex-status
- Elasticsearch reindex explained https://youtu.be/nME4ZjfjwiA
- Elasticsearch reindex needs https://dzone.com/articles/elasticsearch-fault-tolerance-reindexing-need-and-1
- Elasticsearch reindex steps https://www.thirdrocktechkno.com/blog/6-steps-to-reindex-elasticsearch-data/
Friday, 4 November 2022
Elasticsearch
Elasticsearch is the distributed search and analytics engine at the heart of the Elastic Stack.
Reference
- What is Elasticsearch? https://youtu.be/ZP0NmfyfsoM
Thursday, 3 November 2022
AWS OpenSearch Deployment Best Practices
- 3 Dedicated Master Nodes
- Data Nodes deployed in multiples of 3
- Deploy across 3AZs
Offload cluster managment task like health checks and maintain routing and cluster state.
Store the data in shards and perform searches, query request, and CRUD operations.
Destribute the data nodes equally across 3 AZs for the hightest availability.
Subscribe to:
Posts (Atom)