Metric types

• Counter : A counter is a cumulative metric that represents a single monotonically increasing counter whose value can only increase or be reset to zero on restart.
• Gauge: A gauge is a metric that represents a single numerical value that can arbitrarily go up and down.
• Histogram: A histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. It also provides a sum of all observed values.
• Summary: Similar to a histogram, a summary samples observations (usually things like request durations and response sizes).

when to use which metric type?

• Counters are values that can only increase that keep track of the number of events. For example, we would use counters to start a number of requests, number of errors or the number of Finnish transactions.
• Gauge is another type of metric that represents numerical value on like counters. Gauges can go up and down to represent the current state of the system. We can use gauges for information that can both increase and decrease over time. For example, memory usage, CPU usage temperature but also a number of conquering requests.
• Histogram sample observations and count them in configurable buckets. This way we can check how many requests finish within a given time or how many responses fit within a given size, Histogram exposed several metrics for us to query.
• Summaries calculate configurable. lt's over a sliding time window. In general, many use cases of summaries can be covered with Histogram.

Reference

• Documentation: Metric Types

How to setup Grafan on Local using Docker

How to setup Grafan on Local using Docker.

Steps as below

1. Pull docker image:
   docker pull grafana/grafana:6.4.3
2. Run docker image:
   docker run -d --name=grafana -p 3000:3000 grafana/grafana:6.4.3
3. Open/Verify Graphite app:
   localhost:3000
   Note: Default User/Password is admin/admin

Screenshot as below

Reference Material

• Grafana app
• Installing Grafana 6.4.3
• Official Docker image for Grafana
• Grafana 6.4.3 image

Observability

What are Observability?

Observability is the activities that involve measuring, collecting, and analyzing various diagnostics signals from a system. These signals may include metrics, traces, logs, events, profiles and more

• Log aggregation
• Application metrics
• Audit logging
• Distributed tracing
• Exception tracking
• Health check API
• Log deployments and changes

Log aggregation

Definition Use a centralized logging service that aggregates logs from each service instance. The users can search and analyze the logs. They can configure alerts that are triggered when certain messages appear in the logs.

Issue: Handling a large volume of logs requires substantial infrastructure.
Note: Any solution should have minimal runtime overhead.

Application metrics

DefinitionInstrument a service to gather statistics about individual operations. Aggregate metrics in centralized metrics service, which provides reporting and alerting. There are two models for aggregating metrics:

• push - the service pushes metrics to the metrics service
• pull - the metrics services pulls metrics from the service

Examples

• Instrumentation libraries:
  • Prometheus client libraries
• Metrics aggregation services:
  • Prometheus

Benefits:It provides deep insight into application behavior
Drawbacks: Metrics code is intertwined with business logic making it more complicated
Issues: Aggregating metrics can require significant infrastructure

Audit logging

Definition: Record user activity in a database.

Benefits: Provides a record of user actions
Drawbacks: The auditing code is intertwined with the business logic, which makes the business logic more complicated

Distributed Tracing

Definition:

• Assigns each external request a unique external request id
• Passes the external request id to all services that are involved in handling the request
• Includes the external request id in all log messages
• Records information (e.g. start time, end time) about the requests and operations performed when handling a external request in a centralized service

Benefits:

• It provides useful insight into the behavior of the system including the sources of latency
• It enables developers to see how an individual request is handled by searching across aggregated logs for its external request id

Issues: Aggregating and storing traces can require significant infrastructure
Note:

• External monitoring only tells you the overall response time and number of invocations - no insight into the individual operations
• Any solution should have minimal runtime overhead
• Log entries for a request are scattered across numerous logs

Tools:

• Jaeger: open source, end-to-end distributed tracing Monitor and troubleshoot transactions in complex distributed systems
• Open Zipkin - service for recording and displaying tracing information
• Open Tracing - standardized API for distributed tracing

Exception tracking

Definition Report all exceptions to a centralized exception tracking service that aggregates and tracks exceptions and notifies developers.

Benefits: It is easier to view exceptions and track their resolution
Drawbacks: The exception tracking service is additional infrastructure
Note:

• Exceptions must be de-duplicated, recorded, investigated by developers and the underlying issue resolved
• Any solution should have minimal runtime overhead

Health Check API

Definition A service has an health check API endpoint (e.g. HTTP /health) that returns the health of the service. The API endpoint handler performs various checks, such as

• the status of the connections to the infrastructure services used by the service instance
• the status of the host, e.g. disk space
• application specific logic

A health check client - a monitoring service, service registry or load balancer - periodically invokes the endpoint to check the health of the service instance.

Benefits: The health check endpoint enables the health of a service instance to be periodically tested
Drawbacks: The health check might not sufficiently comprehensive or the service instance might fail between health checks and so requests might still be routed to a failed service instance

Log deployments and changes

Definition Log every deployment and every change to the (production) environment.

Benefits: Enables deployments and changes to be easily correlated with issues leading to a faster resolution.

Technology stack for Observability

• https://opentracing.io/ - OpenTracing is not a download or a program. Distributed tracing requires that software developers add instrumentation to the code of an application, or to the frameworks used in the application.

Reference

• Observability challenges in microservices
• https://opentracing.io/registry/

Started with ASP.NET Core Logging

ASP.NET Core using Microsoft.Extensions.Logging

.Net Core Logging : .NET Core supports a logging API that works with a variety of built-in and third-party logging providers.

Video

Quick start with below video


Quick start with below video

Reference

• Logging in .NET Core and ASP.NET Core