Master Software Architecture: Module 7 - Microservices & Distributed Systems

Full Course Link : https://imomins.blogspot.com/2025/08/master-software-architecture-complete.html

Table of Contents

1. Introduction: The Monolithic Mountain and the Microservices Maze

2. Core Principles and Tangible Benefits of Microservices

3. Service Communication: The Nervous System of Your Architecture

   • 3.1. Synchronous Communication with RESTful APIs (ASP.NET Core Web APIs)

   • 3.2. High-Performance gRPC for Internal Services

   • 3.3. Asynchronous Communication with Message Queues (RabbitMQ)

4. Orchestrating Traffic: The API Gateway and Service Discovery

   • 4.1. The API Gateway Pattern (Ocelot Example)

   • 4.2. Service Discovery: Eureka vs. Client-Side (Consul)

5. Building Resilience: Embracing Failure

   • 5.1. The Circuit Breaker Pattern (Polly)

   • 5.2. The Retry Pattern (Polly)

   • 5.3. The Bulkhead Pattern (Polly)

6. Advanced Architectural Patterns

   • 6.1. Command Query Responsibility Segregation (CQRS)

   • 6.2. Event Sourcing: The Single Source of Truth

7. Observability: Seeing in the Dark

   • 7.1. Structured Logging (Serilog)

   • 7.2. Metrics and Monitoring (Prometheus & Grafana)

   • 7.3. Distributed Tracing (OpenTelemetry & Jaeger)

8. Conclusion: Is Microservices the Right Choice for You?

---

1. Introduction: The Monolithic Mountain and the Microservices Maze

Imagine your successful e-commerce application, "MegaShop," built as a single, unified monolithic application. All the code—product catalog, user authentication, order processing, inventory management, payment processing—resides in one massive codebase, is compiled together, and deployed as a single unit. This is the Monolithic Mountain. It's simple to develop, test, and deploy initially. You just run the solution in Visual Studio, and you're good to go.

But as "MegaShop" grows, the mountain becomes harder to climb.

• A small change to the product review system requires building and deploying the entire application.

• A memory leak in the payment module can bring down the entire site, including the product catalog.

• Scaling is all-or-nothing. You can't just scale out the order processing service that's under heavy load during a sale; you have to scale everything.

• Technology lock-in means the entire app is stuck with the same framework (e.g., .NET Framework 4.8), making it difficult to adopt new technologies.

Microservices Architecture is the strategic response to these challenges. Instead of one mountain, you create a range of smaller, independent hills—each representing a single, focused business capability. The "MegaShop" monolith is decomposed into:

• ProductService (Manages products and catalog)

• UserService (Handles authentication and user profiles)

• OrderService (Processes orders and payments)

• InventoryService (Tracks stock levels)

• NotificationService (Sends emails and alerts)

Each service is:

• Loosely coupled: It can be developed, deployed, and scaled independently.

• Independently deployable: A change to ProductService doesn't affect OrderService.

• Owns its domain data: The OrderService has its own database for orders; the ProductService has its own for products.

• Built by a small, focused team: Aligning with Conway's Law.

However, this power comes with immense complexity—the Microservices Maze. You now have to solve problems like service communication, network reliability, distributed data management, and unified monitoring. This module is your map through that maze.

2. Core Principles and Tangible Benefits of Microservices

Let's formalize the principles behind the pattern:

• Single Responsibility Principle (SRP) on a Macro Scale: Each service is responsible for a single business capability (e.g., "Manage Orders," "Calculate Shipping").

• Domain-Driven Design (DDD): Services are organized around Bounded Contexts—explicit boundaries within a domain where a particular model is defined and applicable. This is crucial for defining clear service boundaries and avoiding the "distributed monolith" anti-pattern.

• Decentralized Governance: Teams owning services can choose the best technology for their job (e.g., ProductService in C#, RecommendationService in Python for its ML libraries). This also extends to decentralized data management.

• Design for Failure: Networks fail, servers crash, latency spikes. A microservices architecture must be designed assuming these events will happen and must be resilient to them.

• Infrastructure Automation: Managing dozens of services manually is impossible. CI/CD pipelines, containerization (Docker), and orchestration (Kubernetes) are not optional; they are fundamental requirements.

Benefits:

• Agility & Independent Deployability: Teams can release updates to their service without coordinating with everyone else, drastically increasing release velocity.

• Scalability: You can scale out services that require more resources. The ImageProcessingService can be scaled independently of the mostly-static ProductCatalogService.

• Technological Freedom: Teams are not constrained by the organization's primary technology stack.

• Fault Isolation: A failure in one service does not cascade to bring down the entire system. The checkout can remain operational even if the product review service is failing.

• Small, Focused Teams: Smaller codebases are easier to understand and foster ownership.

3. Service Communication: The Nervous System of Your Architecture

Services are useless if they can't talk to each other. The communication patterns you choose are critical.

3.1. Synchronous Communication with RESTful APIs (ASP.NET Core Web APIs)

REST over HTTP is the most common and familiar pattern, perfect for public APIs and external communication.

Example: OrderService calls ProductService to get product details before creating an order.

In the ProductService (ASP.NET Core Web API):

csharp

// ProductService/Controllers/ProductsController.cs
[ApiController]
[Route("api/products")]
public class ProductsController : ControllerBase
{
    private readonly IProductRepository _repository;

    public ProductsController(IProductRepository repository)
    {
        _repository = repository;
    }

    [HttpGet("{id}")]
    public async Task<ActionResult<ProductDto>> GetProductById(int id)
    {
        var product = await _repository.GetByIdAsync(id);
        if (product == null)
        {
            return NotFound();
        }
        return Ok(new ProductDto { Id = product.Id, Name = product.Name, Price = product.Price, Sku = product.Sku });
    }
}

// ProductDto.cs
public class ProductDto
{
    public int Id { get; set; }
    public string Name { get; set; }
    public decimal Price { get; set; }
    public string Sku { get; set; }
}

In the OrderService, using IHttpClientFactory (Best Practice for resilience and efficiency):

csharp

// OrderService/Services/ProductApiService.cs
public interface IProductApiService
{
    Task<ProductDto> GetProductAsync(int productId);
}

public class ProductApiService : IProductApiService
{
    private readonly HttpClient _httpClient;

    // IHttpClientFactory is injected for best practices (pooling, configuration)
    public ProductApiService(IHttpClientFactory httpClientFactory)
    {
        _httpClient = httpClientFactory.CreateClient("ProductService");
    }

    public async Task<ProductDto> GetProductAsync(int productId)
    {
        var response = await _httpClient.GetAsync($"api/products/{productId}");
        
        if (!response.IsSuccessStatusCode)
        {
            // Handle non-success status codes (404, 500, etc.)
            throw new HttpRequestException($"Failed to get product. Status code: {response.StatusCode}");
        }

        var product = await response.Content.ReadFromJsonAsync<ProductDto>();
        return product;
    }
}

// OrderService/Program.cs (or Startup.cs)
builder.Services.AddHttpClient("ProductService", client =>
{
    client.BaseAddress = new Uri("http://product-service:80"); // Using Docker service name
    client.DefaultRequestHeaders.Add("Accept", "application/json");
});

Pros: Simple, universal, human-readable, firewall-friendly.
Cons: Chatty (overhead of HTTP/1.1), no formal contract (can lead to breaking changes), client needs to know endpoint URLs, can lead to temporal coupling (caller waits).

3.2. High-Performance gRPC for Internal Services

gRPC is a modern RPC framework using HTTP/2 and Protocol Buffers (protobuf). It's ideal for low-latency, high-throughput internal communication.

Step 1: Define the Contract (.proto file)

protobuf

// Protos/product.proto
syntax = "proto3";

option csharp_namespace = "ProductService.Grpc";

service ProductGrpc {
  rpc GetProduct (GetProductRequest) returns (ProductResponse);
}

message GetProductRequest {
  int32 product_id = 1;
}

message ProductResponse {
  int32 id = 1;
  string name = 2;
  decimal price = 3;
  string sku = 4;
}

Step 2: Implement the Server (ProductService)

csharp

// ProductService/GrpcServices/ProductGrpcService.cs
public class ProductGrpcService : ProductGrpc.ProductGrpcBase
{
    private readonly IProductRepository _repository;

    public ProductGrpcService(IProductRepository repository)
    {
        _repository = repository;
    }

    public override async Task<ProductResponse> GetProduct(GetProductRequest request, ServerCallContext context)
    {
        var product = await _repository.GetByIdAsync(request.ProductId);
        if (product == null)
        {
            throw new RpcException(new Status(StatusCode.NotFound, $"Product with ID {request.ProductId} not found."));
        }

        return new ProductResponse { Id = product.Id, Name = product.Name, Price = (double)product.Price, Sku = product.Sku };
    }
}
// In Program.cs, map gRPC service: app.MapGrpcService<ProductGrpcService>();

Step 3: Create the Client (OrderService)

csharp

// OrderService/Services/ProductGrpcClientService.cs
public class ProductGrpcClientService
{
    private readonly ProductGrpc.ProductGrpcClient _grpcClient;

    public ProductGrpcClientService(ProductGrpc.ProductGrpcClient grpcClient)
    {
        _grpcClient = grpcClient;
    }

    public async Task<ProductDto> GetProductAsync(int productId)
    {
        var request = new GetProductRequest { ProductId = productId };
        try
        {
            var response = await _grpcClient.GetProductAsync(request);
            return new ProductDto { Id = response.Id, Name = response.Name, Price = (decimal)response.Price, Sku = response.Sku };
        }
        catch (RpcException ex) when (ex.StatusCode == StatusCode.NotFound)
        {
            return null; // Or handle accordingly
        }
    }
}

// OrderService/Program.cs - Register gRPC Client
builder.Services.AddGrpcClient<ProductGrpc.ProductGrpcClient>(o =>
{
    o.Address = new Uri("https://product-service:5001"); // gRPC port
});

Pros: Extreme performance (binary protobuf, HTTP/2 multiplexing), strong interface contracts, streaming support, built-in code generation.
Cons: Requires HTTP/2, browser support is limited (requires gRPC-Web), harder to debug than REST.

3.3. Asynchronous Communication with Message Queues (RabbitMQ)

For decoupled, eventually consistent workflows, use message queues. A service publishes an event without knowing who will consume it. Other services subscribe to events they care about.

Example: OrderService publishes an OrderPlacedEvent. InventoryService and NotificationService subscribe to it.

Define the Event Message:

csharp

// Shared Class Library or Shared Contracts Project
public class OrderPlacedEvent
{
    public int OrderId { get; set; }
    public DateTime OrderDate { get; set; }
    public string UserEmail { get; set; }
    public List<OrderItemDto> Items { get; set; }
}

public class OrderItemDto
{
    public int ProductId { get; set; }
    public int Quantity { get; set; }
    public decimal UnitPrice { get; set; }
}

Publisher: OrderService (using RabbitMQ.Client)

csharp

// OrderService/Services/MessagePublisherService.cs
public class MessagePublisherService : IMessagePublisherService, IDisposable
{
    private readonly IConnection _connection;
    private readonly IModel _channel;

    public MessagePublisherService(IConfiguration configuration)
    {
        var factory = new ConnectionFactory() { HostName = "rabbitmq" };
        _connection = factory.CreateConnection();
        _channel = _connection.CreateModel();
        
        // Declare an exchange. Fanout broadcasts to all queues.
        _channel.ExchangeDeclare(exchange: "order-events", type: ExchangeType.Fanout, durable: true);
    }

    public void PublishOrderPlacedEvent(OrderPlacedEvent orderEvent)
    {
        var message = JsonSerializer.Serialize(orderEvent);
        var body = Encoding.UTF8.GetBytes(message);

        // Publish to the exchange, not a specific queue
        _channel.BasicPublish(exchange: "order-events",
                             routingKey: "", // Ignored for fanout
                             basicProperties: null,
                             body: body);
    }

    public void Dispose()
    {
        _channel?.Close();
        _connection?.Close();
    }
}

Consumer: InventoryService (Subscribes to update stock)

csharp

// InventoryService/Services/OrderPlacedConsumerService.cs
public class OrderPlacedConsumerService : BackgroundService
{
    private readonly IConnection _connection;
    private readonly IModel _channel;
    private readonly IServiceProvider _serviceProvider; // For scoped dependencies

    public OrderPlacedConsumerService(IConfiguration configuration, IServiceProvider serviceProvider)
    {
        _serviceProvider = serviceProvider;
        var factory = new ConnectionFactory() { HostName = "rabbitmq" };
        _connection = factory.CreateConnection();
        _channel = _connection.CreateModel();

        _channel.ExchangeDeclare(exchange: "order-events", type: ExchangeType.Fanout, durable: true);
        var queueName = _channel.QueueDeclare("inventory-update-queue", durable: true).QueueName;
        _channel.QueueBind(queue: queueName, exchange: "order-events", routingKey: "");
    }

    protected override Task ExecuteAsync(CancellationToken stoppingToken)
    {
        var consumer = new EventingBasicConsumer(_channel);
        consumer.Received += async (model, ea) =>
        {
            var body = ea.Body.ToArray();
            var message = Encoding.UTF8.GetString(body);
            var orderEvent = JsonSerializer.Deserialize<OrderPlacedEvent>(message);

            // Process the message within a scope to get scoped services (like DbContext)
            using (var scope = _serviceProvider.CreateScope())
            {
                var inventoryService = scope.ServiceProvider.GetRequiredService<IInventoryService>();
                foreach (var item in orderEvent.Items)
                {
                    await inventoryService.ReduceStockAsync(item.ProductId, item.Quantity);
                }
            }
            _channel.BasicAck(ea.DeliveryTag, multiple: false); // Acknowledge message processing
        };

        _channel.BasicConsume(queue: "inventory-update-queue", autoAck: false, consumer: consumer);
        return Task.CompletedTask;
    }
}

Pros: Ultimate decoupling, scalability, fault tolerance (messages persist until consumed), enables event-driven architecture.
Cons: Complexity, eventual consistency, message ordering and duplication challenges, requires monitoring of the queue.

4. Orchestrating Traffic: The API Gateway and Service Discovery

With many services, how does a client know where to call? How do we handle cross-cutting concerns?

4.1. The API Gateway Pattern (Ocelot Example)

An API Gateway is a single entry point for all clients. It handles request routing, composition, authentication, rate limiting, and more.

Using Ocelot in .NET:
Create a new ASP.NET Core project, ApiGateway.

csharp

// ApiGateway/Program.cs
using Ocelot.DependencyInjection;
using Ocelot.Middleware;

var builder = WebApplication.CreateBuilder(args);

// Add Ocelot and load configuration from ocelot.json
builder.Configuration.AddJsonFile("ocelot.json", optional: false, reloadOnChange: true);
builder.Services.AddOcelot(builder.Configuration);

var app = builder.Build();
await app.UseOcelot();
app.Run();

Example ocelot.json configuration file:

json

{
  "Routes": [
    {
      "DownstreamPathTemplate": "/api/products",
      "DownstreamScheme": "http",
      "DownstreamHostAndPorts": [
        {
          "Host": "product-service",
          "Port": 80
        }
      ],
      "UpstreamPathTemplate": "/catalog/products",
      "UpstreamHttpMethod": [ "GET" ]
    },
    {
      "DownstreamPathTemplate": "/api/orders",
      "DownstreamScheme": "http",
      "ServiceName": "order-service", // Uses Service Discovery!
      "UpstreamPathTemplate": "/shopping/orders",
      "UpstreamHttpMethod": [ "GET", "POST" ]
    }
  ],
  "GlobalConfiguration": {
    "ServiceDiscoveryProvider": {
      "Host": "consul",
      "Port": 8500,
      "Type": "Consul"
    }
  }
}

The client now only talks to https://mygateway.com/catalog/products, and Ocelot routes it to the internal product-service.

4.2. Service Discovery: Eureka vs. Client-Side (Consul)

In dynamic environments (Docker, Kubernetes), service IPs change. Service Discovery provides a directory where services can register themselves and find each other.

Client-Side Discovery (e.g., with Consul): The client (or API Gateway) is responsible for querying a service registry (Consul) to get the location of a service instance and then making the request directly.

Example: OrderService finds ProductService using Consul.

csharp

// OrderService/Program.cs
builder.Services.AddSingleton<IConsulClient, ConsulClient>(p => new ConsulClient(consulConfig =>
{
    consulConfig.Address = new Uri("http://consul:8500");
}));

// Register itself in Consul on startup
var app = builder.Build();
var consulClient = app.Services.GetRequiredService<IConsulClient>();
var registration = new AgentServiceRegistration()
{
    ID = $"order-service-{app.Environment.ApplicationName}",
    Name = "order-service",
    Address = Dns.GetHostName(), // Gets container hostname
    Port = 80 // Internal port
};
await consulClient.Agent.ServiceRegister(registration);
// Also implement a Health Check endpoint and deregister on shutdown.

Server-Side Discovery (e.g., with Eureka): The client makes a request to a load balancer (e.g., AWS ELB), which queries the registry and routes the request. This is simpler for the client but adds another network hop.

5. Building Resilience: Embracing Failure

Networks are unreliable. Services fail. Timeouts happen. The Polly library is the de-facto standard for handling these transient faults in .NET.

5.1. The Circuit Breaker Pattern (Polly)

The Circuit Breaker prevents an application from repeatedly trying to execute an operation that's likely to fail. It "breaks the circuit" after a threshold of failures, failing fast for a defined period.

csharp

// OrderService/Services/ResilientProductService.cs
// Define the policies
var circuitBreakerPolicy = Policy<HttpResponseMessage>
    .Handle<HttpRequestException>()
    .OrResult(x => !x.IsSuccessStatusCode)
    .CircuitBreakerAsync(handledEventsAllowedBeforeBreaking: 2, // Break after 2 consecutive failures
                        durationOfBreak: TimeSpan.FromSeconds(30)); // Break for 30 seconds

public class ResilientProductService
{
    private readonly HttpClient _httpClient;
    private readonly IAsyncPolicy<HttpResponseMessage> _circuitBreakerPolicy;

    public ResilientProductService(HttpClient httpClient, IAsyncPolicy<HttpResponseMessage> circuitBreakerPolicy)
    {
        _httpClient = httpClient;
        _circuitBreakerPolicy = circuitBreakerPolicy;
    }

    public async Task<ProductDto> GetProductWithCircuitBreaker(int productId)
    {
        // Wrap the HTTP call in the circuit breaker policy
        var response = await _circuitBreakerPolicy.ExecuteAsync(() =>
            _httpClient.GetAsync($"api/products/{productId}")
        );

        if (!response.IsSuccessStatusCode)
        {
            // Handle failure
            return null;
        }
        return await response.Content.ReadFromJsonAsync<ProductDto>();
    }
}

When the circuit is Open, any call will immediately throw a BrokenCircuitException without making the network call, saving resources.

5.2. The Retry Pattern (Polly)

Retries handle transient faults (like a momentary network glitch or a service restarting).

csharp

// Retry with exponential backoff (wait longer between each retry)
var retryPolicy = Policy<HttpResponseMessage>
    .Handle<HttpRequestException>()
    .OrResult(x => !x.IsSuccessStatusCode)
    .WaitAndRetryAsync(retryCount: 3,
                       sleepDurationProvider: retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)), // 2, 4, 8 seconds
                       onRetry: (outcome, timespan, retryAttempt, context) =>
                       {
                           // Log the retry attempt
                           logger.LogWarning($"Retrying attempt {retryAttempt} after {timespan.TotalSeconds}s...");
                       });

5.3. The Bulkhead Pattern (Polly)

The Bulkhead pattern isolates parts of the system to prevent a failure in one service from consuming all resources (like threads) and cascading throughout the application.

csharp

// Limit to 12 parallel executions and queue up to 4 requests for this specific operation.
var bulkheadPolicy = Policy.BulkheadAsync(12, 4);

// Use it to wrap a database call or an external HTTP call.
await bulkheadPolicy.ExecuteAsync(async () =>
{
    await SomeResourceIntensiveOperation();
});

Combining Policies with Polly Wrap:
The real power is combining these policies.

csharp

// OrderService/Program.cs
var resiliencePipeline = Policy.WrapAsync(retryPolicy, circuitBreakerPolicy); // Order matters: Retry -> Circuit Breaker
builder.Services.AddSingleton(resiliencePipeline);

6. Advanced Architectural Patterns

6.1. Command Query Responsibility Segregation (CQRS)

CQRS separates the model for reading (Queries) from the model for writing (Commands). This allows you to optimize each side independently.

Simple CQRS in a Single Service:

csharp

// OrderService/Features/Orders
// Commands (Write)
public class PlaceOrderCommand : IRequest<int> // Returns Order ID
{
    public int UserId { get; set; }
    public List<OrderItemDto> Items { get; set; }
}

public class PlaceOrderCommandHandler : IRequestHandler<PlaceOrderCommand, int>
{
    private readonly OrderDbContext _dbContext;
    public async Task<int> Handle(PlaceOrderCommand command, CancellationToken cancellationToken)
    {
        var order = new Order { UserId = command.UserId, ... };
        // ... add items, calculate total, etc.
        _dbContext.Orders.Add(order);
        await _dbContext.SaveChangesAsync();
        return order.Id; // Return the ID of the created order
    }
}

// Queries (Read)
public class GetOrderByIdQuery : IRequest<OrderDto>
{
    public int OrderId { get; set; }
}

public class GetOrderByIdQueryHandler : IRequestHandler<GetOrderByIdQuery, OrderDto>
{
    private readonly OrderReadDbContext _readDbContext; // Could be a different DbContext optimized for reads!
    public async Task<OrderDto> Handle(GetOrderByIdQuery query, CancellationToken cancellationToken)
    {
        // Use a clean, flat DTO tailored for the UI, perhaps from a denormalized read model.
        return await _readDbContext.Orders
                    .Where(o => o.Id == query.OrderId)
                    .ProjectTo<OrderDto>() // Using AutoMapper
                    .FirstOrDefaultAsync();
    }
}

Advanced CQRS with Separate Read Stores: The write model publishes an event (e.g., OrderPlacedEvent), and a separate handler updates a denormalized SQL table or an Elasticsearch index specifically optimized for complex queries.

6.2. Event Sourcing: The Single Source of Truth

Instead of storing the current state of an entity, Event Sourcing stores a sequence of state-changing events. The current state is rebuilt by replaying these events.

Example: Order Aggregate with Event Sourcing.

csharp

// OrderService/Domain/Order.cs (Aggregate Root)
public class Order : AggregateRoot
{
    public int UserId { get; private set; }
    public OrderStatus Status { get; private set; }
    private readonly List<OrderItem> _items = new List<OrderItem>();
    public IReadOnlyCollection<OrderItem> Items => _items.AsReadOnly();

    // Constructor applies the initial event
    private Order() { } // For EF Core

    public static Order Create(int userId, List<OrderItemDto> items)
    {
        var order = new Order();
        var orderCreatedEvent = new OrderCreatedEvent(userId, items);
        order.ApplyEvent(orderCreatedEvent); // Updates internal state
        order.AddDomainEvent(orderCreatedEvent); // Queues event for persistence
        return order;
    }

    public void Cancel(string reason)
    {
        if (Status != OrderStatus.Placed)
            throw new InvalidOperationException("Can only cancel placed orders.");

        var orderCancelledEvent = new OrderCancelledEvent(reason);
        ApplyEvent(orderCancelledEvent);
        AddDomainEvent(orderCancelledEvent);
    }

    // Apply methods to rebuild state from events
    private void Apply(OrderCreatedEvent @event)
    {
        Id = @event.OrderId;
        UserId = @event.UserId;
        Status = OrderStatus.Placed;
        // ... create order items
    }
    private void Apply(OrderCancelledEvent @event) => Status = OrderStatus.Cancelled;
}

// Events
public abstract class DomainEvent { }
public class OrderCreatedEvent : DomainEvent
{
    public int OrderId { get; set; } // Would be generated
    public int UserId { get; }
    public List<OrderItemDto> Items { get; }
    // ... constructor
}
public class OrderCancelledEvent : DomainEvent
{
    public string Reason { get; }
    // ... constructor
}

The Event Store (Simplified):

csharp

// OrderService/Infrastructure/EventStoreDbContext.cs
public class EventStoreDbContext : DbContext
{
    public DbSet<EventData> Events { get; set; }
    // ...
}

public class EventData
{
    public int Id { get; set; }
    public Guid AggregateId { get; set; } // The Order Id
    public string AggregateType { get; set; } // "Order"
    public string EventType { get; set; } // "OrderCreatedEvent"
    public string Data { get; set; } // JSON serialized event
    public DateTime Timestamp { get; set; }
    public int Version { get; set; } // For optimistic concurrency
}

To get the current state of an Order with ID 123, you retrieve all events for AggregateId=123 and replay the Apply methods.

Pros: Complete audit trail, ability to "time travel" and see state at any point, naturally enables complex event-driven systems.
Cons: Complex learning curve, querying the event stream for data can be difficult (this is where CQRS pairs perfectly with it).

7. Observability: Seeing in the Dark

How do you debug a request that travels through 5 different services? Observability is your answer.

7.1. Structured Logging (Serilog)

Structured logging logs data as key-value pairs, not just strings, enabling powerful querying.

Setup with Serilog and SEQ:

csharp

// In Program.cs of any service
builder.Host.UseSerilog((ctx, lc) => lc
    .WriteTo.Console()
    .WriteTo.Seq("http://seq:5341") // Docker container
    .Enrich.WithProperty("ServiceName", ctx.HostingEnvironment.ApplicationName) // Enrich all logs with service name
    .ReadFrom.Configuration(ctx.Configuration));

// In a Controller or Service
public class OrderController : ControllerBase
{
    private readonly ILogger<OrderController> _logger;

    public OrderController(ILogger<OrderController> logger) => _logger = logger;

    [HttpPost]
    public async Task<IActionResult> CreateOrder([FromBody] CreateOrderRequest request)
    {
        // Log with structured data
        _logger.LogInformation("Creating new order for user {UserId} with {ItemCount} items", request.UserId, request.Items.Count);
        
        try
        {
            // ... logic
            _logger.LogInformation("Order {OrderId} created successfully", newOrderId);
            return Ok(newOrderId);
        }
        catch (Exception ex)
        {
            // Log the full exception with structured data
            _logger.LogError(ex, "Failed to create order for user {UserId}", request.UserId);
            return StatusCode(500);
        }
    }
}

In SEQ, you can then query: ServiceName = 'OrderService' and UserId = 1234.

7.2. Metrics and Monitoring (Prometheus & Grafana)

Metrics are numerical measurements tracked over time (e.g., requests per second, error rate, latency).

Using prometheus-net in ASP.NET Core:

csharp

// In Program.cs
builder.Services.AddMetrics();
app.UseMetricServer(); // Exposes a /metrics endpoint for Prometheus to scrape

// You can also create custom metrics
public class OrderMetrics
{
    private readonly Counter _ordersCreatedCounter;
    public OrderMetrics(IMeterFactory meterFactory)
    {
        var meter = meterFactory.Create("MegaShop.Orders");
        _ordersCreatedCounter = meter.CreateCounter<int>("orders.created", description: "Count of orders created");
    }
    public void OrderCreated() => _ordersCreatedCounter.Add(1);
}

Prometheus scrapes the /metrics endpoint periodically. Grafana then connects to Prometheus to create dashboards.

7.3. Distributed Tracing (OpenTelemetry & Jaeger)

Distributed tracing follows a request (a "trace") as it flows through services. Each operation is a "span." Spans are linked by a traceId.

Setup with OpenTelemetry and Jaeger:

csharp

// In Program.cs of EVERY service
builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing.AddSource("OrderService") // Listen to activity sources in your service
               .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("OrderService"))
               .AddAspNetCoreInstrumentation() // Automatically instrument incoming HTTP requests
               .AddHttpClientInstrumentation() // Automatically instrument outgoing HTTP requests
               .AddEntityFrameworkCoreInstrumentation() // Instrument DB calls
               .AddOtlpExporter(opt => // Export traces to Jaeger
               {
                   opt.Endpoint = new Uri("http://jaeger:4317");
               });
    });

When you call _logger.LogInformation(...), the log is automatically correlated with the current trace ID. In Jaeger, you can search for a traceId and see the entire journey of a request, including how long each service took and any logs from that request.

8. Conclusion: Is Microservices the Right Choice for You?

Microservices are a powerful architectural style that can provide significant benefits in agility, scalability, and resilience. However, they introduce massive operational and cognitive complexity.

Choose Microservices if:

• Your development teams are large and need to scale.

• Different parts of your system have vastly different scalability or technological needs.

• You have a strong need for fault isolation.

• Your organization has mature DevOps and infrastructure automation capabilities.

Stick with a Monolith or a simpler architecture if:

• Your team is small.

• You are building an MVP or a new product where speed of iteration is key.

• The application is simple and will likely remain so.

• You lack the operational expertise to manage distributed systems.

Remember, you can often start with a well-structured monolith and later break it into microservices as boundaries become clear and the need arises. Don't fall into the trap of using a microservices architecture for a problem that doesn't require it. The maze is powerful, but you should only enter it when the mountain becomes too steep to climb.

Full Course Link : https://imomins.blogspot.com/2025/08/master-software-architecture-complete.html