Design Log Processing System

Functional Requirements:

• Ingest logs from multiple sources (e.g., application servers, microservices) in real time.
• Store logs for querying and analysis.
• Support querying logs by fields such as timestamp, log level, source, and message content.
• Allow filtering and searching logs based on criteria (e.g., error logs in the last hour).
• Provide a simple API for log ingestion and retrieval.

Non-Functional Requirements:

• Handle high write throughput (many logs per second).
• Ensure log data is durable and not lost.
• Support reasonable query latency for recent logs (e.g., under a few seconds).
• Scale horizontally to accommodate more log sources and higher volume.
• Provide basic access control for log retrieval APIs.

Out of Scope:

• Advanced analytics or alerting (e.g., anomaly detection, dashboards).
• Complex log parsing or transformation.
• Integration with external notification or monitoring systems.