Design a Top-K Request Analysis System

Functional Requirements:

• The system should track and return the top K most frequent web requests (e.g., URLs) within configurable time intervals (e.g., 1 minute, 10 minutes, 1 hour).
• K and the time interval should be configurable by the client (e.g., K up to 100).
• Support near real-time querying of top K requests for the latest or a specific time window.
• Provide REST APIs for querying top K requests and for configuration.
• Store and allow querying of historical top K data for a configurable retention period (e.g., up to 7 days).
• Collect and store request metadata (e.g., URL, timestamp, user agent).

Non-Functional Requirements:

• Scalability: The system must handle high traffic, e.g., 10 million requests per minute.
• Low Latency: Top K queries should return within a few seconds.
• High Availability: The system should remain operational during node failures or high load.
• Data Retention: Raw and aggregated data should be retained for a configurable period.
• Fault Tolerance: The system should minimize data loss and support recovery from failures.
• Cost Efficiency: Infrastructure and storage costs should be optimized for high throughput.