educative.io

Telemetry 2.0 System

Courses
,
#1

Does the TinyUrl system justify investing in a full Telemetry system? i.e.

  1. Each action (i.e. tinyUrl access request) triggers an “Event”
  2. Each “Event” is sent to a messaging queue system (e.g. Apache Kafka)
    3a. Events are ETLs into a cold storage (e.g. S3/HDFS) and processed offline (MapReduce/Spark) to generate ALL the metrics you want
    3b. Events are processed in near realtime (e.g. Apache Storm) for quick analysis
  3. Visualization/Reporting/Notifications
#2

My $0.02: Investing on full blown storage of data would not be wise as usually aggregated data would be expected as analytical data points. It is of course advisable to buffer the access events using some queue (Kafka would do), and then insert/update the aggregates.

#3

As far as the interview is concerned, designing a Telemetric system would be a separate question. It has its own design complexities.

As far as the requirement is concerned, we definitely need some telemetry in a URL shortening system. These days, most of the existing services make money though it specially through Ad campaigns. For example, a user will create 5 different short URL of their service, share them in different Ads or Posts and count how many clicks they are getting on each short link. Later they can optimize or do machine learning or even steam this data to Kafka/Storm to perform real time analysis to make updates on the fly.

2 Likes