I have question here on KGS service. Where these services are run ? is this on app server or will there be additional servers allotted for this . When KGS wants to cache the keys will it be app server cache?
This is little confusing because in the previous example of tinyurl you mentioned that there will be seperate cache servers for this purpose .
And also what about the cleaning service. Is this also run on app server ?
Hi Shivaprasad,
We are looking into your query and will get back to you soon.
Thank you for reaching out to us. Please let us know if you have any further queries.
I think it is a good question. Any answers?
Disclaimer: I am not an expert but wanted to contribute here as I was evaluating similar questions. Please feel free to poke holes in my responses.
Is KGS a separate server?
I believe Yes. Purpose of having a KGS was to asynchronously making keys available to app servers. Therefore, it would make sense to keep KGS as a separate server.
Caching in relation to keys?
I think we should not use distributed cache for KGS. KGS can use local cache to cache some keys so that they could be quickly allotted to asking application servers. As soon as KGS adds keys to its local cache, it can mark them used in Key-DB.
In addition, KGS gives keys to individual app servers which caches keys again in their local cache so that they can avoid the communication to KGS and directly assign keys.
Which server is used for Cleaning Service?
I think we should not use App server for cleaning service. App server is customer facing and should be responsible to support customer queries. App server will do lazy clean up in case a “to-be” expired record is accessed by user. In general, Cleaning service can be run on KGS server which would delete paster records from db and move keys from used key-db to available key-db.
Again, I am novice and just sharing my views about the question. Feel free to question my findings.
Additional to Cleaning Service : It might be more elegant if recycled keys could be put in cache itself, we will not require db hit.
Based on my knowledge, I am thinking about the KGS service as a separate service than your actual application. The service could be built using any of the backend languages like python, C# …ect. Sending keys could done in two fashioned way pool/push mechanism. Ror faster performance, there should be caches on both ends application, and KGS (KGS should include functionality such as generate new keys, save into DB, send keys to your application)
For cleaning up service yes it should be running separately
The key point you should separate your services in a way when you have a single point of failure your system should still be able to serve users (in system design this is called Decoupling)
You should watch this video about basics of making distributed system https://www.youtube.com/watch?v=SqcXvc3ZmRU
Thanks me later 
Why I am saying both services KGS and cleaning should be separated
Now let’s imagine each server has both KGS service with your application then managing keys will be more complex (will have concurrency problem two server happen to use same key) while when we have KGS separately the KGS can easily decide if a key is used by another app server or not. Not to mention if a one server is down that means one of you KGS is also down along with one of your app as well. Thus, separating them is better because failure of one does not mean failure of the other.
I am sure there lots of downside of combining your app replica, KGS and cleaning service in one server than separating them.
Author! please correct me if I am wrong