An important distinction to be made with Event Metrics – ultimately the "report" data in Event Metrics gets pre-stored in a database, indefinitely. This isn't a problem in terms of storage because they are just numbers. In addition, Event Metrics had a timestamp of when the report was generated, and you as the user will always get that version of the report until you ask for an updated version. In our case, we end up with a epub (or other format), which is not as cheap to store, and also we want to automatically ensure the user is served the latest possible version. So the two systems won't work exactly the same.

In its simplest form, the purpose of a job queue would be to ensure there aren't but so many resource-intensive things going on at the same time. Beyond that, we need to also cache the exported file for some period of time. So for now I'm just going to go off of the investigation at T222936 and recommend a brief period, say 10 minutes.

Combining the system we use for Event Metrics, I envision the system working something like this:

First we need a table in the database to keep track of the jobs. The schema could look something like:

• id – unique ID of the job
• filename – unique filename for the exported book, something like [title]-[lang]-[font].[format]
• submitted – when the job was submitted
• status – status of the job, stored as a smallint but in English it's one of:
  • queued – waiting to be spawned by cron
  • started – currently processing
  • failed_timeout – timed out (we'll come up with a maximum period of time that any job should run, Sam and others probably know what a good value would be)
  • failed_unknown – failed for some other known reason

There's no "completed" status because the system will refer to the file system to determine whether a job is needed.

So the whole pipeline might look something like:

1. Request comes in
2. Check if a file already exists for the desired work/format
   1. If it exists, serve it. Nothing else needs to be done.
3. (no file exists) Check the job table to see if a job is pending for the desired work/format.
   1. If a job exists, the client goes back to step 1 (continually pinging the server until a file becomes available, with some reasonable timeout before failing gracefully)
4. (no job exists) A row is added to the job table and set to "queued"
5. A cron is ran every minute that spawns queued jobs (setting the status to "started"), never allowing more than N jobs to be running at the same time (probably can be a fairly high threshold)

Under this system, we will never be exporting the same thing twice at the same time. This doesn't cover the caching part (more on that below), it just ensures any unique combination of title/format/font is exported synchronously, and ensures that at any given time we have enough resources (RAM and such) to do properly export a work. I don't think we've confirmed that RAM and/or CPU overhead is actually a major problem, but the job queue nonetheless can ensure it stays that way.

Now, more about this cron job. It's workflow could be something like:

1. Start processing
2. Once complete, the file now exists which is the first thing that gets checked, so the corresponding row in the job table can be deleted. The next request will serve the actual file.
3. If the job fails, we set the status to indicate this (whether it was due to timeout or something unknown). This needs to be persisted so that on the next ping from the client, we can tell them the export failed.
   1. We don't have emailing of errors set up yet for WSExport, but when we do, it could email us when this happens.
4. Delete files that over 10 minutes old

#4 is effectively the caching bit. It could be done by a separate cron, but I worry about them competing with each other (one is trying to serve a book and just as it's about to the other deletes it).

A few things I'm unsure about:

• Should we cache just the epub, the final format, or both?
• For small books that export quickly, the job queue system could slow down the experience significantly because you have to wait up to a minute for the next cron run. Perhaps the queue can be bypassed if we can programmatically determine the work isn't very expensive to export (say by number of pages). This is basically what we did for Event Metrics. We also have the option of simulating a cron rate of less than a minute by having multiple cron jobs and putting a sleep in there. Definitely a hack but from quick research this isn't unheard of.

I'm going to stop here for now and let the other engineers read this and we will discuss tomorrow in our meeting. I'm not certain if the above is the best approach but it will allow us to steal a lot of already-written code from Event Metrics.