In our environment we have as many as 8 CDM data sources available and that number will only grow in the future. I have a growing concern in our environment that our users will be able to quickly overload the back-end database servers in our environment. I’m already seeing signs of this as a few users each submit analysis jobs across multiple CDM sources for multiple cohorts.
Our current Spring Batch implementation runs jobs as they are requested and this was certainly a perfectly fine start for our initial release of job management capabilities. I would like to propose the question, should we add a queue to our environment with options on how many concurrent jobs we should allow, perhaps by type even. Is this desirable in other environments? How easy would this be to achieve (@alfranke probably will know best) given what we have today?