We continue to discuss the need for our tools to support many different database plaforms Recently, some, including @lee_evans and @Chris_Knoll have suggested that a way to deal with the complexity that comes from supporting many platforms is to split DatabaseConnector and SqlRender into separate packages, one for each dialect.
At first glance this might seem like a good idea. the expected in- and output of these packages is well defined, and if we split them up, then different developers could own different platforms and implement the required functionality as they see fit, without having to worry about the other platforms. However, I have some arguments against this approach:
No consistent quality. I know some developers are frustrated with the fact that they’ve had to wait for me to review and approve their pull requests. However, in my opinion I was merely preventing calamity later down the road. If we split up the packages, there will be no such oversight. Some of the problems (in my mind) I did not allow to happen include:
- Downright errors that would have prevented all OHDSI applications from working
- Inefficient implementations that would have led to those worst of errors: intermittent ones, that once every n runs would for example throw an ‘out of heap’ error.
- Subtle changes in the semantics of the translated SQL, that would have probably gone unnoticed, but would have forever left us wondering why that particular database was always giving illogical results.
More complex deployment. Already it is pretty hard to write clear instructions on how people should run your study on their data. If we split up the packages there will likely be different instructions for each platform.
Code duplication. Each platform will get its own copy of the code, and these implementations will start to diverge. If we ever want to change something at a higher level (Say, implement an alternative approach to downloading big data), this will be extremely difficult.
DatabaseConnector and SqlRender are just one part of it. Although for platforms that faithfully and completely implement SQL most platform-specific code lives in DatabaseConnector and SqlRender, other platforms don’t play so nice, and require changes in all of our applications. I don’t think having platform-specific forks of each tool is the answer, for reasons 1-3 listed above.
Let me know if you agree or disagree with any of these arguments.