There’s been a request for adding support for our applications for Netezza (see issues in SqlRender and DatabaseConnector. Before we decide whether this is something we can and want to do, here’s my impression of what would be needed to add any DBMS to our list:
Adding the JDBC driver to DatabaseConnector. In the case of Netezza, this should be straightforward since a JDBC driver seems to exist, but it is not publicly available (I can’t find it). We’d need to see how to get it, and what the licensing conditions are.
Add translation rules to SqlRender. Just like RedShift, Netezza seems to claim to use similar SQL syntax as PostgreSQL, but at least for RedShift there are still some major differences. We can start by copying all the rules we have now for PostgreSQL, and see where it breaks.
Create a testing environment for Netezza. I’m currently testing all our code against local SqlServer, Postgres, and Oracle databases before a release, and @mkhayter is our unofficial RedShift guinea pig. (sorry Mark!). Luckily, @lee_evans is building a testing environment that will be available to everyone, although I’m pretty sure for now it won’t support RedShift or Netezza.
Every software we create needs to be debugged using this new testing environment. Just as an example of what you might encounter: I found that RedShift has no way to implement the equivalent of “DELETE TABLE IF EXISTS”, and I therefore had to create a workaround in DatabaseConnector specifically for RedShift for some applications.
In short, adding support for another DBMS would, in my humble opinion, be non-trivial and require additional resources. If someone is willing to make those available, there should be no problem.
This is Aaron Browne, technical lead for the PEDSnet Data Coordinating Center implementation and source of the request for Netezza support. The folks we are working with at Seattle Children’s on PEDSnet use a Netezza database and we had hoped to make work for them. My thoughts in reply to Martijn are:
I’ve asked Nigel, my contact at Seattle, to join this discussion, and hopefully he can shed light on the licensing of the jdbc driver.
I have no way to predict where Netezza will break using PostgreSQL compliant code, but maybe Nigel can take a look at the existing translation rules and spot any incompatibilities.
I obviously can’t help with your testing environment, but maybe Nigel and the Seattle team would be willing to act as your Netezza guinea pig?
See above responses to 2 and 3.
I greatly appreciate your dedication to testing code before releasing it, but I’m also trying to get Achilles working for Seattle pretty quickly, what would you think about a PEDSnet fork of DatabaseConnector and SqlRender as a middle ground (this might also get around any driver licensing issues since the driver wouldn’t be so public)?
Thanks for your time and consideration on this.
Aaron
If the Seattle team is willing to be guinea pigs, then that would give Netezza the same status as RedShift (meaning: you can run it, but we can’t guarantee it will work).
I would suggest you just create Netezza branches in both DatabaseConnector and SqlRender. That way, if things work OK we can merge these branches into the masters, and its easy to keep the branches up to date if needed.
In DatabaseConnector you’d need to put the JDBC driver JAR file in the inst/java folder, and make sure to add the necessary code to the connect and createConnectionDetails functions.
In SqlRender, in the csv file I would suggest you copy all the rules for postgresql and make the target netezza.
I can serve as a guinea pig and get some testing done on a Netezza box as well, if Nigel runs out of steam. I am 90% JDBC comes with Netezza, it’s already expensive enough, charging a $10k for an essential piece makes no sense for a $1M piece of iron.
C
From: Aaron Browne [mailto:notifications@mail132-6.atl131.mandrillapp.com] On Behalf Of Aaron Browne
This is Aaron Browne, technical lead for the PEDSnet Data Coordinating Center implementation and source of the request for Netezza support. The folks we are working with at Seattle Children's on PEDSnet use a Netezza database and we had hoped to make work for them. My thoughts in reply to Martijn are:
I've asked Nigel, my contact at Seattle, to join this discussion, and hopefully he can shed light on the licensing of the jdbc driver.
I have no way to predict where Netezza will break using PostgreSQL compliant code, but maybe Nigel can take a look at the existing translation rules and spot any incompatibilities.
I obviously can't help with your testing environment, but maybe Nigel and the Seattle team would be willing to act as your Netezza guinea pig?
See above responses to 2 and 3.
I greatly appreciate your dedication to testing code before releasing it, but I'm also trying to get Achilles working for Seattle pretty quickly, what would you think about a PEDSnet fork of DatabaseConnector and SqlRender as a middle ground (this might also get around any driver licensing issues since the driver wouldn't be so public)?
I don’t have write permissions on the OHDSI repos, so I’ve forked to my personal account and created branches for netezza support. If you want to make netezza support branches in the OHDSI repos, I’m happy to submit PRs, or testers can get the beta code from my repos. The relevant links are:
I would prefer developing in branches in the OHDSI repo’s. @aaron0browne: I’ve invited you to become a member of the developers team, so you can push to the repositories directly.
I do request that merging into the masters be left to me, so I can run my full suite of tests to make sure nothing breaks somewhere else (unlikely, but still).
New branches created and PRs submitted. The PR is not intended to indicate the code is ready for merging, just to allow everyone to easily review the code changes. I’ll continue to update the branches as comments and test results come back.
This is Nigel Hartell with Seattle Children’s who Aaron mentions in the above thread. I’m pretty sure licensing should be a non-issue with the jdbc driver, but will follow up with IBM to confirm.
Seattle can be the guinea pig Netezza environment if that is required.
I can also take a look at the code to see if there are non-compliant coding issues as we port to Netezza from PostgreSQL.
I merged the Netezza DatabaseConnector branch with master, because I didn’t want them to run too far out of sync and the code looked fine. Feel free to branch again if you want to make further changes.
Has any testing been done on Netezza? I’m expecting more changes will be needed on the SqlRender Netezza branch, so will keep that one separate.
I need to add support for Google BigQuery for OHDSI tools. Based on your comments, I know that I should change both SqlRender and DatabaseConnector. Any idea if it is possible at all, and where I should start?
Note that adding another DBMS to the list of platforms supported by OHDSI will be a major undertaking. My first guess is that just to get the OHDSI R packages working you’ll need a half person-year of an efficient Java + R + SQL programmer. Because there’s no JDBC driver for GBQ, adding GBQ to the WebApi will probably take an equal amount of effort. Make sure you are willing to commit the resources to do so before you proceed.
If you do want to go ahead, here are the steps:
First, support needs to be added to DatabaseConnector. This package relies on the JDBC architecture, but unfortunately there does not appear to be a (viable) JDBC driver for GBQ. This is not the end of the world, but it means that all functions of DatabaseConnector will have to be reimplemented for Google’s specific API, including these functions: connect, executeSql, querySql, querySql.ffdf, and insertTable.
Second, support needs to be added to SqlRender. This means rules need to be added to the translation table. Most importantly, these rules need to be tested with all the OHDSI packages.
Thank you, I have one more question, it seems that GBQ does not support some queries (I could not find any for insert and update!!). Is this a big issue? Is it resolvable?
Yes, that’ll be a big issue. Across the OHDSI tools, we make extensive use
of various ANSI SQL functions, including CREATE, INSERT, SELECT, DELETE
operations, various functions including date comparisons and
partitioning/windowing functions, COUNT/COUNT_BIG/SUM, etc. If GBQ only
provides functionality for a subset of ANSI SQL, you will very likely run
into instances where your platform will not support the OHDSI analysis
capabilities.
I found a JDBC for GBQ that supports other queries. I added required code in DatabaseConnector.R and tested it executing select queries and it works correctly. However, when I try to insert a table using:
I get the following error, am I missing something?
DBMS:
googlebigquery
Error:
execute JDBC update query failed in dbSendUpdate (executeUpdate does not accept SELECT statements. Use executeQuery instead.)
SQL:
IF OBJECT_ID(‘tempdb…#temp’, ‘U’) IS NOT NULL DROP TABLE #temp
