WEBAPI connection to Impala

Implementers
1

Hi Team,

I am newbie to OHDSI. I am trying configure OHDSI in my Hadoop Environment as per the documentation OHDSI 2.x version is compatible with Impala. I am providing all the information required to connect Impala. I am giving below details for flyway database configuration

<datasource.driverClassName>com.cloudera.impala.jdbc41.Driver</datasource.driverClassName>
<datasource.url>jdbc:impala://server_name</datasource.url>
<datasource.username>username</datasource.username>
<datasource.password>password</datasource.password>

<datasource.dialect>impala</datasource.dialect>
<datasource.ohdsi.schema>ohdsi</datasource.ohdsi.schema>
<datasource.dialect.source>impala</datasource.dialect.source>

<flyway.datasource.driverClassName>com.cloudera.impala.jdbc41.Driver</flyway.datasource.driverClassName>
<flyway.datasource.url>jdbc:impala://server_name</flyway.datasource.url>
<flyway.datasource.username>username</flyway.datasource.username>
<flyway.datasource.password>password</flyway.datasource.password>
<flyway.locations>classpath:db/migration/impala</flyway.locations>

<flyway.schemas>${datasource.ohdsi.schema}</flyway.schemas>
<flyway.validateOnMigrate>false</flyway.validateOnMigrate>
<flyway.placeholders.ohdsiSchema>${datasource.ohdsi.schema}</flyway.placeholders.ohdsiSchema>

But i am facing error as below

org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/WebAPI_Impala]]
at org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:441)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:198)
at org.apache.catalina.manager.ManagerServlet.start(ManagerServlet.java:1409)
at org.apache.catalina.manager.HTMLManagerServlet.start(HTMLManagerServlet.java:698)
at org.apache.catalina.manager.HTMLManagerServlet.doPost(HTMLManagerServlet.java:223)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.filters.CsrfPreventionFilter.doFilter(CsrfPreventionFilter.java:136)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:200)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:607)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘shiroFilter’ defined in class path resource [org/ohdsi/webapi/ShiroConfiguration.class]: Unsatisfied dependency expressed through method ‘shiroFilter’ parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘flyway’ defined in class path resource [org/ohdsi/webapi/FlywayConfig.class]: Invocation of init method failed; nested exception is org.flywaydb.core.api.FlywayException: Unsupported Database: Impala 2.11
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:749)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:467)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1173)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1067)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.context.support.PostProcessorRegistrationDelegate.registerBeanPostProcessors(PostProcessorRegistrationDelegate.java:235)
at org.springframework.context.support.AbstractApplicationContext.registerBeanPostProcessors(AbstractApplicationContext.java:702)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:527)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:737)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:370)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:314)
at org.springframework.boot.web.support.SpringBootServletInitializer.run(SpringBootServletInitializer.java:151)
at org.springframework.boot.web.support.SpringBootServletInitializer.createRootApplicationContext(SpringBootServletInitializer.java:131)
at org.springframework.boot.web.support.SpringBootServletInitializer.onStartup(SpringBootServletInitializer.java:86)
at org.springframework.web.SpringServletContainerInitializer.onStartup(SpringServletContainerInitializer.java:169)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5125)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
… 30 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘flyway’ defined in class path resource [org/ohdsi/webapi/FlywayConfig.class]: Invocation of init method failed; nested exception is org.flywaydb.core.api.FlywayException: Unsupported Database: Impala 2.11
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1628)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:296)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
… 52 more
Caused by: org.flywaydb.core.api.FlywayException: Unsupported Database: Impala 2.11
at org.flywaydb.core.internal.dbsupport.DbSupportFactory.createDbSupport(DbSupportFactory.java:151)
at org.flywaydb.core.Flyway.execute(Flyway.java:1420)
at org.flywaydb.core.Flyway.migrate(Flyway.java:971)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1758)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1695)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
… 65 more

Please help me in solving the issue. Thanks in Advance.

Thanks,
Rahul.

2

Hi Rahul,

Taking a look at clause
“Caused by: org.flywaydb.core.api.FlywayException: Unsupported Database: Impala 2.11”

It seems that flywaydb doesn’t support impala. If you see Flywaydb documentation https://flywaydb.org/documentation/ “impala” not appears as a supported DB.

Maybe you have to change your ORM library or search for a workaround for flyway.

Regards
Rubén

3

Thanks Ruben for reply, is there any documentation available to configure impala with OHDSI WebAPI.

Thanks,
Rahul.

4

Hi, @rahulbujji45,
You won’t be able to host the WebAPI database on Hadoop. In fact the only platforms that WebAPI supports is oracle, ms sql server and postgresql. This is because there are features that RDBMs provide (foreign keys, complex constraints, sequences,etc) that are not available on MPP platforms (such as PDW, redshift, BigQuery, etc).

This does not mean that you can’t run analyses on Hadoop! It means that you can set up your CDMs on Hadoop, but WebAPI contains the ‘metadata’ about your CDM network.

So, to get started, I’d suggest you install PostgreSQL following the setup instructions here: http://www.ohdsi.org/web/wiki/doku.php?id=documentation:software:webapi:webapi_installation_guide. and then configure your environment for connections to HadOOP (Impala i believe is supported).

@gregk and @pavgra have better experience with this than I do (I have zero), but I can help you with instructions on setting up your WebAPI environment.

5

By the way: this statement means that the analytical queries are compatable with Impala (cohort generation, characterization, achilles, etc). WebAPI requires ‘full-fledged’ RDBMS features, so only oracle, mssql and postgresql is supported.

6

thanks @Chris_Knoll for reply, i completed setting up the WebAPI. Can you please help me with CDMs setup on Hadoop.

Thanks,
Rahul.

7

@gregk, @pavgra: do you have documentation about getting the Impala JDBC drivers set up, and connection string examples for connecting to Hadoop?

@rahulbujji45: once you know your Hadoop JDBC connection settings, you can follow the instructions for configuring the sources in the WebAPI setup documentation: http://www.ohdsi.org/web/wiki/doku.php?id=documentation:software:webapi:webapi_installation_guide.

8

Hi

Can share any guide to setup hadoop and impala with CDM, WebAPI

thanks
Chidam