Loading Profiles and Cohort Generation in Atlas

hui_xing_tan · September 13, 2018, 11:27pm

Hi! This follows from the another discussion thread

Background is we have set up WebAPI and installed atlas and configured our source and source_daimon tables. We are able to search for the vocabulary using atlas, however, the profile section takes a long time to load and returns no results apart from the gender and number of events.

The error messages upon opening the console (Ctrl+Shift+J) are as follows:

In addition, we do not have the permissions to generate cohorts:

We would really appreciate any assistance we can get here and thanks in advance! Lastly, we would like to check if there is a step-by-step guide to using atlas or some test case/scenario that we can try out to see if Atlas is properly installed and working.

Thank you!
Regards,
Hui Xing

hui_xing_tan · October 8, 2018, 6:53am

Hi,

We are still facing these errors:

Profile is unable to load
No permissions to generate cohort.

Please could anyone advise

Thanks so much in advance

Regards,
Hui Xing

hui_xing_tan · November 1, 2018, 1:39am

@Chris_Knoll @t_abdul_basser @Ajit_Londhe

Hi!
We have re run the steps to set up webapi below and we still are facing the problem of 1. not being able to load profiles and 2. not having the permissions to generate the cohorts.

Steps taken:

create database roles and users: http://www.ohdsi.org/web/wiki/doku.php?id=documentation:software:webapi:postgresql_installation_guide
installed webapi: http://www.ohdsi.org/web/wiki/doku.php?id=documentation:software:webapi:webapi_installation_guide
-configured the sources

As we not familiar with javascript and are having trouble understanding the error messages above, we are still not quite sure where to start debugging Further, we are able to browse the vocabulary and see some achilles results in the Data Sources section in Atlas so the source and source daimon tables should have been configured correctly. Would really appreciate any advice available. Thanks so much!

Regards,
Hui Xing

Chris_Knoll · November 1, 2018, 3:35am

I’m seeing very strange errors in your screenshots: the ‘SyntaxError: illegal character’ is very strange.

I also can’t tell if you are running on Chrome or some other browser. Can you try the latest version of Chrome?

What version of Atlas are you using?
What version of WebAPI have you installed on your java container?

By default, security should be disabled, so it is strange that you are seeing any ‘permission denied’ errors at all. But once you can confirm the version of the browser, WebAPI and Atlas you are using, we can narrow it down.

hui_xing_tan · November 2, 2018, 1:02pm

Hi @Chris_Knoll

We’re using Safari browser, Atlas 2.5.0 and WEBAPI 2.5.0

Tried it on Chrome browser and it produces the HTTP Status 404 error shown below:

Thanks

Chris_Knoll · November 2, 2018, 5:12pm

I am not sure why Safari would return the Atlas application, but Chrome returns a 404 (file not found). It looks like your URLs are case-sensitive, and from your origional error, the URL was localhost:8080/Atlas. Can you try either localhost:8080/Atlas or localhost:8080/Atlas/index.html?

hui_xing_tan · November 4, 2018, 1:24pm

@Chris_Knoll

Thanks for the advice. I’ve tried with localhost:8080/Atlas and it works! However, we are still encountering the same issues:

-Ok: Able to see data sources: Dashboard, Data Density, Person, Visit, Death

-Not ok: Not able to see data sources: Condition, Condition Era, Procedure, Drug, Drug Era, Measurement, Observation

-Not ok: Profile is unable to load

-Not ok: No permissions to generate cohort

Thank you.

Chris_Knoll · November 4, 2018, 5:16pm

Ok, the first error about ‘Invalid or unexpected token’ when loading config-local.js:

You have some kind of invalid character or syntax error in config-local.js. It says line 5. Could you give us the top 10 lines of your config-local.js file?

Also, if you have whitespace (like a space) in your ‘SOURCE_KEY’ value of your WebAPI source table, I’d change it so that your source key has non-whitespace. Ie:
source_name = 'My CDM', source_key = 'MY_CDM'.

If you do update your source table, you will need to clear the WebAPI source cache by going to this URL:

locahost:8080/WebAPI/source/refresh

-Chris

hui_xing_tan · November 4, 2018, 11:13pm

13%20AM

Ah, yes, I had typed in ` instead of ’ in the name field. Have corrected it now (to above) and the invalid character error is gone, thanks

No whitespace in the source key though, here is the tables on pdAgminIII

Also unable to clear cache:

Thanks much!

Chris_Knoll · November 5, 2018, 2:17pm

OK, glad one of the errors was resolved. The problem with your ‘refresh’ call was you didn’t spell ‘localhost’ correct, you put ‘locahost’ not ‘localhost’.

After correcting your config-local.js file, what errors are you seeing now in the console?

hui_xing_tan · November 5, 2018, 3:26pm

i see! OK…here are the errors after correcting config-local.js and clearing the source cache:

With regards to the failed to load resource…status of 500…clicked on it and this was at the end of the pages loaded:

http://localhost:8080/WebAPI/cdmresults/MY_CDM/conditionera

http://localhost:8080/WebAPI/vocabulary/vocab/included-concepts/count

http://localhost:8080/WebAPI/vocabulary/vocab/optimize

and upon clicking on ‘Estimation’:
37%20PM

Thank you!

Chris_Knoll · November 5, 2018, 4:09pm

Ok, the ohdsi.concept_heirarchy error is due to a table not being created in your results schema. I thought this was available in 2.5.0, but you want to execute the following scripts to create and then initialize the table (note you must replace the place-hodlers with your own local cdm/vocabulary schema:

https://github.com/OHDSI/WebAPI/blob/master/src/main/resources/ddl/results/concept_hierarchy.sql

https://github.com/OHDSI/WebAPI/blob/master/src/main/resources/ddl/results/init_concept_hierarchy.sql

The 404 in included-concepts/count and the 405 in /vocab/optimize is (I believe) because those URls get POST’ed to (ie: the browser posts JSON to get procssed) so just trying to open the URL directly results in a GET, which is not found/ method not supported. The POST/GET is the ‘method’ they are referring to in the error. But in the debug console, you can click ont he request to see what the data was that was posted up, and see the actual error (i’m not sure if your output is from the console report or if you opened the URL in a new tab, which would change the POST to a GET)

Finally, the ‘Estimation’ error, it is saying it can’t find the cohort-comparison-browser module, and the URL it tried to access was under /js/cohort-comparison-browser.js. But in 2.5.0, that module is found in /js/components/cohort-comparison-browser.js.

And, according to the main.js file, this line exists:

"cohort-comparison-browser": "components/cohort-comparison-browser",

So, I’m not quite sure where it is thinking that when ‘cohort-comparison-browser’ is being loaded, that ti doesn’t actually try to go to the /js/components path. That’s a bit of a mystery. Could you confirm that in your js/main.js file, you have an entry for ‘cohort-comparison-browser’ around line 17?

hui_xing_tan · November 5, 2018, 11:26pm

hmm ok… these are some of the screenshots from clicking on the error in the console
on clicking concept sets> concept:

in the console where the error is showed, click on the ‘sources’ tab

click on the ‘network’ tab. ‘count’ is highlighted in red

yes, its in the main.js file, shown below:

const bustCache = (() => {
	const key = 'bustCache';
	let hash = localStorage.getItem(key) || (localStorage[key] = Math.random().toString(36).substring(7));
	return '_=' + hash;
})();

const localRefs = {
	"configuration": "components/configuration",
	"concept-manager": "components/concept-manager",
	"conceptset-browser": "components/conceptset/conceptset-browser",
	"conceptset-editor": "components/conceptset/conceptset-editor",
	"conceptset-manager": "components/conceptset/conceptset-manager",
	"conceptset-modal": "components/conceptsetmodal/conceptSetSaveModal",
	"conceptset-list-modal": "components/conceptset/conceptset-list-modal",
	"cohort-comparison-manager": "components/cohort-comparison-manager",
	"job-manager": "components/job-manager",
	"cohort-comparison-browser": "components/cohort-comparison-browser",
	"cohort-comparison-print-friendly": "components/cohort-comparison-print-friendly",
	"cohort-comparison-r-code": "components/cohort-comparison-r-code",
	"cohort-comparison-multi-r-code": "components/cohort-comparison-multi-r-code",
	"user-bar": "components/user-bar",
	"faceted-datatable": "components/faceted-datatable",
	"explore-cohort": "components/explore-cohort",
	"r-manager": "components/r-manager",
	"home": "components/home",
	"welcome": "components/welcome",
	"forbidden": "components/ac-forbidden",
	"unauthenticated": "components/ac-unauthenticated",
	"roles": "components/roles",
	"role-details": "components/role-details",
	"loading": "components/loading",
	"atlas-state": "components/atlas-state",
	"plp-manager": "components/plp-manager",
	"plp-inspector": "components/plp-inspector",
	"plp-browser": "components/plp-browser",
	"plp-roc": "components/plp-roc",
	"plp-calibration": "components/plp-calibration",
	"plp-spec-editor": "components/plp-spec-editor",
	"plp-r-code": "components/plp-r-code",
	"plp-print-friendly": "components/plp-print-friendly",
	"feedback": "components/feedback",
	"conceptsetbuilder": "modules/conceptsetbuilder",
	"conceptpicker": "modules/conceptpicker",
	"webapi": "modules/WebAPIProvider",
	"vocabularyprovider": "modules/WebAPIProvider/VocabularyProvider",
	"css": "plugins/css.min",
};

// set 'optional' path prior to first call to require
requirejs.config({paths: {"optional": "plugins/optional"}});

require([
	'./settings',
	'optional', // require this plugin separately to check in advance whether we have a local config
	'config'
], (settings, optional, appConfig) => {
	const cdnRefs = {};
	Object.entries(settings.paths).forEach(([name, path]) => {
		cdnRefs[name] = appConfig.useBundled3dPartyLibs
			? 'assets/bundle/bundle'
			: path;
	});

	requirejs.config({
		...settings,
		urlArgs: bustCache,
		deps: ['css!styles/jquery.dataTables.min',
			'css!styles/jquery.dataTables.colVis.css'
		],
		paths: {
			...localRefs,
			...cdnRefs,
		},
	});
	require(['bootstrap'], function () { // bootstrap must come first
    $.fn.bstooltip = $.fn.tooltip;
		require([
			'providers/Application',
			'providers/Model',
			'providers/Router',
			'atlas-state',
			'jquery.ui.autocomplete.scroll',
			'loading',
			'user-bar',
			'welcome',
			'components/white-page'
		],
			(
				Application,
				Model,
				Router,
				sharedState,
			) => {
				const app = new Application(new Model(), new Router());

				app.bootstrap()
					.then(() => app.synchronize())
					.then(() => app.run())
					.catch(er => {
						sharedState.appInitializationStatus(Model.applicationStatuses.failed);
						console.error('App initialization failed', er);
					});

				return app;
		});
	});
});

Chris_Knoll · November 6, 2018, 4:16am

On that one, can you flip over to the ‘response’ tab on the right. it will give you a server-side stack trace and we might be able to find out what the underlying cause is. Failing that, we may have to go through the server logs. Do you have access to the tomcat/java server logs?

hui_xing_tan · November 6, 2018, 8:31am

Ok, i have done so!

Below is the text under ‘response’

Well…I went to the tomcat folder and couldn’t find a java folder… sorry is that how I should be accessing the tomcat/java server logs? Thanks!

limadm · November 6, 2018, 4:24pm

Hi, @hui_xing_tan!
I also had some trouble to access tomcat logs inside docker. So I changed log4j.xml to:

?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
  <appender class="org.apache.log4j.FileAppender" name="stdout">
    <param name="file"   value="webapi.log" />
    <param name="append" value="false" />
    <layout class="org.apache.log4j.PatternLayout">
      <param value="%d{yyyy-MM-dd HH:mm:ss.SSS} %p %t %c - %x - %m%n"
        name="ConversionPattern" />
    </layout>
  </appender>

  <logger name="org.ohdsi">
    <level value="debug" />
  </logger>
  <!-- <logger name="org.springframework.orm"> -->
  <!--  <level value="debug" /> -->
  <!-- </logger> -->
  <!-- <logger name="org.springframework.jdbc"> -->
  <!--  <level value="debug" /> -->
  <!-- </logger> -->
  <root>
    <level value="info" />
    <appender-ref ref="stdout" />
  </root>
</log4j:configuration>

After redeploying the tomcat webapp, any org.ohdsi exceptions are appended to webapi.log and can be accessed with
docker exec -it postgresql_broadsea-webtools_1 tail -f webapi.log
(docker container should be running).

Chris_Knoll · November 6, 2018, 11:01pm

So, from the javascript error side, I can’t tell what the real problem is. The serer is reporting a 500 error (meaning that there is a runtime error happening trying to make this request) but it eventually becomes a 404 at the bottom of the error message. I’m not sure what error condition leads to that. Sorry. If you do find where the logs are being written, you can do some calls to the server and see if any additional information can be found in the logs.

t_abdul_basser · November 7, 2018, 7:01pm

Alternatively you can tail the supervisor logs as per the Broadsea docs.

docker-compose exec broadsea-webtools bash

then, in the container shell

tail -f /var/log/supervisor/deploy-script-stdout*.log

This makes changing the log configuration and restarting tomcat in the container unnecessary.

t_abdul_basser · November 7, 2018, 7:38pm

Also try sending us using the Preview tab in the network area (between headers and response) and expanding payload. In addition to a prettified version of the response it often includes the localized message and message from the server-side Java exception.

hui_xing_tan · November 10, 2018, 2:20pm

Thank you all for the helpful responses. We did not implement with docker but managed to find the tomcat logs, let me know which you need:
08%20PM

Ok, here it is!

Thank you! Cheers