New IRIS Dialect and ACHILLES_RESULTS

Hello, first cry for help here, and to be clear I dont think I am looking for a definitive fix for the problem, but due to my newness would like some guidance on which dial or lever to turn to find/diagnose the problem(s), which I face two with Achilles:

  • CTE errors loading the analysis (I dont know much about common table expressions)
  • ACHILLES_RESULTS missing entirely

Who could be the culprit for CTE errors…? I have gotten several of these of different flavors, but now on the latest and greatest seem to only be getting one on select:

java.sql.SQLException: [SQLCODE: <-383>:<CTE is only supported for SELECT statements>]

the java package signal seems to be a little generic.

Predominant in the forums is the missing ACHILLES_RESULTS table… I do get a ACHILLES_ANALYSIS table even tried creating this manually and cant seem to figure it out… security looks good and ruled out from a sql perspective… I assume Achilles creates this, how can I trap the where and see if its failing? Nothing in the output folder in this regard.

Here are the dials I am turning:

cdm5.4
jdbc driver itself (vendor database jar) intersystems-jdbc-3.10.1.jar
sqlrender 1.19.1 github remote
achilles 1.7.12 github remote
achilles parameters:

achilles(connectionDetails = connectionDetails, cdmDatabaseSchema = "OMOPCDM54", cdmVersion = "5.4",resultsDatabaseSchema = "OMOPCDM54_RESULTS", outputFolder = "output", optimizeAtlasCache = TRUE, createTable = TRUE,  scratchDatabaseSchema = "OMOPCDM54_SCRATCH", numThreads = 5)

I update to latest when installing the Achilles package for everything when prompted…

I am just trying to get a “hello world” for deployment purposes with hades,achilles, and atlas that I can use that successfully shows the dashboards and can generate on a cohort against the dialect.

Apologies if my goal aligns with the topic, but the question does not directly.

Hi @sween ! Thank you for this reported issue. We are looking into this issue! I will tag you as soon as there is an update!

1 Like

Hi @sween , which version of IRIS are you using? I suspect you’re using something earlier than 2024.3. The JDBC driver version you’re quoting should support this, hence the server-side error instead of a preparser one.

1 Like

latest, from Maven at least.

https://mvnrepository.com/artifact/com.intersystems/intersystems-jdbc/3.10.3

Also 3.10.2…

I tried 3.7.1 and 3.8.4, and the only noticeable issue I get are varying support operations for CTE’s… some get errors saying QUERY is not supported, some report more operations than that.

I’m asking about the IRIS server version. Any JDBC driver >= 3.10.1 should be fine for this.

Sorry, shoulda had that handy, I know better. The below highlights the varying CTE errors against service versions.

Using intersystems-jdbc-3.10.3.jar from maven and

Service:
IRIS LOCKEDDOWN ARM64 / 2024.2.0.247.0
8.0.1 1.17 01/10/25 iris-lockeddown-arm64:2024.2.0.247.0

Throws:

DBMS:
iris

Error:
java.sql.SQLException: [SQLCODE: <-383>:<CTE is only supported for SELECT statements>]
[Location: <Prepare>]
[%msg: < CTE is only supported for SELECT statements.>]

SQL:
WITH rawData AS (
  select
    year(op1.index_date) - p1.YEAR_OF_BIRTH as stratum_1,
    COUNT(p1.person_id) as count_value
  from OMOPCDM54.person p1
    inner join (select person_id, MIN(observation_period_start_date) as index_date from OMOPCDM54.observation_period group by PERSON_ID) op1
    on p1.PERSON_ID = op1.PERSON_ID
  group by year(op1.index_date) - p1.YEAR_OF_BIRTH
)
CREATE GLOBAL TEMPORARY TABLE OMOPCDM54_RESULTS.ku80ymnms_tmpach_101
 AS SELECT 101 as analysis_id,
  CAST(stratum_1 AS VARCHAR(255)) as stratum_1,
  cast(null as varchar(255)) as stratum_2,
  cast(null as varchar(255)) as stratum_3,
  cast(null as varchar(255)) as stratum_4,
  cast(null as varchar(255)) as stratum_5,
  count_value
 FROM rawData

R version:
R version 4.2.1 (2022-06-23)

Platform:
x86_64-pc-linux-gnu

Attached base packages:
- stats
- graphics
- grDevices
- utils
- datasets
- methods
- base

Other attached packages:
- SqlRender (1.19.1)
- Achilles (1.7.2)
- DatabaseConnector (6.4.0)
- Andromeda (0.6.7)
- dplyr (1.1.4)

Service:
IRIS LOCKEDDOWN ARM64 / 2024.3.0.217.0
8.0.4 1.17 02/03/25 iris-lockeddown-arm64:2024.3.0.217.0 N/A

Throws:

DBMS:
iris

Error:
java.sql.SQLException: [SQLCODE: <-383>:<CTE is only supported for SELECT, INSERT, UPDATE, DELETE statements, or in the query following CREATE TABLE ... AS>]
[Location: <Prepare>]
[%msg: < CTE is only supported for SELECT, INSERT, UPDATE, DELETE statements, or in the query following CREATE TABLE ... AS.>]

SQL:
with rawData (gender_concept_id, age_value) as
(
  select p.gender_concept_id, MIN(YEAR(observation_period_start_date)) - P.YEAR_OF_BIRTH as age_value
	from OMOPCDM54.person p
	JOIN OMOPCDM54.observation_period op on p.person_id = op.person_id
	group by p.person_id,p.gender_concept_id, p.year_of_birth
),
overallStats (gender_concept_id, avg_value, stdev_value, min_value, max_value, total) as
(
  select gender_concept_id,
  CAST(avg(1.0 * age_value) AS FLOAT) as avg_value,
  CAST(STDDEV(age_value) AS FLOAT) as stdev_value,
  min(age_value) as min_value,
  max(age_value) as max_value,
  COUNT(*) as total
  FROM rawData
  group by gender_concept_id
),
ageStats (gender_concept_id, age_value, total, rn) as
(
  select gender_concept_id, age_value, COUNT(*) as total, row_number() over (order by age_value) as rn
  FROM rawData
  group by gender_concept_id, age_value
),
ageStatsPrior (gender_concept_id, age_value, total, accumulated) as
(
  select s.gender_concept_id, s.age_value, s.total, sum(p.total) as accumulated
  from ageStats s
  join ageStats p on s.gender_concept_id = p.gender_concept_id and p.rn <= s.rn
  group by s.gender_concept_id, s.age_value, s.total, s.rn
)
CREATE GLOBAL TEMPORARY TABLE OMOPCDM54_RESULTS.iq6fwdx8tempResults_104
 AS SELECT 104 as analysis_id,
  CAST(o.gender_concept_id AS VARCHAR(255)) as stratum_1,
  o.total as count_value,
  o.min_value,
	o.max_value,
	o.avg_value,
	o.stdev_value,
	MIN(case when p.accumulated >= .50 * o.total then age_value end) as median_value,
	MIN(case when p.accumulated >= .10 * o.total then age_value end) as p10_value,
	MIN(case when p.accumulated >= .25 * o.total then age_value end) as p25_value,
	MIN(case when p.accumulated >= .75 * o.total then age_value end) as p75_value,
	MIN(case when p.accumulated >= .90 * o.total then age_value end) as p90_value
 FROM ageStatsPrior p
join overallStats o on p.gender_concept_id = o.gender_concept_id
GROUP BY o.gender_concept_id, o.total, o.min_value, o.max_value, o.avg_value, o.stdev_value


R version:
R version 4.2.1 (2022-06-23)

Platform:
x86_64-pc-linux-gnu

Attached base packages:
- stats
- graphics
- grDevices
- utils
- datasets
- methods
- base

Other attached packages:
- SqlRender (1.19.1)
- Achilles (1.7.2)
- DatabaseConnector (6.4.0)
- Andromeda (0.6.7)
- dplyr (1.1.4)

Local (EAP Intsall)

USER>zw $ZV "IRIS for UNIX (Ubuntu Server 24.04 LTS for x86-64) 2025.1.0VECANN (Build 124U) Tue Jan 21 2025 17:22:13 EST"

Throws:

DBMS:
iris

Error:
java.sql.SQLException: [SQLCODE: <-383>:<CTE is only supported for SELECT, INSERT, UPDATE, DELETE statements, or in the query following CREATE TABLE ... AS>]
[Location: <Prepare>]
[%msg: < CTE is only supported for SELECT, INSERT, UPDATE, DELETE statements, or in the query following CREATE TABLE ... AS.>]

SQL:
WITH rawData AS (
SELECT 
	m.measurement_concept_id AS stratum_1,
	YEAR(m.measurement_date) AS stratum_2,
	p.gender_concept_id AS stratum_3,
	FLOOR((YEAR(m.measurement_date) - p.year_of_birth) / 10) AS stratum_4,
	COUNT(DISTINCT p.person_id) AS count_value
FROM 
	OMOPCDM54.person p
JOIN 
	OMOPCDM54.measurement m 
ON 
	p.person_id = m.person_id
JOIN 
	OMOPCDM54.observation_period op 
ON 
	m.person_id = op.person_id
AND 
	m.measurement_date >= op.observation_period_start_date
AND 
	m.measurement_date <= op.observation_period_end_date		
GROUP BY 
	m.measurement_concept_id,
	YEAR(m.measurement_date),
	p.gender_concept_id,
	FLOOR((YEAR(m.measurement_date) - p.year_of_birth) / 10)
)
CREATE GLOBAL TEMPORARY TABLE OMOPCDM54_RESULTS.bqtg620qs_tmpach_1804
 AS SELECT 1804 AS analysis_id,
	CAST(stratum_1 AS VARCHAR(255)) AS stratum_1,
	CAST(stratum_2 AS VARCHAR(255)) AS stratum_2,
	CAST(stratum_3 AS VARCHAR(255)) AS stratum_3,
	CAST(stratum_4 AS VARCHAR(255)) AS stratum_4,
	CAST(NULL AS VARCHAR(255)) AS stratum_5,
	count_value
 FROM rawData

R version:
R version 4.2.1 (2022-06-23)

Platform:
x86_64-pc-linux-gnu

Attached base packages:
- stats
- graphics
- grDevices
- utils
- datasets
- methods
- base

Other attached packages:
- SqlRender (1.19.1)
- Achilles (1.7.2)
- DatabaseConnector (6.4.0)
- Andromeda (0.6.7)
- dplyr (1.1.4)

On ALL achilles jobs, after the CTE errors, they all end with this error:

Thread: Main
Message:  line 37 did not have 6 elements
Level:  FATAL
Time:  2025-02-14 17:48:16

Stack trace:
7: scan(file = file, what = what, sep = sep, quote = quote, dec = dec, nmax = nrows, skip = 0, na.strings = na.strings, quiet = TRUE, fill = fill, strip.white = strip.white, blank.lines.skip = blank.lines.skip, multi.line = FALSE, comment.char = comment.ch
6: utils::read.table(file = file.path(outputFolder, "log_achilles.txt"), header = FALSE, sep = "\t", stringsAsFactors = FALSE)
5: .parseLogs(outputFolder)
4: .mergeAchillesScratchTables(resultsTable = table, connectionDetails = connectionDetails, analysisIds = analysisDetails$ANALYSIS_ID, createTable = createTable, schemaDelim = schemaDelim, scratchDatabaseSchema = scratchDatabaseSchema, resultsDatabaseSchem
3: FUN(X[[i]], ...)
2: lapply(resultsTablesToMerge, function(table) {
    .mergeAchillesScratchTables(resultsTable = table, connectionDetails = connectionDetails, analysisIds = analysisDetails$ANALYSIS_ID, createTable = createTable, schemaDelim = schemaDelim, scratchDatabaseSc
1: achilles(connectionDetails = connectionDetails, cdmDatabaseSchema = "OMOPCDM54", cdmVersion = "5.4", resultsDatabaseSchema = "OMOPCDM54_RESULTS", outputFolder = "output", optimizeAtlasCache = TRUE, createTable = TRUE)

R version:
R version 4.2.1 (2022-06-23)

Platform:
x86_64-pc-linux-gnu

Attached base packages:
- stats
- graphics
- grDevices
- utils
- datasets
- methods
- base

Other attached packages:
- SqlRender (1.19.1)
- Achilles (1.7.2)
- DatabaseConnector (6.4.0)
- Andromeda (0.6.7)
- dplyr (1.1.4)



by chance are there some referential versions I can adjust to that have been used in the past with success for the components involved to mitigate the CTE errors?

Hi @sween ,
apologies for the late response. I had this tread as “tracking” and not “watching”, so didn’t receive a notification when you added more detail.

IRIS itself indeed currently does not accept CTEs for DDL (only for DML and queries), but we should have these addressed through SqlRender rules. Let me take a look and report back to you.

thanks,
benjamin

I found the rewrite rule that should address this case (here), but it requires that there is a semicolon separator at the end of the statement, otherwise the rewrite rule is invalid (as per some SqlRender validation thing). When I add a semicolon at the end, the rule correctly rewrites this to the (somewhat blunt) combination of creating temp tables for each CTE and then running the intended CREATE TABLE (and cleaning up afterwards). I’m not yet sure how we can ensure the scripts that go in always end with a semicolon, or find a different solution, as server-side support for this may take a while.

I took a closer look at the most likely source of this statement in the Achilles repo (here) and found it isn’t a WITH .. CREATE TABLE #temp .. statement, but rather a SELECT .. INTO #temp .. command that just happens to have a WITH .. clause in front of it. This gets rewritten by a rule that is placed further down in the rewrite rules, and therefore does not go back to the rewrite rule that works around the lack of CTE support in DDL statements as described above. I have moved the concerned blocks of rules around in the replacementPatterns.csv file and it seems to translate to valid IRIS DDL statements for me now. If you can find a way to test this and confirm it works for you, I’ll issue a new pull request.

Note that you can force-load the replacement pattern file using this code snippet:

library(rJava)
rJava::J('org.ohdsi.sql.SqlTranslate')$setReplacementPatterns('path/to/replacementPatterns.csv')

interim update: I misread the doc and typo’ed my test, and our CTAS command does support CTEs, they just need to follow the AS keyword, as apparently is the case for most other vendors. I’ll need to rework the replacementPatterns.csv a little bit to take advantage of those. Some rules in there predate IRIS support for CTEs in DML/CTAS, so probably this will be a net simplification.