Error zipping results for PLE study

Implementers
1

@schuemie, hitting a bit of an issue with the very last step of a PLE study.

I’m unable to package the results into a Zip. What package executes this function? Maybe I’m missing it.

I’m getting this as my error:

|Packaging results Exporting analyses - cohort_method_analysis table - covariate_analysis table Exporting exposures - exposure_of_interest table Exporting outcomes - outcome_of_interest table - negative_control_outcome table Exporting metadata - database table - exposure_summary table - comparison_summary table - attrition table |===============================================================================================| 100% - covariate table - cm_follow_up_dist table |===============================================================================================| 100% Exporting main results - cohort_method_result table Performing empirical calibration on main effects |===============================================================================================| 100% censoring 0 values (0%) from targetSubjects because value below minimum censoring 0 values (0%) from comparatorSubjects because value below minimum censoring 0 values (0%) from targetOutcomes because value below minimum censoring 0 values (0%) from comparatorOutcomes because value below minimum - cm_interaction_result table |===============================================================================================| 100% Exporting diagnostics - covariate_balance table |===============================================================================================| 100% - preference_score_dist table |===============================================================================================| 100% - propensity_model table |===============================================================================================| 100% - kaplan_meier_dist table Computing KM curves |===============================================================================================| 100% Writing to single csv file

Adding results to zip file
Error in .jcall(“RJavaTools”, “Ljava/lang/Object;”, “invokeMethod”, cl, : java.lang.RuntimeException: java.io.FileNotFoundException: ./IUD/AmbEMR/export/ResultsAmbEMR.zip (No such file or directory) Warning: Error ‘cannot open the connection’ when writing log to file './IUD/AmbEMR/log.txt. Removing file appender from logger.|

It can’t open the results because it is not able to execute writing the results zip. All the other csvs are written. Any thoughts on how to debug this?

traceback()
7: stop(list(message = “java.lang.RuntimeException: java.io.FileNotFoundException: ./IUD/AmbEMR/export/ResultsAmbEMR.zip (No such file or directory)”,
call = .jcall(“RJavaTools”, “Ljava/lang/Object;”, “invokeMethod”,
cl, .jcast(if (inherits(o, “jobjRef”) || inherits(o,
“jarrayRef”)) o else cl, “java/lang/Object”), .jnew(“java/lang/String”,
method), j_p, j_pc, use.true.class = TRUE, evalString = simplify,
evalArray = FALSE), jobj = new(“jobjRef”, jobj = <pointer: 0x6bf5750>,
jclass = “java/lang/RuntimeException”)))
6: .jcheck(silent = FALSE)
5: .jcall(“RJavaTools”, “Ljava/lang/Object;”, “invokeMethod”, cl,
.jcast(if (inherits(o, “jobjRef”) || inherits(o, “jarrayRef”)) o else cl,
“java/lang/Object”), .jnew(“java/lang/String”, method),
j_p, j_pc, use.true.class = TRUE, evalString = simplify,
evalArray = FALSE)
4: .jrcall(x@name, name, …)
3: rJava::J(“org.ohdsi.databaseConnector.Compression”)$createZipFile(files,
rootFolder, zipFile, compressionLevel)
2: DatabaseConnector::createZipFile(zipFile = zipName, files = files) at Export.R#78
1: exportResults(outputFolder, databaseId, databaseName, databaseDescription,
minCellCount = 5, maxCores)

2

The zipping is done (obviously) by DatabaseConnector (didn’t know where else to put it)

It says it can’t find the path to write to. Maybe its the fact that you’re using a relative path? I thought that should not be an issue, but you can try to

  1. use an absolute path, or
  2. update DatabaseConnector
3

I’ve confirmed we have the latest Database Connector. I’m working through suggestion 1.

Now I have a new problem:

  • propensity_model table | | 0%
    Error in setwd(dfile) : cannot change working directory

And I see… (had to truncate the as.ff.default call due to character limits)

traceback()

    BATCHSIZE = BATCHSIZE, BATCHBYTES = BATCHBYTES, VERBOSE = VERBOSE)

17: assign(“ret”, ff(initdata = initdata, length = length, levels = levels,
ordered = ordered, dim = dim, dimorder = dimorder, bydim = bydim,
symmetric = symmetric, fixdiag = fixdiag, names = names,
dimnames = dimnames, ramclass = ramclass, ramattribs = ramattribs,
vmode = vmode, update = update, pattern = pattern, filename = filename,
readonly = FALSE, overwrite = overwrite, pagesize = pagesize,
caching = caching, finalizer = finalizer, finonexit = finonexit,
FF_RETURN = FF_RETURN, BATCHSIZE = BATCHSIZE, BATCHBYTES = BATCHBYTES,
VERBOSE = VERBOSE))
16: clone.ff(x, FF_RETURN = TRUE, filename = filename, overwrite = overwrite,
…)
15: as.ff.default(c(0.119998365602219, 0.0249168094261037, -0.00795928504376532,
-0.305683368735177, 0.0103077348565255, -0.0344566499449007,
0.0148057257155164…), vmode = NULL, pattern = “ffdf”)
13: do.call(“as.ff”, c(list(xi, vmode = vmodes[[i]]), col_args))
12: FUN(X[[i]], …)
11: lapply(seq_along(x), function(i, …) {
xi ← x[[i]]
AsIs ← inherits(xi, “AsIs”)
if (AsIs) {
oldClass(xi) ← oldClass(xi)[-match(“AsIs”, oldClass(xi))]
ret ← do.call(“as.ff”, c(list(xi, vmode = vmodes[[i]]),
col_args))
oldClass(ret) ← c(“AsIs”, oldClass(ret))
ret
}
else {
do.call(“as.ff”, c(list(xi, vmode = vmodes[[i]]), col_args))
}
}, …)
10: as.ffdf.data.frame(coefficients)
9: ff::as.ffdf(coefficients)
8: merge(ff::as.ffdf(coefficients), cohortMethodData$covariateRef)
7: CohortMethod::getPsModel(ps, cmData) at Export.R#799
6: .fun(piece, …)
5: (function (i)
{
piece ← pieces[[i]]
if (.inform) {
res ← try(.fun(piece, …))
if (inherits(res, “try-error”)) {
piece ← paste(utils::capture.output(print(piece)),
collapse = “\n”)
stop("with piece ", i, “: \n”, piece, call. = FALSE)
}
}
else {
res ← .fun(piece, …)
}
progress$step()
res
})(1L)
4: loop_apply(n, do.ply)
3: plyr::llply(split(subset, 1:nrow(subset)), getPsModel, reference = reference,
.progress = “text”) at Export.R#816
2: exportDiagnostics(outputFolder = outputFolder, exportFolder = exportFolder,
databaseId = databaseId, minCellCount = minCellCount, maxCores = maxCores) at Export.R#66
1: exportResults(outputFolder, databaseId, databaseName, databaseDescription,
minCellCount = 5, maxCores)

Hunting around… where can I uncover the setwd(dfile) reference? I’m not sure where this is trying to pull from. Probably need to modify this to be an absolute pathway as well.

4

Might be the fftemp folder. Did you specify it? For example

options(fftempdir = "s:/FFtemp")
5

we saw the same problems and the solution Sigfried used was to close RStudio and re-run it. (and it went away). I know, weird… (cannot change working directory was exactly the error he got and it made no sense for the action he was doing).

6

Hello everyone, my name is Yasmine, I’m a Data Science for health student in the University of Lille (France). I’m currently an intern at a pharmaceutical company.

I had the opportunity to explore the ATLAS features and launch a PLE study.

However, I had the very same error, I’m unable to package the results into a Zip.
I’m getting this error:

Error in .jcall(“RJavaTools”, “Ljava/lang/Object;”, “invokeMethod”, cl, : java.lang.RuntimeException: java.io.FileNotFoundException: myresults/export/ResultsSynpuf.zip (No such file or directory)
Warning: could not find logger DEFAULT

image

Any ideas to solve the problem (except what has been said before)? Thank you.

7

Hi @Yasmine_M. It might help to know the values of files, rootFolder and zipFile at the time of the error.

To get these values try running this code:

# First set an error handling option
options(error = utils::recover)

# Then rerun the execute function with all of the same arguments you used before
execute() # add study arguments like connectionDetails etc

# When the error occurs choose option createZipFile
# you should see a Browser prompt
# Next look up the values 
sapply(c("files", "rootFolder", "zipFile"), function(x) get(x))

You can exit the Browser prompt by typing the letter c.
You can reset the error handling by either restarting R or running options(error = NULL)

If you are using RStudio then you could also look these up using the “rerun with debug” button as well.

1 Like
8

@krfeeney @schuemie
I had the exact same issue as the original report, but then with the PLEE study package.
After some digging around, I might have found the issue.

Lines 72-79 of R/Export.R in PLEE study package:

  ParallelLogger::logInfo("Adding results to zip file")
  zipName <- file.path(exportFolder, sprintf("Results_%s.zip", databaseId))
  files <- list.files(exportFolder, pattern = ".*\\.csv$")
  oldWd <- setwd(exportFolder)
  on.exit(setwd(oldWd))
  DatabaseConnector::createZipFile(zipFile = zipName, files = files)
  ParallelLogger::logInfo("Results are ready for sharing at:", zipName)

Changing the working directory, makes that you are already in the export folder while zipName still points to something like output/export/Results_xyz.zip. Of course, this path does not exist INSIDE the export folder.

The simple fix for me was to manually change the second line to:
zipName <- file.path(sprintf("Results_%s.zip", databaseId))

Happy to make an issue and PR. What is the right repository for that? WebApi?

1 Like
9

It seems the issue is caused by people using relative paths (I tend to use absolute paths everywhere, but we certainly should support relative paths). Perhaps convert the relative path to an absolute one before calling setwd()?

The right repo for this would be SkeletonComparativeEffectStudy.

1 Like
10

Thanks Martijn. I can confirm that the issue was with relative paths. This also causes an issue elsewhere in the execution, with the andromeda temp folder.

And agreed, the study packages ideally would support relative paths. Especially in the case of a remote research environment, where researcher might not even know the absolute path to their home directory.

1 Like