Is there a way to avoid AWS to run https://github.com/OHDSI/ETL-LambdaBuilder ?
We have our own DB in our servers and it could be helpful if I could find a version of this code to run out of AWS.
Thanks
I believe that to avoid AWS I need to specify then next fields:
<connectionStrings>
<add name=“Source” connectionString=“Driver={SQL Server Native Client 11.0};Server=;Database=;Uid=;Pwd=;” />
<add name=“Destination” connectionString=“Driver={SQL Server Native Client 11.0};Server=;Database=;Uid=;Pwd=;” />
<add name=“Vocabulary” connectionString=“Driver={SQL Server Native Client 11.0};Server=;Database=;Uid=;Pwd=;” />
<add name=“Builder” connectionString=“Data Source=;Initial Catalog=;User Id=;Password=;” />
<add name=“CDM_Processing” connectionString=“Data Source=;Initial Catalog=;User Id=;Password=;” />
<add name=“HIX” connectionString=“Data Source=;Initial Catalog=;User Id=;Password=;” />
</connectionStrings>
at the app.config file. My intuition tells me that the source should be for example the truven DB I need to convert, the “Destination” is used to identify where the new DB needs to be saved, the Vocabulary will be the actual -latest?- vocabulary. What are “Builder”, CDM_Processing", “HIX” for? Do I need them for a Truven DB? From the code, these config is used if the program is run without any parameters and this will avoid using AWS settings. Please, let me know.
Thanks
PD: something interesting I found: Truven Marketscan datasets in CDM
While running the builder buildingManager2 application I saw the next error:
Builder: HSC-CTSC-ETL
Time: Nov 10 2020 9:39PM
Error: Exception Found:
Type: System.Data.SqlClient.SqlException
Message: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Source: Core .Net SqlClient Data Provider
Stacktrace: at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error) at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync() at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket() at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer() at System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj) at System.Data.SqlClient.TdsParser.ProcessAttention(TdsParserStateObject stateObj) at System.Data.SqlClient.TdsParserStateObject.WriteSni(Boolean canAccumulate) at System.Data.SqlClient.TdsParserStateObject.WritePacket(Byte flushMode, Boolean canAccumulate) at System.Data.SqlClient.TdsParser.WriteLong(Int64 v, TdsParserStateObject stateObj) at System.Data.SqlClient.TdsParser.WriteUnterminatedValue(Object value, MetaType type, Byte scale, Int32 actualLength, Int32 encodingByteSize, Int32 offset, TdsParserStateObject stateObj, Int32 paramSize, Boolean isDataFeed) at System.Data.SqlClient.TdsParser.WriteBulkCopyValue(Object value, SqlMetaDataPriv metadata, TdsParserStateObject stateObj, Boolean isSqlType, Boolean isDataFeed, Boolean isNull) at System.Data.SqlClient.SqlBulkCopy.ReadWriteColumnValueAsync(Int32 col) at System.Data.SqlClient.SqlBulkCopy.CopyColumnsAsync(Int32 col, TaskCompletionSource1 source)
at System.Data.SqlClient.SqlBulkCopy.CopyRowsAsync(Int32 rowsSoFar, Int32 totalRows, CancellationToken cts, TaskCompletionSource1 source) at System.Data.SqlClient.SqlBulkCopy.CopyBatchesAsyncContinued(BulkCopySimpleResultSet internalResults, String updateBulkCommandText, CancellationToken cts, TaskCompletionSource1 source)
at System.Data.SqlClient.SqlBulkCopy.CopyBatchesAsync(BulkCopySimpleResultSet internalResults, String updateBulkCommandText, CancellationToken cts, TaskCompletionSource1 source) at System.Data.SqlClient.SqlBulkCopy.WriteToServerInternalRestContinuedAsync(BulkCopySimpleResultSet internalResults, CancellationToken cts, TaskCompletionSource1 source)
at System.Data.SqlClient.SqlBulkCopy.WriteToServerInternalRestAsync(CancellationToken cts, TaskCompletionSource1 source) at System.Data.SqlClient.SqlBulkCopy.WriteToServerInternalAsync(CancellationToken ctoken) at System.Data.SqlClient.SqlBulkCopy.WriteRowSourceToServerAsync(Int32 columnCount, CancellationToken ctoken) at System.Data.SqlClient.SqlBulkCopy.WriteToServer(IDataReader reader) at org.ohdsi.cdm.framework.desktop.Savers.Saver.AddChunk(List1 chunk, Int32 index)
Inner Exception Found:
Type: System.ComponentModel.Win32Exception
Message: The wait operation timed out
Source:
Stacktrace:
What this error means? However, this application continues running bt apparently stuck just at the beggining of the chunks creation step.
I just saw:
Chunks creation in progress, please do not close the window…
Bucket =;CDMFolder=cdm
SaveOnlyToS3=True; StorageType=Parquet;
Settings initialized
Vendor=Truven_CCAE; restart=False
BuildingId=1; Chunk size=512
Source connection string=Driver={SQL Server Native Client 11.0};Server=localhost;Database=Tru …
What are
SaveOnlyToS3=True; StorageType=Parquet;
for?