dbptk stops unexpectedly "Found an unsupported datatype value, and an attempt was made to save it as text."

[Laurira]

How to turn on more detailed log? At the moment dbptk stops with no errors and in the log the last line is
"Found an unsupported datatype value, and an attempt was made to save it as text."

What to do?

[hmiguim]

When DPBTK Developer detects a datatype that is not supported by the JDBC driver it logs the information about the datatype. One time during the process of converting the JDBC datatype into the internal DBPTK datatype.

dbptk-developer/dbptk-modules/dbptk-module-jdbc/src/main/java/com/databasepreservation/modules/jdbc/in/JDBCDatatypeImporter.java

Lines 468 to 473 in 6605b4b

	@Override
	protected Type getUnsupportedDataType(int dataType, String typeName, int columnSize, int decimalDigits,
	int numPrecRadix) throws UnknownTypeException {
	LOGGER.debug("Unsupported JDBC type, code: " + dataType);
	return new UnsupportedDataType(dataType, typeName, columnSize, decimalDigits, numPrecRadix);
	}

And another during the process of writting the metadata.xml file.

dbptk-developer/dbptk-modules/dbptk-module-siard/src/main/java/com/databasepreservation/modules/siard/out/content/Sql2008toXSDType.java

Lines 111 to 115 in 6605b4b

	} else if (type instanceof UnsupportedDataType) {
	reporter.savedAsString();
	LOGGER.debug("Found an unsupported datatype and saved it as xs:string: {}", type);
	ret = "xs:string";

[Laurira]

the dba sent me the report not log and in the end of log there is something more:

2021-11-30 15:52:35,272 [main] INFO (c.d.m.m.f.ProgressLoggerObserver) Completed table 'vastus' (17 of 17 tables completed)
2021-11-30 15:52:35,272 [main] INFO (c.d.m.m.f.ProgressLoggerObserver) Completed schema 'egt' (1 of 1 schemas completed)
2021-11-30 15:52:35,272 [main] DEBUG (c.d.m.j.i.JDBCImportModule) Freeing resources
2021-11-30 15:52:35,820 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10}
2021-11-30 15:52:35,820 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10}
2021-11-30 15:52:35,822 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10}
2021-11-30 15:52:35,826 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10}
2021-11-30 15:52:35,826 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10}
2021-11-30 15:52:35,829 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10}
2021-11-30 15:52:35,829 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10}
2021-11-30 15:52:35,830 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10}
2021-11-30 15:52:35,831 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10}
2021-11-30 15:52:35,832 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10}
2021-11-30 15:52:35,833 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10}
2021-11-30 15:52:35,833 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10}
2021-11-30 15:52:35,833 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10}
2021-11-30 15:52:35,834 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10}
2021-11-30 15:52:35,834 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10}
2021-11-30 15:52:35,835 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10}
2021-11-30 15:52:35,835 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10}
2021-11-30 15:52:35,835 [main] DEBUG (c.d.m.s.o.c.Sql2008toXSDType) Found an unsupported datatype and saved it as xs:string: Type{description='null', originalTypeName='geometry', sql99TypeName='VARCHAR(2147483647)', sql2008TypeName='VARCHAR(2147483647)'}-->UnsupportedDataType{dataType=1111, typeName='geometry', columnSize=2147483647, decimalDigits=0, numPrecRadix=10}
2021-11-30 15:52:36,059 [main] INFO (c.d.m.m.f.ProgressLoggerObserver) Completed database migration
2021-11-30 15:52:36,060 [main] DEBUG (c.d.m.j.i.JDBCImportModule) Closing connection to source database

Why does it print out this message 18 times?

[Laurira]

And it is weird that siard file were not created without any other error. We will try to re-run it on thursday and then give additional feedback.

[hmiguim]

Any news on this?

[Laurira]

Yes, they made a new attampt on Friday. There were slight changes in the yml and it seems that dbptk did not crash any more. But I have log as well and there are 66 weird SQLExceptions.

2022-02-27 19:16:34,720 [main] DEBUG (c.d.m.r.Reporter) dataTypeChangedOnImport, invoker: com.databasepreservation.modules.oracle.in.Oracle12cJDBCDatatypeImporter; message: - Type conversion in import module: in KOLA.HA_ISIKUD.ROA_TAGASI_NOUTUD (format: schema.table.column) has original type NUMBER and was converted to the standard type NUMERIC(10,2); and type: Type{description='null', originalTypeName='NUMBER', sql99TypeName='NUMERIC(10,2)', sql2008TypeName='NUMERIC(10,2)'}-->SimpleTypeNumericExact{precision=10, scale=2}
2022-02-27 19:16:34,720 [main] DEBUG (c.d.m.j.i.JDBCImportModule) Data type: 2
Type name: NUMBER
Column size: 10
Decimal digits: 2
Radix: 10
Remarks: null
Default value: null
Index: 28
Is Nullable: true
Is auto increment: false
Calculated type: SimpleTypeNumericExact
ColumnType hash: 1455568852

2022-02-27 19:16:34,798 [main] DEBUG (c.d.m.j.i.JDBCImportModule) handled SQLException
java.sql.SQLException: Invalid column name
at oracle.jdbc.driver.OracleStatement.getColumnIndexPrimitive(OracleStatement.java:3899)
at oracle.jdbc.driver.OracleStatement.getColumnIndex(OracleStatement.java:3857)
at oracle.jdbc.driver.InsensitiveScrollableResultSet.findColumn(InsensitiveScrollableResultSet.java:270)
at oracle.jdbc.driver.GeneratedResultSet.getString(GeneratedResultSet.java:596)
at com.databasepreservation.modules.jdbc.in.JDBCImportModule.getTriggers(JDBCImportModule.java:1582)
at com.databasepreservation.modules.jdbc.in.JDBCImportModule.getTableStructure(JDBCImportModule.java:886)
at com.databasepreservation.modules.oracle.in.Oracle12cJDBCImportModule.getTableStructure(Oracle12cJDBCImportModule.java:186)
at com.databasepreservation.modules.jdbc.in.JDBCImportModule.getTables(JDBCImportModule.java:674)
at com.databasepreservation.modules.jdbc.in.JDBCImportModule.getSchemaStructure(JDBCImportModule.java:511)
at com.databasepreservation.modules.oracle.in.Oracle12cJDBCImportModule.getSchemas(Oracle12cJDBCImportModule.java:156)
at com.databasepreservation.modules.jdbc.in.JDBCImportModule.getDatabaseStructure(JDBCImportModule.java:409)
at com.databasepreservation.modules.jdbc.in.JDBCImportModule.migrateDatabaseTo(JDBCImportModule.java:2075)
at com.databasepreservation.DatabaseMigration.migrate(DatabaseMigration.java:123)
at com.databasepreservation.Main.runMigration(Main.java:294)
at com.databasepreservation.Main.internalMain(Main.java:125)
at com.databasepreservation.Main.main(Main.java:81)
2022-02-27 19:16:35,126 [main] INFO (c.d.m.j.i.JDBCImportModule) Ignoring table KOLA.HA_ISIKUD_ASENDAMINE
2022-02-27 19:16:35,126 [main] INFO (c.d.m.j.i.JDBCImportModule) Obtaining table structure for KOLA.HA_LAHENDITE_SISUD
2022-02-27 19:16:35,141 [main] DEBUG (c.d.m.r.Reporter) dataTypeChangedOnImport, invoker: com.databasepreservation.modules.oracle.in.Oracle12cJDBCDatatypeImporter; message: - Type conversion in import module: in KOLA.HA_LAHENDITE_SISUD.KOOD (format: schema.table.column) has original type NUMBER and was converted to the standard type NUMERIC(10); and type: Type{description='null', originalTypeName='NUMBER', sql99TypeName='NUMERIC(10)', sql2008TypeName='NUMERIC(10)'}-->SimpleTypeNumericExact{precision=10, scale=0}
2022-02-27 19:16:35,141 [main] DEBUG (c.d.m.j.i.JDBCImportModule) Data type: 2
Type name: NUMBER
Column size: 10
Decimal digits: 0
Radix: 10
Remarks: null
Default value: null
Index: 1
Is Nullable: false
Is auto increment: false
Calculated type: SimpleTypeNumericExact
ColumnType hash: -1642140362

This column has letter "õ" missing from its description. It should be "Riigi õigusabi (RÕA)...":

ROA_TAGASI_NOUTUD
NUMERIC(10,2)
NUMBER
true
Riigi ?igusabi (R?A) tagasi n?utud summa, alates jaanuar 2005

[dosubot]

Hi, @Laurira. I'm Dosu, and I'm helping the keeps/dbptk-developer team manage their backlog and am marking this issue as stale.

Issue Summary:

• You reported that dbptk stops unexpectedly with repeated warnings about unsupported "geometry" datatype being saved as text.
• You asked how to enable more detailed logging to diagnose the issue.
• You shared logs showing repeated warnings and noted the SIARD file was not created.
• A retry with YAML changes avoided crashes but caused many SQLExceptions related to invalid column names and character encoding issues.
• The issue is still under investigation with ongoing attempts to resolve datatype and logging problems.

Next Steps:

• Please let me know if this issue is still relevant with the latest version of keeps/dbptk-developer by commenting here.
• If I don’t hear back within 7 days, I will automatically close this issue.

Thank you for your understanding and contribution!

[Laurira]

But the estonian letter õ is still not displayed correctly. Can you update your characterset or encoding so that letter "õ" is not displayed as questionmark.

Thanks

[dosubot]

@hmiguim The user Laurira has pointed out that the Estonian letter õ is still not displayed correctly and suggests updating the character set or encoding to fix this issue. Could you please assist with this?