DBTimes

Home

Products

Frameworks

Contact us

JSON Schema documentation

Source Comparer Configuration v1

Defines required fields for the application that uses DW Source Comparer library. The application configuration can have other root level fields.

Dialect
https://json-schema.org/draft/2020-12/schema

Root schema

object

Defines required fields for the application that uses DW Source Comparer library. The application configuration can have other root level fields.

Top-level properties

Properties declared directly by the root schema. Nested properties are documented within their parent property.

sourceCompare objectRequired

Configuration for source compare

JSON Pointer: /properties/sourceCompare

Nested schema details (4)

sourceCompareConfigVersion stringRequired

Version of a Source Comparer json configuration

JSON Pointer: /properties/sourceCompare/properties/sourceCompareConfigVersion

Additional schema keywords (1)
enum
["1"]

compareScenarios arrayRequired

Specifications for comparing sources

JSON Pointer: /properties/sourceCompare/properties/compareScenarios

Nested schema details (1)
Array items object

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items

Nested schema details (12)
name stringRequired

A name of the compare scenario to differentiate multiple scenarios. Names must be unique.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/name

isActive booleanRequired

If the value of isActive is false, the compare scenario will not be processed.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/isActive

isDebugDwLib booleanOptional

If the value is true, there will be some dataframe related output sent to the standard output from the library causing slowdown in job execution.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/isDebugDwLib

floatThreshold numberOptional

When comparing two floating point values, they will be considered equal if the absolute value of a difference is less than floatThreshold.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/floatThreshold

timestampTruncateUnit stringOptional

When comparing two timestamp fields they will be truncated using defined unit prior to being compared.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/timestampTruncateUnit

String constraints
pattern
"^(?i)(year|yyyy|yy|month|mon|mm|day|dd|microsecond|millisecond|second|minute|hour|week|quarter)$"
Additional schema keywords (1)
customMessage
{"pattern":"The allowed values for timestampTruncateUnit are year,yyyy,yy,month,mon,mm,day,dd,microsecond,millisecond,second,minute,hour,week and quarter."}
dateTruncateUnit stringOptional

When comparing two date fields they will be truncated using defined unit prior to being compared.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/dateTruncateUnit

String constraints
pattern
"^(?i)(year|yyyy|yy|month|mon|mm|week|quarter)$"
Additional schema keywords (1)
customMessage
{"pattern":"The allowed values for dateTruncateUnit are year,yyyy,yy,month,mon,mm,week and quarter."}
maxSameDifferences numberOptional

When defined, the number of reported differences for each column will not exceed the specified value. That will reduce the size of resulting differences file.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/maxSameDifferences

compareResult objectRequired

Specification for the result of the comparison

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult

Nested schema details (4)
file objectRequired

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file

Nested schema details (4)
dir stringRequired

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/properties/dir

csv objectRequired

Properties for csv file. By default the saved file is multiline

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/properties/csv

Nested schema details (9)
mode stringOptional

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/properties/csv/properties/mode

header booleanOptional

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/properties/csv/properties/header

encoding stringOptional

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/properties/csv/properties/encoding

dateFormat stringOptional

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/properties/csv/properties/dateFormat

nullValue stringOptional

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/properties/csv/properties/nullValue

escape stringOptional

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/properties/csv/properties/escape

quote stringOptional

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/properties/csv/properties/quote

delimiter stringOptional

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/properties/csv/properties/delimiter

Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/properties/csv/unevaluatedProperties

^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/patternProperties/^__

Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/unevaluatedProperties

schema arrayRequired

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/schema

Array constraints
minItems
1
Additional schema keywords (1)
customMessage
{"minItems":"Compare result's schema does not have any columns defined. It must have at least one column"}
Nested schema details (2)
Array items object

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/schema/items

Nested schema details (5)
colName stringRequired

Column name

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/schema/items/properties/colName

isUniqueKey booleanOptional

At least one column must have this property set to true.
Multiple columns can be designated as unique key to create a compound unique key.
By default the unique key is included unless isInclude value is set to false

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/schema/items/properties/isUniqueKey

isInclude booleanOptional

By default all columns are compared and included in comparison result, unless at least one column has explicit isInclude flag set (to true or false).
If that's the case only columns with isInclude flag set to true will be compared and included into comparison result.
If columns are listed that do not have isInclude flag, they will be ignored, unless they are keys.
For keys (isUniqueKey is set to true) the default for isInclude is true

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/schema/items/properties/isInclude

^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/schema/items/patternProperties/^__

Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/schema/items/unevaluatedProperties

allOf option 1 Any type

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/schema/allOf/0

Array constraints
minContains
1
Additional schema keywords (1)
customMessage
{"contains":"Compare result's schema does not have unique key defined. At least one column must have \\"isUniqueKey\\" flag set to true, i.e., ... \\"isUniqueKey\\" : true, ... "}
Nested schema details (1)
Contains object

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/schema/allOf/0/contains

Nested schema details (1)
isUniqueKey Any typeRequired

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/schema/allOf/0/contains/properties/isUniqueKey

Additional schema keywords (1)
const
true
^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/patternProperties/^__

Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/unevaluatedProperties

leftSource Any typeRequired

First or left source to compare. Both left and right sections are identical in terms of structure, but they can have different types of sources. For example, one can be DBMS and another a file. The sets being compared must have common columns matched compareResult.schema.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/leftSource

References
rightSource Any typeRequired

Second or right source to compare. Both left and right sections are identical in terms of structure, but they can have different types of sources. For example, one can be DBMS and another a file. The sets being compared must have common columns matched compareResult.schema.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/rightSource

References
^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/patternProperties/^__

Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/unevaluatedProperties

^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /properties/sourceCompare/patternProperties/^__

Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /properties/sourceCompare/unevaluatedProperties

sourceCompare objectRequired

Configuration for source compare

JSON Pointer: /properties/sourceCompare

Nested schema details (4)

sourceCompareConfigVersion stringRequired

Version of a Source Comparer json configuration

JSON Pointer: /properties/sourceCompare/properties/sourceCompareConfigVersion

Additional schema keywords (1)
enum
["1"]

compareScenarios arrayRequired

Specifications for comparing sources

JSON Pointer: /properties/sourceCompare/properties/compareScenarios

Nested schema details (1)
Array items object

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items

Nested schema details (12)
name stringRequired

A name of the compare scenario to differentiate multiple scenarios. Names must be unique.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/name

isActive booleanRequired

If the value of isActive is false, the compare scenario will not be processed.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/isActive

isDebugDwLib booleanOptional

If the value is true, there will be some dataframe related output sent to the standard output from the library causing slowdown in job execution.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/isDebugDwLib

floatThreshold numberOptional

When comparing two floating point values, they will be considered equal if the absolute value of a difference is less than floatThreshold.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/floatThreshold

timestampTruncateUnit stringOptional

When comparing two timestamp fields they will be truncated using defined unit prior to being compared.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/timestampTruncateUnit

String constraints
pattern
"^(?i)(year|yyyy|yy|month|mon|mm|day|dd|microsecond|millisecond|second|minute|hour|week|quarter)$"
Additional schema keywords (1)
customMessage
{"pattern":"The allowed values for timestampTruncateUnit are year,yyyy,yy,month,mon,mm,day,dd,microsecond,millisecond,second,minute,hour,week and quarter."}
dateTruncateUnit stringOptional

When comparing two date fields they will be truncated using defined unit prior to being compared.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/dateTruncateUnit

String constraints
pattern
"^(?i)(year|yyyy|yy|month|mon|mm|week|quarter)$"
Additional schema keywords (1)
customMessage
{"pattern":"The allowed values for dateTruncateUnit are year,yyyy,yy,month,mon,mm,week and quarter."}
maxSameDifferences numberOptional

When defined, the number of reported differences for each column will not exceed the specified value. That will reduce the size of resulting differences file.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/maxSameDifferences

compareResult objectRequired

Specification for the result of the comparison

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult

Nested schema details (4)
file objectRequired

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file

Nested schema details (4)
dir stringRequired

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/properties/dir

csv objectRequired

Properties for csv file. By default the saved file is multiline

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/properties/csv

Nested schema details (9)
mode stringOptional

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/properties/csv/properties/mode

header booleanOptional

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/properties/csv/properties/header

encoding stringOptional

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/properties/csv/properties/encoding

dateFormat stringOptional

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/properties/csv/properties/dateFormat

nullValue stringOptional

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/properties/csv/properties/nullValue

escape stringOptional

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/properties/csv/properties/escape

quote stringOptional

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/properties/csv/properties/quote

delimiter stringOptional

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/properties/csv/properties/delimiter

Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/properties/csv/unevaluatedProperties

^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/patternProperties/^__

Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/file/unevaluatedProperties

schema arrayRequired

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/schema

Array constraints
minItems
1
Additional schema keywords (1)
customMessage
{"minItems":"Compare result's schema does not have any columns defined. It must have at least one column"}
Nested schema details (2)
Array items object

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/schema/items

Nested schema details (5)
colName stringRequired

Column name

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/schema/items/properties/colName

isUniqueKey booleanOptional

At least one column must have this property set to true.
Multiple columns can be designated as unique key to create a compound unique key.
By default the unique key is included unless isInclude value is set to false

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/schema/items/properties/isUniqueKey

isInclude booleanOptional

By default all columns are compared and included in comparison result, unless at least one column has explicit isInclude flag set (to true or false).
If that's the case only columns with isInclude flag set to true will be compared and included into comparison result.
If columns are listed that do not have isInclude flag, they will be ignored, unless they are keys.
For keys (isUniqueKey is set to true) the default for isInclude is true

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/schema/items/properties/isInclude

^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/schema/items/patternProperties/^__

Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/schema/items/unevaluatedProperties

allOf option 1 Any type

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/schema/allOf/0

Array constraints
minContains
1
Additional schema keywords (1)
customMessage
{"contains":"Compare result's schema does not have unique key defined. At least one column must have \\"isUniqueKey\\" flag set to true, i.e., ... \\"isUniqueKey\\" : true, ... "}
Nested schema details (1)
Contains object

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/schema/allOf/0/contains

Nested schema details (1)
isUniqueKey Any typeRequired

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/properties/schema/allOf/0/contains/properties/isUniqueKey

Additional schema keywords (1)
const
true
^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/patternProperties/^__

Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/compareResult/unevaluatedProperties

leftSource Any typeRequired

First or left source to compare. Both left and right sections are identical in terms of structure, but they can have different types of sources. For example, one can be DBMS and another a file. The sets being compared must have common columns matched compareResult.schema.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/leftSource

References
rightSource Any typeRequired

Second or right source to compare. Both left and right sections are identical in terms of structure, but they can have different types of sources. For example, one can be DBMS and another a file. The sets being compared must have common columns matched compareResult.schema.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/properties/rightSource

References
^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/patternProperties/^__

Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /properties/sourceCompare/properties/compareScenarios/items/unevaluatedProperties

^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /properties/sourceCompare/patternProperties/^__

Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /properties/sourceCompare/unevaluatedProperties

Definitions

jdbcOptions object

These options are applicable to all JDBC databases - currently sql server and oracle. See https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html for details.

JSON Pointer: /$defs/jdbcOptions

Additional schema keywords (1)
customMessage
{"not":"The field \\"partitionColumn\\" is not allowed within dbms object as it must be defined in the \\"schema\\" with \\"isPartitionColumn\\" flag set to true (that is because the tool needs column type for internal logic)."}
Nested schema details (23)

url stringRequired

JSON Pointer: /$defs/jdbcOptions/properties/url

dbtable Any typeOptional

JSON Pointer: /$defs/jdbcOptions/properties/dbtable

Nested schema details (2)
anyOf option 1 string

JSON Pointer: /$defs/jdbcOptions/properties/dbtable/anyOf/0

anyOf option 2 array

JSON Pointer: /$defs/jdbcOptions/properties/dbtable/anyOf/1

Nested schema details (1)
Array items string

JSON Pointer: /$defs/jdbcOptions/properties/dbtable/anyOf/1/items

query Any typeOptional

JSON Pointer: /$defs/jdbcOptions/properties/query

Nested schema details (2)
anyOf option 1 string

JSON Pointer: /$defs/jdbcOptions/properties/query/anyOf/0

anyOf option 2 array

JSON Pointer: /$defs/jdbcOptions/properties/query/anyOf/1

Nested schema details (1)
Array items string

JSON Pointer: /$defs/jdbcOptions/properties/query/anyOf/1/items

prepareQuery Any typeOptional

JSON Pointer: /$defs/jdbcOptions/properties/prepareQuery

Nested schema details (2)
anyOf option 1 string

JSON Pointer: /$defs/jdbcOptions/properties/prepareQuery/anyOf/0

anyOf option 2 array

JSON Pointer: /$defs/jdbcOptions/properties/prepareQuery/anyOf/1

Nested schema details (1)
Array items string

JSON Pointer: /$defs/jdbcOptions/properties/prepareQuery/anyOf/1/items

lowerBound stringOptional

JSON Pointer: /$defs/jdbcOptions/properties/lowerBound

upperBound stringOptional

JSON Pointer: /$defs/jdbcOptions/properties/upperBound

numPartitions numberOptional

JSON Pointer: /$defs/jdbcOptions/properties/numPartitions

queryTimeout numberOptional

JSON Pointer: /$defs/jdbcOptions/properties/queryTimeout

fetchsize numberOptional

JSON Pointer: /$defs/jdbcOptions/properties/fetchsize

sessionInitStatement stringOptional

JSON Pointer: /$defs/jdbcOptions/properties/sessionInitStatement

pushDownPredicate booleanOptional

JSON Pointer: /$defs/jdbcOptions/properties/pushDownPredicate

pushDownAggregate booleanOptional

JSON Pointer: /$defs/jdbcOptions/properties/pushDownAggregate

pushDownLimit booleanOptional

JSON Pointer: /$defs/jdbcOptions/properties/pushDownLimit

pushDownOffset booleanOptional

JSON Pointer: /$defs/jdbcOptions/properties/pushDownOffset

pushDownTableSample booleanOptional

JSON Pointer: /$defs/jdbcOptions/properties/pushDownTableSample

keytab stringOptional

JSON Pointer: /$defs/jdbcOptions/properties/keytab

principal stringOptional

JSON Pointer: /$defs/jdbcOptions/properties/principal

refreshKrb5Config booleanOptional

JSON Pointer: /$defs/jdbcOptions/properties/refreshKrb5Config

preferTimestampNTZ booleanOptional

JSON Pointer: /$defs/jdbcOptions/properties/preferTimestampNTZ

hint stringOptional

JSON Pointer: /$defs/jdbcOptions/properties/hint

Not Any type

"partitionColumn" is excluded from the list of options for jdbc source since this column has to be defined in "schema" with "isPartitionColumn" flag

JSON Pointer: /$defs/jdbcOptions/not

^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /$defs/jdbcOptions/patternProperties/^__

Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /$defs/jdbcOptions/unevaluatedProperties

compareSource object

This definition is used for both - left and right - compare sources

JSON Pointer: /$defs/compareSource

Nested schema details (7)

oneOf option 1 object

JSON Pointer: /$defs/compareSource/oneOf/0

Nested schema details (1)
file objectRequired

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file

Nested schema details (5)
oneOf option 1 object

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/0

Nested schema details (1)
parquet objectRequired

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/0/properties/parquet

oneOf option 2 object

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1

Nested schema details (1)
csv objectRequired

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv

Nested schema details (13)
mode stringOptional

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/properties/mode

header booleanOptional

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/properties/header

encoding stringOptional

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/properties/encoding

dateFormat stringOptional

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/properties/dateFormat

nullValue stringOptional

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/properties/nullValue

escape stringOptional

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/properties/escape

quote stringOptional

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/properties/quote

delimiter stringOptional

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/properties/delimiter

multiLine booleanOptional

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/properties/multiLine

inferSchema booleanOptional

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/properties/inferSchema

trueValue stringOptional

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/properties/trueValue

falseValue stringOptional

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/properties/falseValue

Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/unevaluatedProperties

path stringRequired

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/properties/path

^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/patternProperties/^__

Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/unevaluatedProperties

oneOf option 2 object

JSON Pointer: /$defs/compareSource/oneOf/1

Nested schema details (1)
dbms objectRequired

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms

Nested schema details (7)
oneOf option 1 object

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/0

Nested schema details (2)
sqlServer Any typeRequired

Options for SQL Server jdbc connection.

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/0/properties/sqlServer

References
^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/0/patternProperties/^__

oneOf option 2 object

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/1

Nested schema details (2)
oracle Any typeRequired

Options for Oracle jdbc connection.

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/1/properties/oracle

References
^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/1/patternProperties/^__

oneOf option 3 object

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2

Nested schema details (2)
mongoDb objectRequired

Properties for mongoDb can be any of the options supported by MongoDb Spark connector in batch mode. Schema must be defined

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb

Nested schema details (17)
connection.uri stringRequired

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/connection.uri

database stringOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/database

collection stringRequired

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/collection

comment stringOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/comment

mode stringOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/mode

columnNameOfCorruptRecord stringOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/columnNameOfCorruptRecord

mongoClientFactory stringOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/mongoClientFactory

partitioner stringOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/partitioner

sampleSize numberOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/sampleSize

sql.inferSchema.mapTypes.enabled booleanOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/sql.inferSchema.mapTypes.enabled

sql.inferSchema.mapTypes.minimum.key.size numberOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/sql.inferSchema.mapTypes.minimum.key.size

aggregation.pipeline Any typeOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/aggregation.pipeline

Nested schema details (2)
anyOf option 1 string

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/aggregation.pipeline/anyOf/0

anyOf option 2 array

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/aggregation.pipeline/anyOf/1

Nested schema details (1)
Array items string

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/aggregation.pipeline/anyOf/1/items

aggregation.allowDiskUse booleanOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/aggregation.allowDiskUse

outputExtendedJson booleanOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/outputExtendedJson

^partitioner\\.options\\. string

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/patternProperties/^partitioner\\.options\\.

^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/patternProperties/^__

Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/unevaluatedProperties

^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/patternProperties/^__

user stringOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/properties/user

password stringOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/properties/password

^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/patternProperties/^__

Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/unevaluatedProperties

moniker stringRequired

A meaningful name of the source to show in compare results. If none defined – the default is “left” or “right”

JSON Pointer: /$defs/compareSource/properties/moniker

schema arrayOptional

JSON Pointer: /$defs/compareSource/properties/schema

Nested schema details (1)
Array items object

JSON Pointer: /$defs/compareSource/properties/schema/items

Nested schema details (6)
colName stringRequired

Column nmae

JSON Pointer: /$defs/compareSource/properties/schema/items/properties/colName

colType stringOptional

Column type

JSON Pointer: /$defs/compareSource/properties/schema/items/properties/colType

isPartitionColumn booleanOptional

For DBMS source one column can be defined as partition column to read data on multiple connections in parallel. isPartitionColumn must be defined if numPartitions is defined.

JSON Pointer: /$defs/compareSource/properties/schema/items/properties/isPartitionColumn

^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /$defs/compareSource/properties/schema/items/patternProperties/^__

allOf option 1 Any type

JSON Pointer: /$defs/compareSource/properties/schema/items/allOf/0

Nested schema details (2)
If Any type

JSON Pointer: /$defs/compareSource/properties/schema/items/allOf/0/if

Nested schema details (1)
isPartitionColumn Any typeRequired

JSON Pointer: /$defs/compareSource/properties/schema/items/allOf/0/if/properties/isPartitionColumn

Additional schema keywords (1)
const
true
Then Any type

If "isPartitionColumn" attribute of a column is set to true, colType must be defined (needed to determine the type of upper and lower bounds.

JSON Pointer: /$defs/compareSource/properties/schema/items/allOf/0/then

Additional schema keywords (1)
customMessage
{"required":"\\"colType\\" is required when \\"isPartitionColumn\\" is true."}
Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /$defs/compareSource/properties/schema/items/unevaluatedProperties

subsetQuery Any typeOptional

Query to create a subset of columns and/or rows to compare. It can be any valid select statement using Spark SQL syntax to be executed on a data loaded from the Source. Source table must be desinated by three underscores, e.g., SELECT CAST(id AS INT) AS id FROM ___ .If not defined all fields and all rows from the source will be used

JSON Pointer: /$defs/compareSource/properties/subsetQuery

Nested schema details (2)
anyOf option 1 string

JSON Pointer: /$defs/compareSource/properties/subsetQuery/anyOf/0

String constraints
pattern
"^.*___.*$"
Additional schema keywords (1)
customMessage
{"type":"\\"subsetQuery\\" must be a string or an array of strings (this message may be displayed even if type is string or array, when there are other errors - ignore it in that case).","pattern":"The query must have a three underscores ( ___ ) placeholder for a source table."}
anyOf option 2 array

JSON Pointer: /$defs/compareSource/properties/subsetQuery/anyOf/1

Additional schema keywords (1)
customMessage
{"type":"\\"subsetQuery\\" must be a string or an array of strings (this message may be displayed even if type is string or array, when there are other errors - ignore it in that case).","contains":"At least one element in the array must have a three underscores ( ___ ) placeholder for a source table."}
Nested schema details (2)
Array items string

JSON Pointer: /$defs/compareSource/properties/subsetQuery/anyOf/1/items

Contains Any type

JSON Pointer: /$defs/compareSource/properties/subsetQuery/anyOf/1/contains

String constraints
pattern
"^.*___.*$"

^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /$defs/compareSource/patternProperties/^__

Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /$defs/compareSource/unevaluatedProperties

jdbcOptions object

These options are applicable to all JDBC databases - currently sql server and oracle. See https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html for details.

JSON Pointer: /$defs/jdbcOptions

Additional schema keywords (1)
customMessage
{"not":"The field \\"partitionColumn\\" is not allowed within dbms object as it must be defined in the \\"schema\\" with \\"isPartitionColumn\\" flag set to true (that is because the tool needs column type for internal logic)."}
Nested schema details (23)

url stringRequired

JSON Pointer: /$defs/jdbcOptions/properties/url

dbtable Any typeOptional

JSON Pointer: /$defs/jdbcOptions/properties/dbtable

Nested schema details (2)
anyOf option 1 string

JSON Pointer: /$defs/jdbcOptions/properties/dbtable/anyOf/0

anyOf option 2 array

JSON Pointer: /$defs/jdbcOptions/properties/dbtable/anyOf/1

Nested schema details (1)
Array items string

JSON Pointer: /$defs/jdbcOptions/properties/dbtable/anyOf/1/items

query Any typeOptional

JSON Pointer: /$defs/jdbcOptions/properties/query

Nested schema details (2)
anyOf option 1 string

JSON Pointer: /$defs/jdbcOptions/properties/query/anyOf/0

anyOf option 2 array

JSON Pointer: /$defs/jdbcOptions/properties/query/anyOf/1

Nested schema details (1)
Array items string

JSON Pointer: /$defs/jdbcOptions/properties/query/anyOf/1/items

prepareQuery Any typeOptional

JSON Pointer: /$defs/jdbcOptions/properties/prepareQuery

Nested schema details (2)
anyOf option 1 string

JSON Pointer: /$defs/jdbcOptions/properties/prepareQuery/anyOf/0

anyOf option 2 array

JSON Pointer: /$defs/jdbcOptions/properties/prepareQuery/anyOf/1

Nested schema details (1)
Array items string

JSON Pointer: /$defs/jdbcOptions/properties/prepareQuery/anyOf/1/items

lowerBound stringOptional

JSON Pointer: /$defs/jdbcOptions/properties/lowerBound

upperBound stringOptional

JSON Pointer: /$defs/jdbcOptions/properties/upperBound

numPartitions numberOptional

JSON Pointer: /$defs/jdbcOptions/properties/numPartitions

queryTimeout numberOptional

JSON Pointer: /$defs/jdbcOptions/properties/queryTimeout

fetchsize numberOptional

JSON Pointer: /$defs/jdbcOptions/properties/fetchsize

sessionInitStatement stringOptional

JSON Pointer: /$defs/jdbcOptions/properties/sessionInitStatement

pushDownPredicate booleanOptional

JSON Pointer: /$defs/jdbcOptions/properties/pushDownPredicate

pushDownAggregate booleanOptional

JSON Pointer: /$defs/jdbcOptions/properties/pushDownAggregate

pushDownLimit booleanOptional

JSON Pointer: /$defs/jdbcOptions/properties/pushDownLimit

pushDownOffset booleanOptional

JSON Pointer: /$defs/jdbcOptions/properties/pushDownOffset

pushDownTableSample booleanOptional

JSON Pointer: /$defs/jdbcOptions/properties/pushDownTableSample

keytab stringOptional

JSON Pointer: /$defs/jdbcOptions/properties/keytab

principal stringOptional

JSON Pointer: /$defs/jdbcOptions/properties/principal

refreshKrb5Config booleanOptional

JSON Pointer: /$defs/jdbcOptions/properties/refreshKrb5Config

preferTimestampNTZ booleanOptional

JSON Pointer: /$defs/jdbcOptions/properties/preferTimestampNTZ

hint stringOptional

JSON Pointer: /$defs/jdbcOptions/properties/hint

Not Any type

"partitionColumn" is excluded from the list of options for jdbc source since this column has to be defined in "schema" with "isPartitionColumn" flag

JSON Pointer: /$defs/jdbcOptions/not

^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /$defs/jdbcOptions/patternProperties/^__

Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /$defs/jdbcOptions/unevaluatedProperties

compareSource object

This definition is used for both - left and right - compare sources

JSON Pointer: /$defs/compareSource

Nested schema details (7)

oneOf option 1 object

JSON Pointer: /$defs/compareSource/oneOf/0

Nested schema details (1)
file objectRequired

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file

Nested schema details (5)
oneOf option 1 object

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/0

Nested schema details (1)
parquet objectRequired

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/0/properties/parquet

oneOf option 2 object

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1

Nested schema details (1)
csv objectRequired

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv

Nested schema details (13)
mode stringOptional

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/properties/mode

header booleanOptional

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/properties/header

encoding stringOptional

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/properties/encoding

dateFormat stringOptional

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/properties/dateFormat

nullValue stringOptional

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/properties/nullValue

escape stringOptional

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/properties/escape

quote stringOptional

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/properties/quote

delimiter stringOptional

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/properties/delimiter

multiLine booleanOptional

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/properties/multiLine

inferSchema booleanOptional

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/properties/inferSchema

trueValue stringOptional

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/properties/trueValue

falseValue stringOptional

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/properties/falseValue

Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/oneOf/1/properties/csv/unevaluatedProperties

path stringRequired

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/properties/path

^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/patternProperties/^__

Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /$defs/compareSource/oneOf/0/properties/file/unevaluatedProperties

oneOf option 2 object

JSON Pointer: /$defs/compareSource/oneOf/1

Nested schema details (1)
dbms objectRequired

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms

Nested schema details (7)
oneOf option 1 object

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/0

Nested schema details (2)
sqlServer Any typeRequired

Options for SQL Server jdbc connection.

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/0/properties/sqlServer

References
^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/0/patternProperties/^__

oneOf option 2 object

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/1

Nested schema details (2)
oracle Any typeRequired

Options for Oracle jdbc connection.

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/1/properties/oracle

References
^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/1/patternProperties/^__

oneOf option 3 object

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2

Nested schema details (2)
mongoDb objectRequired

Properties for mongoDb can be any of the options supported by MongoDb Spark connector in batch mode. Schema must be defined

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb

Nested schema details (17)
connection.uri stringRequired

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/connection.uri

database stringOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/database

collection stringRequired

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/collection

comment stringOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/comment

mode stringOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/mode

columnNameOfCorruptRecord stringOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/columnNameOfCorruptRecord

mongoClientFactory stringOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/mongoClientFactory

partitioner stringOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/partitioner

sampleSize numberOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/sampleSize

sql.inferSchema.mapTypes.enabled booleanOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/sql.inferSchema.mapTypes.enabled

sql.inferSchema.mapTypes.minimum.key.size numberOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/sql.inferSchema.mapTypes.minimum.key.size

aggregation.pipeline Any typeOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/aggregation.pipeline

Nested schema details (2)
anyOf option 1 string

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/aggregation.pipeline/anyOf/0

anyOf option 2 array

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/aggregation.pipeline/anyOf/1

Nested schema details (1)
Array items string

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/aggregation.pipeline/anyOf/1/items

aggregation.allowDiskUse booleanOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/aggregation.allowDiskUse

outputExtendedJson booleanOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/properties/outputExtendedJson

^partitioner\\.options\\. string

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/patternProperties/^partitioner\\.options\\.

^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/patternProperties/^__

Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/properties/mongoDb/unevaluatedProperties

^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/oneOf/2/patternProperties/^__

user stringOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/properties/user

password stringOptional

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/properties/password

^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/patternProperties/^__

Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /$defs/compareSource/oneOf/1/properties/dbms/unevaluatedProperties

moniker stringRequired

A meaningful name of the source to show in compare results. If none defined – the default is “left” or “right”

JSON Pointer: /$defs/compareSource/properties/moniker

schema arrayOptional

JSON Pointer: /$defs/compareSource/properties/schema

Nested schema details (1)
Array items object

JSON Pointer: /$defs/compareSource/properties/schema/items

Nested schema details (6)
colName stringRequired

Column nmae

JSON Pointer: /$defs/compareSource/properties/schema/items/properties/colName

colType stringOptional

Column type

JSON Pointer: /$defs/compareSource/properties/schema/items/properties/colType

isPartitionColumn booleanOptional

For DBMS source one column can be defined as partition column to read data on multiple connections in parallel. isPartitionColumn must be defined if numPartitions is defined.

JSON Pointer: /$defs/compareSource/properties/schema/items/properties/isPartitionColumn

^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /$defs/compareSource/properties/schema/items/patternProperties/^__

allOf option 1 Any type

JSON Pointer: /$defs/compareSource/properties/schema/items/allOf/0

Nested schema details (2)
If Any type

JSON Pointer: /$defs/compareSource/properties/schema/items/allOf/0/if

Nested schema details (1)
isPartitionColumn Any typeRequired

JSON Pointer: /$defs/compareSource/properties/schema/items/allOf/0/if/properties/isPartitionColumn

Additional schema keywords (1)
const
true
Then Any type

If "isPartitionColumn" attribute of a column is set to true, colType must be defined (needed to determine the type of upper and lower bounds.

JSON Pointer: /$defs/compareSource/properties/schema/items/allOf/0/then

Additional schema keywords (1)
customMessage
{"required":"\\"colType\\" is required when \\"isPartitionColumn\\" is true."}
Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /$defs/compareSource/properties/schema/items/unevaluatedProperties

subsetQuery Any typeOptional

Query to create a subset of columns and/or rows to compare. It can be any valid select statement using Spark SQL syntax to be executed on a data loaded from the Source. Source table must be desinated by three underscores, e.g., SELECT CAST(id AS INT) AS id FROM ___ .If not defined all fields and all rows from the source will be used

JSON Pointer: /$defs/compareSource/properties/subsetQuery

Nested schema details (2)
anyOf option 1 string

JSON Pointer: /$defs/compareSource/properties/subsetQuery/anyOf/0

String constraints
pattern
"^.*___.*$"
Additional schema keywords (1)
customMessage
{"type":"\\"subsetQuery\\" must be a string or an array of strings (this message may be displayed even if type is string or array, when there are other errors - ignore it in that case).","pattern":"The query must have a three underscores ( ___ ) placeholder for a source table."}
anyOf option 2 array

JSON Pointer: /$defs/compareSource/properties/subsetQuery/anyOf/1

Additional schema keywords (1)
customMessage
{"type":"\\"subsetQuery\\" must be a string or an array of strings (this message may be displayed even if type is string or array, when there are other errors - ignore it in that case).","contains":"At least one element in the array must have a three underscores ( ___ ) placeholder for a source table."}
Nested schema details (2)
Array items string

JSON Pointer: /$defs/compareSource/properties/subsetQuery/anyOf/1/items

Contains Any type

JSON Pointer: /$defs/compareSource/properties/subsetQuery/anyOf/1/contains

String constraints
pattern
"^.*___.*$"

^__ objectarraystringnumberintegerbooleannull

Comment field begins with two underscores. There can be any number of comment fields.

JSON Pointer: /$defs/compareSource/patternProperties/^__

Unevaluated properties

Boolean schema: false - no instance value is accepted at this location.

JSON Pointer: /$defs/compareSource/unevaluatedProperties

Schema structure

Unevaluated properties

Boolean schema: true - all instance values are accepted at this location.

JSON Pointer: /unevaluatedProperties

Unevaluated properties

Boolean schema: true - all instance values are accepted at this location.

JSON Pointer: /unevaluatedProperties

Diagram

This structural view complements the reader-focused documentation above.

Use the scrollable region to explore the complete schema structure. Wide diagrams stay inside this panel instead of widening the page.


Multiple columns can be designated as unique key to create a compound unique key.
By default the unique key is included unless isInclude value is set to false
 isInclude object array string integer number boolean null description By default all columns are compared and included in comparison result, unless at least one column has explicit isInclude flag set (to true or false).
If that's the case only columns with isInclude flag set to true will be compared and included into comparison result.
If columns are listed that do not have isInclude flag, they will be ignored, unless they are keys.
For keys (isUniqueKey is set to true) the default for isInclude is true
 patternProperties ^__ object array string integer number boolean null description Comment field begins with two underscores. There can be any number of comment fields. required colName unevaluatedProperties false allOf 1 { } object array string integer number boolean null minContains 1 contains object array string integer number boolean null required isUniqueKey properties 1 isUniqueKey required object array string integer number boolean null const value true customMessage "contains": Compare result's schema does not have unique key defined. At least one column must have \"isUniqueKey\" flag set to true, i.e., ... \"isUniqueKey\" : true, ... customMessage "minItems": Compare result's schema does not have any columns defined. It must have at least one column patternProperties ^__ object array string integer number boolean null description Comment field begins with two underscores. There can be any number of comment fields. required file schema unevaluatedProperties false leftSource required object array string integer number boolean null $ref #/$defs/compareSource description First or left source to compare. Both left and right sections are identical in terms of structure, but they can have different types of sources. For example, one can be DBMS and another a file. The sets being compared must have common columns matched compareResult.schema. rightSource required object array string integer number boolean null $ref #/$defs/compareSource description Second or right source to compare. Both left and right sections are identical in terms of structure, but they can have different types of sources. For example, one can be DBMS and another a file. The sets being compared must have common columns matched compareResult.schema. patternProperties ^__ object array string integer number boolean null description Comment field begins with two underscores. There can be any number of comment fields. required name isActive compareResult leftSource rightSource unevaluatedProperties false patternProperties ^__ object array string integer number boolean null description Comment field begins with two underscores. There can be any number of comment fields. required sourceCompareConfigVersion compareScenarios unevaluatedProperties false required sourceCompare unevaluatedProperties true

Named-property index

Back to documentation