sourceLoad objectRequired
Configuration for Source Loader
JSON Pointer: /properties/sourceLoad
Nested schema details (4)
sourceLoadConfigVersion stringRequired
Version of a Source Loader json configuration
JSON Pointer: /properties/sourceLoad/properties/sourceLoadConfigVersion
Additional schema keywords (1)
enum["1"]
loadActions arrayRequired
Specifications for loading files or database tables into staging files. A load action is used to load one database table or one file. A single load action can also be used to load multiple files if their names follow the same pattern with a date being a variable part
JSON Pointer: /properties/sourceLoad/properties/loadActions
Nested schema details (1)
Array items object
JSON Pointer: /properties/sourceLoad/properties/loadActions/items
Nested schema details (13)
oneOf option 1 object
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0
Nested schema details (2)
fileSource objectRequired
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource
Nested schema details (9)
oneOf option 1 object
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/oneOf/0
Nested schema details (1)
fileSystemLocation objectRequired
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/oneOf/0/properties/fileSystemLocation
Nested schema details (4)
dir stringRequired
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/oneOf/0/properties/fileSystemLocation/properties/dir
namePattern stringRequired
A regular expression of file names to process. The files can have effective date as part of the name. In that case the files are processed in a date ascending order.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/oneOf/0/properties/fileSystemLocation/properties/namePattern
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/oneOf/0/properties/fileSystemLocation/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/oneOf/0/properties/fileSystemLocation/unevaluatedProperties
oneOf option 2 object
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/oneOf/1
Nested schema details (1)
internetLocation objectRequired
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/oneOf/1/properties/internetLocation
Nested schema details (6)
url stringRequired
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/oneOf/1/properties/internetLocation/properties/url
user stringOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/oneOf/1/properties/internetLocation/properties/user
password stringOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/oneOf/1/properties/internetLocation/properties/password
isDisableSslVerification booleanOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/oneOf/1/properties/internetLocation/properties/isDisableSslVerification
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/oneOf/1/properties/internetLocation/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/oneOf/1/properties/internetLocation/unevaluatedProperties
oneOf option 3 object
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/oneOf/2
Nested schema details (1)
customLocation objectRequired
This attribute must be defined when a user function is used to provide a path of the file to be loaded. The user function must be implemented in Scala object or Java static method. The method must take one parameter of type Option[Config]and returns a String with file path or file content (depending on "methodReturns" value - "FilePath" or "FileData”, e.g.,
def sampleForCsvFileCustomLocation( params: Option[Config] ): String = {
s"/some-path/csv-file.csv"
}
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/oneOf/2/properties/customLocation
Nested schema details (7)
packageName stringOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/oneOf/2/properties/customLocation/properties/packageName
moduleName stringRequired
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/oneOf/2/properties/customLocation/properties/moduleName
methodName stringRequired
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/oneOf/2/properties/customLocation/properties/methodName
methodReturns stringRequired
The type of data that the method returns: FilePath - the path of the file, FileData - the string with the file content
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/oneOf/2/properties/customLocation/properties/methodReturns
Additional schema keywords (1)
enum["FilePath","FileData"]
applicationSpecific objectOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/oneOf/2/properties/customLocation/properties/applicationSpecific
Nested schema details (1)
Unevaluated properties
Boolean schema: true - all instance values are accepted at this location.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/oneOf/2/properties/customLocation/properties/applicationSpecific/unevaluatedProperties
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/oneOf/2/properties/customLocation/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/oneOf/2/properties/customLocation/unevaluatedProperties
isRemoveDuplicateRows booleanOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/properties/isRemoveDuplicateRows
effectiveDate stringOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/properties/effectiveDate
String constraints
format"date"
csv objectRequired
Specifies the file being loaded is .csv. This object can optionally specify any valid Spark read .csv file options used for loading csv file.
Example of a configuration with no options,
"csv" : { }
Example of a configuration with some options,
"csv" :
{
"mode" : "PERMISSIVE",
"header" : true,
"encoding" : "ISO-8859-1"
}
Properties for csv file can be any of the options for Spark csv file processing. "schema" array must be defined if inferSchema is not set to true
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/properties/csv
Nested schema details (13)
mode stringOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/properties/csv/properties/mode
header booleanOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/properties/csv/properties/header
encoding stringOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/properties/csv/properties/encoding
dateFormat stringOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/properties/csv/properties/dateFormat
nullValue stringOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/properties/csv/properties/nullValue
escape stringOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/properties/csv/properties/escape
quote stringOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/properties/csv/properties/quote
delimiter stringOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/properties/csv/properties/delimiter
multiLine booleanOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/properties/csv/properties/multiLine
inferSchema booleanOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/properties/csv/properties/inferSchema
trueValue stringOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/properties/csv/properties/trueValue
falseValue stringOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/properties/csv/properties/falseValue
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/properties/csv/unevaluatedProperties
processedFilesDir stringRequired
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/properties/processedFilesDir
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/properties/fileSource/unevaluatedProperties
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/0/patternProperties/^__
oneOf option 2 object
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1
Nested schema details (2)
dbmsSource objectRequired
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource
Nested schema details (12)
oneOf option 1 object
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/0
Nested schema details (2)
sqlServer Any typeRequired
Defines SQL Server as source DBMS. The fields in sqlServer object can be any valid fields defined by spark specification for jdbc database source – see https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html.
For example,
"url" : "jdbc:sqlserver://my-server\\\\SQLSERVER2019:1433;database=my-db-name;integratedSecurity=true;".
If "numPartitions" is defined then one of the columns in "schema" must have attribute "isPartitionColumn" set to true. The "lowerBound" and "upperBound" are optional – if absent they will be determined by the loader by querying the source table.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/0/properties/sqlServer
References
$ref:#/$defs/jdbcOptions
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/0/patternProperties/^__
oneOf option 2 object
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/1
Nested schema details (2)
oracle Any typeRequired
Defines Oracle as source DBMS. The fields inside sqlServer object can be any valid field defined by spark specification for jdbc database source – see https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html
For example,
"url" : "jdbc:oracle:thin@//host-name:port/service-name"
.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/1/properties/oracle
References
$ref:#/$defs/jdbcOptions
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/1/patternProperties/^__
oneOf option 3 object
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/2
Nested schema details (2)
mongoDb objectRequired
Properties for dbmsSource mongoDb can be any of the options supported by MongoDb Spark connector in batch mode. Schema must be defined
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/2/properties/mongoDb
Nested schema details (17)
connection.uri stringRequired
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/2/properties/mongoDb/properties/connection.uri
database stringOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/2/properties/mongoDb/properties/database
collection stringRequired
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/2/properties/mongoDb/properties/collection
comment stringOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/2/properties/mongoDb/properties/comment
mode stringOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/2/properties/mongoDb/properties/mode
columnNameOfCorruptRecord stringOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/2/properties/mongoDb/properties/columnNameOfCorruptRecord
mongoClientFactory stringOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/2/properties/mongoDb/properties/mongoClientFactory
partitioner stringOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/2/properties/mongoDb/properties/partitioner
sampleSize numberOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/2/properties/mongoDb/properties/sampleSize
sql.inferSchema.mapTypes.enabled booleanOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/2/properties/mongoDb/properties/sql.inferSchema.mapTypes.enabled
sql.inferSchema.mapTypes.minimum.key.size numberOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/2/properties/mongoDb/properties/sql.inferSchema.mapTypes.minimum.key.size
aggregation.pipeline Any typeOptional
When load action defines incremental load, the attribute, if used, should have a statement that loads data incrementally based on a watermark column or columns. The watermark column is a column whose previously loaded MAX value is used to determine the new rows for the incremental load.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/2/properties/mongoDb/properties/aggregation.pipeline
Nested schema details (2)
anyOf option 1 string
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/2/properties/mongoDb/properties/aggregation.pipeline/anyOf/0
anyOf option 2 array
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/2/properties/mongoDb/properties/aggregation.pipeline/anyOf/1
Nested schema details (1)
Array items string
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/2/properties/mongoDb/properties/aggregation.pipeline/anyOf/1/items
aggregation.allowDiskUse booleanOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/2/properties/mongoDb/properties/aggregation.allowDiskUse
outputExtendedJson booleanOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/2/properties/mongoDb/properties/outputExtendedJson
^partitioner\\.options\\. string
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/2/properties/mongoDb/patternProperties/^partitioner\\.options\\.
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/2/properties/mongoDb/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/2/properties/mongoDb/unevaluatedProperties
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/oneOf/2/patternProperties/^__
isRemoveDuplicateRows booleanOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/properties/isRemoveDuplicateRows
effectiveDate stringOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/properties/effectiveDate
String constraints
format"date"
user stringOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/properties/user
password stringOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/properties/password
incrementalLoad objectOptional
Incremental load defines loading of newly added or changed data from the source. The incremetal load uses a field designated as watermark column. Watermark column can be any value that grows when the row is inserted or updated, e.g., row update timestamp or in case of SQL Server a ROWVERSION column. The loader determines MAX value for the watermark column in current data and and injects it in to the query to allow selection of data rows that were added or changed since the last load.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/properties/incrementalLoad
Nested schema details (4)
uniqueKeys Any typeOptional
Unique keys are needed to delete records from the destination file for keys that are no longer in the source. If unique keys are not defined they will be generated programmatically.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/properties/incrementalLoad/properties/uniqueKeys
Nested schema details (2)
anyOf option 1 string
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/properties/incrementalLoad/properties/uniqueKeys/anyOf/0
anyOf option 2 array
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/properties/incrementalLoad/properties/uniqueKeys/anyOf/1
Nested schema details (1)
Array items string
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/properties/incrementalLoad/properties/uniqueKeys/anyOf/1/items
delimForWatermarkColumns stringRequired
Watermark columns are the columns in the destination file that are used in incremental load. The query to get data incrementally uses the MAX current value of watermark column to get new records with the value of the watermark column greater that the one in the destination file already. Delimiter designates a column in the query as a watermark column. For example, with the delimiter as three colons, the query for incremental load may look like this: SELECT OrderNumber, OrderTime From Orders WHERE OrderTime > :::OrderTime:::
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/properties/incrementalLoad/properties/delimForWatermarkColumns
String constraints
maxLength5pattern"^[:@#!-]+$"
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/properties/incrementalLoad/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/properties/incrementalLoad/unevaluatedProperties
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/patternProperties/^__
sqlServer Any type
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/dependentSchemas/sqlServer
Additional schema keywords (1)
customMessage{"required":{"uniqueKeys":"\\"uniqueKeys\\" attribute is missing. It is required for incremental load for sqlServer"}}
Nested schema details (1)
incrementalLoad Any typeOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/dependentSchemas/sqlServer/properties/incrementalLoad
oracle Any type
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/dependentSchemas/oracle
Additional schema keywords (1)
customMessage{"required":{"uniqueKeys":"\\"uniqueKeys\\" attribute is missing. It is required for incremental load for oracle"}}
Nested schema details (1)
incrementalLoad Any typeOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/dependentSchemas/oracle/properties/incrementalLoad
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/properties/dbmsSource/unevaluatedProperties
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/oneOf/1/patternProperties/^__
name stringRequired
A name of the load action to differentiate multiple load actions. The name of a load action must be unique within the configuration.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/name
isActive booleanRequired
If the value of isActive is false, the load action will not be processed.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/isActive
isDebugDwLib booleanOptional
If the value is true, there will be some dataframe related output sent to the standard output from the library causing slowdown in job execution.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/isDebugDwLib
fileLoadControl objectOptional
When this attribute is defined, the loader will create a file with summary info for each load. The info includes record counts, description of source and destination, and the action timing.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/fileLoadControl
Nested schema details (4)
parquet objectRequired
File type is parquet for load control.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/fileLoadControl/properties/parquet
dir stringRequired
A directory with the load control file.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/fileLoadControl/properties/dir
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/fileLoadControl/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/fileLoadControl/unevaluatedProperties
fileDestination objectRequired
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/fileDestination
Nested schema details (7)
isVersioned booleanRequired
When the value of isVersioned is true, the loader will maintain versioned destination file. The new version of the row is inserted based on unique key when any of the field’s values change. The unique key can be a single column or comprised of multiple columns (compound key). The granularity of the versioning is a calendar day defined by effective date. If a source has a date field that is designated as effective date then its value will be used to create versions. In the absence of such a field, the loader will use the current date.
When the data source has a value of isVersioned set to false, the loaded will maintain non-versioned destination file. Non-versioned file saves every effective date data in full.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/fileDestination/properties/isVersioned
metadataColumnsPrefix stringOptional
Metadata columns are the columns with reserved names that serve special functions. If the name of the metadata column collides with data column, use metadata columns prefix to differentiate the names.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/fileDestination/properties/metadataColumnsPrefix
String constraints
pattern"^[A-Za-z_][A-Za-z0-9_]*$"
parquet objectRequired
Defines parquet as a type of destination file
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/fileDestination/properties/parquet
path stringRequired
Full path of the destination file.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/fileDestination/properties/path
previousCopyPath stringOptional
If defined the Source Loader will save a copy of a current file to this name. This can be useful to go back to previous version of the file if needed.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/fileDestination/properties/previousCopyPath
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/fileDestination/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/fileDestination/unevaluatedProperties
schema arrayOptional
Shema defines the structure of the destination file. Schema can be absent for DBMS source. It can also be absent for file source if “inferSchema” option is set to true.
If fileDestination.isVersioned set to true the schema must have one or more columns with isUniqueKey attribute set to true.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/schema
Nested schema details (1)
Array items object
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/schema/items
Nested schema details (8)
colName stringRequired
Column name
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/schema/items/properties/colName
colType stringOptional
Column data type
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/schema/items/properties/colType
isUniqueKey booleanOptional
Designates column as a unique key. This column attribute cannot be derived from the DBMS, so to designate a column as a unique key the column has to be included in schema.
Multiple columns can be designated as unique key to create a composite unique key.
Unique key is used for incremental load from dbms sources or for creating a versioned destination file.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/schema/items/properties/isUniqueKey
isPartitionColumn booleanOptional
For DBMS source one column can be defined as partition column to read data on multiple connections in parallel. isPartitionColumn must be defined if loadAction.dbmsSource.numberOfPartitions is defined.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/schema/items/properties/isPartitionColumn
isExcludeFromVersioning booleanOptional
This field is used for versioned destination file. When this flag is set to true then even if the field changed in the source it will not trigger the creation of a new version of the row.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/schema/items/properties/isExcludeFromVersioning
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/schema/items/patternProperties/^__
allOf option 1 Any type
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/schema/items/allOf/0
Nested schema details (2)
If Any type
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/schema/items/allOf/0/if
Nested schema details (1)
isPartitionColumn Any typeRequired
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/schema/items/allOf/0/if/properties/isPartitionColumn
Additional schema keywords (1)
consttrue
Then Any type
If "isPartitionColumn" attribute of a column is set to true, colType must be defined (it is needed to determine the type of upper and lower bounds).
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/schema/items/allOf/0/then
Additional schema keywords (1)
customMessage{"required":"\\"colType\\" is required when \\"isPartitionColumn\\" is true."}
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/schema/items/unevaluatedProperties
postProcess objectOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/postProcess
Nested schema details (7)
packageName stringOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/postProcess/properties/packageName
moduleName stringRequired
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/postProcess/properties/moduleName
methodName stringRequired
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/postProcess/properties/methodName
stgSources arrayOptional
Staging sources that are used for post processing current staging source. These sources will be preloaded and made available in the post process method as views
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/postProcess/properties/stgSources
Nested schema details (1)
Array items object
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/postProcess/properties/stgSources/items
Nested schema details (5)
moniker stringRequired
Staging source moniker is a unique identifier of a staging source. It is used as a view name in Spark SQL.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/postProcess/properties/stgSources/items/properties/moniker
description stringOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/postProcess/properties/stgSources/items/properties/description
fileSource objectRequired
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/postProcess/properties/stgSources/items/properties/fileSource
Nested schema details (4)
parquet objectRequired
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/postProcess/properties/stgSources/items/properties/fileSource/properties/parquet
path stringRequired
A parquet file path.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/postProcess/properties/stgSources/items/properties/fileSource/properties/path
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/postProcess/properties/stgSources/items/properties/fileSource/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/postProcess/properties/stgSources/items/properties/fileSource/unevaluatedProperties
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/postProcess/properties/stgSources/items/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/postProcess/properties/stgSources/items/unevaluatedProperties
applicationSpecific objectOptional
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/postProcess/properties/applicationSpecific
Nested schema details (1)
Unevaluated properties
Boolean schema: true - all instance values are accepted at this location.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/postProcess/properties/applicationSpecific/unevaluatedProperties
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/postProcess/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/properties/postProcess/unevaluatedProperties
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/patternProperties/^__
allOf option 1 Any type
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/allOf/0
Nested schema details (2)
If Any type
If destination file defined with "isVersioned"=true, then the schema must be defined with at least one column as primary key, i.e. "isUniqueKey": true
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/allOf/0/if
Nested schema details (1)
fileDestination Any typeRequired
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/allOf/0/if/properties/fileDestination
Nested schema details (1)
isVersioned Any typeRequired
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/allOf/0/if/properties/fileDestination/properties/isVersioned
Additional schema keywords (1)
consttrue
Then Any type
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/allOf/0/then
Additional schema keywords (1)
customMessage{"required":"\\"schema\\" must be defined with at least one column as primary key when destination has \\"isVersioned\\" set to true"}
Nested schema details (1)
schema Any typeRequired
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/allOf/0/then/properties/schema
Additional schema keywords (1)
customMessage{"contains":"The schema must have at least one column as primary key when destination has \\"isVersioned\\" set to true, i.e. \\"isUniqueKey\\": true"}
Nested schema details (1)
Contains Any type
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/allOf/0/then/properties/schema/contains
Nested schema details (1)
isUniqueKey Any typeRequired
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/allOf/0/then/properties/schema/contains/properties/isUniqueKey
Additional schema keywords (1)
consttrue
allOf option 2 Any type
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/allOf/1
Nested schema details (2)
If Any type
If incrementalLoad is defined then destination must be versioned, i.e., with "isVersioned"=true
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/allOf/1/if
Nested schema details (1)
dbmsSource Any typeRequired
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/allOf/1/if/properties/dbmsSource
Then Any type
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/allOf/1/then
Nested schema details (1)
fileDestination Any typeRequired
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/allOf/1/then/properties/fileDestination
Additional schema keywords (1)
customMessage{"required":"\\"isVersioned\\" flag must be defined and set to true when \\"incrementalLoad\\" is defined for DBMS source","const":"\\"isVersioned\\" flag must set to true when \\"incrementalLoad\\" is defined for DBMS source"}
Nested schema details (1)
isVersioned Any typeRequired
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/allOf/1/then/properties/fileDestination/properties/isVersioned
Additional schema keywords (1)
consttrue
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/sourceLoad/properties/loadActions/items/unevaluatedProperties
^__ objectarraystringnumberintegerbooleannull
Comment field begins with two underscores. There can be any number of comment fields.
JSON Pointer: /properties/sourceLoad/patternProperties/^__
Unevaluated properties
Boolean schema: false - no instance value is accepted at this location.
JSON Pointer: /properties/sourceLoad/unevaluatedProperties