The order of the configuration parameters is important, they must be ordered as stated in this documentation.

1. delimiter

On the first line the delimiter has to be specified. You can put any string you want as delimiter in your data files as the value of the parameter or any number and combination of 'tab' and 'space' to use one or more tab or space characters.

Example:

delimiter:tab

delimiter:space

2. surrounding_char

With this you can specify a character that surrounds data for a field. If specified it will only affect the data that has that character at the start and end of it. Nothing happens if the character is not there.

There are two usable characters for now: " and '.

If anything else is specified as the character or there is no specified character, then nothing will be done.

Example:

surrounding_char:"

3. decimal_separator

This can be used, if you want to use something else than period (.).

Example:

decimal_separator:,

4. mode

This line specifies the mode. There are six modes, new, unique_new, add, upsert , modify and dynamic.

•new - New objects will be created. Will not check if object already exists, thus new objects (with different DP_OID) will be created with the same data.

•unique_new - New objects will be created. If it already exist it will be ignored. Should be configured like the new mode but with id parameter.

•add - New components will be added to existing objects.

•upsert - Create a new object if it does not exist, else update existing object. Should be configured like the new mode but with id parameter.

•modify - Modify or add components to existing objects, rows for non existing objects will be logged. Should be configured like the add mode with the addition of one or more lines with cid parameters for each component type, see below.

•dynamic - Choose what mode to use for each line depending on the content of some field in the line being processed. See more parameters further down for dynamic.

5. otype

The otype can be specified with a fixed value or a column in the data to read the value from.

Example, import all rows into object with otype 50001:

otype:50001

Example, import rows into objects depending on the otype specified in field 3:

otype:col:3

6. subtype

Should only be specified if the mode is new, upsert or dynamic (if the latter can handle new and/or upsert).

subtype:col:7

7. state

Should only be specified if the mode is new, upsert or dynamic (if the latter can handle new and/or upsert).

state:1

8. forcencolumns

Forced check of number of columns. You can force the importer to complain if a row contains a number of columns differing from the number defined by forcencolumns.

forcencolumns:12

9. date

If you want to use a special date format you can specify that.

Example:

date:yyyyMMdd

The date format should follow the format for java.util.SimpleDateFormat. If you do not specify a format the yyyy-MM-dd will be used.

10. verify_fields

Set to true if you want to verify data before importing. Fields are verified one by one while importing. If the data does not comply, an error message will be appended to the log file and the data will be replaced with null, so not null fields will generate two errors.

Example:

verify_fields:true

11. encoding

The character encoding of the import data. The default is ISO-8859-1. If more than one ImportFormats are used on the same data file, the encoding of the first is used.

Example:

encoding:ISO-8859-1

12. class

Specifies the parameter class.

Example:

class:sys.nis.servlets.imp.LinkImporter

The value should be a colon separated list of classes implementing the interface PostImporter. The method PostImporter.postImport () is invoked for each class when a row has been imported.  

13. linestringcol

Read further down under Mapping about linestring for explanation.

14. id

If mode is add, upsert, modify or unique_new the next line should contain a specification on how to find the object to add to.

Example:

id:0:100000:CUSTOMER_NO

The keyword should be id, the first part of the parameter is the column or columns of the data to match, the second part is the ctype of the component to match to and the third is the field of the component to match to. You can specify many id conditions. If you want to find a object with a codelisted value you have to do like this:

id:1:22302:OBS_CODE:CODELISTED:OBS_TEXT

You can also define a specific class that defines a more involved method of fetching objects for update using, for example:

idclass:sys.nis.servlets.imp.ParentLinkIdFetcher

The keyword should be idclass, the next part is the name of the class that implements the ImportedIdFetcher interface. Any such id fetcher may also need additional parameters specific to its own requirements. They are specified using the idclassparam keyword:

idclassparam:PARENT_LNK_CTYPE:5000

These parameters can be accessed by the id fetching class as a String to String map that stores paramName->paramValue pairs. Please refer to the specific class documentation for details.

15. cid

If mode is modify, the next keyword tell how to find the components to modify. The format looks the same as for id. If no component is found for an object, a new will be created.

16. preprocessor

Configuration about preprocessors is a semicolon separated list. Each element of the list is a Config file and one or more sections of this config separated with commas. Each section must contain the parameter class, which value names a class that implements ImporterPreprocessor. For each of the classes the method process will be called for each line of the import data before an import attempt is performed. The classes are used in the order they are specified.

Example:

preprocessor:^app/fpgas/conf/cust_fov/customer_import/preprocessor.conf,equipment_translator

17. constant

Some component fields may be set to a constant value. The format is similar to that for the column mappings except the column number is replaced with the keyword constant and an extra colon at the end followed by the constant data.

Example:

constant:23000:IMP_SOURCE:STRING:FPGAS

An arbitrary number of constant fields may be specified. Accepted data types are STRING, DATA, DATE, BOOLEAN and INTEGER. The type DATE supports the keyword SYSDATE which yields a date following the provided date format or default if none is provided of the current time when reading the config, any precision beyond days might not be useful.

18. constant_set_on_upsert

Set field to a constant value only when a row is being inserted/updated. This value will not be compared to the existing value in the database to trigger an update. Other than that, it is identical to constant.

19. dynamic

If mode is set to dynamic each line should contain some field with some known token to indicate which mode it should use. It is important to know which modes are expected because other configuration parameters need to be specified. E.g. if add can be chosen for a line the id parameter is needed, for modify the cid parameter is needed. dynamic:column:# - specifies in which column the token should be found. It counts from zero. Should appear before the following modes. dynamic:mode:? - One or many for each mode (so different tokens can be used for the same action). At least one mode is required (but not all). Example:

dynamic:column:8

dynamic:new:I

dynamic:modify:Up

dynamic:modify:Rm

The tokens will be looked for in the 9th column, the string "I" will apply the mode new. Strings "Up" and "Rm" will both apply the modify mode.

20. skip

Skip specifies which rows should be skipped when certain columns equals certain string. Using skip is optional and it is possible to call it multiple times. It is mandatory to specify both column and search string.

Example 1 (skip every row where 0th column equals "10"):

skip:0:10

Example 2 (skip every row where 5th column equals "test"):

skip:5:test

Example 3 (skip every row where 15th column equals "message"):

skip:15:message

Example 4 (skip every row where 20th column is empty i e equals ""):

skip:20:

If the row only has 19 values, it will not be skipped.

21. take

Take specifies which rows should be imported when certain columns equals certain string. All other rows are skipped. i e not imported. Using take is optional and it is possible to call it multiple times. It is mandatory to specify both column and search string.

Example 1 (import only rows where 0th column equals "10"):

take:0:10

Example 2 (import only rows where 5th column equals "A" or "B"):

take:5:A

take:5:B

Example 3 (import only rows where 5th column equals "A" or 6th column equals "B"):

take:5:A

take:6:B

If the row only has 4 values, it will not be imported.

22. skipfirst

Skipfirst can be used when a number of rows in the beginning of the data file needs to be skipped. Using skipfirst is optional.

Example (skip first 4 rows):

skipfirst:4

23. Mapping

A specification of how the columns should be mapped. One row per column.

Example 1:

1:100001:ADDRESS_TYPE:STRING

2:100001:STREET:STRING

5:100001:COUNTRY:STRING

6,7:101010:SHAPE:SYMBOL:ST74:RT90

Example 2 (Excel column positions):

B:100001:ADDRESS_TYPE:STRING

C:100001:STREET:STRING

F:100001:COUNTRY:STRING

G,7:101010:SHAPE:SYMBOL:ST74:RT90

Example using field names:

SPRINKLER:7201:SPRINKLER:STRING

"LSO 2:3":7201:LSO_2_3:STRING

PERMIT\, DATE:7201:PERMIT_DATE:STRING

N,E:451:SHAPE:SYMBOL

As with id specification, the first part of the parameter is the column or columns of the data to match, column numbering is zero indexed if integers are used. It is also possible to use Excel style column positions where A = 0. If field names are used, ordering doesn't matter. Fields containing : can be double-quoted. Commas can be escaped. The second part is which ctype of the object that the mapping should affect. The third part of the mapping is the field where to put the data. The last part describes the type of mapping that should be done. The Importer understands the types:

•STRING: The data is imported as it is, sent as a string to the database.

•DATA: The data is imported as it is, sent as a string to the database. Use STRING for new mapping. Use INTEGER for numerical values.

•INTEGER: The input string is parsed as an integer value.

•DOUBLE: The input string is parsed as a double. Configure decimal_separator if other than period (.).

•DATE: The data is parsed according to the dateformat specified, see section above.

•BOOLEAN: The data is parsed to a boolean value, 1 and true is interpreted to TRUE (1 in oracle), everything else to FALSE (0 in oracle). 1 is interpreted as TRUE to facilitate imports of data from other system using 1 as true-value.

•NULL_CONSTANT_TYPE: Use this type in connection with a constant to set specific column-values to null in the customer import.

•SYMBOL: The data values are interpreted as coordinates and a symbol component is created. The coordinates are fetched from the columns specified like x,y,[z,rotation]. Examples:

6,7:101010:SHAPE:SYMBOL:ST74:RT90

1,2,3:101017:SHAPE:SYMBOL:ST74:RT90

1,2,3:101017:SHAPE:SYMBOL:ST74:GET_CS_FROM_CTYPE

1,2,-1:101017:SHAPE:SYMBOL:ST74:GET_CS_FROM_CTYPE

1,2,-1,4:101017:SHAPE:SYMBOL:ST74:GET_CS_FROM_CTYPE

The z-column is optional unless a rotation is specified. If there is no z-column but a rotation is wanted the z-column should be set to -1. Default value for rotation is 0.0. If the coordinates should be transformed during import you can specify which coordinate system to translate from and to as the fifth and a sixth part of the mapping. You can also set a flag "GET_CS_FROM_CTYPE". In this case the database table name for the ctype is fetched and this table is checked for the target coordinate system. In the example above the symbols are expressed in ST74 in the data file, but are stored in RT90 in the database. In the third example the targrt coordystem is fetched automatically.

•TEXT: The data values are interpreted as coordinates and a text component is created. The coordinates/rotation/text are fetched from the columns specified like x,y,[z],r,t where r = rotation column, t = text column. If r = -1 then rotation = 0.0. If t = -1 then the text is fetched from text trigger information. Examples:

1,2,3,7,6:16597:SHAPE:TEXT:SWEREF99 13 30:SWEREF99 13 30 # with z value

1,2,3,-1,-1:16610:SHAPE:TEXT:SWEREF99 13 30:SWEREF99 13 30 # with z value

1,2,-1,-1:16610:SHAPE:TEXT:SWEREF99 13 30:SWEREF99 13 30 # without z value

For transformations, see SYMBOL.

•LINESTRING: The values of several rows are interpreted as coordinates of a linestring component. To know how many rows with coordinates that makes one linestring a property specifying the column with an unique id for each linestring should be specified before the column mappings. Example of format:

otype:533

subtype:1

state:1

linestringcol:3

0,1,2:1024:SHAPE:LINESTRING

3:1025:NAME:STRING

Corresponding data:

98989898,2363243,12,NOR78

98989894,2363245,12,NOR78

98989568,2363246,12,NOR78

98989413,2363234,12,NOR79

98989884,2363291,12,NOR79

98989814,2363276,12,NOR81

98989766,2363209,12,NOR81

98982345,2363288,12,NOR81

This mapping and data will result in 3 linestring objects of otype 533. The first will have a data component of ctype 1025 with field NAME set to NOR78 and a graphic component of ctype 1024 with two points. The second will have a data component of ctype 1025 with field NAME set to NOR79 and a graphic component of ctype 1024 with two points. The third will have a data component of ctype 1025 with field NAME set to NOR81 and a graphic component of ctype 1024 with three points.You can specify coordinate systems in the same way for a LINESTRING as for a SYMBOL

•OGC WKT (Well Known Text): The value should be a valid OGC WKB geometry.

Example of format:

otype:533

subtype:1

state:1

0:1024:SHAPE:WKT

1:1025:NAME:STRING

Corresponding data:

LINESTRING (98989898 2363243, 98989894 2363245, 98989568 2363246),NOR78

LINESTRING (98989413 2363234, 98989884 2363291),NOR79

LINESTRING (98989814 2363276, 98989766 2363209, 98982345 2363288),NOR81

This mapping and data will result in 3 linestrings the same as in the LINESTRING example but with Z component in all vertices set to 0.0. For 3D geometries use "Z" letter after the geometry type as stated in the examples for OGC (Open Geospatial Consortium) WKT at https://postgis.net/docs/using_postgis_dbmanagement.html#OpenGISWKBWKT. The data corresponding with the mapping will then look like this:

LINESTRING Z (98989898 2363243 12, 98989894 2363245 12, 98989568 2363246 12),NOR78

LINESTRING Z (98989413 2363234 12, 98989884 2363291 12),NOR79

LINESTRING Z (98989814 2363276 12, 98989766 2363209 12, 98982345 2363288 12),NOR81

You can specify coordinate systems in the same way for a WKT as for a SYMBOL. The EWKT (Extended WKT) standard is not supported. The LRS "M" component is not supported.

•OGC WKB (Well Known Binary): The value should be a valid OGC WKB geometry.

Example of format:

otype:533

subtype:1

state:1

0:1024:SHAPE:WKB

1:1025:NAME:STRING

Corresponding data:

000000000200000003419799DD28000000414207B580000000419799DD18000000414207B680000000419799D800000000414207B700000000,NOR78

000000000200000002419799D594000000414207B100000000419799DCF0000000414207CD80000000,NOR79

000000000200000003419799DBD8000000414207C600000000419799DB18000000414207A4800000004197996724000000414207CC00000000,NOR81

This mapping and data will result in 3 linestrings the same as in the above WKT (Well Known Text) example. You can specify coordinate systems in the same way for a WKT as for a SYMBOL. The EWKB (Extended WKB) standard is also supported, but the SRID will be ignored. The LRS "M" component is not supported. The "\\x" characters at the beginning of the hexadecimal representation of bytes is optional.

•EWKB PostGIS (Extended Well Known Binary for PostGIS): The value should be a valid OGC WKB geometry with Postgis extension. The description of the format and differences between OGC standard could be found here.

Example of format:

otype:533

subtype:1

state:1

0:1024:SHAPE:WKB_POSTGIS

1:1025:NAME:STRING

Corresponding data:

\x01020000000300000000000028dd99974100000080b507424100000018dd99974100000080b607424100000000d899974100000000b7074241,NOR78

\x01020000000200000000000094d599974100000000b1074241000000f0dc99974100000080cd074241,NOR79

\x010200000003000000000000d8db99974100000000c607424100000018db99974100000080a4074241000000246799974100000000cc074241,NOR81

This mapping and data will result in 3 linestrings the same as in the above WKT (Well Known Text) example. You can specify coordinate systems in the same way for a WKT as for a SYMBOL. The EWKB (Extended WKB) standard is also supported, but the SRID will be ignored. The LRS "M" component is not supported. The "\\x" characters at the beginning of the hexadecimal representation of bytes is optional.

•CODELISTED: Another special form that tells the import to take the data from a codelist.

Example:

8:19033:SUBSCR_TYPE:CODELISTED:CODETEXT

The last field tells the field name to use as key. There can be several columns and key fields:

3,4,5:19211:EQUIPMENT:CODELISTED:EQUIP_MAKE,EQUIP_MODEL,EQUIP_TYPE

•CODELISTED_ADD: Basically the same as CODELISTED with the difference that codelist texts that are not in the codelist are created and a codenum is assigned to this text.