Skip to main content

Commands

Genesis has numerous built-in commands that have their own individual functions. This page goes through them and details the function, parameters and use cases of those commands.

Once an application has been built and zipped, you can install it in any another system that contains the Genesis Application Platform.

To ensure a correct installation, you must follow the product installation procedure.

Logging level for commands

The logging level for Genesis commands is set to "WARN" by default.

To change the level for a command, set the environment variable GENESIS_LOGGING_LEVEL to any valid level ("ERROR", "WARN", "INFO", "DEBUG", "TRACE").

If you set GENESIS_LOGGING_LEVEL to a non-valid level, it is automatically reset to the default.

AppGen

AppGen can be used to generate a fully working application from a dictionary file.

Usually when creating a application, you would start with a schema; you then build Data Servers, Request Servers and Event Handlers on top to create your application. AppGen automates all this, and will generate the following new files:

  • Data Server: application-dataserver.kts
  • Request Server: application-reqrep.kts
  • Event Handler: application-eventhandler.kts
  • System processes: application-processes.xml
  • Service definitions: application**-service-definitions.xml
Data Server and Request Server

One block will be generated for each table in the database, complete with metadata (all fields on the table) and a field block that returns all fields on the table. For Request Servers, the inbound metadata will be based on the primary key.

Event Handler

The file for the Event Handler contains insert, amend and delete transactions for every table in the database. All transactions support validation and meta data. If a field is marked as a sequence in the dictionary (i.e. generated ID) then the field is not specified on the metadata for inserts, but it will be specified on modifies and deletes.

Deletes will have reduced metadata, as it is only necessary for the columns to satisfy the primary key to perform the delete.

Parameters
ArgumentArgument long nameMandatoryDescriptionRestricted Values
-d dictionary file nametruethe name of the dictionary to read at startupNo
-t table name(s)falsethe table name(s) within the dictionary to read at startup. Multiple tables must be separated by spacesNo
-p port offsettruethe port range to use when generating the services fileNo
-pn product nametruethe name of the product to createNo
Examples

This example has no -t option.

AppGen -d tas-dictionary.xml -p 4000 -pn tas

In this case, the dictionary to read is tas-dictionary.xml, the port offset is 4000 and the product name to generate is tas. Running this command results in the following structure being created:

tas/
/bin
/cfg/tas-dataserver2.xml
/tas-fields-dictionary.kts
/tas-processes.xml
/tas-reqrep.xml
/tas-service-definitions.xml
/tas-system-definition.kts
/tas-tables-dictionary.kts
/scripts/tas-tnHandler.gy

This example has a -t option, which specifies that only the table ORDER USER should be read from the source dictionary.

AppGen -d tas-dictionary.xml -t ORDER USER -p 4000 -pn tas

The ORDER USER table will be appended to the files that were created in the tas folder.

CountRecords

This counts the number of records in the database, grouped by table; the results are printed to screen.

By default, a count is given for every table in the database.

If you only want the record count for specific tables, you can specify the table name or names.

Syntax
ArgumentArgument long nameMandatoryDescriptionRestricted valuesDefault
[TABLES] nothe name of the table whose rows are to be counted; for more than one table, this must be a space-separated listnonea record count is provided for all tables in the database

This example gives a record count for all tables:

CountRecords

The example below gives a record count for two specific tables:

CountRecords CHF_ORDERS CHF_TRADES

CreateMissingSqlSequences

warning

Stop all your application's processes before using this command.

This command creates sequences in the database for ID fields that rely on auto-generated sequence values.

The command checks to see which sequences exist already, and only creates new sequences where they do not exist. Consequently, if you run the command more than once, you will get the same result each time.

This command is applicable only for SQL databases and should only be executed when you are enabling Sequences for SQL databases.

Syntax

The CreateMissingSqlSequences command has no parameters.

Working with different databases

The behavior of this command depends on which database implementation your application uses.

  • If you are using a NOSQL database, such as Foundation DB, auto-incremented values are assigned in blocks of 100 in order to improve performance. This command sets the value in the database, which corresponds to the first value in the next range to be allocated.

  • If you are using Oracle, you can not set a sequence value directly. This command increments the sequence value by the difference between the current counter value and the desired value. This can have unexpected effects on sequence values that are already assigned in the cache, as the increment is also applied to these values.

And remember, only use this command when all your applications have been stopped. After running SetAutoIncrement, you need to restart the server.

DbMon

DbMon is the Genesis database client, which provides its own command line. From here, you can navigate through the database tables in your application.

DbMon itself has a help command, which shows all the available commands. To get help on a specific command, run help _command_.

DbMon --quietMode performs database changes without triggering real-time updates in the update queue layer.

For full details, see our page on DbMon.

DictionaryBuilder

DictionaryBuilder connects to an RDBMS, parses schemas and uses this information to generate a Genesis tables dictionary. It supports SQLServer and PostgreSQL databases.

The script accepts a series of arguments to establish a connection to the database and to control the behaviour of the script:

ArgLong argumentMandatoryDescription
-j--jdbcUrl <jdbcUrl>YesJDBC URL of the database.
-p--product <productName>Yesrepresents the product name and affects the output files
-c--createFormatgenerates JSON ready for use with Genesis Create, instead of the standard tables dictionary
-e--exportexports table data to CSV

e | -t | --tables <table1,table2> | | table list to include, default is all tables | | -o | --output <outputDir> | | output directory for files; if the directory does not exist, it will be created. Defaults to project root. | | -s | --schema <schemaId> | | Schema name / System ID | | | --tableid <tableIdStart> | | Table ID start number, default = 2000; ignored if reading from Genesis DB. |

Example

Generates a Genesis tables dictionary from a PostgreSQL database and exports data to a TABLE_NAME.csv file in the root of the project:

DictionaryBuilder --jdbcUrl jdbc:postgresql://localhost:5432/?user=postgres&password=docker --product my-app --export

The example below generates JSON for use with Genesis Create from an SqlServer database:

DictionaryBuilder --jdbcUrl jdbc:sqlserver://localhost:1433;trustServerCertificate=true;username=SA;password=VerySecure123 --product my-app --createFormat

The example below generates a Genesis tables dictionary from a PostgreSQL database called myDatabase

DictionaryBuilder --jdbcUrl jdbc:postgresql://localhost:5432/myDatabase?user=postgres&password=docker --product my-app --export --tables user_trades
How the script behaves

The script connects to the specified JDBC Url and retrieves the information it requires to build a Genesis tables dictionary.

There are a few considerations you should be aware of:

  • Table and column names parsed in lowerCamelCase format will be transformed to UPPER_SNAKE_CASE
  • If there is no mapping available for the java.sql.Type the column will be ignored and the name logged in the output.
  • Exporting extremely large datasets with the --export flag is not supported.
Keys and indexes

Primary keys will be parsed as primary keys in Genesis, whether they are single-column-based or multiple-column-based.

Only unique indexes will be parsed as secondary keys.

There is no concept of foreign keys in Genesis, so these are ignored.

DropTable

The DropTable command removes all rows from the specified tables.

warning

If you do not specify any tables, then all rows will be removed from all tables in the database.

Use with care!

Syntax
ArgumentArgument long nameMandatoryDescriptionRestricted valuesDefault
-t [TABLES]nothe name of the table to have its rows removed; for more than one table, this must be a space-separated listnoneall records are removed from all tables

The example below removes all the rows from three separate tables:

DropTable -t EQUITY_ORDERS_GB EQUITY_ORDERS_DE EQUITY_ORDERS_IT

The command will ask you to confirm the removal of the content from each table.

DumpIt

To copy data from a Genesis database, use the DumpIt command.

Syntax

The DumpIt command can take the following arguments:

ArgumentArgument long nameMandatoryDescriptionRestricted valuesDefault
-a --all noexports all tables to csvnonenone
-f --file (arg) noname of the csv file where table is exportednonenone
-fields (arg) nospace separated field list e.g. "FIRST_NAME LAST_NAME"nonenone
-h --help noshow help on how to use the commandnonenone
-s --sql (arg) noname of the sql file to export the table tononenone
-t --table (arg) nospecify the name of the table you want to export to CSV. If there are multiple tables to include, provide a list separated by spaces.nonenone
-cem --criteriaEvaluatorMode (arg) nothe type of criteria evaluator to be usedTYPE_AWARE or LEGACYLEGACY
-fm --formatMode (arg) noindicates whether field formats should be taken into accountFORMATTED (takes field formats into account) or LEGACY (does not take field formats into account)none
-qi --quoteIdentifiers noif present, all sql identifiers (e.g. column names) will be quotednonenone
-where (arg) nomatch criteria e.g. "USER_NAME=='John'"nonenone

Here are some examples:

DumpIt -t GBP_TRADES -f gbp-trades

This copies all records in the GBP_TRADES table to the file gbp-trades.csv.

DumpIt -t USER -where "USER_NAME=='John'" -fields "USER_NAME"

This copies the USER_NAME of every record in the USER table where the USER_NAME is John. This useful if you want to know if the user name John is in the database.

DumpIt -t FUND -f FUND -fields "FUND_ID NAME" -where "NAME == 'FUND_FUND' && YEARS_IN_SERVICE >= 10"

This copies the FUND_ID and NAME fields of every record that has "FUND_FUND" for a name, and ten or more years in service.

DumpIt --all

This copies all the tables in the system, creating one .csv file for each table in the database. The files are saved in the current directory. It is useful for taking a back-up of the current system database.

Interactive mode

You can run DumpIt without any arguments to enter interactive mode.

FixEnumValues

Converts non-matching enum values in the database to SNAKE_CASE. This command is intended for use after a dictionary change that adds new enum values. It will only update the data if the converted value matches the list of enum values in the dictionary.

Changes will only be applied to the database if the command is run with the --commit flag.

warning

Stop all processes before using this command.

Syntax

The FixEnumValues command can take the following arguments:

ArgumentArgument long nameMandatoryDescriptionRestricted valuesDefault
-c --commit noapplies dictionary changes to the databasenonenone
[TABLES] noa space-separated list of specific tables to be changed; if no list is supplied, all tables are changednonenone
-h --help noshows help on how to use the commandnonenone

In the example below, the changes are applied to the database for two tables: TRADE and POSITION.

FixEnumValues --commit TRADE POSITION

Using an installHook

To automate this process, you can use an installHook to call the script before remap is performed - be aware that it will only run successfully once.

The following example finds all String to Enum changes in all tables and commits any valid updates to the database before remap is performed.

#!/bin/bash
source "$HOME"/.bashrc
shopt -s expand_aliases

FixEnumValues --commit

exit $?

To implement this:

  1. Navigate to the appName\server\jvm\appName-config\src\main\resources\scripts\installHooks folder

  2. Create a file called nextInstallHookNumber_ConvertData.sh or similar and add the bash script above.

The installHook will run before remap on your next deploy.

genesisInstall

This command validates all system and product configuration, checking for things such as field duplication.

genesisInstall looks at all the folders (apart from runtime and generated), all the modules, and all files in the cfg directory. It copies the config files from the cfg directory into the generated folder.

In the files collected, the command examines the installation environment and looks for system definition tokens (file names with suffix .tmplt.xml). The generated cfg file names have their token placeholders replaced with the environment's system definition value for the token, and the suffix will be changed to .auto.xml.

The command also checks the system-specific definitions and uses these to replace any definitions that have the same name in any of the modules. Where a file in site-specific/cfg has the same name as a file in a module's cfg, the version in site-specific/cfg will always be used.

Following this, when you start any process, the startProcess command reads from the cfg directory in the generated folder.

genesisInstall also completes config checking, looking out for mistakes in the configured code and providing warnings and error messages. If an error is encountered, the configuration will not be propagated to the run/generated/cfg area.

Syntax

The genesisInstall command can take the following arguments:

ArgumentArgument long nameMandatoryDescriptionRestricted valuesDefault
--ignore noIf supplied, will ignore errors in the configuration filesnonenone
--ignoreHooks noIf supplied, will ignore any install hooks foundnonenone
--compactProcesses noWhen set to true, combines compatible services into fewer JVM processes (depending on the selected strategy); overall, this reduces the total number of services required to runnonenone
--compactProcessesStrategy noThe strategy to be used when compactProcesses is set; defaults to DEFAULT; valid values are: DEFAULT, DEV. WARNING: the DEV strategy optimizes (and sometimes removes) processes to provide the optimal developer experience with the fewest possible resources; DO NOT use this for production deploymentsnonenone
--repeatedHooks noIf supplied, will repeat the specified install hooksnonenone
--hostDiff noIf supplied, will compare all the files in every Genesis server in the cluster to make sure that the files are in syncnonenone

Once complete, all configuration files will be copied and, where necessary, merged into the ~/run/generated/cfg file, which we alias as $GC.

If any problems are found in the generated configuration files, they will be deleted. This forces you to correct the errors in the original configuration files.

To ignore errors in the configuration files, use the --ignore argument. This leaves the configuration files undeleted, even if errors are found.

All process configuration is stored within $GC.

The example below ignores errors and leaves the configuration files undeleted, even if errors are found.

genesisInstall --ignore

The example combines all compatible processes into a single process.

genesisInstall --compactProcesses

Install hooks

Install hooks run as part of genesisInstall. You can specify the relevant scripts as part of genesisInstall then add those scripts to the file applicationName_config/resources/scripts/installHooks, where applicationName is the name of application you are developing.

The scripts (hooks) you add will only run once, unless their execution fails. If you run genesisInstallagain, previously successful executions of installHook scripts will not be run as part of the install.

The scripts must be implemented to work in an idempotent way, and the end result of executing a script means the system is (or already was) in the expected target state, whether you run it on a pre-existing environment (e.g. upgrading a server) or you run it in a completely new environment.

On the server, it is located in the GENESIS_HOME/applicationName/scripts/installHooks directory. Logs are located in GENESIS_HOME/runtime/installHooks

Install hook file-name conventions:

  • We only use shell script for install hooks and inside the shell script you can call a Python script, a Kotlin script or whatever is necessary.
  • The install hook name must be unique.
  • It must contain a priority number at the beginning of the file name. This number should be unique. For example: 1_migrateLogFiles.sh, 2_migrateDictionary.sh.
  • If you need to create a new install hook that has to execute before priority number 1 or number 2, you must increase the numbers for all the other scripts (e.g. rename 1_migrateLogFiles.sh to 9_migrateLogFiles.sh).

Practical examples

You can create a new script and add it to the folder mentioned above to perform any particular functionality as part of genesisInstall.

Any script exiting with value "0" is considered successful by the installHooks system, and any script exiting with a non-zero value is considered to have failed execution.

In a Java or Kotlin world, a simple implementation could look like this:

@JvmStatic
fun main(args: Array<String>) {
ScriptUtils.initRootLogLevel(Level.WARN)

val genesisHome = System.getenv("GENESIS_HOME")

if (genesisHome == null || "" == genesisHome) {
val message = "System environment variable GENESIS_HOME is not set. Aborting migration process..."
// installHooks FAILURE
System.err.println(message)
exitProcess(1)
}

try {
run(args)
// installHooks SUCCESS
exitProcess(0)
} catch (e: Exception) {
// installHooks FAILURE
System.err.println(e.message)
exitProcess(1)
}
}

Consider another example; we have a migration script called migrateDictionary.sh as an install hook; this internally executes MigrateDictionary as shown below:

#!/bin/bash

MigrateDictionary -dst DB

exit $?

GenesisRun

This is a Python script wrapper for Genesis scripts.

'GenesisRun` will attempt to find a script to execute within the Genesis folder structure (site-specific or scripts).

There are two environment variables that can be used to configure how much RAM the scripts will use:

  • SCRIPT_MAX_HEAP
  • REMAP_MAX_HEAP

GenesisRun can execute code in two different modes: Groovy script and GPAL Kotlin script. GenesisRun builds the necessary classpath, so you don't need to build it in each script.

  • Groovy script: GenesisRun SetLogLevelScript.groovy
  • GPAL Kotlin script: GenesisRun customPurger-script.kts

There is a separate wrapper, JvmRun for Java main class scripts.

GetAutoIncrementCount

This gets the current auto increment INT values defined in dictionaries for all the sequences in the system. By default, the values are printed on screen (to the terminal), but they can be written to a file so they can be reused by the SetAutoIncrement script (see below).

warning

Stop all your application's processes before using this command.

Syntax

The GetAutoIncrementCount command can take the following arguments:

ArgumentArgument long nameMandatoryDescriptionRestricted valuesDefault
-f --file (arg) noname of file to receive the valuesnoneAutoIncrementValues
-h --help nodisplays help on the commandnonenone
-p --print noif sending to a file, then use this if you also want to print to screennonetrue (unless -f is supplied)

The behavior of this command depends on which database implementation your application uses.

  • If you are using a NOSQL database, such as Foundation DB, auto-incremented values are assigned in blocks of 100 in order to improve performance. This command retrieves the value of the counter stored on disk. If the system is currently active, this value might not correspond to the value of the next record inserted that references the value.

  • Similarly, if you are using Oracle, auto-incremented values are cached in memory in configurable block sizes. This command only retrieves the current value of the counter stored on disk.

  • If you are using an SQL implementation, this command returns the last value assigned by the sequence, not the next to be assigned.

For example, this command puts all the auto increment values in a file called AutoIncVals. The values are not displayed on screen.

GetAutoIncrementCount -f=AutoIncVals

And remember: only use this command when all the application's processes have been stopped.

GetSequenceCount

This gets the current sequence number for all the sequences in the system. The values can be printed on screen or written to a file so they can be reused by the SetSequence script (see below). For example, if you have 120 rows in the table DE_ORDERS, the sequence count is for that table is 120.

Syntax

The GetSequenceCount command can take the following arguments:

ArgumentArgument long nameMandatoryDescriptionRestricted valuesDefault
-f --file (arg) noname of the file to contain the sequence numbersnoneSEQUENCE.csv
-h --help noshow help on how to use thus commandnonenone
-p --print nononetrue (unless -f is supplied)

The example below puts the numbers for all sequences in the database in the file /home/user/run/sequenceCount.

GetSequenceCount --file=/home/user/run/sequenceCount

GetNextSequenceNumbers

This gives you the next sequence number of every table in the application. The numbers are provided in table format (csv), for example:

"Table","Sequence","Value"
"USER_AUDIT","UA","104"
"PROFILE_AUDIT","PR","804"
"PROFILE_USER_AUDIT","PA","104"

In the example above, the table USER_AUDIT has 103 rows, so the next sequence number for that table is 104.

Syntax

The GetSequenceNumbers command can take the following arguments:

ArgumentArgument long nameMandatoryDescriptionRestricted valuesDefault
-f --file (arg) noname of the file to contain the sequence numbersnoneSequenceValues.csv
-h --help noshow help on how to use this commandnonenone
-p --print nononetrue (unless -f is supplied)

The example below displays the next sequence number of each table, using the format described above.

GetNextSequenceNumbers

By default, the details are sent to the screen, but you can redirect the output to a file. For this, use either the -f argument followed by the filename. Alternatively, you can use the >> annotation followed by a filename, as in the example below:

GetNextSequenceNumbers >> /tmp/NextSeqNumbers.txt

Correcting errors in tables

The GetNextSequenceNumbers command is often used with the SetSequence script see below, for example, if you suspect that you have an error in one of your tables:

  1. Stop all the processes and run GetNextSequenceNumbers to find the next sequence numbers of the tables.
  2. Check the table contents. You might find that a row is missing or needs to be added. Make this change on the database manually. This affects the sequence numbers in those tables.
  3. Run SetSequence to reset the sequence numbers where relevant.
  4. Now you can restart your processes.

killProcess

This command is used to terminate a specified process.

Syntax

The killProcess command can take the following arguments:

ArgumentArgument long nameMandatoryDescriptionRestricted valuesDefault
-s HOSTNAME [HOSTNAME ...] --hostname HOSTNAME HOSTNAME [HOSTNAME ...] [HOSTNAME ...]noWhere the application is running on more than one node, this identifies the node where you want to kill the process (so you can kill a process on a different node). Specify the Host Name.nonenone
-f --force forcefully kills a process (using kill -9)nonenone
-w WAIT --wait WAIT nospecifies how many seconds to wait before forcing the killnone10
-c --cluster nokills the process on every node in the clusternonenone

The example below kills the GENESIS_AUTH_PERMS process.

killProcess GENESIS_AUTH_PERMS

In the example below, there is a cluster; this command kills the GENESIS_ROUTER process on the nodes HARRY and GARY.

killProcess GENESIS_ROUTER --hostname HARRY, GARY

In the example below, there is a cluster; this command kills the GENESIS_ROUTER process on ALL nodes.

killProcess GENESIS_ROUTER --cluster

killServer

This command kills the server. It reads the $GC/processes.xml file to determine which processes to kill. It will prompt for confirmation (Are you sure you want to kill server? (y/n):), unless you specify --force.

Syntax

The killServer command can take the following arguments:

ArgumentArgument long nameMandatoryDescriptionRestricted valuesDefault
-s HOSTNAME [HOSTNAME ...] --hostname HOSTNAME [HOSTNAME ...] noWhere the application is running on more than one node, this identifies the node where you want to kill the server (so you can kill a server on a different node). Specify the Host Name, Host Names or "cluster" for all hostsnonenone
-f --force noforcefully kills a process (using kill -9)nonenone
--all nokills all processes, including GENESIS_CLUSTERnonenone
-c --cluster nokills the server on all the nodes in the clusternonenone

In the example below, there is a single node; this command kills all processes for the application, including GENESIS_CLUSTER.

killServer GENESIS_ROUTER --all

In the example below, there is a cluster; this command kills the nodes HARRY and GARY.

killServer GENESIS_ROUTER --hostname HARRY, GARY

LogLevel

To change the logging levels by code (dynamically) on any Genesis process, use the LogLevel command.

Syntax

The LogLevel command can take the following arguments:

ArgumentArgument long nameMandatoryDescriptionRestricted valuesDefault
-c --class <class name> nochanges log level on the defined classnonenone
-DATADUMP_NACK_OFF nochanges log level to INFO for Genesis messagesnonenone
-DATADUMP_NACK_ON nochanges log level to TRACE and captures only _NACK messages from Genesis messagesnonenone
-DATADUMP_OFF nochanges the log level to info for Genesis messagesnonenone
-DATADUMP_ON nochanges the log level to trace for Genesis messagesnonenone
-h --help noshow help on the commandnonenone
-l --level <log level> nolog level - if log level is not correct, it will be set automatically to DEBUG levelnonenone
-p process-name,..,process-name noattaches processes to the commandnonenone
-r process-name,..,process-name`noremove processesnonenone
-STATUSDUMP_OFF nochanges the log level to info for status updatesnonenone
-STATUSDUMP_ON nochanges the log level to trace for status updatesnonenone
-t <time> noduration of log level change in min/sec Eg: 1m, 1000snonenone

The example below sets the logging level for the GENESIS_AUTH_DATASERVER process to TRACE.

LogLevel -p GENESIS_AUTH_DATASERVER -l TRACE

The example below resets the LogLevel for the GENESIS_CLUSTER process.

LogLevel -r GENESIS_CLUSTER

The example below changes the log level to TRACE and captures only _NACK messages from Genesis messages for all processes.

LogLevel -DATADUMP_NACK_ON

MigrateAliases

This migrates the Genesis alias store from database storage to file storage and vice versa. This is useful for debugging when you have FDB database technology.

Syntax

The MigrateAliases command can take the following arguments:

ArgumentArgument long nameMandatoryDescriptionRestricted valuesDefault
-h --help noshow help on the commandnonenone
-o --override nooverwrite existing alias store if it exists in the destinationnonenone
-dst --destination yesdestination dictionary storeDB or FILEnone

The example below migrates the Genesis alias store to DB. It overwrites any existing dictionary in the destination.

MigrateAliases -dst=DB

Database technology

FDB implementations use internal aliases for fields and tables. Migrating these aliases from database to a file will help to debug problems in the data storage.

  • If you are running Genesis on a single node, use a file store.
  • If you are running Genesis on more than one node, use database mode.

The remap operation updates the alias store, so if you are running a Genesis cluster it is better to use a database storage mode; this is less open to error, and you won't have to copy the alias storage file to the remaining nodes manually.

MigrateDictionary

This migrates the Genesis dictionary from the Database Dictionary Store to the File Dictionary Store storage and vice versa.

The MigrateAliases command can take the following arguments:

ArgumentArgument long nameMandatoryDescriptionRestricted valuesDefault
-h --help noshow help on the commandnonenone
-o --override nooverwrite existing alias store if it exists in the destinationnonenone
-dst --destination yesdestination dictionary storeDB or FILEnone

The example below migrates the Genesis dictionary from DB to FILE, it also overwrites any existing dictionary in the destination

 MigrateDictionary -dst=FILE -o

The script uses the system definition file to discover the DictionarySource property:

  • If the property is DB (if the server uses a Database Dictionary Store), the MigrateDictionary script saves the dictionary to a file.
  • If the DictionarySource is FILE (if the server uses a File Dictionary Store), the dictionary is saved to a database. The target database type - DbLayer - is also retrieved from the system definitions file.

Here is a recommendation:

  • Use a file store (set by default) if you are running Genesis on a single node
  • Use database store if you are running Genesis on more than one node

The remap operation updates the dictionary, so if you are running a Genesis cluster, it is better to use a Database Dictionary Store; this is more robust and you won't have to copy the dictionary file manually to the remaining nodes.

warning

It is potentially dangerous to switch the DictionarySource property.

If you run remap (which modifies the dictionary) after MigrateDictionary and before switching the DictionarySource property, the file store and database store could contain different dictionaries and it is not safe to switch between them.

mon

This command shows the status of the overall system, so you can see if the server is up or not. By default, it displays a snapshot. If you want to the screen to be refreshed, simply add a flag with the number of seconds between refreshes, such as mon -10.


***********************************************************************************************

Start: 2023-11-09 10:04:03 Uptime: 3 minutes
Date: 2023-11-09 10:07:23 Daemon status: OK

PID Process Name Port Status CPU Memory Message
===============================================================================================
547477 GENESIS_AUTH_CONSOLIDATOR 8005 STANDBY 27.50 2.80
547348 GENESIS_AUTH_DATASERVER 8002 HEALTHY 39.90 3.70 ;Lmdb currently uses 1% of available space (total size 2GB)
547308 GENESIS_AUTH_MANAGER 8001 HEALTHY 65.50 4.60
547390 GENESIS_AUTH_PERMS 8003 HEALTHY 35.00 3.50
547436 GENESIS_AUTH_REQUEST_SERVER 8004 HEALTHY 34.10 3.40
547520 GENESIS_CLUSTER 9000 HEALTHY 43.20 4.10 ;Lmdb currently uses 1% of available space (total size 2GB)
547567 GENESIS_ROUTER 9017 HEALTHY 33.20 3.50
===============================================================================================
n/a ALPHA_CONSOLIDATOR 11002 MISSING n/a n/a Daemon missing
547644 ALPHA_DATASERVER 11000 HEALTHY 34.20 3.90 ;Lmdb currently uses 1% of available space (total size 2GB)
547771 ALPHA_EVALUATOR 11003 STANDBY 25.50 2.80
547686 ALPHA_EVENT_HANDLER 11001 RUNNING 47.90 4.10

By default, the command shows the following columns of information:

ColumnDescription
PIDThe process ID, which is generated by the system
Process NameThe process name
PortThe port where the process is operating; you can change the defaults in the process.xml file
StatusThe status of the process (see below)
CPUThe percentage of CPU being used by the process. For multi-core CPUs or multiple CPUs, the total percentage of all processes can exceed 100
MemoryThe percentage of RAM being used by the process
MessageDisplays any relevant message generated by the process

Process statuses

The key column in the display is Status, which can have the following values:

  • RUNNING the process has just started and is still initializing.
  • STARTING the process has completed initialization; it is running but hasn't yet signalled to the monitor that it has completed its internal health checks.
  • HEALTHY the process is fully available.
  • STANDBY the process is in standby mode; it will take it over if the node becomes Primary
  • MISSING there has been an error and the process has not started correctly.
info

If you are using a development environment and you are running a local build via the IntelliJ plugin, you cannot run the mon command. The plugin provides a Mon tab where you can monitor the status of each process and start and stop each one.

Syntax

The mon command can take the following arguments:

ArgumentArgument long nameMandatoryDescriptionRestricted valuesDefault
-h --help noshow help on the mon commandsnonenone
-v --version noshow installed product versionsnonenone
-c --cfg noshow the config files used by each processnonenone
-a --all noshow all informationnonenone
-m --monitors noshow state monitors (the Message column heading becomes Monitor)nonenone
-u --unhealthy noshow unhealthy processes onlynonenone
--down noshow DOWN processesnonenone
--warning noshow WARNING processesnonenone
--error noshow ERROR processesnonenone
--unknown noshow UNKNOWN processesnonenone
--missing noshow MISSING processesnonenone
--running noshow RUNNING processesnonenone
--starting noshow STARTING processesnonenone
--standby noshow STANDBY processesnonenone
--healthy noshow HEALTHY processesnonenone
info

Unhealthy processes includes all processes that are not HEALTHY or STANDBY. --unhealthy can not be used with other status-filtering arguments.

The example below runs mon with a polling interval of one second.

mon 5

The example below runs mon; it shows the status of every process and outputs the config files used by each process.

mon --cfg

The example below runs mon, but only shows processes that are unhealthy.

mon --unhealthy

The example below runs mon, but only shows processes that are missing.

mon --missing

PopulateHolidays

This command populates the Holidays table with holidays, based on a specific year(s), country(ies) and region(s).

Syntax

The PopulateHolidays command can take the following arguments:

ArgumentArgument long nameMandatoryDescriptionRestricted values
-c --country (arg) Nothe country name to search for holidaysNo
-h --help Noshow help on this commandNo
-r --region (arg) Nothe region name to search for holidaysNo
-y --year (arg) Nothe year of holidaysNo

For example:

PopulateHolidays -y 2020,2021 -c BR,GB -r rj,en

PurgeTables

This enables you to define the data-retention policy. Data will be removed from the database where the defined criteria are fulfilled.

Example: For the TRADE table, you could decide to keep allocated trades for 60 days, and 30 days for the rest.

To use this tool, you must have a *-purger.kts file in the application's cfg`` folder. Invoking the **PurgeTables** command will pick up all files that are found in the $GENESIS_HOME/generated/cfg/` directory.

In order to enable syntax highlighting and autocompletion for purger files, you must add genesis-environment as a dependency of your application's -config module. See simple examples below for purger definitions:

A log file called purge_<time_of_run> will be created under the $GENESIS_HOME/runtime/logs/ folder.

The functions and filters below give you different ways of purging data.

Inject repository

You can inject any repository into the purger script; for example:

purgers{
val userSessionRepo = inject<UserSessionRx2Repository>()
}

Purge by date

The purger supports purging based on days. You can specify the max age of a record in terms of either calendar days or business days.

  • Business days disregard weekends and public holidays.
  • Calendar days disregard weekends by default, but can be set to include them, as shown in the example below.

To use this script, you need to supply

  • a LONG or DATETIME field in the table to be used, or you can supply just the table name, in which case the TIMESTAMP field of the table will be used to calculate the age of the record.
  • The max age of the record, which indicates all the records older than these days will be purged
  • Optional fields when you use calendar days: boolean flag ignoreWeekends (defaults to true).
  • Optional fields when you use business days: country and region name.
purgers {
// purge trades over 180 days old based on field TRADE_DATE, not taking weekends into account (180 week-days)
daysPurger(TRADE.TRADE_DATE, 180)
// purge trades over 6 months old based on field TIMESTAMP of table which will be used internally, including weekends (180 full calendar days)
daysPurger(TRADE, 180, ignoreWeekends = false)

// purge prices older than 5 business days based on PRICE_DATETIME
businessDaysPurger(PRICE.PRICE_DATETIME, 5)
// purge prices older than 5 business days based on PRICE_DATETIME in country SPAIN
businessDaysPurger(PRICE.PRICE_DATETIME, 5, "SPAIN")
}

Purge by range

Range purger is similar to bulkPurger, but it performs efficient index range searches in database and speed up purger performance compared to bulkPurger.

  • You need to give index name of the table you want to purge data on
  • Purger has special function called whereRange which is used to filter based on index value
import global.genesis.gen.dao.repository.UserSessionRx2Repository
purgers {
// Inject repo
val userSessionRepo = inject<UserSessionRx2Repository>()

// Range purger with whereRange
rangePurger(USER_SESSION.BY_USER_NAME).whereRange("JohnDoe")

// Combination of filters for maximum flexibility
rangePurger(USER_SESSION.BY_USER_NAME)
.whereRange("JohnDoe")
.filterBusinessDays(maxDays = 5, country = "SPAIN", region = "NATIONAL", field = USER_SESSION.LAST_ACCESS_TIME)

rangePurger(USER_SESSION.BY_USER_NAME)
.whereRange("JohnDoe")
.filterDays(maxDays = 5)

// finally clause usage
rangePurger(USER_SESSION.BY_USER_NAME)
.whereRange("JohnDoe")
.finally { userSession ->
println("Purged record: ${userSession.toGenesisSet()}")
}
}

Purge bulk data

You can purge data from the whole table using:

bulkPurger(USER_SESSION)

You can purge data from the table based on some conditions:

purgers {
// Purger that reads the whole user session every time it runs and deletes all sessions for JohnDoe
bulkPurger(USER_SESSION).filter { it.userName == "JohnDoe" }
// Purge USER_SESSION older than 8 business days in country India
bulkPurger(USER_SESSION)
.filterBusinessDays(8, "INDIA")
}

Filters

The following filters are used in the examples explained above:

whereRange: This filter can contain value of the table-index parameter or parameters if there are more than one field in the table-index. Parameter list can range from 1-10. It filters the records based on the index parameter value/values

filter: filter based on predicate provided

filterBusinessDays: this method allows you to purge data based on business date. You need to provide:

  • max age of record
  • country name
  • region which defaults to "NATIONAL"
  • and optional LONG or DATETIME field of table you want to purge and if not specified TIMESTAMP field of table is used

filterDays: this method allows to purge data based on calendar date You need to provide:

  • max age of record
  • and optional LONG or DATETIME field of table you want to purge and if not specified TIMESTAMP field of table is used

finally clause: It is run for every record that is purged and used to add some extra functionality if needed

ReconcileDatabaseSync

This is used to check if there are differences between a local DB and a remote DB with common dictionary tables.

Typically, this would be used to reconcile tables that are being kept in sync by the GENESIS_SYNC process.

The local DB's details (host, port, user, etc) are read from the system definition file in the local environment. The remote DB's details are specified as options to the command.

The tables to check are specified in the genesis-sync-definition.xml file. Here is a simple example that specifies two tables:

<sync>
<tables>
<table name="TRADE"/>
<table name="ORDER"/>
</tables>
</sync>

If there are any differences found between the local and remote tables, then the result will be output to a text file. The location of this text file will be displayed on screen. There are 3 categories in the output: "Records Missing From Local", "Records Missing From Remote" and "Records Which Differ Between Remote and Local".

Records are compared using the primary key field. The TIMESTAMP and RECORD_ID fields are extremely likely to differ, because the tables being compared are in separate databases. Therefore, these fields are not compared.

Arguments
ArgumentArgument long nameMandatoryDescriptionRestricted Values
-d --dblayer trueDatabase Layer typeYes: FDB, FDB2, SQL, SIMPLE
-f --fdb falseFDB cluster file nameNo
-H --host falseRemote DB hostnameNo
-P --port falseRemote DB portYes: Number > 0
-u --username falseDB user usernameNo
-p --password falseDB user passwordNo
-s --nullstring falseEvaluate null and empty strings as equalNo value required
-n --numdays falseOnly compare records with timestamps between now and the number of days specified. If either DB has matching records outside this timestamp range, this will not be flagged as a reconciliation differenceYes: Number > 0
-i --ignorefields falseComma-separated list of additional fields to ignore ("RECORD_ID" and "TIMESTAMP" are always ignored)No
-h --help falseShow usage informationNo
Examples

This is a simple run with a remote Postgres DB:

ReconcileDatabaseSync -d SQL -H "jdbc:postgresql://dbhost:5432/" -u dbuser -p dbpass

This example runs with a remote Postgres DB. it evaluates null and empty strings as equal; it compares records up to 2 days ago, and it ignores the field STATUS.

ReconcileDatabaseSync -d SQL -H "jdbc:postgresql://dbhost:5432/" -u dbuser -p dbpass -s -n 2 -i STATUS

remap

Remap is a schema-migration tool used to apply the current schema (defined in the deployed field and table GPAL dictionaries) to the underlying database layer used by the Genesis Application Platform.

The remap command performs the following tasks:

  • It reads all dictionary files (fields.kts, tables.kts and view.kts) from $GC and compares these to the previously generated schema. It uses these changes to remap the memory-resident database.
  • It generates dao objects based on the dictionary tables, so you can perform database operations in a type-safe way.
  • If you are running FDB, it updates the Genesis alias store.

If you run remap with no arguments, it simply gives a report of changes that exist in the configuration.

If you want to commit the changes to the database, you must use the --commit argument:

remap [-c | --commit]

For full details, see our page on the remap command.

RenameFields

This command is used to rename a field name in a database without changing the dictionary or config files.

Syntax

The RenameFields command takes two arguments; both of which are mandatory:

RenameFields [-i <[current name of field]>] [-o  <[new name of field]>]
ArgumentArgument long nameMandatoryDescriptionRestricted valuesDefault
-i --input yesname of field that you want to changenonenone
-o --output yesname you want the field to be changed tononenone
-h --help noshow help on using this commandnonenone

The --input argument represents the name of the field you would like to change. The argument must be an existing field name in the database.

The --output argument represents the name of the field you would like to change to. The argument must also be an existing field name in the database.

Both arguments must also be of the same type.

If both arguments are in the same table, it would result in the --output field being deleted.

All changes using RenameFields can be changed back to the original database schema by using the command remap --commit.

For example:

RenameFields -i SYMBOL -o TRADE_ID

This changes the name of the SYMBOL field to TRADE_ID.

Another example:

RenameFields --input FIRST_NAME --output FNAME

This changes the name of the field FIRST_NAME to FNAME.

Invalid example:

RenameFields -i PRICE -o FIRST_NAME

This would result in an error, as PRICE is of type DOUBLE while FIRST_NAME is of type STRING.

SendIt

To send data into the database (inserts, modifies and upserts), use the SendIt command.

The data you want to send must be in a .csv file or files:

  • The file must be in the directory that you are running the command from.
  • The name of the file (or files) should match the name of the table where the new or amended data is to be sent. If you want to use a different filename, you must specify this using the -f argument.
info

Make sure your data is in the correct format before sending to your target table. For example:

  • DATE values should be in the format of yyyy-MM-dd (for example 2026-05-25).
  • DATETIME values should be in the format of yyyy-MM-dd HH:mm:ss.SSS Z (for example 2026-05-25 12:30:01.123 +0000).

If you're having trouble sending in data, use the verbose option with -v to get a row-by-row report!

Syntax

The SendIt command can take the following arguments:

ArgumentArgument long nameMandatoryDescriptionRestricted valuesDefault
-a --all noimport all the tables from all the csv files to the databasenonone
-d --delete noperform delete operations on all recordsnonone
-cf --columnFormat noset specific date format for column, for example -cf DATE=yyyyMMddnonone
-f --file (arg) noname of the csv file that contains the datanoGenesis looks for a new .csv file whose name matches the name of the source table
-fm --formatMode (arg) noFORMATTED takes field formats into account; LEGACY does notFORMATTED and LEGACYLEGACY
-h --help noshow help on how to use this commandnonone
-m --modify (arg) nokey name used to find original recordnonone
-mf --modifyFields (arg) nospecifies fields to modify (only used with -m)nonone
-quiet --quietMode nomake database changes without triggering real-time updates in update queue layernonone
-r --recover noperform recover operations on all records; this is a special operation meant to preserve the original timestamps; use with caution. Only use this when you want to restore a system after completely erasing the database tables. You must use only untouched files from a real back-up of the original dataset. There are no other circumstances in which you should use this option. Evernonone
-t --table (arg) yesthe name of the database table to be updated or amendedmust be a valid tablenone
-u --upsert (arg) noupserts records from a single csv filenonone
-ua --upsertAll (arg) noupserts records from multiple csv filesnonone
-v --verbose nolog every error line to outputnonone

For example:

SendIt -t FUND -f FUND.csv

This reads the FUND.csv file in the local directory and inserts the data from the file into the FUND table.

Modifying and upserting

info

You can use SendIt to make changes to the database while the application is running; Data Servers and Consolidators will process any changes in the normal way (for example). But if you are in any doubt that your changes might clash with other changes happening in the application, you should switch off all processes before using SendIt.

To modify records, create a .csv file of the records that need to be modified, including the key value and the modified values for each record. Then use SendIt with the -m flag to specify the key that identifies the record or records to be modified. You cannot modify the key that you supply here - although you can modify other key fields (with care). For example, you can't change an ID in the file and then modify on_BY_ID key.

The following example:

  • looks for a file called FUND.csv by default.
  • compares each record in the FUND table with the record that has the same ID in the csv file
  • applies any changes in the csv record to the record in the table
SendIt -t FUND -m FUND_BY_ID

Modify fields (-mf) is an extra parameter that can be added to -m operations. SendIt only attempts to modify the record fields specified in this comma-separated list. For example:

SendIt -t ALL_TRADES -m TRADE_BY_TRADE_ID -mf TRADE_PRICE, TRADE_QUANTITY

To upsert records, create a .csv file of the records that need to be modified or inserted, including the key value and the relevant values for each record. Then use SendIt with the -u flag to specify the key that identifies the original record from each row in the csv file.

The following example:

  • looks for a file called FUND.csv by default
  • for each record in the FUND table that has a matching ID in the csv, it applies the csv changes to the table
  • for any record in the csv file that has no matching ID in the FUND table, the record is inserted into the table
SendIt -t FUND -u FUND_BY_ID

Deleting

To delete records, specify -d (or --delete)

SendIt -t FUND -d

Verbose mode

Verbose mode additionally outputs line-by-line operation outcome, and a final summary of error lines to be corrected and resubmitted. This makes SendIt useful for scheduled or automated jobs (e.g. daily data loads).

warning

Do not use SendIt to update User details in any way. This can easily cause database errors. To update User profiles or User attributes, only use Genesis user entity management.

SetAutoIncrement

warning

Stop all your application's processes before using this command.

This command enables you to set the next number to be generated for one or more auto-incrementing fields.

You can set the value for a single field using the -v argument.

You can set values for more than one field by supplying the details in a csv file and using the -f argument. The file should take the following format:

table, field, value
TRADE, TRADE_ID, 1
ORDER, ORDER_ID, 10538

By default, this command expects the file to be called AutoIncrementValues.csv, but you can specify a different file name.

Syntax

The SetAutoIncrement command can take the following arguments:

ArgumentArgument long nameMandatoryDescriptionRestricted valuesDefault
-f --file (arg) noname of the csv file containing the new sequence numbers (this overrides any --value option supplied)noneAutoIncrementValues.csv
-h --help nodisplays help on using this commandnonenone
-s --field (arg) noname of the auto-increment field (when setting the next sequence number for a single field)nonenone
-t --table (arg) noname of the table that has the auto-increment (when setting the next sequence number for a single field)nonenone
-v --value (arg) nonew sequence number (integer) for the auto-increment (when setting the auto-increment for a single field)nonenone

Note that all fields are optional in principle. But when setting a single field value, you must provide -t -s and -v.

The example below sets the number 101 as the next number to be generated for the TRADE_ID field in the TRADE table.

SetAutoIncrement -t TRADE -s TRADE_ID -v 101

The example below uses the file autoIncVals to set auto-increment values for multiple fields.

SetAutoIncrement -f autoIncVals.csv

SetSequence

warning

Stop all your application's processes before using this command.

This command enables you to set the next sequence number for one or more sequence fields.

  • To set the value for a single sequence, use the -v argument.

  • To set values for more than one sequence, supply the details in a csv file (for example, a file that you have exported using either GetNextSequenceNumbers or GetSequenceCount) and use the -f argument. The file should take the following format:

"Table","Sequence","Value"
"USER","US","1303"
"PROFILE","PR","0"
"RIGHT","RI","0"

SetSequence must only be run when the system processes have been stopped. After running SetSequence, you need to restart the server.

Syntax

The SetSequence command can take the following arguments:

ArgumentArgument long nameMandatoryDescriptionRestricted valuesDefault
-f --file (arg) noname of csv file containing table-sequence-value sets (these override any --sequence and --value options supplied)nonenone
-h --help nodisplays help on how to use the commandnonenone
-s --sequence (arg) notwo-character ID for the sequence (if setting an individual value)nonenone
-t --table (arg) noname of the table that contains the sequence (if setting an individual value)nonenone
-v --value (arg) nonew integer value to be set (if setting an individual value)nonenone

All fields are optional in principle. But if you are setting the value for a single sequence, you must provide a -t -s and -v.

The example below sets the sequence AA in the table TRADE for the TRADE_ID field; the next sequence number to be generated is set to 1001.

SetSequence-t TRADE -s AA -v 1001

The example below uses the file SeqVals to set the next sequence number for multiple sequences.

SetSequence -f SeqVals.csv

startProcess

This command starts a Genesis process.

Syntax

The command must be followed by the name of the process that you want to start.

The following arguments are also available:

ArgumentArgument long nameMandatoryDescriptionRestricted valuesDefault
-s HOSTNAME [HOSTNAME ...] --hostname HOSTNAME [HOSTNAME ...] nowhere the application is running on more than one node, this identifies the node where you want to start the process (so you can start a process on a different node). Specify the Host Namenonenone
-v --verbose nostarts in verbose mode; logs to the terminal so you can view on screennonenone
--dump nodisplays progress of the process, which is useful for debuggingnonenone

The following example starts the process called APP_EVALUATOR:

startProcess APP_EVALUATOR

The following example starts the GENESIS_CLUSTER process and displays any useful debugging information in the console:

startProcess GENESIS_CLUSTER --dump

The following example starts the APP_NOTIFY process on node1 only:

startProcess APP_NOTIFY -s node1

The script looks in the processes.xml file (see startServer below) to find out how to start the process. For example startProcess AUTH_DATASERVER starts the process with the correct classpath and extra arguments. Something similar to:

java -Xmx256m -DXSD_VALIDATE=false global.genesis.dta.dta_process.DtaProcessBootstrap -name AUTH_DATASERVER -scan global.genesis.dta.dataserver -module dataserver -config auth-dataserver.xml -loggingLevel INFO,DATADUMP_OFF >/dev/null 2> $L/AUTH_DATASERVER.log.err &

startServer

This command starts the server of an application. It reads the $GC/processes.xml file to determine which processes to start and how to start them.

Syntax

The startServer command can take the following arguments:

ArgumentArgument long nameMandatoryDescriptionRestricted valuesDefault
-s HOSTNAME [HOSTNAME ...] --hostname HOSTNAME [HOSTNAME ...] noif the application is running on more than one node, this identifies the node where you want to start the server (so you can start a server on a different node). Specify the Host Namenonenone
-c --cluster nostarts the server on every node in the clusternonenone
-i --ignoreDaemon noavoids killing/starting the daemonnonenone
-v --verbose nostarts in verbose mode; logs to the terminal so you can view on screennonenone

The example below starts the server.

startServer

The example below starts the server on node1 without starting/killing the daemon.

startServer  -s node1 -i

The processes.xml file looks like this:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<configuration>
<process name="GENESIS_AUTH_MANAGER">
<start>true</start>
<options>-Xmx512m -DXSD_VALIDATE=false</options>
<module>auth-manager</module>
<package>global.genesis.dta.auth.manager</package>
<classpath>quickfixj-core-2.1.0.jar</classpath>
</process>
<process name="GENESIS_AUTH_DATASERVER">
<start>true</start>
<options>-Xmx512m -DXSD_VALIDATE=false</options>
<module>dataserver</module>
<package>global.genesis.dataserver</package>
<config>auth-dataserver.xml</config>
<loggingLevel>INFO,DATADUMP_OFF</loggingLevel>
</process>
<process name="GENESIS_AUTH_PERMS">
<start>true</start>
<options>-Xmx512m -DXSD_VALIDATE=false</options>
<module>auth-perms</module>
<package>global.genesis.dta.auth.perms</package>
<dependency>AUTH_MANAGER,AUTH_DATASERVER</dependency>
</process>
</configuration>

Each process property is defined in here, including Java arguments, configuration files, and scripts.

The dependency tag defines the processes that the current process is dependent on. In the above example, the GENESIS_AUTH_PERMS process will start after all its dependencies have started.

The loggingLevel tag defines the default log level for the process, which is based on slf4j levels. It also accepts DATADUMP_ON/DATADUMP_OFF to declare explicitly that you would like to log all the received/sent network messages.

The classpath tag defines additional jar files that might be needed by the microservices. The jar files declared in this section have to be comma-separated; they need to exist within a lib folder for one of the genesis products in the environment. A use case would be to use the quickfixj library to parse a fix message within a query definition.