Building Dynamic Data Mapping Solution for Parsing JSON Payloads with Varied Field Names Across Different Client Systems in WSO2 Micro Integrator

When developing integration solutions, it is often necessary to handle input source keys or field names that vary between different systems. Unfortunately, the current data mapping solution offered by WSO2 requires defining the input and output fields during the developement phase, without the flexibility to load a dynamic input configuration file during runtime. As a result, this limitation forces the creation of multiple datamapper projects for each integrating system, adding extra development and deployment efforts for each integrating client system.

While there is no direct method available, we can accomplish this task through a non-standard approach. The data mapper operates using a “.dmc” file, which contains JavaScript code to facilitate data mapping. By modifying this “.dmc” file, we can insert the necessary code to load a list of dynamic field names, achieving the desired functionality.

Step 1: Create sample JSON file

Create a JSON file with below data

{

“number”:”110001″,

“description”: “unable to create order”

}

<property expression=”json-eval($)” name=”payload” scope=”default” type=”STRING”/>

Step 2: Define Datamapper

Begin the data mapping process by providing the payload file from step 1 as input. Use the same sample file for both input and output purposes. Next, navigate to the file created in the registry resource project, having the “.dmc” extension, and open it. Inside, you will find the code provided below.

map_S_root_S_root = function(){ 

var outputroot={};

outputroot = {};

outputroot.number= inputroot.number;

outputroot.description= inputroot.description;

return outputroot;

};

Step 3: Modify .dmc file

Now, change the JavaScript code in the “.dmc” file to enable dynamic retrieval of input field names from a JSON key-value map.

Note: After making modifications to the .dmc file, refrain from opening the file or making changes in Integration Studio, as doing so will override custom mapping code with the default settings.

map_S_root_S_root = function(){ 

var outputroot={};

outputroot = {};

outputroot.number=getDynamicFieldName (DM_PROPERTIES.DEFAULT[‘payload’],’number‘,DM_PROPERTIES.DEFAULT[‘mappingData’]);

outputroot.description= getDynamicFieldName (DM_PROPERTIES.DEFAULT[‘payload’],’description‘,DM_PROPERTIES.DEFAULT[‘mappingData’]);

return outputroot;

};

getDynamicFieldName = function(payload, canonicalFieldName, mappingData){
var mappingJSON = JSON.parse(mappingData);
fieldName = mappingJSON[canonicalFieldName];
obj = JSON.parse(payload);
return obj[fieldName];

};

The above provided code introduces a fresh function called getDynamicFieldName, requiring three inputs: the payload to be converted, the mapping data, and the canonical field name to extract from the map. To incorporate this function, substitute the original inputroot.fieldnames in the initial .dmc code. By doing so, the JavaScript logic will retrieve dynamic field names from the mapping data and appropriately process the input payload.

Step 4: Create Dynamic Field mapping payloadfactory

Within the API/Sequence file, before invoking the datamapper mediator, generate a Key-Value payload to establish a correspondence between input field names of integrating client system and the canonical field names specified in the .dmc file. Store this payload as a property. This property will be utilized within the .dmc file to facilitate the parsing of the input payload.

Please take note: In this post, we have directly hardcoded mapping payload here; however, in an ideal scenario, it should be stored in a database. It is recommended to maintain separate input mapping payload for each client system in database and retrieve them based on the input system of the respective client.

<payloadFactory media-type=”json”>
<format>
{
“number”:”incident_no”,
“description”:”desc”

}

</format>
<args/>
</payloadFactory>

<property expression=”json-eval($)” name=”mappingData” scope=”default” type=”STRING”/>

Step 5: Testing the program

Time to test the implementation, call the API with below input payload

{

“incident_no”:”INC_12801″,

“desc” : “Unable to open the website”

}

The API with help of custom dynamic datamapper will convert the above payload as below

{

“number”:”INC_12801″,

“description” : “Unable to open the website”

}

Using this approach, you can generate a fresh Datamapping payload for each individual client system and carry out the conversion. This eliminates the creation of New Datamapper code for each integrating system.