A good design of a solution involving application or data integration should account for handling extraneous conditions resulting from encountering bad data or process execution level errors at runtime. Throwing off these errors into a file or a database and hoping someone would get to it later is typically how these issues are handled. What you really need is a way to resolve these errors while the data is being processed in realtime or in other words during the execution of the process flow. Not able to resolve errors at realtime would result in data integrity issues in the target application such as out of sequence or partial record updates, downstream failures in dependent systems and much more.
So how does one handle realtime data exceptions in Adeptia?
Well the answer is simple. Adeptia has multi-pronged approach by providing sophisticated error-handling features in dealing with exceptions and I will discuss these features in detail in this article. First let’s review the various types of data integration errors and let's try to categorize them into two broad categories.
Type of Errors in Application Integration
If you think about it, the errors in data integration thrown at runtime are of various types such as:
- Data does not match the schema defined for the incoming/outgoing message
- Expecting a flat delimited file but received fixed width data
- Web Service request format is different than how you have defined in REST JSON or in the SOAP XML
- Overall format is correct but some of the records fail due to data type mismatch issues (i.e., date field contains text rather than date, data truncation, invalid record identifiers, special characters, invalid encoding and so on)
- Data fields don’t match with what your mapping was expecting
- EDI message type is not compliant with a particular X12 standard
- Data from UDP is not in the correct binary/hex format
- And there are so many other examples that can relate to these semantic errors
- Failure to connect to source or target applications such as database not available, web service not responding, expiry of cloud API token, authentication errors and so on
- Invalid key to encrypt / decrypt data
- More importantly data is not compliant with all the business rules that are applied on it at runtime
- During the process flow execution you encounter data records that do no meet certain rules such as a Purchase Order’s total amount is less than what your system was expecting and so on
- Data is incomplete because it’s missing key field values
- Enumeration value in the data field does not match the accepted list of codes that are defined by your business
- The aggregate sum of all the amounts in the detail records do not balance with the total value in the trailer record
- And there can be so many other errors that are thrown when you apply your business rules on the data
Based on these two types of errors as explained above we can thus summarize the errors into two broad categories:
- Semantic errors - that result from the structural discrepancies in the data
- Process execution level errors - that result from applying business rule validations on the data or due to failure in connecting to systems at runtime
So how does one handle these errors in Adeptia? Let’s go through different error-handling features that are available in Adeptia Integration Suite.
Different ways of resolving Errors in Application Integration
Adeptia Integration Suite provides several features that can be used while designing the process flow to handle runtime errors. I will list them here:
- Automatic email notifications if any step in the process flow fails with description as to the root cause of the exceptions
- Route errors as a data stream to an error correction or reporting activity such as a database, sub-process, web service, files etc.
- Route errors to a workflow task and visually show these errors in a web form in order to allow the end-user correct and resubmit the data back into the process flow
- Automatic connection retries after initial attempt to connect to an application fails
- Gateway conditions to verify if certain business rules are being met else route the data to a different sub-process or activity
- Custom exception handling scripts that can be used at runtime to cleanse bad data
- Logging errors in the Monitoring dashboards with error descriptions
In the following section we will pick four examples that relate to some of the features listed above.
Example 1: Routing errors to a workflow task for correction and resubmission
In this example we will take Customer Enrollment data files arriving from source application and map them to a back-end database. As part of the validating the inbound data we will apply transformation rules to map the incoming data to the desired output format expected by the database. If there are any errors in the data such as missing Coverage Code, those errors will be routed to a workflow task assignment for a user to correct those errors and resubmit.
Workflow task highlighted above shows the data in a web form. User opens up the task and sees the errors and makes corrections as shown below.
Here's a video that describes how to build this solution (~25 minutes).
Example 2: Routing errors to an email notification
In this use case we will show how to identify and route data errors to an email notification. As part of error handling scenarios users can build additional types of error handling activities such as routing exceptions to workflows, sub processes, status table in a database or to a web service. The process designer provides developers with flexible design capabilities to automate their error handling as per their business rules.
Here’s a video that describes the above solution in detail (~30 minutes).
Example 3: Routing errors to dashboards for monitoring and for taking corrective actions
Another common use case is to take those errors and put them into a database with a specific status so that your team is alerted on the existence of these errors and can fix them right away.
Here’s an example of a database view using the web form that shows the number of transactions in the incoming message and their status. Clicking on ‘Failure’ would allow user to see the diagnostics and corrective actions needed to fix the error.
Here’s a video that shows the different dashboards users can see to monitor runtime status of transactions (~5 minutes).
Example 4: Logging Error records and transport errors
There are additional features that are available out-of-the-box that do not require further configuration in the orchestration design. For example, if you select Email notification in the End Event of the flow it would automatically send error notification email in case of process flow failure. You can enable ErrorIntermediateEvent property that would allow you to skip the rest of the activities and route the error to another activity in the flow. Process Flow logs capture all types of errors; it also shows error record count and the root cause of those failures. Using Gateway "decision" nodes one can use these context variables to determine which process route to take at runtime.
By using these automated error handling features you can eliminate the time and effort involved in fixing data errors manually. I hope this article was helpful in understanding the types of errors one can face during application integration scenarios and by using a product like Adeptia you can easily onboard your customer data, handle different data issues, and deliver services to your customers quickly.
Support forum provides a rich reference material on how to configure these error handling features. Please use the search feature to explore and learn. Also refer to the Developer Guide.