Handling Large JSON Payload Errors (Jackson StreamReadConstraints Limit)

While processing large JSON payloads (especially those containing Base64 data), you may encounter errors like:

String value length exceeds the maximum allowed (20000000)

or

Unable to parse source data as per defined schema, please verify your source schema definition:
com.fasterxml.jackson.core.exc.StreamConstraintsException:
String value length (20051112) exceeds the maximum allowed (20000000, from `StreamReadConstraints.getMaxStringLength()`): Error while reading the JSON

🔍 Why this happens

By default, Jackson has a limit of around 20MB for string values (via StreamReadConstraints).
This is done to prevent excessive memory usage.

If your JSON contains large Base64 data (e.g., ~20MB+), it will fail once this limit is exceeded.

✅ Solution

This can be resolved by increasing the Jackson string length limit using a custom plugin.

✔ Steps:

1. Create a custom plugin (e.g., ExtractJsonCP)
2. Add the following code:

import com.fasterxml.jackson.core.StreamReadConstraints;

try {
    // Set global Jackson string length limit to 25 MB
    StreamReadConstraints.overrideDefaultStreamReadConstraints(
        StreamReadConstraints.builder()
            .maxStringLength(25000000) // 25 MB in bytes
            .build()
    );

} catch (Exception t) {
    // Log any exception that occurs while setting the limit
    service.getLogger().error("Failed to set Jackson limit: " + t.getMessage());
}

3. Call this plugin before the JSON Layout activity in the process flow

⚠️ Important Note

• The custom plugin should be used only to set the Jackson limit and should not consume or generate any stream
• Ensure “Consume Stream” and “Generate Stream” are disabled in the plugin configuration
• The source stream should be directly passed to the JSON Layout activity

👉 The plugin should act only as a configuration step, without modifying the data flow.

⚙️ Configuration

The limit can be adjusted as needed:

.maxStringLength(25000000)

• Value is in bytes
• Example: 25000000 ≈ 25MB

⚠️ Important Considerations

• This is a global setting and applies to all JSON parsing in the same JVM
• Avoid setting very high values unnecessarily
• Increasing the limit too much may lead to:
  • Higher memory usage
  • Performance impact under heavy load

✅ Recommendation

Set the limit slightly above your actual payload size (for example, add a small buffer of a few MB) instead of using very large values.