Introducing the Synplity Platform through the Camunda Experience
Recognizable general concepts
Recognizable general concepts Camunda has standard tools (Modeler and Cockpit) for process design and administration. Projects on Camunda rarely use only the standard administration functionality, because Cockpit’s capabilities do not cover all the needs of projects, therefore, either their own analogues are created, or Cockpit is being finalized. These extensions are offered as standalone products or purchased with engine expertise. Camunda’s free tools cover most basic needs. In Synplity Platform, the basic needs of process monitoring are covered by standard Kibana dashboards and metrics available through Prometheus/Grafana. An analogue of Modeler and Cockpit in the Synplity Platform is a separate Angular application with an administration interface that uses a universal API and has the ability to customize or extend the standard solution with the necessary functionality for the needs of the customer, which happens often. In addition to front-end applications for the Synplity Platform, the method of modeling and administration through code and CLI is also relevant — using a sandbox project with a special library and a maven plugin connected. Camunda has a Process-engine that allows to use it in 3 versions: servlet-container app, embedded app (spring-boot) and standalone, as well as consuming BPMN schemas and launching processes based on them. In Synplity Platform, the same functionality is provided by SP-workbench and SP-core microservices. SP-workbench allows to manage the process graph (vertices and edges) and describe the process context (accessible context variables, data types, structures, access control). If each Process-engine keeps its process diagrams to itself, then the SP-workbench is a common repository of all diagrams, replicated and available for process modifications at runtime (in the case of Camunda, it will be necessary to ensure seamless switching to new versions of microservices, even with minor changes to the process logic). SP-core creates and executes instances of the processes available in SP-workbench. The basis of the state machine model in the SP-core is the Spring State Machine — an open and well- documented project with high quality source code, and Spring projects are close to many programmers. The vertices of the graph turn into states (states), and the edges of the graph — into transitions (transitions) between states. You can attach any logic (actions) to the vertices and transitions in the data model. The model is enriched with specific functionality for designing business processes, such as cancellation and error pseudostates, so as not to create redundant transitions in template situations. Thus, the state machine model in the Synplity Platform is created and changed on the fly, immediately starting to process instances of processes and run related logic. Camunda and Synplity Platform, as business process automation systems, have many similarities and the same features, for example, call/task delegation, interaction between processes and subprocesses, working with variables in the context of a process instance, event timers, error compensation-handling. The differences are in the details, especially in development, testing, and production experience.
The Path to Implementing a Stable Solution Infrastructure
Without third-party components or Enterprise subscriptions, Camunda is a bare engine of libraries and tools for working with the specific model and format of the BPMN standard. For a production-ready cluster, it is highly likely that you will need to either refine or purchase a ready-made set of admin panels, monitoring with tracing and logging, implement pipelines for building Docker images that are assembled into a cluster, design Process-engines with different business requirements for business requirements. — processes to interact with each other and with external interfaces, organize general management of module configurations. You also often have to work out the configuration for user authorization and Security restrictions with engine components and in the processes themselves. Thus, the advantages of Camunda include complete freedom in customization, but at the same time, the development of the project will have to be carried out practically from scratch. To help developers, there are a number of readymade tools for working with the model and concepts of the engine, as well as integrating parts of the engine with Spring. Having chosen independent development, Synplity Platform builds processes on the general principles of finite automaton and directed graph models, fully based on Spring projects. The platform comes as a ready-made infrastructure using popular tools for microservice solutions, which takes care of CI / CD needs, and adds flexible options for Security configuration in processes and interactions between them.
Working with API and engine tools
What made us disappointed in Camunda? The presence of XML in process diagrams, in resources and module configs, in writing logic, connectors, the presence of redundant and legacy functionality with a mixture of Backend & Frontend development. The Synplity Platform project is focused on SPA, and the generation of HTML templates on the Backend side is needed only in exceptional cases. In Camunda, the built-in capabilities for developing interfaces and APIs are not relevant, and all new projects simply ignore this functionality, encountering it only when extending Cockpit (this is working with Tasklist and various Forms and plugins). Camunda has a cumbersome and complex process instance model, which translates into the complexity of handling the REST API for process control and complicates the tasks of Frontend developers when developing custom UI. Another example of actually not used functionality in Camunda — we have not met a single case with extensive use of scripts and built-in EL in logic. This is obviously because these tools are difficult to understand, inconvenient to use in XML format, difficult to test, and there are many restrictions on the available functions. Development on Camunda actually becomes hardcoded based on Java Delegate and External Task in each process, which reduces development efficiency.
Diagrams and business process design
Initially, there were no complaints about BPMN 2.0 — the schemes look logical and beautiful. However, in practice, only developers fully understand them, business is not motivated to learn this notation, and analysts are very limited in their ability to understand the logic of processes — in order to understand, it is often necessary to look into the context of variables, not all logic can be seen on the diagram or figure out what it does or other challenge. To implement processes and complete immersion, you will have to memorize the meanings of shapes and icons, which becomes like learning a language based on hieroglyphs, while often only a small part of the notation is used. Often, to simplify, the scheme has to be divided into smaller and simpler blocks, which is why it grows. Complex processes often don’t fit on the screen (or are hard to see when zooming out), causing one process to be split into smaller ones. In practice, it turned out that to describe the process of a simple graph with points and connections, a basic understanding of business, analysts and developers is enough, and in a simple script / pseudocode it is often easier to understand the logic, work with context variables and call external points than in a complex scheme with icons and mapping tables. This is especially important when efforts to make everything simple and understandable do not lead to the expansion of the scheme and an increase in the number of subprocesses, but result in a simple graph with short and readable instructions in sections that follow the order of execution and are reused in different processes. The lack of a visual BPMN scheme in the Synplity Platform is compensated by the ability to conveniently search in a project with scripts and process graphs by any key identifiers: system/subprocess/attribute name, interaction type, reusable function, passed parameters, places where parallelization is used or a callback is expected, by the resulting type of asynchronous operation — all this allows you to quickly move to all parts of the logic when studying processes or solving problems, which is important for developers.
Integrations
Our frequent case in projects is the interaction of several dozen systems. This is a sequential call to systems through the steps of a process, or several calls in one step to aggregate data displayed to the user or subsequent request. What Camunda offers is a "strange" choice: add and configure a plugin in Process-engine with Rest / Soap connectors and compose XML files with request-response configuration from intended services, or manually write Java code with a service call, implementing another JavaDelegate or ExternalTask (ExternalTask will need to be adapted on the other side so that the result goes to the Process-engine queue). XML files are inconvenient to configure and test because of the lack of flexibility, which pushes you to manually program the call to external systems directly and the context parameter mappings to and from the request. This approach is costly when making edits and debugging processes, and also complicates reuse for different processes. Updates to the API of the external system will lead to a cascade of edits in the JavaDelegate and ExternalTask set. All this shifts the focus from integration for business purposes to adaptation of an external system. for the technical needs of Camunda. At the same time, the same code for accessing external systems will be duplicated in different Process-engines, and if the system is complex, then it will definitely be decided to move the main repeating or complex blocks to a separate adapter, which Process-engine instances will reuse. The Synplity Platform has some analogies with the way you work with integrations in Camunda. Instead of a plugin and XML configs for simple calls, you can use standard Rest / Soap clients familiar to Java developers directly in the scripting logic of the process, for example, Apache (HttpClient) or Spring (RestTemplate) without the need to configure anything, but this method is not optimal and is suitable only for the simplest calls to external resources without restrictions. Due to the fact that scripts are easily replaceable, manual programming in scripts is cheaper than inside an application/microservice. If accessing the external system becomes more complicated (for example, authentication by temporary tokens, adaptation of mapping models, error handling, data caching, setting connection parameters, dynamic routing / balancing, nonstandard protocol, complex calculations), then a microservice adapter is created, which is similar to ExternalTask executors may be independent. There are more advantages for adapter microservices in Java/Kotlin, because Synplity Platform provides ready-made libraries and tools for fast implementation of adapters — SOAP/REST/MQ component templates, as well as automatic configuration of the adapter for unified monitoring, logging, configuration, cluster registration, resource access control and administration of the entire platform. The result of the development of a microservice adapter is a universal REST API that can be used not only within processes, but also separately for research purposes during development, point testing, or to display data on the interface directly, and not from the context of the process. Asynchronous interaction, if it is necessary to obtain a delayed result, is adapted in a way similar to ExternalTask — the adapter that implements asynchronous interaction with expectation of a result takes the responsibility to provide a unique identifier in response to a method call, and attach the same identifier to the asynchronous result that is placed in the topic Kafka. To fulfill this contract, there is a separate API library and a Starter library with a service for sending asynchronous results to Kafka. Access to external systems through adapter microservices in the Synplity Platform from process scripts is standardized for ease of reuse: Data classes and an interface are required that will be used directly in the script code. A project is created, to which the dependency on everything necessary for describing the API is connected — documenting annotations, annotations for Json mapping and descriptions of the Rest interface (Spring-web annotations). The project is compiled, and the resulting dependency can be included in the sandbox to develop and test process scripts, and using the sandbox plugin, subsequently deliver the JAR with the adapter API and process scripts directly to the test or production environment. The described approach in the Synplity Platform separates work on integrations from work on process logic, leaves behind a reusable and documented API that is not mixed with the business process engine used. Thus, engines can be changed or not used at all, and the remaining infrastructure and services can become the basis for further development of the microservice architecture. Obviously, this approach would be a logical choice when developing on Camunda, but it is not thought out of the box in it. Embedding integration interactions into processes on the Synplity Platform engine, building logic with lightweight scripts, requires much less action and knowledge of the nuances of the platform, and instead requires knowledge of more generally accepted and standard tools in the industry, such as the Spring Framework — based on them, there are ready-made solutions for the Synplity Platform, so each integration is easily reusable and scalable.
Development flexibility
As already mentioned, scripts in Camunda provide limited features, are developed and tested inconveniently, which ultimately forces you to do everything through JavaDelegate / ExternalTask, and this slows down the development of simple things, for example, moving data from one place to another, smears validations and makes it harder to fix bugs. At the same time, scripts in the Synplity Platform are a very powerful tool, the functionality of which is easily extended and supplemented with the necessary services and methods, and those already available out of the box will definitely prove useful in a large number of cases. The scripts themselves can be developed in a separate sandbox, in IDEs familiar to developers. In order for the project to become a sandbox for developing scripts, it is enough to include a library with the necessary development and testing API, as well as a plugin for CI / CD. Scripts are developed in separate directories corresponding to the structure of the process graph, and they can be tested as a whole or by separate methods from a common script. There are not so many popular projects with examples of successful use of Groovy, and at the same time, this scripting language really fits into the Synplity Platform and is effective for developing lightweight process logic. The Groovy script scope integrates Synplity Platform's own tools and models: process context, process management service, process repository access service, timer management and creation/cancellation of deferred events, tool for asynchrony and parallelization of tasks, service for converting synchronous calls adapters to asynchronous and vice versa, a service for handling distributed caches. Based on the presented tools, one can guess about the intersections with the functionality of Processengine Camunda. Indeed, everything that can be done on the Process-engine in different ways (in BPMN notations, scripts, ServiceTask, JavaDelegate) can be done uniformly using scripts in the Synplity Platform. It is worth noting that scripts can be written not only for processes, but also as separate utilities/libraries in order to reuse their methods and structures in different parts of the process, or even in different processes. Camunda allows logic to be reused only in the form of subprocesses, which is also possible in the Synplity Platform, but this does not work well on small blocks of logic, it is difficult to universalize. And most importantly, why everything turned out this way in the Synplity Platform - all changes can be made on the fly. Any process can be modified without restarting or stopping modules. Scripts are compiled at runtime, adapter models and APIs are added at runtime as well. This SIGNIFICANTLY speeds up development, testing, bug fixing. In the case of Camunda, simple errors like NPE/type conversion errors (everyone happens at least once) and other unforeseen things like adding a new process step or handling an unexpected error lead to a new version of the process, restarting the application. With small errors, this is especially annoying. In Synplity Platform, this is quickly fixed directly from the sandbox or from the administration interface - we edit the script code in the right place, update the script on the stand and overcome the error. Splitting process scripts in the Synplity Platform allows you to point-fix places with errors, make temporary changes to bypass errors (in case of breakdowns in external systems), flexibly update processes, creating logical conditions for updating data in the process and moving to new steps instead of old ones, which can be a little later clean out of the process. Errors are easily investigated in the Synplity Platform, the entire log chain and stack trace can be seen by the process ID, Exception data is stored in the context of the process, and there is a special error handling script for each process in which you can describe the general logic for handling uncaught exceptions, for example, canceling transactions or adding a human-readable description of the error to an attribute.
Work on performance
At the base level, both Synplity Platform and Camunda perform simple calculations in a common way: request processing occurs on a web thread. application, the engine service is triggered, which launches the business logic, changes in the process instance are saved in the database. Setting up a web connection pool application and the database connection pool can be ignored, because they are the same approach. The launch of business logic in the case of Synplity Platform will always lead to the launch of scripts, and in the case of Camunda it can be a lot of different scenarios with different delegates that execute the logic, and, accordingly, with different performance impacts and ways to solve problems that arise. In this regard, only the Camunda script delegate can be correctly compared with the Synplity Platform, but it is almost never used in real projects, because limited in possibilities. For both platforms, there is a reasonable recommendation to move processor-intensive logic (and / or a complex of many different operations) into thirdparty modules so as not to clog servlet threads and Executor wait queues — therefore, comparing the speed of scripts relative to Java delegates will not show significant the difference is when in both cases only a few lines of identical code work, launching heavy processes on the side and putting changes into the context of the process. The approach to storage is different. Camunda uses a relational database, everything engine tables are tied to the same storage. Synplity Platform is tied to non-relational MongoDB, collections are distributed in accordance with the functionality of microservices and can be tied to different storages — initially this was worked out for the possibility of scaling and fault tolerance of critical storages separately, in accordance with priorities. In contrast to linking the platform to MongoDB specifically, a lot of work has been done on all possible optimization of collections, indexes, queries and the amount of data transferred, replication and sharding cases have been developed to work on large amounts of data (about 1−2 TB of data per day). instance process in Synplity Platform contains all the necessary data in one document (context variables, external tasks, system data of a running instance), without requiring transactions when working with a process instance. A model adapted immediately to the Json document format is more versatile for storing a variety of nested structures, and nested parameters can be indexed. Thus, in the Synplity Platform there are no problems that require attention in Camunda — it is easier to handle nested complex structures in context, group as you like, change through the API; fewer errors related to JSON serialization/deserialization; there are no problems with data inconsistency, with competitive access to the context of processes or with transaction management; there are no inconveniences due to storing parameters in one universal SQL table and, as a result, degradation the performance of SQL queries as they become more complex and/or the table grows. For Synplity Platform and Camunda, there is one common limitation — the amount of data attached to the process, and the general recommendation is to move this data to a separate storage / microservice. For example, in Camunda it is not recommended to store heavy files or large data structures in the context of a process, but it is better to store only links to them — this is reasonable to reduce the load on the table in the database and the application memory consumption for byte array allocation. Anticipating this in the Synplity Platform, we created an optional microservice for optimal handling of files from the process, as well as storing the results of asynchronous tasks outside the context of the process when they exceed a given size. Camunda has many different mechanisms for waiting, asynchrony and parallelization: Waiting states, separate notations for interacting with parallel running subprocesses, moving calculations and batch processing to separate subprocesses, and launching subprocesses as collections of instances of a single subprocess. In Synplity Platform, the use of parallelization and asynchrony fit into a simple choice:
1) If it is possible to speed up the execution of business logic without a timeout, then a thread pool is used to run tasks directly in the script logic, and the thread executing the process instance is held until the end of the execution of all parallel tasks.
2) If there is a risk of getting a timeout, then calculations and queries are started by asynchronous tasks, releasing the thread executing the process instance, and the results obtained asynchronously are checked for compliance with the conditions for moving to the next state with processing results or errors.
3) If the calculations are costly, time-consuming and complex, then this logic is formed in the form of a Spring Batch task and taken out into a separate microservice, connecting it to a ready-made infrastructure and a ready-made module for managing Spring Batch task executors.
Thus, all operations in the Synplity Platform are treated as simple function calls or function calls with a callback, which is closer to programmers, not analysts or businesses. Any operation in the Synplity Platform can be wrapped as a callback through ready-made scripting API methods, and operations that were originally made as callbacks can be wrapped as a call to a simple synchronous operation. For analysts, this is simply a division into receiving data / moving immediately or through a stage with a wait. Thus, what in Camunda is represented in the diagram by different notations of parallel subprocesses and separate waiting states, in Synplity Platform it is implemented in scripts by launching tasks / subprocesses in the thread pool in a batch, and in the form of launching a batch of external tasks / subprocesses with a callback. Such a simple choice of approach in the Synplity Platform (instead of various schemes and methods in Camunda) simplifies the development of solutions for the distribution of calculations in processes, and also narrows the scope of finding and solving problems. Very little attention is paid to horizontal scaling in the Camunda documentation: routing and load balancing of incoming events are not included in the engine and are not considered, synchronization between engine instances is provided only through the database. Also, a separate Job-executor of the engine captures tasks through the same lock at the database level. Optimistic locks are used to avoid process deadlocks. In total, the entire load is shifted to the database, and optimistic lock errors and cleaning up the consequences of synchronization errors — compensatory logic and retries, control of non-idempotent calls become a common problem. There is no possibility to control the distribution of tasks from the queue between engines. Synplity Platform was designed from the start with scaling in mind, with load balancing and routing. The core of the Synplity Platform includes an application-level synchronization tool — locking by process id and a guarantee that the process instance can be thread-safely executed only on the event source thread, in the order in which events are received. Additionally, it is possible to block between instances by an arbitrary identifier, as well as use shared caches — these tools are available in scripts. The load on the database is lower, the process instance without acquiring a lock at the interservice level is not requested and does not execute logic once again, the process instance document is self-sufficient, there are no Joins and transactions on several tables, there are no errors due to mutual locks or contention with optimistic locks. The deferred task queue, triggers and asynchronous calculations are not in the core of the Synplity Platform engine, but in a separate microservice that uses a separate DB instance if desired — this allows you to scale important parts separately, redistributing resources according to the frequency of use in processes. When a task module accesses a kernel module, it is possible to distribute processing between kernel instances via an interservice layer.
Goal setting
Camunda is a project with a huge history, existing in the concept of an academic theoretical business process management model, also known as BPMS. Its main emphasis is on modeling business processes in a transparent manner, merging technical and business sense through the semantics of the specification, and as close as possible to the model immediately to the executable workflow. The concept is captivating when immersed, but beyond it there are still many other tasks, more applied level. Synplity Platform is an application-level project based on simple concepts from mathematics and logic to describe processes: a graph and a finite state machine. The main emphasis is on the accelerated development of solutions in a supportive framework that simplifies the work and interaction of teams. The concept consists of the following assumptions:
1. Outside of developed solutions and templates, there will always be areas that require manual development, not taken into account by ready-made solutions and templates — this niche will always be open.
2. If you create and gain expertise, then in basic, universal, demanded and long-term tools so that developers of different levels are available and applicable from the labor market, there is an interest in experience and technologies, and competencies are easily were transmitted.
3. Systems are aging, workloads are growing, data sources and business tools are expanding, new competitive solutions need to be brought to market at lightning speed, and only developers solve these problems, which means that the task is to ensure that developers have a ready to expand and update a free basis, as well as reliance on a toolkit that solves template tasks for developers.
At Camunda, a noticeable emphasis is placed on building the logic of the work of everything the whole process. For the Synplity Platform — for the development of individual steps (with a reserve for reusing small sections of logic or a whole set of actions in one function) and interaction with the interface at each of the steps (specialized and contractually validated attributes, informative errors in business logic, redirects to other steps or processes, Websocket notifications). Camunda does not have a Frontendfriendly contract for process logic. Synplity Platform specializes in one architecture option — microservice — and immediately offers a ready-made set of tools for interacting with existing and creating new microservices: administration, logging, configuration, API documentation, monitoring and metrics. Libraries for integrations, batch tasks, and auditing are also provided. For small projects, this will be redundant and resource intensive. Within the Synplity Platform, development becomes flexible — a simple process diagram with test data can be made and completed in minutes without restarting components. The development of a new business process for the readiness of the formulated requirements is built as follows:
All the above qualities of the Synplity Platform do not require specialized engine or platform competencies and fit into the average requirements for developers of different levels. Our current approach to help plan the distribution of team resources:
Synplity Platform removes barriers specific to enterprise systems and solves most of the tasks of any system on a microservice architecture. The developer focuses on business logic and the usual development of web applications with REST API, having the opportunity to replace tools and frameworks or improve existing ones. For novice developers or developers with experience in other technologies, the platform is a good example to get acquainted with all popular technologies, the most commonly used development approaches and microservice architecture.
Recognizable general concepts
Recognizable general concepts Camunda has standard tools (Modeler and Cockpit) for process design and administration. Projects on Camunda rarely use only the standard administration functionality, because Cockpit’s capabilities do not cover all the needs of projects, therefore, either their own analogues are created, or Cockpit is being finalized. These extensions are offered as standalone products or purchased with engine expertise. Camunda’s free tools cover most basic needs. In Synplity Platform, the basic needs of process monitoring are covered by standard Kibana dashboards and metrics available through Prometheus/Grafana. An analogue of Modeler and Cockpit in the Synplity Platform is a separate Angular application with an administration interface that uses a universal API and has the ability to customize or extend the standard solution with the necessary functionality for the needs of the customer, which happens often. In addition to front-end applications for the Synplity Platform, the method of modeling and administration through code and CLI is also relevant — using a sandbox project with a special library and a maven plugin connected. Camunda has a Process-engine that allows to use it in 3 versions: servlet-container app, embedded app (spring-boot) and standalone, as well as consuming BPMN schemas and launching processes based on them. In Synplity Platform, the same functionality is provided by SP-workbench and SP-core microservices. SP-workbench allows to manage the process graph (vertices and edges) and describe the process context (accessible context variables, data types, structures, access control). If each Process-engine keeps its process diagrams to itself, then the SP-workbench is a common repository of all diagrams, replicated and available for process modifications at runtime (in the case of Camunda, it will be necessary to ensure seamless switching to new versions of microservices, even with minor changes to the process logic). SP-core creates and executes instances of the processes available in SP-workbench. The basis of the state machine model in the SP-core is the Spring State Machine — an open and well- documented project with high quality source code, and Spring projects are close to many programmers. The vertices of the graph turn into states (states), and the edges of the graph — into transitions (transitions) between states. You can attach any logic (actions) to the vertices and transitions in the data model. The model is enriched with specific functionality for designing business processes, such as cancellation and error pseudostates, so as not to create redundant transitions in template situations. Thus, the state machine model in the Synplity Platform is created and changed on the fly, immediately starting to process instances of processes and run related logic. Camunda and Synplity Platform, as business process automation systems, have many similarities and the same features, for example, call/task delegation, interaction between processes and subprocesses, working with variables in the context of a process instance, event timers, error compensation-handling. The differences are in the details, especially in development, testing, and production experience.
The Path to Implementing a Stable Solution Infrastructure
Without third-party components or Enterprise subscriptions, Camunda is a bare engine of libraries and tools for working with the specific model and format of the BPMN standard. For a production-ready cluster, it is highly likely that you will need to either refine or purchase a ready-made set of admin panels, monitoring with tracing and logging, implement pipelines for building Docker images that are assembled into a cluster, design Process-engines with different business requirements for business requirements. — processes to interact with each other and with external interfaces, organize general management of module configurations. You also often have to work out the configuration for user authorization and Security restrictions with engine components and in the processes themselves. Thus, the advantages of Camunda include complete freedom in customization, but at the same time, the development of the project will have to be carried out practically from scratch. To help developers, there are a number of readymade tools for working with the model and concepts of the engine, as well as integrating parts of the engine with Spring. Having chosen independent development, Synplity Platform builds processes on the general principles of finite automaton and directed graph models, fully based on Spring projects. The platform comes as a ready-made infrastructure using popular tools for microservice solutions, which takes care of CI / CD needs, and adds flexible options for Security configuration in processes and interactions between them.
Working with API and engine tools
What made us disappointed in Camunda? The presence of XML in process diagrams, in resources and module configs, in writing logic, connectors, the presence of redundant and legacy functionality with a mixture of Backend & Frontend development. The Synplity Platform project is focused on SPA, and the generation of HTML templates on the Backend side is needed only in exceptional cases. In Camunda, the built-in capabilities for developing interfaces and APIs are not relevant, and all new projects simply ignore this functionality, encountering it only when extending Cockpit (this is working with Tasklist and various Forms and plugins). Camunda has a cumbersome and complex process instance model, which translates into the complexity of handling the REST API for process control and complicates the tasks of Frontend developers when developing custom UI. Another example of actually not used functionality in Camunda — we have not met a single case with extensive use of scripts and built-in EL in logic. This is obviously because these tools are difficult to understand, inconvenient to use in XML format, difficult to test, and there are many restrictions on the available functions. Development on Camunda actually becomes hardcoded based on Java Delegate and External Task in each process, which reduces development efficiency.
Diagrams and business process design
Initially, there were no complaints about BPMN 2.0 — the schemes look logical and beautiful. However, in practice, only developers fully understand them, business is not motivated to learn this notation, and analysts are very limited in their ability to understand the logic of processes — in order to understand, it is often necessary to look into the context of variables, not all logic can be seen on the diagram or figure out what it does or other challenge. To implement processes and complete immersion, you will have to memorize the meanings of shapes and icons, which becomes like learning a language based on hieroglyphs, while often only a small part of the notation is used. Often, to simplify, the scheme has to be divided into smaller and simpler blocks, which is why it grows. Complex processes often don’t fit on the screen (or are hard to see when zooming out), causing one process to be split into smaller ones. In practice, it turned out that to describe the process of a simple graph with points and connections, a basic understanding of business, analysts and developers is enough, and in a simple script / pseudocode it is often easier to understand the logic, work with context variables and call external points than in a complex scheme with icons and mapping tables. This is especially important when efforts to make everything simple and understandable do not lead to the expansion of the scheme and an increase in the number of subprocesses, but result in a simple graph with short and readable instructions in sections that follow the order of execution and are reused in different processes. The lack of a visual BPMN scheme in the Synplity Platform is compensated by the ability to conveniently search in a project with scripts and process graphs by any key identifiers: system/subprocess/attribute name, interaction type, reusable function, passed parameters, places where parallelization is used or a callback is expected, by the resulting type of asynchronous operation — all this allows you to quickly move to all parts of the logic when studying processes or solving problems, which is important for developers.
Integrations
Our frequent case in projects is the interaction of several dozen systems. This is a sequential call to systems through the steps of a process, or several calls in one step to aggregate data displayed to the user or subsequent request. What Camunda offers is a "strange" choice: add and configure a plugin in Process-engine with Rest / Soap connectors and compose XML files with request-response configuration from intended services, or manually write Java code with a service call, implementing another JavaDelegate or ExternalTask (ExternalTask will need to be adapted on the other side so that the result goes to the Process-engine queue). XML files are inconvenient to configure and test because of the lack of flexibility, which pushes you to manually program the call to external systems directly and the context parameter mappings to and from the request. This approach is costly when making edits and debugging processes, and also complicates reuse for different processes. Updates to the API of the external system will lead to a cascade of edits in the JavaDelegate and ExternalTask set. All this shifts the focus from integration for business purposes to adaptation of an external system. for the technical needs of Camunda. At the same time, the same code for accessing external systems will be duplicated in different Process-engines, and if the system is complex, then it will definitely be decided to move the main repeating or complex blocks to a separate adapter, which Process-engine instances will reuse. The Synplity Platform has some analogies with the way you work with integrations in Camunda. Instead of a plugin and XML configs for simple calls, you can use standard Rest / Soap clients familiar to Java developers directly in the scripting logic of the process, for example, Apache (HttpClient) or Spring (RestTemplate) without the need to configure anything, but this method is not optimal and is suitable only for the simplest calls to external resources without restrictions. Due to the fact that scripts are easily replaceable, manual programming in scripts is cheaper than inside an application/microservice. If accessing the external system becomes more complicated (for example, authentication by temporary tokens, adaptation of mapping models, error handling, data caching, setting connection parameters, dynamic routing / balancing, nonstandard protocol, complex calculations), then a microservice adapter is created, which is similar to ExternalTask executors may be independent. There are more advantages for adapter microservices in Java/Kotlin, because Synplity Platform provides ready-made libraries and tools for fast implementation of adapters — SOAP/REST/MQ component templates, as well as automatic configuration of the adapter for unified monitoring, logging, configuration, cluster registration, resource access control and administration of the entire platform. The result of the development of a microservice adapter is a universal REST API that can be used not only within processes, but also separately for research purposes during development, point testing, or to display data on the interface directly, and not from the context of the process. Asynchronous interaction, if it is necessary to obtain a delayed result, is adapted in a way similar to ExternalTask — the adapter that implements asynchronous interaction with expectation of a result takes the responsibility to provide a unique identifier in response to a method call, and attach the same identifier to the asynchronous result that is placed in the topic Kafka. To fulfill this contract, there is a separate API library and a Starter library with a service for sending asynchronous results to Kafka. Access to external systems through adapter microservices in the Synplity Platform from process scripts is standardized for ease of reuse: Data classes and an interface are required that will be used directly in the script code. A project is created, to which the dependency on everything necessary for describing the API is connected — documenting annotations, annotations for Json mapping and descriptions of the Rest interface (Spring-web annotations). The project is compiled, and the resulting dependency can be included in the sandbox to develop and test process scripts, and using the sandbox plugin, subsequently deliver the JAR with the adapter API and process scripts directly to the test or production environment. The described approach in the Synplity Platform separates work on integrations from work on process logic, leaves behind a reusable and documented API that is not mixed with the business process engine used. Thus, engines can be changed or not used at all, and the remaining infrastructure and services can become the basis for further development of the microservice architecture. Obviously, this approach would be a logical choice when developing on Camunda, but it is not thought out of the box in it. Embedding integration interactions into processes on the Synplity Platform engine, building logic with lightweight scripts, requires much less action and knowledge of the nuances of the platform, and instead requires knowledge of more generally accepted and standard tools in the industry, such as the Spring Framework — based on them, there are ready-made solutions for the Synplity Platform, so each integration is easily reusable and scalable.
Development flexibility
As already mentioned, scripts in Camunda provide limited features, are developed and tested inconveniently, which ultimately forces you to do everything through JavaDelegate / ExternalTask, and this slows down the development of simple things, for example, moving data from one place to another, smears validations and makes it harder to fix bugs. At the same time, scripts in the Synplity Platform are a very powerful tool, the functionality of which is easily extended and supplemented with the necessary services and methods, and those already available out of the box will definitely prove useful in a large number of cases. The scripts themselves can be developed in a separate sandbox, in IDEs familiar to developers. In order for the project to become a sandbox for developing scripts, it is enough to include a library with the necessary development and testing API, as well as a plugin for CI / CD. Scripts are developed in separate directories corresponding to the structure of the process graph, and they can be tested as a whole or by separate methods from a common script. There are not so many popular projects with examples of successful use of Groovy, and at the same time, this scripting language really fits into the Synplity Platform and is effective for developing lightweight process logic. The Groovy script scope integrates Synplity Platform's own tools and models: process context, process management service, process repository access service, timer management and creation/cancellation of deferred events, tool for asynchrony and parallelization of tasks, service for converting synchronous calls adapters to asynchronous and vice versa, a service for handling distributed caches. Based on the presented tools, one can guess about the intersections with the functionality of Processengine Camunda. Indeed, everything that can be done on the Process-engine in different ways (in BPMN notations, scripts, ServiceTask, JavaDelegate) can be done uniformly using scripts in the Synplity Platform. It is worth noting that scripts can be written not only for processes, but also as separate utilities/libraries in order to reuse their methods and structures in different parts of the process, or even in different processes. Camunda allows logic to be reused only in the form of subprocesses, which is also possible in the Synplity Platform, but this does not work well on small blocks of logic, it is difficult to universalize. And most importantly, why everything turned out this way in the Synplity Platform - all changes can be made on the fly. Any process can be modified without restarting or stopping modules. Scripts are compiled at runtime, adapter models and APIs are added at runtime as well. This SIGNIFICANTLY speeds up development, testing, bug fixing. In the case of Camunda, simple errors like NPE/type conversion errors (everyone happens at least once) and other unforeseen things like adding a new process step or handling an unexpected error lead to a new version of the process, restarting the application. With small errors, this is especially annoying. In Synplity Platform, this is quickly fixed directly from the sandbox or from the administration interface - we edit the script code in the right place, update the script on the stand and overcome the error. Splitting process scripts in the Synplity Platform allows you to point-fix places with errors, make temporary changes to bypass errors (in case of breakdowns in external systems), flexibly update processes, creating logical conditions for updating data in the process and moving to new steps instead of old ones, which can be a little later clean out of the process. Errors are easily investigated in the Synplity Platform, the entire log chain and stack trace can be seen by the process ID, Exception data is stored in the context of the process, and there is a special error handling script for each process in which you can describe the general logic for handling uncaught exceptions, for example, canceling transactions or adding a human-readable description of the error to an attribute.
Work on performance
At the base level, both Synplity Platform and Camunda perform simple calculations in a common way: request processing occurs on a web thread. application, the engine service is triggered, which launches the business logic, changes in the process instance are saved in the database. Setting up a web connection pool application and the database connection pool can be ignored, because they are the same approach. The launch of business logic in the case of Synplity Platform will always lead to the launch of scripts, and in the case of Camunda it can be a lot of different scenarios with different delegates that execute the logic, and, accordingly, with different performance impacts and ways to solve problems that arise. In this regard, only the Camunda script delegate can be correctly compared with the Synplity Platform, but it is almost never used in real projects, because limited in possibilities. For both platforms, there is a reasonable recommendation to move processor-intensive logic (and / or a complex of many different operations) into thirdparty modules so as not to clog servlet threads and Executor wait queues — therefore, comparing the speed of scripts relative to Java delegates will not show significant the difference is when in both cases only a few lines of identical code work, launching heavy processes on the side and putting changes into the context of the process. The approach to storage is different. Camunda uses a relational database, everything engine tables are tied to the same storage. Synplity Platform is tied to non-relational MongoDB, collections are distributed in accordance with the functionality of microservices and can be tied to different storages — initially this was worked out for the possibility of scaling and fault tolerance of critical storages separately, in accordance with priorities. In contrast to linking the platform to MongoDB specifically, a lot of work has been done on all possible optimization of collections, indexes, queries and the amount of data transferred, replication and sharding cases have been developed to work on large amounts of data (about 1−2 TB of data per day). instance process in Synplity Platform contains all the necessary data in one document (context variables, external tasks, system data of a running instance), without requiring transactions when working with a process instance. A model adapted immediately to the Json document format is more versatile for storing a variety of nested structures, and nested parameters can be indexed. Thus, in the Synplity Platform there are no problems that require attention in Camunda — it is easier to handle nested complex structures in context, group as you like, change through the API; fewer errors related to JSON serialization/deserialization; there are no problems with data inconsistency, with competitive access to the context of processes or with transaction management; there are no inconveniences due to storing parameters in one universal SQL table and, as a result, degradation the performance of SQL queries as they become more complex and/or the table grows. For Synplity Platform and Camunda, there is one common limitation — the amount of data attached to the process, and the general recommendation is to move this data to a separate storage / microservice. For example, in Camunda it is not recommended to store heavy files or large data structures in the context of a process, but it is better to store only links to them — this is reasonable to reduce the load on the table in the database and the application memory consumption for byte array allocation. Anticipating this in the Synplity Platform, we created an optional microservice for optimal handling of files from the process, as well as storing the results of asynchronous tasks outside the context of the process when they exceed a given size. Camunda has many different mechanisms for waiting, asynchrony and parallelization: Waiting states, separate notations for interacting with parallel running subprocesses, moving calculations and batch processing to separate subprocesses, and launching subprocesses as collections of instances of a single subprocess. In Synplity Platform, the use of parallelization and asynchrony fit into a simple choice:
1) If it is possible to speed up the execution of business logic without a timeout, then a thread pool is used to run tasks directly in the script logic, and the thread executing the process instance is held until the end of the execution of all parallel tasks.
2) If there is a risk of getting a timeout, then calculations and queries are started by asynchronous tasks, releasing the thread executing the process instance, and the results obtained asynchronously are checked for compliance with the conditions for moving to the next state with processing results or errors.
3) If the calculations are costly, time-consuming and complex, then this logic is formed in the form of a Spring Batch task and taken out into a separate microservice, connecting it to a ready-made infrastructure and a ready-made module for managing Spring Batch task executors.
Thus, all operations in the Synplity Platform are treated as simple function calls or function calls with a callback, which is closer to programmers, not analysts or businesses. Any operation in the Synplity Platform can be wrapped as a callback through ready-made scripting API methods, and operations that were originally made as callbacks can be wrapped as a call to a simple synchronous operation. For analysts, this is simply a division into receiving data / moving immediately or through a stage with a wait. Thus, what in Camunda is represented in the diagram by different notations of parallel subprocesses and separate waiting states, in Synplity Platform it is implemented in scripts by launching tasks / subprocesses in the thread pool in a batch, and in the form of launching a batch of external tasks / subprocesses with a callback. Such a simple choice of approach in the Synplity Platform (instead of various schemes and methods in Camunda) simplifies the development of solutions for the distribution of calculations in processes, and also narrows the scope of finding and solving problems. Very little attention is paid to horizontal scaling in the Camunda documentation: routing and load balancing of incoming events are not included in the engine and are not considered, synchronization between engine instances is provided only through the database. Also, a separate Job-executor of the engine captures tasks through the same lock at the database level. Optimistic locks are used to avoid process deadlocks. In total, the entire load is shifted to the database, and optimistic lock errors and cleaning up the consequences of synchronization errors — compensatory logic and retries, control of non-idempotent calls become a common problem. There is no possibility to control the distribution of tasks from the queue between engines. Synplity Platform was designed from the start with scaling in mind, with load balancing and routing. The core of the Synplity Platform includes an application-level synchronization tool — locking by process id and a guarantee that the process instance can be thread-safely executed only on the event source thread, in the order in which events are received. Additionally, it is possible to block between instances by an arbitrary identifier, as well as use shared caches — these tools are available in scripts. The load on the database is lower, the process instance without acquiring a lock at the interservice level is not requested and does not execute logic once again, the process instance document is self-sufficient, there are no Joins and transactions on several tables, there are no errors due to mutual locks or contention with optimistic locks. The deferred task queue, triggers and asynchronous calculations are not in the core of the Synplity Platform engine, but in a separate microservice that uses a separate DB instance if desired — this allows you to scale important parts separately, redistributing resources according to the frequency of use in processes. When a task module accesses a kernel module, it is possible to distribute processing between kernel instances via an interservice layer.
Goal setting
Camunda is a project with a huge history, existing in the concept of an academic theoretical business process management model, also known as BPMS. Its main emphasis is on modeling business processes in a transparent manner, merging technical and business sense through the semantics of the specification, and as close as possible to the model immediately to the executable workflow. The concept is captivating when immersed, but beyond it there are still many other tasks, more applied level. Synplity Platform is an application-level project based on simple concepts from mathematics and logic to describe processes: a graph and a finite state machine. The main emphasis is on the accelerated development of solutions in a supportive framework that simplifies the work and interaction of teams. The concept consists of the following assumptions:
1. Outside of developed solutions and templates, there will always be areas that require manual development, not taken into account by ready-made solutions and templates — this niche will always be open.
2. If you create and gain expertise, then in basic, universal, demanded and long-term tools so that developers of different levels are available and applicable from the labor market, there is an interest in experience and technologies, and competencies are easily were transmitted.
3. Systems are aging, workloads are growing, data sources and business tools are expanding, new competitive solutions need to be brought to market at lightning speed, and only developers solve these problems, which means that the task is to ensure that developers have a ready to expand and update a free basis, as well as reliance on a toolkit that solves template tasks for developers.
At Camunda, a noticeable emphasis is placed on building the logic of the work of everything the whole process. For the Synplity Platform — for the development of individual steps (with a reserve for reusing small sections of logic or a whole set of actions in one function) and interaction with the interface at each of the steps (specialized and contractually validated attributes, informative errors in business logic, redirects to other steps or processes, Websocket notifications). Camunda does not have a Frontendfriendly contract for process logic. Synplity Platform specializes in one architecture option — microservice — and immediately offers a ready-made set of tools for interacting with existing and creating new microservices: administration, logging, configuration, API documentation, monitoring and metrics. Libraries for integrations, batch tasks, and auditing are also provided. For small projects, this will be redundant and resource intensive. Within the Synplity Platform, development becomes flexible — a simple process diagram with test data can be made and completed in minutes without restarting components. The development of a new business process for the readiness of the formulated requirements is built as follows:
- The analyst makes a list of required data sources/integrations for the process;
- The analyst makes a list and sequence of steps of the future process, describes the logic of filling in the attributes corresponding to the steps for Frontend and Backend developers;
- Backend developers share the implementation of individual steps and data exchange endpoints (integrated methods);
- Frontend developers divide the development of individual screens into steps. Without repeating the mentioned difference of engines, in the context of the Synplity Platform specifically, Groovy scripts and the ability to change them on the fly can reduce the cost of development and testing.
- The dependence of developers on the readiness of the tasks of colleagues is reduced — in dependent sections of the code, you can replace the functionality with a stub in the right place and continue development, subsequently substituting the finished functionality in a couple of minutes. The last step of the process can be implemented without the previous steps being ready in the same way, Frontend developers can quickly get test interaction with Backend. Simplifies testing of complex and rare cases — logic branches can be temporarily rebuilt for testing;
- Scripts, like code, are universalized and reused, strengthening the codebase exponentially over time. In the future, a scenario for describing processes in a format close to low-code solutions is possible. Integration methods in adapter microservices can be developed without regard to processes and without restrictions from the platform engine. Utility scripts and specialized tools built into the script scope allow you to smooth out the inconvenience of the API and highlight individual functions for the specific use of integrations in a particular process.
- Writing boilerplate code, making changes, and fixing bugs is simplified. Modern IDEs perform well in tandem with a sandbox project for developing scripts — you can quickly search and switch to the right sections of code, write unit tests and use code hints. Errors are detected and localized sufficiently transparent — according to the stack trace from the script, engine exceptions are informative — an analyst or a tester can immediately attach a server response or a log with an exception, and the developer will find an erroneous method through the search or will correct the model at the prompt of the process validator. When using the capabilities of Groovy, work with mapping, collections, text formatting is simplified, Null-safe constructions are added.
- The lightness of the script code allows you to flexibly distribute the load between the Backend and Frontend, microservices and the engine core, as well as make changes faster later. All data for display can be passed through process attributes, or only part of the data can be passed — it depends on how much code the Backend or Frontend developer writes. You can make many requests to the data source, or you can make the request API more universal and allow you to do everything with one request. You can first make a minimally working process, and then optimize it as much as possible. In any case, eliminating the script code, changing or adding a new one is much cheaper and faster.
All the above qualities of the Synplity Platform do not require specialized engine or platform competencies and fit into the average requirements for developers of different levels. Our current approach to help plan the distribution of team resources:
- Scripts are handled by Junior developers. If development is required in a short time — Middle-developers, if in addition complete independence is required and there are gaps in analytics — Senior-developers.
- Creation of new microservices and adapters, integration methods are tasks for Middle-developers. If time permits — junior developer under mentoring, if integration using a rare protocol and/or requires interaction restrictions — senior developer.
- Optimization of processes and methods of adapter microservices, configuration tuning, resource allocation — Senior developers.
Synplity Platform removes barriers specific to enterprise systems and solves most of the tasks of any system on a microservice architecture. The developer focuses on business logic and the usual development of web applications with REST API, having the opportunity to replace tools and frameworks or improve existing ones. For novice developers or developers with experience in other technologies, the platform is a good example to get acquainted with all popular technologies, the most commonly used development approaches and microservice architecture.