Upcoming Release: Process data pipeline

We are soon releasing the first version of a long-anticipated feature - the process data pipeline. Think about it as a database for your robots, but which is also convenient for business users to interact

Most automated processes revolve around processing external data or files (e.g., Excel sheets, PDFs) as input or provide them as an output. Also, data flows in processes can be asynchronous, so data storage can be used as an intermediary between process phases. Process data pipeline addresses these problems by allowing storing of data used in processes outside of the execution time. No need to host and maintain your own MongoDB or SQL!

Work items are the entities used in Robocorp Cloud to store any kind of data meant to be processed by robots. By using the data pipeline to handle every transaction in the process by work item basis, the state of processing is always up-to-date. For example, splitting handling of 1000 customers to 1000 work items, even if the robot crashes in an unexpected way, the processing state of each item stays known. This also allows splitting the work to multiple execution environments in one-to-many style of processing. Think about it as “work queues”, but with much more added functionality.

Old processes in Cloud are backward compatible with the data pipeline, and changes are mostly visual in status indicators.In the first version, we are rolling out some basic functionality:

  • Improved process dashboard
  • Processing of multiple work items in one process run
  • Tracking of work items in Input, In-progress, and Output storages
  • Monitoring of work item success rate
  • Process run status indicator showing if processing or work items have succeeded

During the H1 of 2021, we will be extending functionality by adding more features like:

  • Processing multiple items from the queue one-by-one (looping through the queue)
  • Processing work items with environment groups (parallel processing with self-hosted environments)
  • Extensive API:s to interact with the inputs and outputs

More information coming next week - stay tuned!!

4 Likes

Hi @Erno, thanks for this announcement

I think this new functionality is great for tracking the robot success rate and the numbers of processed workitems

I have had the opportunity to test it and I believe one of the best new functionalities is the ability to reprocess some workitems, all of them or the failed ones. This is great news!!

Another thing I believe is great is the functionality to incorporate a new workitem from the cloud to be processed.

One thing I don’t like very much it’s that we have lost the “indented view” of the workitem JSON. Now, with this new version, they are shown in a single line and with no indentation at all. That makes it difficult to recognize some of them

Another suggestion I would like to add is if it would be possible to launch only one or more steps to be run with the selected workitems.

Imagine you have a process with three steps: the first steps retrieves some info and creates the workitem, the second creates a transaction in some system with the workitem information and the third step saves some information in a Content Management system and you need to rerun only the third step because it failed due to a failure in the target system.

It would be great to have that ability. Choose a workitem, choose an step … and re-run

Please, let me know if I have properly exposed the case and let me know if you have any doubt

Thanks for your work around the product. I really believe this platform is a fantastic RPA enterprise solution.

Jose

1 Like

Thanks Jose for the feedback, really delighted to hear that you find the feature useful!

I’ll forward the JSON view UX improvement to the team.

Thanks also for providing another suggestion about re-runs with an example use case, I can say that this is already on our roadmap and will be delivering it later on along with other improvements to the CRUD of work items and exception handling :slightly_smiling_face:

1 Like

This is great @erno

Happy to hear all these new features coming

Jose

1 Like

Hi @erno,

Thanks for the announcement.

Given that “Currently only one work item per execution is supported
by Robocorp Cloud, which should be loaded automatically.”, Does that mean in order to handle 1000 transactions, we need to run the bot 1000 times?

Has the functionality “Processing of multiple work items in one process run” been implemented yet?

Thanks

Richard

1 Like

Hello @richard.yeh !

Currently, it is not possible to “loop” multiple work items with a robot run, and a new robot run is created for each work item. So the robot will do a full restart for each item. Looping will be implemented in the future for better efficiency.

It is possible to start a process run with multiple items, documentation for the API call can be found here https://robocorp.com/docs/robocorp-cloud/api/process-api under “Start a process with multiple work item data payloads”.

Hope this helps!

1 Like

Hi @erno,

Thank you so much for the information.

Very looking forward to see the loop feature for work items in the future!

Richard

1 Like