Developer Docs
Build Your Own Custom Experiences
Welcome to Tw4i Documentation! We are here to provide you with a comprehensive guide on how to utilize the powerful features of W4IDataFlow. Whether you need assistance in automating your recurring tasks or building efficient workflow processes, our open source and cross-platform workflow engine and automation platform has got you covered. With Tw4i, you can easily streamline your operations and boost productivity. Join us today and experience the seamless integration of Workflow 4.
1. Introduction
W4IDataFlow is an open source and cross-platform workflow engine and automation platform that aims to automate recurring tasks. With the help of W4IDataFlow, building automation and workflow processes become easy.
W4IDataFlow provides a cross-platform workflow server, a backend for designing, managing and tracking workflows, sequential workflows, flowchart workflows, and approval workflows on generic business objects called records.
W4IDataFlow automates the process of organizing, storing, manipulating files, provides more than 100+ activities, and allows custom integration with other solutions through custom activities or W4IDataFlow API.
Here are some real life use cases:
- Orchestration engine
- Batch recording live video feeds
- Batch transcoding audio and video files
- Batch uploading videos to YouTube
- Automatically send tweets
- Automatically send SMS messages
- Batch encrypting and decrypting large files
- Batch converting, resizing and cropping images
- Creating and sending reports and invoices by email
- Connecting systems and applications
- Batch downloading files over FTP/FTPS/SFTP/HTTP/Torrent
- Batch uploading files over FTP/FTPS/SFTP
- Database administration and maintenance
- Synchronizing the content of local or remote directories
2. Installation
W4IDataFlow is easy to install and needs zero configuration. It can be installed and configured quickly.
This section shows how to install the .NET version on Windows, and how to install the .NET Core version on Windows, Linux or macOS.
Windows (.NET)
Download the latest release of W4IDataFlow
Right click on the installer, click on properties, check unblock then click OK.
Launch the installer and follow the instructions
After W4IDataFlow is installed a Windows Service named W4IDataFlow is installed and starts automatically.
The following menus are added in the start menu:
The “Backend” menu opens the backend.
The “Configuration” menu opens the configuration folder of W4IDataFlow.
The “Documentation” menu opens the documentation folder of W4IDataFlow.
The “Logs” menu opens the log file of the day.
The “Manager” menu opens W4IDataFlow Manager GUI.
The “Install SQLite samples” menu installs workflow samples in SQLite.
The “Install MongoDB samples” menu installs workflow samples in MongoDB.
The “Install SQL Server samples” menu installs workflow samples in SQL Server.
Here are the credentials to sign in from the backend or W4IDataFlow Manager:
Username: admin
Password: W4IDataFlow2018
Once logged in, you can change the password from the backend.
You can host the backend on a web server if you want to.
You can choose from 3 database systems that W4IDataFlow uses to store its data:
SQLite (Default)
MongoDB
SQL Server
See configration page to see how to change the database system.
If you want to install MongoDB or SQL Server samples, you have to set the connection string:
MongoDB: C:\Program Files\W4IDataFlow\W4IDataFlow.Scripts.MongoDB\W4IDataFlow.Scripts.MongoDB.exe.config
SQL Server: C:\Program Files\W4IDataFlow\W4IDataFlow.Scripts.SQLServer\W4IDataFlow.Scripts.SQLServer.exe.config
And don’t forget to restart W4IDataFlow Windows Service if you install workflow samples.
If you want to use SQL Server Trusted_Connection=true in the connection string, you have to run W4IDataFlow Windows Service as a user who has privileges to connect to SQL Server.
Windows (.NET Core)
Download and install ASP.NET Core 7.0 Runtime
Download and extract W4IDataFlow’s .NET Core package
Double-click on “install.bat” to install the configuration files of W4IDataFlow
That’s it. Double-click on “run.bat” to start W4IDataFlow workflow server
To open the backend, go to “Backend” folder and double-click on the file “index.html”.
The credentials are the same listed in Windows .NET section.
Linux (.NET Core)
Download and install ASP.NET Core 7.0 Runtime
Download and extract the W4IDataFlow’s .NET Core package in /opt/
Add permissions:
sudo chown -R $USER:$USER /opt/W4IDataFlow
That’s it. You can run W4IDataFlow as follows:
cd /opt/W4IDataFlow/W4IDataFlow.Server
dotnet W4IDataFlow.Server.dll
You can open the backend by opening /opt/W4IDataFlow/Backend/index.html on a browser.
The credentials are the same listed in Windows .NET section.
macOS (.NET Core)
Download and install ASP.NET Core 7.0 Runtime
Download and extract the W4IDataFlow’s .NET Core package in /Applications/
That’s it. You can run W4IDataFlow as follows:
cd /Applications/W4IDataFlow/W4IDataFlow.Server
dotnet W4IDataFlow.Server.dll
You can open the backend by opening /Applications/W4IDataFlow/Backend/index.html on a browser.
The credentials are the same listed in Windows .NET section.
3. Docker
Docker
This section describes how build and run W4IDataFlow Docker image.
- Download and unzip the latest version of W4IDataFlow-linux-netcore.zip:
unzip W4IDataFlow-linux-netcore.zip
The folder ./W4IDataFlow/ will be created on your current folder.
- Download Dockerfile and docker-compose.yml files and copy them on your current folder.
- Run the following command on your current folder:
sudo docker compose up
That’s it W4IDataFlow will start and the backend will be accessible from http://localhost:8011/.
If you have already deployed W4IDataFlow image and want to upgrade to another version, run the following commands to make sure you get all features updated:
sudo docker compose build --no-cache sudo docker compose up
If you want to use MongoDB instead of SQLite, follow the instructions in docker-compose.yml file.
4. Configuration
Configuration
W4IDataFlow needs zero-configuration. However, if you want to change the configuration you can read this page.
W4IDataFlow Server
.NET
You can configure W4IDataFlow Server from C:\Program Files\W4IDataFlow\W4IDataFlow.Server.exe.config:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<configSections>
<section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler,log4net"/>
</configSections>
<appSettings>
<add key="W4IDataFlowSettingsFile" value="C:\W4IDataFlow\W4IDataFlow.xml"/>
<!-- LogLevel: Debug | All | Severely | Minimum | None -->
<add key="LogLevel" value="All"/>
<add key="W4IDataFlowServicePort" value="8000"/>
<add key="SuperAdminUsername" value="admin"/>
<add key="EnableWorkflowsHotFolder" value="false"/>
<add key="EnableRecordsHotFolder" value="true"/>
<add key="EnableEmailNotifications" value="false"/>
<add key="DateTimeFormat" value="dd-MM-yyyy HH:mm:ss"/>
<add key="Smtp.Host" value="smtp.gmail.com"/>
<add key="Smtp.Port" value="587"/>
<add key="Smtp.EnableSsl" value="true"/>
<add key="Smtp.User" value="user"/>
<add key="Smtp.Password" value="password"/>
<add key="Smtp.From" value="user"/>
<add key="ClientSettingsProvider.ServiceUri" value=""/>
</appSettings>
<log4net>
<appender name="RollingFile" type="log4net.Appender.RollingFileAppender">
<file value="W4IDataFlow.log"/>
<encoding value="utf-8"/>
<appendToFile value="true"/>
<rollingStyle value="Date"/>
<datePattern value="yyyyMMdd"/>
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%date %5level [%thread] - %message%newline"/>
</layout>
</appender>
<root>
<level value="INFO"/>
<appender-ref ref="RollingFile"/>
</root>
</log4net>
...
</configuration>
LogLevel options:
- Debug: All logs and debug logs.
- All: All logs without debug logs (Default).
- Severely: Only last workflow log and error logs.
- Minimum: Only last workflow log.
- None: No logs.
.NET Core
You can configure W4IDataFlow Server from W4IDataFlow.Server/appsettings.json:
{
"W4IDataFlowSettingsFile": "C:\\W4IDataFlow-netcore\\W4IDataFlow.xml",
"LogLevel": "All",
"W4IDataFlowServicePort": 8000,
"SuperAdminUsername": "admin",
"EnableWorkflowsHotFolder": false,
"EnableRecordsHotFolder": true,
"EnableEmailNotifications": false,
"DateTimeFormat": "dd-MM-yyyy HH:mm:ss", /* Date and time format in the backend. */
"Smtp.Host": "smtp.gmail.com",
"Smtp.Port": 587,
"Smtp.EnableSsl": true,
"Smtp.User": "user",
"Smtp.Password": "password",
"Smtp.From": "user"
}
W4IDataFlow.xml
W4IDataFlow.xml is the main configuration file of W4IDataFlow server. Its path can be configured from C:\Program Files\W4IDataFlow\W4IDataFlow.Server.exe.config for the .NET version and from W4IDataFlow.Server\appsettings.json for the .NET Core version.
.NET
Below is the configuration file W4IDataFlow.xml for the .NET version:
<?xml version="1.0" encoding="UTF-8" ?>
<W4IDataFlow>
<Setting name="workflowsFolder" value="C:\W4IDataFlow\Workflows" />
<Setting name="recordsFolder" value="C:\W4IDataFlow\Records" />
<Setting name="recordsHotFolder" value="C:\W4IDataFlow\Records\_HotFolder" />
<Setting name="tempFolder" value="C:\W4IDataFlow\Temp" />
<Setting name="tasksFolder" value="C:\W4IDataFlow\Tasks" />
<Setting name="approvalFolder" value="C:\W4IDataFlow\Approval" />
<Setting name="xsd" value="C:\W4IDataFlow\Workflow.xsd" />
<Setting name="tasksNamesFile" value="C:\W4IDataFlow\TasksNames.json" />
<Setting name="tasksSettingsFile" value="C:\W4IDataFlow\TasksSettings.json" />
<Setting name="globalVariablesFile" value="C:\W4IDataFlow\GlobalVariables.xml" />
<!-- SQLite or MongoDB or SQLServer -->
<Setting name="dbType" value="SQLite" />
<!-- SQLite -->
<Setting name="connectionString" value="Data Source=C:\W4IDataFlow\Database\W4IDataFlow.sqlite;Version=3;" />
<!-- MongoDB -->
<!--<Setting name="connectionString" value="Database=W4IDataFlow;MongoUrl=mongodb://localhost:27017;EnabledSslProtocols=false;SslProtocols=None" />-->
<!-- SQLServer -->
<!--<Setting name="connectionString" value="Server=localhost;Trusted_Connection=True;Database=W4IDataFlow;" />-->
</W4IDataFlow>
W4IDataFlow supports 3 database systems for storing its data. You can choose from the following dbType options:
- SQLite (Default)
- MongoDB
- SQLServer
If you change W4IDataFlow’s database system, don’t forget to update connectionString setting.
.NET Core
Windows
Below is the configuration file W4IDataFlow.xml for the .NET Core version:
<?xml version="1.0" encoding="UTF-8" ?>
<W4IDataFlow>
<Setting name="workflowsFolder" value="C:\W4IDataFlow-netcore\Workflows" />
<Setting name="recordsFolder" value="C:\W4IDataFlow-netcore\Records" />
<Setting name="recordsHotFolder" value="C:\W4IDataFlow-netcore\Records\_HotFolder" />
<Setting name="tempFolder" value="C:\W4IDataFlow-netcore\Temp" />
<Setting name="tasksFolder" value="C:\W4IDataFlow-netcore\Tasks" />
<Setting name="approvalFolder" value="C:\W4IDataFlow-netcore\Approval" />
<Setting name="xsd" value="C:\W4IDataFlow-netcore\Workflow.xsd" />
<Setting name="tasksNamesFile" value="C:\W4IDataFlow-netcore\TasksNames.json" />
<Setting name="tasksSettingsFile" value="C:\W4IDataFlow-netcore\TasksSettings.json" />
<Setting name="globalVariablesFile" value="C:\W4IDataFlow-netcore\GlobalVariables.xml" />
<!-- SQLite or MongoDB or SQLServer -->
<Setting name="dbType" value="SQLite" />
<!-- SQLite -->
<Setting name="connectionString" value="Data Source=C:\W4IDataFlow-netcore\Database\W4IDataFlow.sqlite;Version=3;" />
<!-- MongoDB -->
<!--<Setting name="connectionString" value="Database=W4IDataFlow_netcore;MongoUrl=mongodb://localhost:27017;EnabledSslProtocols=false;SslProtocols=None" />-->
<!-- SQLServer -->
<!--<Setting name="connectionString" value="Server=localhost;Trusted_Connection=True;Database=W4IDataFlow_netcore;" />-->
</W4IDataFlow>
Linux
Below is the configuration file W4IDataFlow.xml for the .NET Core version:
<?xml version="1.0" encoding="UTF-8" ?>
<W4IDataFlow>
<Setting name="workflowsFolder" value="/opt/W4IDataFlow/W4IDataFlow/Workflows" />
<Setting name="recordsFolder" value="/opt/W4IDataFlow/W4IDataFlow/Records" />
<Setting name="recordsHotFolder" value="/opt/W4IDataFlow/W4IDataFlow/Records/_HotFolder" />
<Setting name="tempFolder" value="/opt/W4IDataFlow/W4IDataFlow/Temp" />
<Setting name="tasksFolder" value="/opt/W4IDataFlow/W4IDataFlow/Tasks" />
<Setting name="approvalFolder" value="/opt/W4IDataFlow/W4IDataFlow/Approval" />
<Setting name="xsd" value="/opt/W4IDataFlow/W4IDataFlow/Workflow.xsd" />
<Setting name="tasksNamesFile" value="/opt/W4IDataFlow/W4IDataFlow/TasksNames.json" />
<Setting name="tasksSettingsFile" value="/opt/W4IDataFlow/W4IDataFlow/TasksSettings.json" />
<Setting name="globalVariablesFile" value="/opt/W4IDataFlow/W4IDataFlow/GlobalVariables.xml" />
<!-- SQLite or MongoDB or SQLServer -->
<Setting name="dbType" value="SQLite" />
<!-- SQLite -->
<Setting name="connectionString" value="Data Source=/opt/W4IDataFlow/W4IDataFlow/Database/W4IDataFlow.sqlite;Version=3;" />
<!-- MongoDB -->
<!--<Setting name="connectionString" value="Database=W4IDataFlow_netcore;MongoUrl=mongodb://localhost:27017;EnabledSslProtocols=false;SslProtocols=None" />-->
<!-- SQLServer -->
<!--<Setting name="connectionString" value="Server=localhost;Trusted_Connection=True;Database=W4IDataFlow_netcore;" />-->
</W4IDataFlow>
macOS
Below is the configuration file W4IDataFlow.xml for the .NET Core version:
<?xml version="1.0" encoding="UTF-8" ?>
<W4IDataFlow>
<Setting name="workflowsFolder" value="/Applications/W4IDataFlow/W4IDataFlow/Workflows" />
<Setting name="recordsFolder" value="/Applications/W4IDataFlow/W4IDataFlow/Records" />
<Setting name="recordsHotFolder" value="/Applications/W4IDataFlow/W4IDataFlow/Records/_HotFolder" />
<Setting name="tempFolder" value="/Applications/W4IDataFlow/W4IDataFlow/Temp" />
<Setting name="tasksFolder" value="/Applications/W4IDataFlow/W4IDataFlow/Tasks" />
<Setting name="approvalFolder" value="/Applications/W4IDataFlow/W4IDataFlow/Approval" />
<Setting name="xsd" value="/Applications/W4IDataFlow/W4IDataFlow/Workflow.xsd" />
<Setting name="tasksNamesFile" value="/Applications/W4IDataFlow/W4IDataFlow/TasksNames.json" />
<Setting name="tasksSettingsFile" value="/Applications/W4IDataFlow/W4IDataFlow/TasksSettings.json" />
<Setting name="globalVariablesFile" value="/Applications/W4IDataFlow/W4IDataFlow/GlobalVariables.xml" />
<!-- SQLite or MongoDB or SQLServer -->
<Setting name="dbType" value="SQLite" />
<!-- SQLite -->
<Setting name="connectionString" value="Data Source=/Applications/W4IDataFlow/W4IDataFlow/Database/W4IDataFlow.sqlite;Version=3;" />
<!-- MongoDB -->
<!--<Setting name="connectionString" value="Database=W4IDataFlow_netcore;MongoUrl=mongodb://localhost:27017;EnabledSslProtocols=false;SslProtocols=None" />-->
<!-- SQLServer -->
<!--<Setting name="connectionString" value="Server=localhost;Trusted_Connection=True;Database=W4IDataFlow_netcore;" />-->
</W4IDataFlow>
Backend
It is possible to format date and time in the backend through DateTimeFormat setting option. The date is local and can be formatted however you want. The default format is dd-MM-yyyy HH:mm:ss.
If this setting option is modified, you must restart W4IDataFlow server to take it into consideration.
.NET
To change the setting option DateTimeFormat simply open the settings file C:\Program Files\W4IDataFlow\W4IDataFlow.Server.exe.config and edit the setting option.
.NET Core
To change the setting option DateTimeFormat simply open the settings file W4IDataFlow.Server/appsettings.json and edit the setting option.
5. Get Started
Get Started
.NET
After installing W4IDataFlow, the folders C:\W4IDataFlow\ and C:\W4IDataFlowTesting\ are created.
The folder C:\W4IDataFlow\ is the main configuration folder of W4IDataFlow and contains the following elements:
- W4IDataFlow.xml which is the main configuration file of W4IDataFlow server. Its path can be configured from C:\Program Files\W4IDataFlow\W4IDataFlow.Server.exe.config for the .NET version and from W4IDataFlow.Server\appsettings.json for the .NET Core version.
- Database/ which contains the database of W4IDataFlow workflow engine.
- Workflows/ which is workflows hot folder if hot reloading is enabled through EnableWorkflowsHotFolder setting.
- Temp/ which is the temporary folder of W4IDataFlow.
- Tasks/ which is an optional folder that can contain the dlls of custom tasks.
- Workflow.xsd which is the XML Schema Definition of a workflow.
- GlobalVariables.xml which contains the global variables for workflows.
- TasksNames.json which contains the names of the tasks. This configuration file is used by the designer.
- TasksSettings.json which contains the settings of the tasks. This configuration file is used by the designer.
The folder C:\W4IDataFlowTesting\ contains data of testing scenarios.
The logs are written in C:\Program Files\W4IDataFlow\W4IDataFlow.log. There is one log file per day. The old log files are saved in this format W4IDataFlow.logyyyyMMdd.
.NET Core
In the .NET Core version:
- Windows: The folders C:\W4IDataFlow-dotnet-core\ and C:\W4IDataFlowTesting\ are created. The path of the main configuration file C:\W4IDataFlow-dotnet-core\W4IDataFlow.xml can be configured from W4IDataFlow.Server\appsettings.json. The logs are written in W4IDataFlow.Server\W4IDataFlow.log.
- Linux: The folders /opt/W4IDataFlow/W4IDataFlow/ and /opt/W4IDataFlow/W4IDataFlowTesting/ are created. The path of the main configuration file /opt/W4IDataFlow/W4IDataFlow/W4IDataFlow.xml can be configured from /opt/W4IDataFlow/W4IDataFlow.Server/appsettings.json. The logs are written in /opt/W4IDataFlow/W4IDataFlow.Server/W4IDataFlow.log.
- macOS: The folders /Applications/W4IDataFlow/W4IDataFlow/ and /Applications/W4IDataFlow/W4IDataFlowTesting/ are created. The path of the main configuration file /Applications/W4IDataFlow/W4IDataFlow/W4IDataFlow.xml can be configured from /Applications/W4IDataFlow/W4IDataFlow.Server/appsettings.json. The logs are written in /Applications/W4IDataFlow/W4IDataFlow.Server/W4IDataFlow.log.
Designer
Workflows can be Designed through the designer, XML or JSON. It is highly recommended to understand workflows syntax (XML/JSON) in order to become familiar with this workflow engine.
Below is the configuration file of a workflow in XML:
<?xml version="1.0" encoding="utf-8" ?>
<!--
This is the configuration file of a workflow.
A workflow is composed of:
- An id which is an integer that must be unique.
- A name which is a string that must be unique.
- A description which is a string.
- A Settings section which is composed of the following elements:
- A launchType which is one of the following options:
- startup: The workflow is launched when W4IDataFlow Engine starts.
- trigger: The workflow is launched manually from W4IDataFlow Manager.
- periodic: The workflow is launched periodically.
- cron: The workflow is launched depending on a cron expression.
- A period which is necessary for the periodic launchType. It is
a timeSpan in this format dd.hh:mm:ss. For example the period
00.00:02:00 will launch the workflow every 2 minutes.
- A cron expression which is necessary for the cron launchType.
For example '0 0/2 * * * ?' will launch the workflow every 2 minutes.
- The enabled option which allows to enable or disable a workflow.
The possible values are true or false.
- The approval option which marks the current workflow as an approval workflow.
The possible values are true or false. An approval workflow must contain
at least one Approval task or more.
- The enableParallelJobs option Shows whether workflow jobs are executed in parallel.
Otherwise jobs are queued. Defaults to true.
- A LocalVariables section which contains local variables.
- A Tasks section which contains the tasks that will be executed by
the workflow one after the other.
- A Task is composed of:
- An id which is an integer that must be unique.
- A name which is one of the options described in the tasks documentation.
- A description which is a string.
- The enabled option which allows to enable or disable a task. The possible
values are true or false.
- A collection of settings.
- An ExecutionGraph section which contains the execution graph of the workflow.
-->
<Workflow xmlns="urn:W4IDataFlow-schema" id="$int" name="$string" description="$string">
<Settings>
<Setting name="launchType" value="startup|trigger|periodic|cron" />
<Setting name="period" value="dd.hh:mm:ss" />
<Setting name="cronExpression" value="$string" />
<Setting name="enabled" value="true|false" />
<Setting name="approval" value="true|false" />
<Setting name="enableParallelJobs" value="true|false" />
</Settings>
<LocalVariables>
<Variable name="$string" value="$string" />
<Variable name="$string" value="$string" />
<!-- You can add as many variables as you want. -->
</LocalVariables>
<Tasks>
<Task id="$int" name="$string" description="$string" enabled="true|false">
<Setting name="$string" value="$string" />
<Setting name="$string" value="$string" />
<!-- You can add as many settings as you want. -->
</Task>
<Task id="$int" name="$string" description="$string" enabled="true|false">
<Setting name="$string" value="$string" />
<Setting name="$string" value="$string" />
</Task>
<!-- You can add as many tasks as you want. -->
</Tasks>
<!-- This section is optional and described in the samples section. -->
<ExecutionGraph />
</Workflow>
For cron workflows, read the following documentation for more details.
Local variables are explained here.
Global variables are explained here.
The name option of a Task must be one of the names listed in the following documentation. You can find the documentation of each task in the Documentation folder of W4IDataFlow.
The execution graph is explained in the samples section.
To learn how to make your own workflows, you can check out the workflow samples availabe in the Designer, in the samples section, and read the tasks documentation available in Configuration folder.
If a new workflow is created or if an existing workflow is deleted or modified, you don’t have to restart W4IDataFlow Windows Service so that these modifications take effect. W4IDataFlow engine will automatically detect the changes and reload, add or delete the workflow.
To disable a workflow, you can set the enabled settings option of the workflow to false.
How tasks communicate between each other?
State is transferred between tasks through selectFiles and through selectEntities settings.
This works the following way:
- A task in a workflow does its job and produces files which it stores in a collection.
- Another task (must be in the same workflow) can afterwards reference those files with the selectFiles XML property, specifying the ID of the task that produced the required files. It then can use these files to do its own job.
More visually (from the examples):
<Workflow xmlns="urn:W4IDataFlow-schema" id="1" name="Workflow_Invoices" description="Workflow_Invoices">
<Settings>
<Setting name="launchType" value="trigger" />
<Setting name="enabled" value="true" />
</Settings>
<Tasks>
<Task id="1" name="FilesLoader" description="Loading invoices" enabled="true">
<Setting name="folder" value="C:\W4IDataFlowTesting\Invoices\" />
</Task>
<!-- some more tasks here -->
<Task id="6" name="FilesMover" description="Moving invoices" enabled="true">
<Setting name="selectFiles" value="1" />
<Setting name="destFolder" value="C:\W4IDataFlowTesting\Invoices_sent\" />
</Task>
</Tasks>
</Workflow>
selectEntities setting works the same way as selectFiles. The only difference is that selectEntities is designed to be used for tasks that manipulate custom objects from a database or from web services for example. To go further, read this documentation regarding entities.
W4IDataFlow Manager
When you open W4IDataFlow Manager for the first time, you will get a login window.
Here are the credentials to sign in:
- Username: admin
- Password: W4IDataFlow2018
You can change the password from the backend.
W4IDataFlow Manager is a simple application that allows you to do the following things:
- See all the workflows loaded by W4IDataFlow Engine.
- See the status of the selected workflow (running, suspended, waiting for approval or disabled).
- Start a workflow.
- Stop a workflow.
- Suspend a workflow.
- Resume a workflow.
- Approve an approval workflow.
- Disapprove an approval workflow.
- The “Backend” button opens the backend from which you can manage workflows, design workflows, track workfows and have real-time statistics on workflows.
- The “Logs” button allows to view the logs of the day.
- The “Refresh” button allows to reload the list of workflows.
- The “Restart server” button allows to restart W4IDataFlow Server.
- The “Search” button allows to search for workflows.
- The “Help” menu opens the help page.
- The “About” menu shows the version of W4IDataFlow and checks if a new version is available.
To see what’s going on in W4IDataFlow, open the log file C:\Program Files\W4IDataFlow\W4IDataFlow.log in a text editor like Notepad++. Notepad ++ will update the log file as it fills up.
Backend
The backend is a website that you can host on IIS, Apache, NGINX or any other web server. The backend can also run locally.
The backend gives real-time statistics on workflows. It will let you manage, design and track your workflows with ease and flexibility. You can use the backend to access, configure, manage, administer, and develop your workflows with ease.
Login
When you open the backend for the first time, you will get a login window.
Here are the credentials to sign in:
- Username: admin
- Password: W4IDataFlow2018
After you sign in, you can change the password from the “Users” page.
Password reset
If a user forgot his password, he can click on “Forgot password?” link to reset his password.
When the user clicks on “Submit” button, an email is sent to him with a temporary password that he can change after he signs in.
To allow the backend sending emails, the SMPT configuration must be set in the configuration.
Dashboard
After you sign in, you will arrive on the dashboard page. W4IDataFlow gives you a beautiful dashboard to view real-time statistics on your workflows. Indeed, the “Dashboard” page gives you real-time statistics on workflows and will let you track your workflow server with ease and detail. From the dashboard, you can also filter the workflow entries by a keyword or by date. You can also order the workflow entries by date, by name, etc.
Manager
The “Manager” page will let you manage your workflows. From this page you can start a workflow, suspend a running workflow, resume a suspended workflow, stop a running workflow and search for workflows.
Designer
The “Designer” page will let you design your workflows. From this page you can create a new workflow, edit an existing workflow or delete a workflow. Using the “Designer” page, we get a nice visual overview of the dependency graph of the workflow. Each node represents a task which has to be run.
Furthermore, the “Designer” page allows to edit workflow files through its Web XML or JSON editor.
Press Ctrl+S to save your workflow.
Press Ctrl+Alt+H in XML or JSON view for keyboard shortcuts.
Approval
The “Approval” page will let you view all approval workflows and will let you approve or disapprove workflows.
History
The “History” page will let you track all your workflows and everything that happens on the workflow server.From this page you will have an overview of all the workflow instances executed on the workflow server. Furthermore, you can filter the entries by keywords or date. You can also order the entries by date, by name, etc.
Users
The “Users” page allows to create new users, change passwords and user’s informations, and delete users who have restricted access.
A user who has restricted rights has only access to the “Dashboard” page and the “History” page.
Profiles
The “Profiles” page allows to assign workflows to users. Once the workflow assigned, the user can run it, modify it and delete it.
Installation
To install the backend on a web server, simply copy the content of the folder “C:\Program Files\W4IDataFlow\Backend\” in the web server. Then, configure the workflow server URI if you want to install the backend on another machine. Simply edit the configuration file “js/settings.js”:
Settings = {
Uri: "http://localhost:8000/W4IDataFlow/"
};
Instead of localhost, put the IP or the DNS of the machine where W4IDataFlow server is installed.
6. Samples
Samples
Workflows can be designed through the Designer, through XML or JSON. However, it is highly recommended to understand W4IDataFlow workflows syntax in order to become familiar with this workflow engine.
Each workflow in W4IDataFlow has a configuration (XML/JSON). Each configuration contains a set of settings and tasks to do depending on a specified schedule and a specified configuration.
Workflow samples
In this section, few workflow samples will be presented in order to make the end user familiar with W4IDataFlow workflow syntax:
Sequential workflows
A sequential workflow executes a set of tasks in order, one by one. Tasks are executed in a sequential manner until the last task finishes. The order of the execution of the tasks can be altered by modifying the execution graph of the workflow.
Workflow 1
This workflow uploads invoices to an SFTP server, then waits for 2 days and then notifies the customers.
<Workflow xmlns="urn:W4IDataFlow-schema" id="99" name="Workflow_Invoices" description="Workflow_Invoices">
<Settings>
<Setting name="launchType" value="trigger" />
<Setting name="enabled" value="true" />
</Settings>
<Tasks>
<Task id="1" name="FilesLoader" description="Loading invioces" enabled="true">
<Setting name="folder" value="C:\W4IDataFlowTesting\Invoices\" />
</Task>
<Task id="2" name="Ftp" description="Uploading invoices" enabled="true">
<Setting name="protocol" value="sftp" /> <!-- ftp|ftps|sftp -->
<Setting name="command" value="upload" /> <!-- list|upload|download|delete -->
<Setting name="server" value="127.0.1" />
<Setting name="port" value="21" />
<Setting name="user" value="user" />
<Setting name="password" value="password" />
<Setting name="path" value="/" />
<Setting name="selectFiles" value="1" />
</Task>
<Task id="3" name="Wait" description="Waiting for 2 days" enabled="true">
<Setting name="duration" value="2.00:00:00" />
</Task>
<Task id="4" name="FilesLoader" description="Loading emails" enabled="true">
<Setting name="file" value="C:\W4IDataFlowTesting\Emails\Invoices.xml" />
</Task>
<Task id="5" name="MailsSender" description="Notifying customers" enabled="true">
<Setting name="selectFiles" value="4" />
<Setting name="host" value="127.0.0.1" />
<Setting name="port" value="587" />
<Setting name="enableSsl" value="true" />
<Setting name="user" value="user" />
<Setting name="password" value="password" />
</Task>
<Task id="6" name="FilesMover" description="Moving invoices" enabled="true">
<Setting name="selectFiles" value="1" />
<Setting name="destFolder" value="C:\W4IDataFlowTesting\Invoices_sent\" />
</Task>
</Tasks>
</Workflow>
First of all, the FilesLoader task loads all the invoices located in the folder C:\W4IDataFlowTesting\Invoices\, then the Ftp task uploads them to the SFTP server, then the Wait task waits for 2 days, then the FilesLoader task loads the emails in XML format and then the MailsSender task sends the emails. Finally, the FilesMover task moves the invoices to the folder C:\W4IDataFlowTesting\Invoices_sent.
Workflow 2
This workflow waits for files to arrive in C:\W4IDataFlowTesting\Watchfolder1\ and C:\W4IDataFlowTesting\Watchfolder2\ then uploads them to an FTP server then moves them to C:\W4IDataFlowTesting\Sent\ folder. This workflow starts every 2 minutes.
<Workflow xmlns="urn:W4IDataFlow-schema" id="6" name="Workflow_FilesSender" description="Workflow_FilesSender">
<Settings>
<Setting name="launchType" value="periodic" />
<Setting name="period" value="00.00:02:00.00" />
<Setting name="enabled" value="true" />
</Settings>
<Tasks>
<Task id="1" name="FilesLoader" description="Loading files" enabled="true">
<Setting name="folder" value="C:\W4IDataFlowTesting\Watchfolder1\" />
<Setting name="folder" value="C:\W4IDataFlowTesting\Watchfolder2\" />
</Task>
<Task id="2" name="Ftp" description="Uploading files" enabled="true">
<Setting name="protocol" value="ftp" /> <!-- ftp|ftps|sftp -->
<Setting name="command" value="upload" /> <!-- list|upload|download|delete -->
<Setting name="server" value="127.0.1" />
<Setting name="port" value="21" />
<Setting name="user" value="user" />
<Setting name="password" value="password" />
<Setting name="path" value="/" />
<Setting name="selectFiles" value="1" />
</Task>
<Task id="3" name="FilesMover" description="Moving files to Sent folder" enabled="true">
<Setting name="selectFiles" value="1" />
<Setting name="destFolder" value="C:\W4IDataFlowTesting\Sent\" />
</Task>
</Tasks>
</Workflow>
First of all, the FilesLoader task loads all the files located in the folders C:\W4IDataFlowTesting\Watchfolder1\ and C:\W4IDataFlowTesting\Watchfolder2\ then the Ftp task loads the files and uploads them to the FTP server. Finally, the FilesMover task moves the files to the folder C:\W4IDataFlowTesting\Sent.
If you want to trigger tasks on file events, you should use FileSystemWatcher task to avoid having a lot of logs. Here is a sample workflow.
Workflow 3
This workflow transcodes the WAV files located in C:\W4IDataFlowTesting\WAV\ to MP3 format through FFMPEG and moves the transcoded files to C:\W4IDataFlowTesting\MP3.
<Workflow xmlns="urn:W4IDataFlow-schema" id="12" name="Workflow_ffmpeg" description="Workflow_ffmpeg">
<Settings>
<Setting name="launchType" value="trigger" />
<Setting name="enabled" value="true" />
</Settings>
<Tasks>
<Task id="1" name="FilesLoader" description="Loading WAV files" enabled="true">
<Setting name="folder" value="C:\W4IDataFlowTesting\WAV\" />
</Task>
<Task id="2" name="ProcessLauncher" description="WAV to MP3" enabled="true">
<Setting name="selectFiles" value="1" />
<!-- You need to install FFMPEG -->
<Setting name="processPath" value="C:\Program Files\ffmpeg\bin\ffmpeg.exe" />
<!-- variables: {$filePath},{$fileName},{$fileNameWithoutExtension}-->
<Setting name="processCmd" value="-i {$filePath} -codec:a libmp3lame -qscale:a 2 {$output:$fileNameWithoutExtension.mp3}" />
<Setting name="hideGui" value="true" />
<Setting name="generatesFiles" value="true" />
</Task>
<Task id="3" name="FilesMover" description="Moving MP3 files from temp folder" enabled="true">
<Setting name="selectFiles" value="2" />
<Setting name="destFolder" value="C:\W4IDataFlowTesting\MP3\" />
</Task>
</Tasks>
</Workflow>
First of all, the FilesLoader task loads all the files located in the folder C:\W4IDataFlowTesting\WAV\ then the ProcessLauncher task launches FFMPEG process on every file by specifying the right command in order to create the MP3 file. Finally, the FilesMover task moves the MP3 files to the folder C:\W4IDataFlowTesting\MP3.
Workflow 4
This workflow waits for WAV files to arrive in C:\W4IDataFlowTesting\WAV\ then transcodes them to MP3 files through VLC then uploads the MP3 files to an FTP server then moves the WAV files to C:\W4IDataFlowTesting\WAV_processed. This workflow starts every 2 minutes.
<Workflow xmlns="urn:W4IDataFlow-schema" id="13" name="Workflow_vlc" description="Workflow_vlc">
<Settings>
<Setting name="launchType" value="periodic" />
<Setting name="period" value="00.00:02:00.00" />
<Setting name="enabled" value="true" />
</Settings>
<Tasks>
<Task id="1" name="FilesLoader" description="Loading WAV files" enabled="true">
<Setting name="folder" value="C:\W4IDataFlowTesting\WAV\" />
</Task>
<Task id="2" name="ProcessLauncher" description="WAV to MP3" enabled="true">
<Setting name="selectFiles" value="1" />
<!-- You need to install VLC-->
<Setting name="processPath" value="C:\Program Files\VideoLAN\VLC\vlc.exe" />
<!-- variables: {$filePath},{$fileName},{$fileNameWithoutExtension}-->
<Setting name="processCmd" value="-I dummy {$filePath} :sout=#transcode{acodec=mpga}:std{dst={$output:$fileNameWithoutExtension.mp3},access=file} vlc://quit" />
<Setting name="hideGui" value="true" />
<Setting name="generatesFiles" value="true" />
</Task>
<Task id="3" name="Ftp" description="Uploading MP3 files" enabled="true">
<Setting name="protocol" value="ftp" />
<Setting name="command" value="upload" />
<Setting name="server" value="127.0.1" />
<Setting name="port" value="21" />
<Setting name="user" value="user" />
<Setting name="password" value="password" />
<Setting name="path" value="/" />
<Setting name="selectFiles" value="2" />
</Task>
<Task id="4" name="FilesMover" description="Moving WAV files" enabled="true">
<Setting name="selectFiles" value="1" />
<Setting name="destFolder" value="C:\W4IDataFlowTesting\WAV_processed\" />
</Task>
</Tasks>
</Workflow>
First of all, the FilesLoader task loads all the files located in the folder C:\W4IDataFlowTesting\WAV\ then the ProcessLauncher task launches VLC process on every file by specifying the right command in order to create the MP3 file. Then, the Ftp task loads the MP3 files generated by the ProcessLauncher task and then uploads them to the FTP server. Finally, the FilesMover task moves the processed WAV files to the folder C:\W4IDataFlowTesting\WAV_processed.
If you want to trigger tasks on file events, you should use FileSystemWatcher task to avoid having a lot of logs. Here is a sample workflow.
Workflow 5
This workflow downloads specific files from an FTP server. This workflow starts by listing all the files located at the root folder of the server, then the specific files that will be downloaded are tagged through an XSLT (LisFiles.xslt), then the files are downloaded by the Ftp task through todo=”toDownload” and from=”app4″ tags, then the downloaded files are moved to the folder C:\W4IDataFlowTesting\Ftp_download.
<Workflow xmlns="urn:W4IDataFlow-schema" id="40" name="Workflow_Ftp_download_tag" description="Workflow_Ftp_download_tag">
<Settings>
<Setting name="launchType" value="trigger" /> <!-- startup|trigger|periodic -->
<Setting name="enabled" value="true" /> <!-- true|false -->
</Settings>
<Tasks>
<Task id="1" name="Ftp" description="Listing files (FTP)" enabled="true">
<Setting name="command" value="list" />
<Setting name="protocol" value="ftp" /> <!-- ftp|ftps|sftp -->
<Setting name="server" value="127.0.1" />
<Setting name="port" value="21" />
<Setting name="user" value="user" />
<Setting name="password" value="password" />
<Setting name="path" value="/" />
</Task>
<Task id="2" name="ListFiles" description="Listing files" enabled="true">
</Task>
<Task id="3" name="Xslt" description="Renaming and tagging files" enabled="true">
<Setting name="selectFiles" value="2" />
<Setting name="xsltPath" value="C:\W4IDataFlow\Xslt\ListFiles.xslt" />
<Setting name="version" value="2.0" /> <!-- 1.0|2.0 -->
<Setting name="removeW4IDataFlowProcessingNodes" value="false" />
</Task>
<Task id="4" name="Ftp" description="Downloading files" enabled="true">
<Setting name="command" value="download" />
<Setting name="protocol" value="ftp" /> <!-- ftp|ftps|sftp -->
<Setting name="server" value="127.0.1" />
<Setting name="port" value="21" />
<Setting name="user" value="user" />
<Setting name="password" value="password" />
<Setting name="path" value="/" />
<Setting name="selectFiles" todo="toDownload" from="app4" />
</Task>
<Task id="5" name="FilesMover" description="Moving files to Ftp_download" enabled="true">
<Setting name="selectFiles" value="4" />
<Setting name="destFolder" value="C:\W4IDataFlowTesting\Ftp_download\" />
<Setting name="overwrite" value="true" />
</Task>
</Tasks>
</Workflow>
Roughly speaking, the Ftp task loads the list of files located at the root folder of the FTP server in the running instance of the workflow, then the ListFiles task outputs and XML file that contains all the files loaded then the Xslt task takes as input this XML and generates an XML which contains a system node called W4IDataFlowProcessing which contains the list of files to be tagged and/or renamed.
To understand how tagging and renaming files work, refer to the documentation of the ListFiles and Xslt tasks.
Below is the XSLT ListFiles.xslt used for tagging files:
<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" indent="yes"/>
<xsl:template match="/">
<root>
<W4IDataFlowProcessing>
<xsl:for-each select="//W4IDataFlowProcessing/Workflow/Files//File">
<xsl:choose>
<xsl:when test="@name = 'file1.txt'">
<File taskId="{@taskId}" name="{@name}" renameTo="file1_renamed.txt"
todo="toRename"
from="app1" />
</xsl:when>
<xsl:when test="@name = 'file2.txt'">
<File taskId="{@taskId}" name="{@name}" renameTo="file2_renamed.txt"
todo="toSend"
from="app2" />
</xsl:when>
<xsl:when test="@name = 'file3.txt'">
<File taskId="{@taskId}" name="{@name}" renameTo="file3_renamed.txt"
todo="toDownload"
from="app3" />
</xsl:when>
<xsl:when test="@name = 'file4.txt'">
<File taskId="{@taskId}" name="{@name}" renameTo="file4_renamed.txt"
todo="toDownload"
from="app4" />
</xsl:when>
</xsl:choose>
</xsl:for-each>
</W4IDataFlowProcessing>
</root>
</xsl:template>
</xsl:stylesheet>
Execution graph
This workflow loads the file C:\W4IDataFlowTesting\file1.txt then uploads it to an FTP server then moves it to C:\W4IDataFlowTesting\Sent\ folder.
<Workflow xmlns="urn:W4IDataFlow-schema" id="6" name="Workflow_Ftp_upload" description="Workflow_Ftp_upload">
<Settings>
<Setting name="launchType" value="trigger" />
<Setting name="enabled" value="true" />
</Settings>
<Tasks>
<Task id="1" name="FilesLoader" description="Loading files" enabled="true">
<Setting name="file" value="C:\W4IDataFlowTesting\file1.txt" />
</Task>
<Task id="2" name="Ftp" description="Uploading files" enabled="true">
<Setting name="protocol" value="ftp" />
<Setting name="command" value="upload" />
<Setting name="server" value="127.0.1" />
<Setting name="port" value="21" />
<Setting name="user" value="user" />
<Setting name="password" value="password" />
<Setting name="path" value="/" />
<Setting name="selectFiles" value="1" />
</Task>
<Task id="3" name="FilesMover" description="Moving files to Sent folder" enabled="true">
<Setting name="selectFiles" value="1" />
<Setting name="destFolder" value="C:\W4IDataFlowTesting\Sent\" />
</Task>
</Tasks>
<ExecutionGraph>
<Task id="1"><Parent id="-1" /></Task>
<Task id="2"><Parent id="1" /></Task>
<Task id="3"><Parent id="2" /></Task>
</ExecutionGraph>
</Workflow>
First of all, the FilesLoader task loads the file C:\W4IDataFlowTesting\file1.txt then the Ftp task loads that file and uploads it to the FTP server. Finally, the FilesMover task moves that file to the folder C:\W4IDataFlowTesting\Sent.
By convention, the parent task id of the first task to be executed must always be -1.
However, if the execution graph is modified as follows:
<ExecutionGraph>
<Task id="1"><Parent id="-1" /></Task>
<Task id="3"><Parent id="1" /></Task>
<Task id="2"><Parent id="3" /></Task>
</ExecutionGraph>
Task 3 will be executed after task 1.
If the execution graph is modified as follows:
<ExecutionGraph>
<Task id="3"><Parent id="-1" /></Task>
<Task id="2"><Parent id="3" /></Task>
<Task id="1"><Parent id="2" /></Task>
</ExecutionGraph>
Task 3 will be executed first, then task 2 then task 1.
Two things are forbidden in the execution graph:
- Infinite loops.
- Parallel tasks.
Here is an example of infinite loops:
<ExecutionGraph>
<Task id="1"><Parent id="-1" /></Task>
<Task id="2"><Parent id="1" /></Task>
<Task id="1"><Parent id="2" /></Task>
</ExecutionGraph>
Here is an example of parallel tasks:
<ExecutionGraph>
<Task id="1"><Parent id="-1" /></Task>
<Task id="2"><Parent id="1" /></Task>
<Task id="3"><Parent id="1" /></Task>
</ExecutionGraph>
Flowchart workflows
A flowchart workflow is a workflow that contains at least one flowchart node (If/While/Switch) in its execution graph. A flowchart node takes as input a flowchart task and a set of tasks to execute in order, one by one. The order of the execution of the tasks can be altered by modifying the execution graph of the flowchart node.
If
The following workflow is a flowchart workflow that is triggered by the file file.trigger. If the file file.trigger is found on the file system then this workflow will upload the file file1.txt to an FTP server then it will notify customers that the upload was successful. Otherwise, if the trigger file.trigger is not found on the file system then the workflow will notify customers that the upload failed.
<Workflow xmlns="urn:W4IDataFlow-schema" id="7" name="Workflow_If" description="Workflow_If">
<Settings>
<Setting name="launchType" value="trigger" />
<Setting name="enabled" value="true" />
</Settings>
<Tasks>
<Task id="1" name="FilesLoader" description="Loading files" enabled="true">
<Setting name="file" value="C:\W4IDataFlowTesting\file1.txt" />
</Task>
<Task id="2" name="Ftp" description="Uploading files" enabled="true">
<Setting name="protocol" value="ftp" />
<Setting name="command" value="upload" />
<Setting name="server" value="127.0.1" />
<Setting name="port" value="21" />
<Setting name="user" value="user" />
<Setting name="password" value="password" />
<Setting name="path" value="/" />
<Setting name="selectFiles" value="1" />
</Task>
<Task id="3" name="FilesLoader" description="Loading emails (OK)" enabled="true">
<Setting name="file" value="C:\W4IDataFlowTesting\Emails\Emails.xml" />
</Task>
<Task id="4" name="MailsSender" description="Notifying customers (OK)" enabled="true">
<Setting name="selectFiles" value="3" />
<Setting name="host" value="127.0.0.1" />
<Setting name="port" value="587" />
<Setting name="enableSsl" value="true" />
<Setting name="user" value="user" />
<Setting name="password" value="password" />
</Task>
<Task id="5" name="FilesLoader" description="Loading emails (KO)" enabled="true">
<Setting name="file" value="C:\W4IDataFlowTesting\Emails\Emails.xml" />
</Task>
<Task id="6" name="MailsSender" description="Notifying customers (KO)" enabled="true">
<Setting name="selectFiles" value="5" />
<Setting name="host" value="127.0.0.1" />
<Setting name="port" value="587" />
<Setting name="enableSsl" value="true" />
<Setting name="user" value="user" />
<Setting name="password" value="password" />
</Task>
<Task id="99" name="FileExists" description="Checking trigger file" enabled="true">
<Setting name="file" value="C:\W4IDataFlowTesting\file.trigger" />
</Task>
</Tasks>
<ExecutionGraph>
<If id="100" parent="-1" if="99">
<Do>
<Task id="1"><Parent id="-1" /></Task>
<Task id="2"><Parent id="1" /></Task>
<Task id="3"><Parent id="2" /></Task>
<Task id="4"><Parent id="3" /></Task>
</Do>
<Else>
<Task id="5"><Parent id="-1" /></Task>
<Task id="6"><Parent id="5" /></Task>
</Else>
</If>
</ExecutionGraph>
</Workflow>
By convention, the parent task id of the first task to execute in
You can add If flowchart nodes pretty much wherever you want in the execution graph. Also, you can add as mush as you want. You can also add them in the event nodes OnSuccess, OnWarning and OnError.
An If can be inside an If, a While and a Switch.
While
This workflow is triggered by the file file.trigger. While the file file.trigger exists, this workflow will upload the file file1.txt to an FTP server then it will notify customers then it will wait for 2 days then it will start again.
<Workflow xmlns="urn:W4IDataFlow-schema" id="8" name="Workflow_While" description="Workflow_While">
<Settings>
<Setting name="launchType" value="trigger" />
<Setting name="enabled" value="true" />
</Settings>
<Tasks>
<Task id="1" name="FilesLoader" description="Loading files" enabled="true">
<Setting name="file" value="C:\W4IDataFlowTesting\file1.txt" />
</Task>
<Task id="2" name="Ftp" description="Uploading files" enabled="true">
<Setting name="protocol" value="ftp" />
<Setting name="command" value="upload" />
<Setting name="server" value="127.0.1" />
<Setting name="port" value="21" />
<Setting name="user" value="user" />
<Setting name="password" value="password" />
<Setting name="path" value="/" />
<Setting name="selectFiles" value="1" />
</Task>
<Task id="3" name="FilesLoader" description="Loading emails" enabled="true">
<Setting name="file" value="C:\W4IDataFlowTesting\Emails\Emails.xml" />
</Task>
<Task id="4" name="MailsSender" description="Notifying customers" enabled="true">
<Setting name="selectFiles" value="3" />
<Setting name="host" value="127.0.0.1" />
<Setting name="port" value="587" />
<Setting name="enableSsl" value="true" />
<Setting name="user" value="user" />
<Setting name="password" value="password" />
</Task>
<Task id="5" name="Wait" description="Waiting for 2 days..." enabled="true">
<Setting name="duration" value="02.00:00:00" />
</Task>
<Task id="99" name="FileExists" description="Checking trigger file" enabled="true">
<Setting name="file" value="C:\W4IDataFlowTesting\file.trigger" />
</Task>
</Tasks>
<ExecutionGraph>
<While id="100" parent="-1" while="99">
<Task id="1"><Parent id="-1" /></Task>
<Task id="2"><Parent id="1" /></Task>
<Task id="3"><Parent id="2" /></Task>
<Task id="4"><Parent id="3" /></Task>
<Task id="5"><Parent id="4" /></Task>
</While>
</ExecutionGraph>
</Workflow>
By convention, the parent task id of the first task to be executed in the
You can add While flowchart nodes pretty much wherever you want in the execution graph. Also, you can add as mush as you want. You can also add them in the event nodes OnSuccess, OnWarning and OnError.
A While can be inside a While, an If and a Switch.
Switch
This workflow starts every 24 hours. On Monday, it uploads files to an FTP server and on Wednesday it notifies customers.
<Workflow xmlns="urn:W4IDataFlow-schema" id="43" name="Workflow_Switch" description="Workflow_Switch">
<Settings>
<Setting name="launchType" value="periodic" />
<Setting name="period" value="1.00:00:00" />
<Setting name="enabled" value="true" />
</Settings>
<Tasks>
<Task id="1" name="Now" description="Getting current day" enabled="true">
<Setting name="culture" value="en-US" />
<Setting name="format" value="dddd" />
</Task>
<Task id="2" name="FilesLoader" description="Loading files" enabled="true">
<Setting name="file" value="C:\W4IDataFlowTesting\file1.txt" />
</Task>
<Task id="3" name="Ftp" description="Uploading files" enabled="true">
<Setting name="protocol" value="ftp" />
<Setting name="command" value="upload" />
<Setting name="server" value="127.0.1" />
<Setting name="port" value="21" />
<Setting name="user" value="user" />
<Setting name="password" value="password" />
<Setting name="path" value="/" />
<Setting name="selectFiles" value="1" />
</Task>
<Task id="4" name="FilesLoader" description="Loading emails" enabled="true">
<Setting name="file" value="C:\W4IDataFlowTesting\Emails\Emails.xml" />
</Task>
<Task id="5" name="MailsSender" description="Notifying customers" enabled="true">
<Setting name="selectFiles" value="3" />
<Setting name="host" value="127.0.0.1" />
<Setting name="port" value="587" />
<Setting name="enableSsl" value="true" />
<Setting name="user" value="user" />
<Setting name="password" value="password" />
</Task>
</Tasks>
<ExecutionGraph>
<Switch id="100" parent="-1" switch="1">
<Case value="Monday">
<Task id="2"><Parent id="-1" /></Task>
<Task id="3"><Parent id="2" /></Task>
</Case>
<Case value="Wednesday">
<Task id="4"><Parent id="-1" /></Task>
<Task id="5"><Parent id="4" /></Task>
</Case>
<Default />
</Switch>
</ExecutionGraph>
</Workflow>
By convention, the parent task id of the first task to be executed in the Case/Default nodes must always be -1.
You can add Switch flowchart nodes pretty much wherever you want in the execution graph. Also, you can add as mush as you want. You can also add them in the event nodes OnSuccess, OnWarning and OnError.
A Switch can be inside a While, an If and a Switch.
Approval workflows
Approval workflows are workflows marked as approval through approval setting option. They can be marked as approval whether from the Designer page in the back end or by XML editing:
<Workflow xmlns="urn:W4IDataFlow-schema" id="125" name="Workflow_Approval" description="Workflow_Approval">
<Settings>
<Setting name="launchType" value="trigger" />
<Setting name="enabled" value="true" />
<Setting name="approval" value="true" />
</Settings>
<LocalVariables />
<Tasks />
</Workflow>
Approval workflows must contain at least one Approval task. Approval tasks can be put wherever you want in the workflow and can be multiple. You can create workflows where some tasks get done then the workflow waits for approval then the users are notified for example.
Workflows are being approved whether from W4IDataFlow Manager or from Approval page in the backend.
If the workflow is rejected the OnRejected workflow event is raised and the tasks after Approval task are not executed.
The rejection of workflows can be done by clicking on reject button whether from Approval page in the back end or from W4IDataFlow Manager.
Simple approval workflow
To give you a hint on how approval workflows work, here is a very simple example:
<Workflow xmlns="urn:W4IDataFlow-schema" id="131" name="Workflow_Approval" description="Workflow_Approval">
<Settings>
<Setting name="launchType" value="trigger" />
<Setting name="enabled" value="true" />
<Setting name="approval" value="true" />
</Settings>
<LocalVariables />
<Tasks>
<Task id="1" name="Approval" description="Waiting for approval" enabled="true" />
<Task id="2" name="Wait" description="Waiting for 2 seconds" enabled="true">
<Setting name="duration" value="00.00:00:02" />
</Task>
</Tasks>
</Workflow>
This simple workflow is an approval workflow that waits for approval in order to start. Once approved, this workflow waits for 2 seconds. This workflow can be approved or rejected whether from W4IDataFlow Manager or from Approval page in the backend.
OnRejected workflow event
Here is another simple approval workflow:
<Workflow xmlns="urn:W4IDataFlow-schema" id="132" name="Workflow_Approval_Disapprove" description="Workflow_Approval_Disapprove">
<Settings>
<Setting name="launchType" value="trigger" />
<Setting name="enabled" value="true" />
<Setting name="approval" value="true" />
</Settings>
<LocalVariables />
<Tasks>
<Task id="1" name="Approval" description="Waiting for approval" enabled="true" />
<Task id="2" name="Wait" description="Waiting for 2 seconds" enabled="true">
<Setting name="duration" value="00.00:00:02" />
</Task>
<Task id="3" name="Wait" description="Waiting for 3 seconds" enabled="true">
<Setting name="duration" value="00.00:00:03" />
</Task>
</Tasks>
<ExecutionGraph>
<Task id="1"><Parent id="-1" /></Task>
<Task id="2"><Parent id="1" /></Task>
<OnRejected>
<Task id="3"><Parent id="-1" /></Task>
</OnRejected>
</ExecutionGraph>
</Workflow>
This simple workflow is an approval workflow that waits for approval in order to start. Once approved, this workflow waits for 2 seconds. If this workflow is rejected, the task 2 is not executed and the task 3 is executed. In other words, if this workflow is rejected it waits for 3 seconds. This workflow can be approved or rejected whether from W4IDataFlow Manager or from Approval page in the backend.
YouTube approval workflow
Here is a more professional approval workflow:
<Workflow xmlns="urn:W4IDataFlow-schema" id="132" name="Workflow_YouTube" description="Workflow_YouTube">
<Settings>
<Setting name="launchType" value="trigger" />
<Setting name="enabled" value="true" />
<Setting name="approval" value="true" />
</Settings>
<LocalVariables />
<Tasks>
<Task id="1" name="FilesLoader" description="Loading YouTube videos" enabled="true">
<Setting name="file" value="C:\W4IDataFlowTesting\YouTube\YouTube.xml" />
</Task>
<Task id="2" name="YouTube" description="Uploading YouTube videos" enabled="true">
<Setting name="selectFiles" value="1" />
<Setting name="user" value="username" />
<Setting name="applicationName" value="W4IDataFlow" />
<Setting name="clientSecrets" value="C:\W4IDataFlow-dotnet-core\client_secrets.json" />
</Task>
<Task id="3" name="Approval" description="Waiting for approval" enabled="true" />
<Task id="4" name="FilesLoader" description="Loading notification mails" enabled="true">
<Setting name="file" value="C:\W4IDataFlowTesting\Mails\mails.xml" />
</Task>
<Task id="5" name="MailsSender" description="Sending notification mails" enabled="true">
<Setting name="selectFiles" value="4" />
<Setting name="host" value="smtp.gmail.com" />
<Setting name="port" value="587" />
<Setting name="enableSsl" value="true" />
<Setting name="user" value="user" />
<Setting name="password" value="password" />
</Task>
</Tasks>
</Workflow>
This workflow starts by uploading videos to YouTube then it waits for approval in order to check that videos have been effectively uploaded with success to YouTube and edited by the management team. Then, if this workflow is approved from Approval page in the backend or from W4IDataFlow Manager.
When this workflow arrives to Approval task, it suspends its jobs and waits for approval process until it’s being approved and then continues its tasks.
Form submission approval workflow
Here is another interesting approval workflow:
<Workflow xmlns="urn:W4IDataFlow-schema" id="134" name="Workflow_FormSubmission" description="Workflow_FormSubmission">
<Settings>
<Setting name="launchType" value="trigger" />
<Setting name="enabled" value="true" />
<Setting name="approval" value="true" />
</Settings>
<Tasks>
<Task id="1" name="ProcessLauncher" description="Form submission" enabled="true">
<Setting name="processPath" value="C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" />
<Setting name="processCmd" value="https://docs.google.com/forms/d/1sHnPCJ05GLecqvZb0MNeFkFK0eMuVqBUyWAo5uurEQ8/prefill" />
<Setting name="hideGui" value="false" />
<Setting name="generatesFiles" value="false" />
</Task>
<Task id="2" name="Approval" description="Waiting for approval" enabled="true" />
<Task id="3" name="Wait" description="Waiting for 2 seconds" enabled="true">
<Setting name="duration" value="00.00:00:02" />
</Task>
<Task id="4" name="Wait" description="Waiting for 3 seconds" enabled="true">
<Setting name="duration" value="00.00:00:03" />
</Task>
</Tasks>
<ExecutionGraph>
<Task id="1"><Parent id="-1" /></Task>
<Task id="2"><Parent id="1" /></Task>
<!-- You can add other tasks here depending on your need. -->
<Task id="3"><Parent id="2" /></Task>
<OnRejected>
<!-- You can add other tasks here depending on your need. -->
<Task id="4"><Parent id="-1" /></Task>
</OnRejected>
</ExecutionGraph>
</Workflow>
This approval workflow opens a submission form and waits for approval. If the submission is correct, the workflow is approved and waits for 2 seconds (this is just a simple task for testing but you can add email tasks or whatever). Otherwise, if the submission is incorrect, the workflow is rejected and waits for 3 seconds (this is just a simple task for testing but you can add email tasks or whatever). This workflow works on the .NET Core version of W4IDataFlow only because the .NET version of W4IDataFlow does not support opening GUI from ProcessLauncher task since W4IDataFlow server is running in a Windows service in the .NET version.
Approval workflows are very useful when some tasks get done then you have to wait for approval to check that previous tasks have been done with success then users are notified for example. This is just an example, but you can create and imagine other examples as you want and as you need.
Records
W4IDataFlow allows approval workflows on generic assets called records. A record is an entity that refers to a file. Each record has a name, a description, file versions, comments, and approval start and end dates.
A manager can assign a record to a user. If the record is updated with required info, the manajer can approve or reject the workflow.
Here is a simple approval workflow on records:
<?xml version="1.0" encoding="utf-8"?>
<Workflow xmlns="urn:W4IDataFlow-schema" id="146" name="Workflow_ApproveDocument" description="Workflow_ApproveDocument">
<Settings>
<Setting name="launchType" value="trigger" />
<Setting name="enabled" value="true" />
<Setting name="approval" value="true" />
<Setting name="enableParallelJobs" value="true" />
</Settings>
<LocalVariables />
<Tasks>
<Task id="1" name="ApproveRecord" description="Approve document" enabled="true">
<Setting name="record" value="1" />
<Setting name="assignedTo" value="W4IDataFlow" />
</Task>
</Tasks>
</Workflow>
The manager assigns the record 1 to the user W4IDataFlow. The user W4IDataFlow will receive a notification in W4IDataFlow and an email if email settings are properly configured to edit the record in question. After the record is updated, the manager will receive a notification and can then check the latest version of the recod. If the record info is good, the manager can approve the workflow. Otherwise, he can reject it.
Workflow events
After a workflow finishes its job, its final result is either success, or warning or error or rejected. If its final result is success, the OnSuccess event is triggered. If its final result is warning, the OnWarning event is triggered. If its final result is error, the OnError event is triggered. If the workflow is rejected, the OnRejected event is triggered. An event contains a set of tasks and/or flowchart nodes to execute in order, one by one. The order of the execution of the tasks and/or flowchart nodes can be altered by modifying the execution graph of the event.
This workflow uploads the file1.txt to an FTP server then notifies customers in case of success.
<Workflow xmlns="urn:W4IDataFlow-schema" id="9" name="Workflow_Events" description="Workflow_Events">
<Settings>
<Setting name="launchType" value="trigger" />
<Setting name="enabled" value="true" />
</Settings>
<Tasks>
<Task id="1" name="FilesLoader" description="Loading files" enabled="true">
<Setting name="file" value="C:\W4IDataFlowTesting\file1.txt" />
</Task>
<Task id="2" name="Ftp" description="Uploading files" enabled="true">
<Setting name="protocol" value="ftp" />
<Setting name="command" value="upload" />
<Setting name="server" value="127.0.1" />
<Setting name="port" value="21" />
<Setting name="user" value="user" />
<Setting name="password" value="password" />
<Setting name="path" value="/" />
<Setting name="selectFiles" value="1" />
</Task>
<Task id="3" name="FilesLoader" description="Loading emails" enabled="true">
<Setting name="file" value="C:\W4IDataFlowTesting\Emails\Emails.xml" />
</Task>
<Task id="4" name="MailsSender" description="Notifying customers" enabled="true">
<Setting name="selectFiles" value="3" />
<Setting name="host" value="127.0.0.1" />
<Setting name="port" value="587" />
<Setting name="enableSsl" value="true" />
<Setting name="user" value="user" />
<Setting name="password" value="password" />
</Task>
</Tasks>
<ExecutionGraph>
<Task id="1"><Parent id="-1" /></Task>
<Task id="2"><Parent id="1" /></Task>
<OnSuccess>
<Task id="3"><Parent id="-1" /></Task>
<Task id="4"><Parent id="3" /></Task>
</OnSuccess>
</ExecutionGraph>
</Workflow>
The flowchart event nodes OnWarning, OnError and OnRejected can be used in the same way. You can put If and While flowchart nodes in event nodes.
For OnRejected workflow event, the workflow must be an approval workflow and must contain at least one Approval task. The OnRejected workflow event is raised once the end user clicks on reject button whether from the Approval page in the backend or from W4IDataFlow Manager.
These are simple and basic workflows to give an idea on how to make your own workflows. However, if you have multiple systems, applications and automations involved in a workflow, the workflow could be very interesting.
7. Tasks
Tasks
W4IDataFlow is modular. A workflow executes a set of tasks. The user can choose between the built-in tasks that come with W4IDataFlow or create his own custom tasks.
Every task is a module which can be enabled, disabled or replaced. W4IDataFlow provides 100+ built-in tasks.
*: The task is not available in the .NET Core version.
**: The task is available in the .NET Core version only.
- File system tasks
- Encryption tasks
- Compression tasks
- Iso tasks
- Speech tasks
- Hashing tasks
- Process tasks
- Network tasks
- XML tasks
- SQL tasks
- WMI tasks
- Image tasks
- Audio and video tasks
- Email tasks
- Workflow tasks
- Social media tasks
- Waitable tasks
- Reporting tasks
- Web tasks
- Script tasks
- JSON and YAML tasks
- Entities tasks
- Flowchart tasks
- Approval tasks
- Notification tasks
- SMS tasks
File system tasks
These tasks allow to create, copy, move, rename or delete files and directories on a file system. These tasks allow also to check whether a collection of remote or local files and/or directories exists. These tasks allow also to check whether two files are the same and allow also to calculate the diff of two files.
- FilesLoader: This task loads a collection of files located in folders or through the file option.
- FilesLoaderEx: This task loads a collection of files located in folders or through the file option. This task is inherited from FilesLoader task, but by default result is empty, you must configure file system attributes rules to populate result.
- FilesCopier: This task copies a collection of files to a destination folder.
- FilesMover: This task moves a collection of files to a destination folder.
- FilesRemover: This task deletes a collection of files.
- FilesRenamer: This task allows to rename a collection of files on a file system. The Xslt task can be used along with the ListFiles task to create new file names.
- FilesExist: This task checks whether a collection of files and/or directories exists.
- FilesEqual: This task checks whether two files are the same.
- FilesDiff*: This task calculates the diff of two files.
- FilesConcat: This task concatenates a collection of files.
- FilesJoiner: This task concatenates a collection of files. This task should be used to join and restore original file splitted with “FilesSplitter” task. Original file name is restored from splitted file name with end part “_N”.
- FilesConcat: This task splits files into chunks.
- FilesInfo: This task generates files information of a collection of files and writes the results in an XML file. The format of the output XML file is described in the documentation of the task.
- Touch: This task creates a collection of empty files.
- ListFiles: This task lists all the files loaded by the workflow tasks in the logs. This task is useful for resolving issues.
- Mkdir: This task creates a collection of folders.
- Rmdir: This task deletes a collection of folders.
- Movedir: This task moves a folder and allows to overwrite the destination folder.
- FileSystemWatcher: This task watches a hot folder and triggers tasks on file created, changed or deleted.
Encryption tasks
These tasks allow to encrypt and decrypt any type of files of any size. These tasks allow also to encrypt and decrypt text based files.
- FilesEncryptor: This task encrypts a collection of files.
- FilesEncryptor: This task decrypts the files crypted by the task FilesEncryptor.
- TextsEncryptor: This task encrypts a collection of text based files.
- TextsDecryptor: This task decrypts the files crypted by the task TextsEncryptor.
Compression tasks
These tasks allow to create a .zip, a .tar, a .tar.gz or a .7z from a collection of files. These tasks allow also to extract .zip, .tar, .tar.gz, .rar or .7z archives.
- Zip: This task creates a zip archive from a collection of files.
- Tar: This task creates a tar archive from a collection of files.
- Tgz: This task creates a tar.gz archive from a collection of files.
- SevenZip*: This task creates a .7z archive from a collection of files.
- Unzip: This task extracts ZIP archives.
- Untar: This task extracts TAR archives.
- Untgz: This task extracts TAR.GZ archives.
- Unrar*: This task extracts RAR archives.
- UnSevenZip*: This task extracts 7Z archives.
Iso tasks
These tasks allow to create a .iso from a source folder and to extract a .iso to a destination folder.
- IsoCreator*: This task creates a .iso from a source folder.
- IsoExtractor*: This task extracts .iso files.
Speech tasks
These tasks allow to convert text to speech and speech to text.
- TextToSpeech*: This task converts text files to speech.
- SpeechToText*: This task converts audio files to text files.
Hashing tasks
These tasks allow to generate MD5, SHA-1, SHA-256 and SHA-512 hashes of a collection of files.
- Md5: This task generates MD5 sums of a collection of files and writes the results in an XML file. The format of the output XML file is described in the documentation of the task.
- Sha1: This task generates SHA-1 hashes of a collection of files and writes the results in an XML file. The format of the output XML file is described in the documentation of the task.
- Sha256: This task generates SHA-256 hashes of a collection of files and writes the results in an XML file. The format of the output XML file is described in the documentation of the task.
- Sha512: This task generates SHA-512 hashes of a collection of files and writes the results in an XML file. The format of the output XML file is described in the documentation of the task.
Process tasks
These tasks allow to start or kill processes on the workflow server. They also allow to retrieve information about a process.
- ProcessLauncher: This task launches a process. If the process generates a file as output It is possible to pass a collection of files to the task so that for each file an output file will be generated through the process. Read the documentation of the task for further informations.
- ProcessKiller*: This task kills a process.
- ProcessInfo: This task shows information about a process.
- SshCmd: This task executes an SSH command.
Network tasks
These task allow to list, upload, download or delete files over FTP, FTPS or SFTP. These tasks allow also to download files over HTTP or HTTPS. These tasks allow also to download torrent files, to ping servers and to execute GET/POST/PUT/PATCH/DELETE requests.
- Ftp: This task allows to list, upload, download or delete files over FTP, FTPS or SFTP.
- Http: This task allows to downoad files over HTTP or HTTPS.
- HttpGet: This task executes a GET request.
- HttpPost: This task executes a POST request.
- HttpPut: This task executes a PUT request.
- HttpPatch: This task executes a PATCH request.
- HttpDelete: This task executes a DELETE request.
- Torrent: This task downloads torrent files.
- Ping: This is a flowchart task that checks whether a server responds to a ping request or not.
XML tasks
These tasks allow to work with XML and CSV data. XSLT can be used along with XPath to generate XML documents. XSLT 1.0 and XSLT 2.0 are supported.
- CsvToXml: This task transforms a CSV file to an XML file.
- XmlToCsv: This task transforms an XML file to a CSV file. The format of the input XML file is described in the documentation of the task.
- Xslt: This task transforms a collection of XML files. It is possible to use XSLT 1.0 processor or XSLT 2.0 processor.
- Guid: This task generates Guids and outputs the result in an XML file.
SQL tasks
These tasks allow to execute SQL scripts. These tasks supports Microsoft Sql Server, Microsoft Access, Oracle, MySql, SQLite, PostGreSql and Teradata. These tasks can be used for bulk insert, for database updates, for database cleanup, for rebuilding indexes, for reorganizing indexes, for shrinking databases, for updating statistics, for transfering database data and so on. These tasks allow also to export SQL data to XML or CSV and to import CSV data to a database. These tasks allow also to backup and restore databases.
- Sql: This task executes SQL scripts. It supports Microsoft Sql Server, Microsoft Access, Oracle, MySql, SQLite, PostGreSql and Teradata.
- SqlToXml: This task executes SQL scripts and outputs the results in XML files. It supports Microsoft Sql Server, Microsoft Access, Oracle, MySql, SQLite, PostGreSql and Teradata.
- SqlToCsv: This task executes SQL scripts and outputs the results in CSV files. It supports Microsoft Sql Server, Microsoft Access, Oracle, MySql, SQLite, PostGreSql and Teradata.
- CsvToSql: This task converts CSV files to SQL scripts (SQL Server Insert only).
WMI tasks
- Wmi*: This task executes a WMI query and outputs the results in an XML file. The format of the output XML file is described in the documentation of the task.
Image tasks
These tasks allow to convert images to the following formats Bmp, Emf, Exif, Gif, Icon, Jpeg, Png, Tiff and Wmf. These tasks allow also to resize or to crop or to concatenate or to overlay images.
- ImagesTransformer: This task transforms a collection of image files to a specified format. The output format can be one of the followings Bmp, Emf, Exif, Gif, Icon, Jpeg, Png, Tiff or Wmf.
- ImagesResizer: This task resizes a collection of images.
- ImagesCropper: This task crops a collection of images.
- ImagesConcat: This task concatenates a collection of images.
- ImagesOverlay: This task overlays a collection of images.
Audio and video tasks
These tasks allow to convert, cut or edit audio and video files through FFMEG, VLC or any other audio/video software. These tasks can also be used to perform custom operations such as generating images and thumbnails from video files. These tasks allow also to generate the most relevant technical and tag data for video and audio files.
- MediaInfo*: This task generates the most relevant technical and tag data for video and audio files and outputs the results in an XML file. The format of the output XML file is described in the documentation of the task.
- YouTube**: This task uploads videos to YouTube.
- YouTubeSearch**: This task searches for content on YouTube.
- YouTubeListUploads**: This task retrieves a list of videos uploaded to a YouTube channel.
- Vimeo: This task uploads videos to Vimeo.
- VimeoListUploads: This task retrieves a list of videos uploaded to a Vimeo channel.
The task ProcessLauncher can be used along with FFMPEG, VLC or any other software in order to perform audio and video tasks.
Email tasks
This task allows to send or fetch a collection of emails.
- MailsSender: This task sends a collection of emails from XML files. The format of the input XML files is described in the documentation of the task.
- MailsReceiver: This task fetches a collection of emails.
Workflow tasks
- Workflow*: This task allows to start, suspend, resume, stop, approve or disapprove a list of workflows.
- SubWorkflow: This task kicks off a sub workflow.
Social media tasks
- Twitter: This task sends tweets.
- InstagramUploadImage: This task uploads images to Instagram.
- InstagramUploadVideo: This task uploads videos to Instagram.
- Reddit**: This task sends posts and links to Reddit.
- RedditListPosts**: This task retrieves Reddit post history.
- RedditListComments**: This task retrieves Reddit comment history.
Waitable tasks
- Wait: This task waits for a specified duration of time.
Reporting tasks
These tasks allow to generate reports in PDF format from HTML or XML or TXT files.
- TextToPdf*: This task generates PDF files from TEXT files.
- HtmlToPdf*: This task generates PDF files from HTML files.
- PdfToText*: This task extracts TEXT from PDF files.
The task Xslt can be used to generate HTML reports from XML files. Then, the HTML reports can be transformed into PDF reports through HtmlToPdf task.
Web tasks
These tasks allow to take screenshots from urls and to download the HTML source code from urls after the pages are rendered. These tasks allow also to uglify, minify and compress JavaScript, CSS and HTML files. These tasks allow also to extract text from HTML files. These tasks allow also to convert SCSS files to CSS files.
- WebToScreenshot*: This task takes screenshots from urls.
- WebToHtml*: This task retrieves HTML sources from urls.
- UglifyJs*: This task uglifys JavaScript files.
- UglifyCss*: This task compresses and minifies CSS files.
- UglifyHtml*: This task compresses and minifies HTML files.
- HtmlToText*: This task extracts text from HTML files.
- ScssToCss*: This task converts SCSS files to CSS files.
Script tasks
These tasks allow to execute C# and VB scripts.
JSON and YAML tasks
These tasks allow to convert YAML files to JSON files, JSON files to YAML files, CSV files to JSON files and CSV files to YAML files.
- YamlToJson*: This task converts YAML files to JSON files.
- JsonToYaml*: This task converts JSON files to YAML files.
- CsvToJson*: This task converts CSV files to JSON files.
- CsvToYaml*: This task converts CSV files to YAML files.
Entities tasks
- ListEntities: This task lists all the entities loaded by the workflow tasks in the logs. This task is useful for resolving issues.
Flowchart tasks
These tasks can be used within flowchart workflows to perform specific jobs.
- FileExists: This is a flowchart task that checks whether a given file exists on a file system or not.
- FileMatch: This is a flowchart task that checks whether a file exists or not in a directory by using a regex pattern.
- Now: This is a flowchart task that retrieves the current date in the specified format. This task is designed to be used in a Switch flowchart node.
- Ping: This is a flowchart task that checks whether a server responds to a ping request or not.
- EnvironmentVariable: This is a flowchart task that retrieves the value of an environment variable.
- MessageCorrect: This is a flowchart task that checks whether a message is in the memory of the task having as key the value of the setting option checkString.
- FolderExists: This is a flowchart task that checks whether a given folder exists on a file system or not.
- FileContentMatch: This task checks whether the content of a file matches a regex pattern.
Approval tasks
- Approval: This task marks the current workflow as needing approval.
- ApproveRecord: This task assigns a record to a user and launches the approval process on that record.
- ApprovalRecordsCreator: This task creates records from files.
- ApprovalWorkflowsCreator: This task creates approval workflows for records from shared memory and starts them. The record ids are sent from ApprovalRecordsCreator task.
Notification tasks
- Slack: This task sends Slack messages.
SMS tasks
- Twilio: This task sends SMS messages.
8. Local variables
Local Variables
It is possible to declare local variables in a workflow.
The syntax is as follows:
<Workflow xmlns="urn:W4IDataFlow-schema" id="115" name="Workflow_FilesLoaderLocalVariables" description="Workflow_FilesLoaderLocalVariables">
<Settings>
<Setting name="launchType" value="trigger" />
<Setting name="enabled" value="true" />
</Settings>
<LocalVariables>
<Variable name="myFile1" value="C:\W4IDataFlowTesting\file1.txt" />
<Variable name="myFile2" value="C:\W4IDataFlowTesting\file2.txt" />
<Variable name="myFile3" value="C:\W4IDataFlowTesting\file3.txt" />
</LocalVariables>
<Tasks>
<Task id="1" name="FilesLoader" description="Loading files" enabled="true">
<Setting name="file" value="$myFile1" />
<Setting name="file" value="$myFile2" />
<Setting name="file" value="$myFile3" />
</Task>
<Task id="2" name="ListFiles" description="Listing files" enabled="true">
</Task>
</Tasks>
</Workflow>
When W4IDataFlow server loads the workflow file, the workflow file will be parsed and thus the local variables will be replaced by their respective values.
It is also possible to combine global variables with local variables, here is an example:
GlobalVariables.xml:
<?xml version="1.0" encoding="utf-8" ?>
<GlobalVariables>
<Variable name="W4IDataFlowTesting" value="C:\W4IDataFlowTesting" />
<Variable name="fileName1" value="file1.txt" />
<Variable name="fileName2" value="file2.txt" />
<Variable name="fileName3" value="file3.txt" />
</GlobalVariables>
Below a sample workflow which contains global variables and local variables:
<Workflow xmlns="urn:W4IDataFlow-schema" id="115" name="Workflow_FilesLoaderLocalVariables" description="Workflow_FilesLoaderLocalVariables">
<Settings>
<Setting name="launchType" value="trigger" />
<Setting name="enabled" value="true" />
</Settings>
<LocalVariables>
<Variable name="myFile1" value="$W4IDataFlowTesting\$fileName1" />
<Variable name="myFile2" value="$W4IDataFlowTesting\$fileName2" />
<Variable name="myFile3" value="$W4IDataFlowTesting\$fileName3" />
</LocalVariables>
<Tasks>
<Task id="1" name="FilesLoader" description="Loading files" enabled="true">
<Setting name="file" value="$myFile1" />
<Setting name="file" value="$myFile2" />
<Setting name="file" value="$myFile3" />
</Task>
<Task id="2" name="ListFiles" description="Listing files" enabled="true">
</Task>
</Tasks>
</Workflow>9. Global Variables
Global Variables
Global variables are declared by default in the file C:\W4IDataFlow\GlobalVariables.xml
The path of this file can be edited from the configuration file C:\W4IDataFlow\W4IDataFlow.xml
Here is an example of GlobalVariables.xml:
<?xml version="1.0" encoding="utf-8" ?>
<GlobalVariables>
<Variable name="file1" value="C:\W4IDataFlowTesting\file1.txt" />
<Variable name="file2" value="C:\W4IDataFlowTesting\file2.txt" />
<Variable name="file3" value="C:\W4IDataFlowTesting\file3.txt" />
</GlobalVariables>
The variables can then be used in workflow files as follow:
<Workflow xmlns="urn:W4IDataFlow-schema" id="114" name="Workflow_FilesLoaderGlobalVariables" description="Workflow_FilesLoaderGlobalVariables">
<Settings>
<Setting name="launchType" value="trigger" />
<Setting name="enabled" value="true" />
</Settings>
<Tasks>
<Task id="1" name="FilesLoader" description="Loading files" enabled="true">
<Setting name="file" value="$file1" />
<Setting name="file" value="$file2" />
<Setting name="file" value="$file3" />
</Task>
<Task id="2" name="ListFiles" description="Listing files" enabled="true">
</Task>
</Tasks>
</Workflow>
When W4IDataFlow server loads the workflow file, the workflow file is parsed so that the global variables are replaced by their values.
10. Cron Scheduling
Cron Scheduling
Introduction
cron is a UNIX tool that has been around for a long time, so its scheduling capabilities are powerful and proven. W4IDataFlow provides the ability to create workflows that start depending on a cron expression.
Cron workflows are often more useful than trigger or periodic workflows, if you need a job-firing schedule that recurs based on calendar-like notions, rather than on the exactly specified intervals.
With cron workflows, you can specify firing-schedules such as “every Friday at noon”, or “every weekday and 9:30 am”, or even “every 5 minutes between 9:00 am and 10:00 am on every Monday, Wednesday and Friday”.
A cron expression is a string comprised of 6 or 7 fields separated by white space. Fields can contain any of the allowed values, along with various combinations of the allowed special characters for that field. The fields are as follows:
* * * * * * *
┬ ┬ ┬ ┬ ┬ ┬ ┬
│ │ │ │ │ │ └ Year (1970 - 2099) (Optional) (Allowed Special Characters: , - * /)
│ │ │ │ │ └────── Day of week (1 - 7) (Sunday=1) (Allowed Special Characters: , - * ? / L #)
│ │ │ │ └─────────── Month (1 - 12) (Jan=1) (Allowed Special Characters: , - * /)
│ │ │ └──────────────── Day of month (1 - 31) (Allowed Special Characters: , - * ? / L W)
│ │ └───────────────────── Hours (0 - 23) (Allowed Special Characters: , - * /)
│ └────────────────────────── Minutes (0 - 59) (Allowed Special Characters: , - * /)
└─────────────────────────────── Seconds (0 - 59) (Allowed Special Characters: , - * /)
So cron expressions can be as simple as this: ? *
or more complex, like this: 0/5 14,18,3-39,52 * ? JAN,MAR,SEP MON-FRI 2002-2010
Pay attention
W4IDataFlow is using Quartz.NET cron expressions. UNIX Cron expressions and Quartz ones are different. Simply:
- In Unix:
(minute, hour, day, month, day_of_week, year)
- In Quartz:
(second, minute, hour, day, month, day_of_week, year)
Special characters
- “” (“all values”) – used to select all values within a field. For example, “” in the minute field means * “every minute”.
- ? (“no specific value”) – useful when you need to specify something in one of the two fields in which the character is allowed, but not the other. For example, if I want my workflow to fire on a particular day of the month (say, the 10th), but don’t care what day of the week that happens to be, I would put “10” in the day-of-month field, and “?” in the day-of-week field. See the examples below for clarification.
- – used to specify ranges. For example, “10-12” in the hour field means “the hours 10, 11 and 12”.
- , used to specify additional values. For example, “MON,WED,FRI” in the day-of-week field means “the days Monday, Wednesday, and Friday”.
- / – used to specify increments. For example, “0/15” in the seconds field means “the seconds 0, 15, 30, and 45”. And “5/15” in the seconds field means “the seconds 5, 20, 35, and 50”. You can also specify ‘/’ after the ‘’ character – in this case ‘’ is equivalent to having ‘0’ before the ‘/’. ‘1/3’ in the day-of-month field means “fire every 3 days starting on the first day of the month”.
- L (“last”) – has different meaning in each of the two fields in which it is allowed. For example, the value “L” in the day-of-month field means “the last day of the month” – day 31 for January, day 28 for February on non-leap years. If used in the day-of-week field by itself, it simply means “7” or “SAT”. But if used in the day-of-week field after another value, it means “the last xxx day of the month” – for example “6L” means “the last friday of the month”. You can also specify an offset from the last day of the month, such as “L-3” which would mean the third-to-last day of the calendar month. When using the ‘L’ option, it is important not to specify lists, or ranges of values, as you’ll get confusing/unexpected results.
- W (“weekday”) – used to specify the weekday (Monday-Friday) nearest the given day. As an example, if you were to specify “15W” as the value for the day-of-month field, the meaning is: “the nearest weekday to the 15th of the month”. So if the 15th is a Saturday, the workflow will fire on Friday the 14th. If the 15th is a Sunday, the workflow will fire on Monday the 16th. If the 15th is a Tuesday, then it will fire on Tuesday the 15th. However if you specify “1W” as the value for day-of-month, and the 1st is a Saturday, the workflow will fire on Monday the 3rd, as it will not ‘jump’ over the boundary of a month’s days. The ‘W’ character can only be specified when the day-of-month is a single day, not a range or list of days. ** The ‘L’ and ‘W’ characters can also be combined in the day-of-month field to yield ‘LW’, which translates to “last weekday of the month”.
- # used to specify “the nth” XXX day of the month. For example, the value of “6#3” in the day-of-week field means “the third Friday of the month” (day 6 = Friday and “#3” = the 3rd one in the month). Other examples: “2#1” = the first Monday of the month and “4#5” = the fifth Wednesday of the month. Note that if you specify “#5” and there is not 5 of the given day-of-week in the month, then no firing will occur that month. ** The legal characters and the names of months and days of the week are not case sensitive. MON is the same as mon.
Examples
Here are some examples:
0 0 12 * * ? Fire at 12pm (noon) every day.
0 15 10 ? * * Fire at 10:15am every day.
0 15 10 * * ? Fire at 10:15am every day.
0 15 10 * * ? * Fire at 10:15am every day.
0 15 10 * * ? 2019 Fire at 10:15am every day during the year 2019.
0 * 14 * * ? Fire every minute starting at 2pm and ending at 2:59pm, every day.
0 0/5 14 * * ? Fire every 5 minutes starting at 2pm and ending at 2:55pm, every day.
0 0/5 14,18 * * ? Fire every 5 minutes starting at 2pm and ending at 2:55pm, AND fire every 5 minutes starting at 6pm and ending at 6:55pm, every day.
0 0-5 14 * * ? Fire every minute starting at 2pm and ending at 2:05pm, every day.
0 10,44 14 ? 3 WED Fire at 2:10pm and at 2:44pm every Wednesday in the month of March.
0 15 10 ? * MON-FRI Fire at 10:15am every Monday, Tuesday, Wednesday, Thursday and Friday.
0 15 10 15 * ? Fire at 10:15am on the 15th day of every month.
0 15 10 L * ? Fire at 10:15am on the last day of every month.
0 15 10 L-2 * ? Fire at 10:15am on the 2nd-to-last last day of every month.
0 15 10 ? * 6L Fire at 10:15am on the last Friday of every month.
0 15 10 ? * 6L Fire at 10:15am on the last Friday of every month.
0 15 10 ? * 6L 2019-2020 Fire at 10:15am on every last friday of every month during the years 2019 and 2020.
0 15 10 ? * 6#3 Fire at 10:15am on the third Friday of every month.
0 0 12 1/5 * ? Fire at 12pm (noon) every 5 days every month, starting on the first day of the month.
0 11 11 11 11 ? Fire every November 11th at 11:11am.
Pay attention to the effects of ‘?’ and ‘*’ in the day-of-week and day-of-month fields!
Cron launch type
It is possible to create this type of workflows from W4IDataFlow Designer or through XML editing. However, here is a sample workflow that starts every two minutes:
<?xml version="1.0" encoding="utf-8"?>
<Workflow xmlns="urn:W4IDataFlow-schema" id="75" name="Workflow_Cron" description="Workflow_Cron">
<Settings>
<Setting name="launchType" value="cron" />
<Setting name="cronExpression" value="0 0/2 * * * ?" /> <!-- Every two minutes -->
<Setting name="enabled" value="true" />
</Settings>
<Tasks>
<Task id="1" name="FilesLoader" description="Loading files" enabled="true">
<Setting name="file" value="C:\W4IDataFlowTesting\file1.txt" />
</Task>
<Task id="2" name="Wait" description="Wait for 10 seconds..." enabled="true">
<Setting name="duration" value="00.00:00:10" />
</Task>
<Task id="3" name="FilesCopier" description="Copying files" enabled="true">
<Setting name="selectFiles" value="1" />
<Setting name="destFolder" value="C:\W4IDataFlowTesting\Cron" />
<Setting name="overwrite" value="true" />
</Task>
</Tasks>
</Workflow>11. Logging
Logging
Everything that happens in W4IDataFlow is traced and logged. With W4IDataFlow’s logging system, you can track your workflows with ease and stay informed with real-time monitoring and email notifications.
W4IDataFlow’s logs are written in C:\Program Files\W4IDataFlow\W4IDataFlow.log. There is one log file per day. The old log files are saved in following format: W4IDataFlow.logyyyyMMdd
It is possible to configure W4IDataFlow to send incident reports when an error occurs by using log4net.Appender.SmtpAppender in the configuration file of W4IDataFlow C:\Program Files\W4IDataFlow\W4IDataFlow.Clients.WindowsService.exe.config. Below a sample configuration:
<appender name="SmtpAppender" type="log4net.Appender.SmtpAppender">
<to value="foo@bar.com" />
<from value="baz@bar.com" />
<subject value="Some subject" />
<smtpHost value="smtp.gmail.com" />
<authentication value="Basic" />
<port value="587" />
<username value="gmail user name" />
<password value="gmail password" />
<bufferSize value="1" />
<EnableSsl value="true"/>
<lossy value="true" />
<evaluator type="log4net.Core.LevelEvaluator">
<threshold value="ERROR"/>
</evaluator>
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%newline%date [%thread] %-5level %logger [%property{NDC}] - %message%newline%newline%newline" />
</layout>
</appender>
W4IDataFlow gives you a beautiful Dashboard to view real-time statistics on your workflows. The Dashboard will let you track your workflow server with ease.
The History page in the backend will also let you track all your workflows and everything that happens on the workflow server. Indeed, from this page you will have an overview of all the workflow instances executed on the workflow server.
12. Custom Tasks
Custom Tasks
Custom tasks are a must in a workflow engine and allow systems and applications to interact.
How to create a custom task?
General
To create a custom task MyTask for example you will need to proceed as follows:
- Create a new class library project in Visual Studio and name it W4IDataFlow.Tasks.MyTask. For .NET, you must target .NET 4.8. For .NET Core, you must target .NET 7.0.
- Reference W4IDataFlow dependencies through nuget package manager:
PM> Install-Package W4IDataFlow
Or by using .NET CLI (.NET Core version):
dotnet add package W4IDataFlow
- Create a public class MyTask that implements the abstract class W4IDataFlow.Core.Task.
W4IDataFlow.Tasks.MyTask code should look like as follows:
using System.Threading;
using System.Xml.Linq;
using W4IDataFlow.Core;
namespace W4IDataFlow.Tasks.MyTask
{
public class MyTask : Task
{
public MyTask(XElement xe, Workflow wf) : base(xe, wf)
{
// Task settings goes here
}
public override TaskStatus Run()
{
try
{
// Task logic goes here
return new TaskStatus(Status.Success);
}
catch (ThreadAbortException)
{
throw;
}
}
}
}
Each task returns a TaskStatus object when it finishes performing its job. TaskStatus is composed of the following elements:
public Status Status { get; set; }
public bool Condition { get; set; }
public string SwitchValue { get; set; }
The Status can be one of the followings:
public enum Status
{
Success,
Warning,
Error
}
For example, if a task performs an opetation on a collection of files and if this operation succeeds for all the files then its Status should be Success. Otherwise if this operation succeeds for some files and fails for others then its Status should be Warning. Otherwise if this operation fails for all the files then its Status should be Error.
The Condition property is designed for flowchart tasks. In addition to the Status of the task, a flowchart task returns either true or false after performing its operation.
The Condition property should always be set to false for sequential tasks.
The SwitchValue is designed to be used by Switch flowchart nodes. If you set a value in the SwitchValue property and use this task in a Switch flowchart node, the case corresponding to the value will be executed. Otherwise, if the Default case is set, it will be executed.
You can use the TaskStatus constructor that suits your needs.
To retrieve settings, you can use the following methods:
string settingValue = this.GetSetting("settingName");
string settingValue = this.GetSetting("settingName", defaultValue);
string[] settingValues = this.GetSettings("settingName");
To load a file within a task, you can do it as follows:
this.Files.Add(new FileInf(path, this.Id));
To load an entity within a task, you can do it as follows:
this.Entities.Add(myEntity);
Finally if you finished coding your custom task, compile the class library project and copy the assembly W4IDataFlow.Tasks.MyTask.dll in C:\Program Files\W4IDataFlow\ or in C:\W4IDataFlow\Tasks. The path of the folder C:\W4IDataFlow\Tasks\ can be configured through tasksFolder setting in the configuration file C:\W4IDataFlow\W4IDataFlow.xml.
Your custom task is then ready to be used as follows:
<Task id="$int" name="MyTask" description="My task description" enabled="true">
<Setting name="settingName" value="settingValue" />
</Task>
That’s it. That’s all the things you need to know to start coding your own custom tasks.
To test the custom task, create a new workflow (new XML file) and put the configuration of the custom task in it as follows:
<Workflow xmlns="urn:W4IDataFlow-schema" id="99" name="Workflow_MyWorkflow" description="Workflow_MyWorkflow">
<Settings>
<Setting name="launchType" value="trigger" /> <!-- startup|trigger|periodic|cron -->
<Setting name="enabled" value="true" /> <!-- true|false -->
</Settings>
<Tasks>
<Task id="1" name="MyTask" description="My task description" enabled="true">
<Setting name="settingName" value="settingValue" />
</Task>
</Tasks>
</Workflow>
Then, place that XML file in C:\W4IDataFlow\Workflows\.
The workflow will then appear in the list of workflows in W4IDataFlow Manager. You can then launch it from there.
Logging
The following methods are available from the Task class for logging:
public void Info(string msg);
public void InfoFormat(string msg, params object[] args);
public void Debug(string msg);
public void DebugFormat(string msg, params object[] args);
public void Error(string msg);
public void ErrorFormat(string msg, params object[] args);
public void Error(string msg, Exception e);
public void ErrorFormat(string msg, Exception e, params object[] args);
Files
Files can be loaded in a task by calling the methods Add or AddRange:
this.Files.Add(myFile);
this.Files.AddRange(myFiles);
Then the files loaded can be selected in other tasks by their task Id as follows:
<Setting name="selectFiles" value="$taskId" />
To select the files loaded by the running instance of a workflow through the selectFiles settings option, you can do it as follows:
FileInf[] files = this.SelectFiles();
Entities
Entity is an abstract class having the Id of the task as property:
namespace W4IDataFlow.Core
{
public abstract class Entity
{
public int TaskId { get; set; }
}
}
The entity class is designed to be inherited by other classes such as objects retrieved from a database or a web service or an API or whatever. Then, these objects can be loaded in a task by calling the methods Add or AddRange:
this.Entities.Add(myEntity);
this.Entities.AddRange(myEntities);
Then, the entities loaded can be selected in other tasks by their task Id as follows:
<Setting name="selectEntities" value="$taskId" />
Entities are designed to be used in custom tasks.
To select entities loaded by the running instance of a workflow through the selectEntities settings option, you can do it as follows:
Entity[] entities = this.SelectEntities();
The Entity class could be very useful when working with custom tasks that manipulate objects from a database or Web Services for example.
Shared memory
Tasks contains a Hashtable that can be used as a shared memory between them.
To add an object to the Hashtable, simply proceed as follows:
this.Hashtable.Add("myKey", myObject);
To retrieve an object from the Hashtable, simply proceed as follows:
var myObject = this.Hashtable["myKey"];
To remove an object from the Hashtable, simply proceed as follows:
this.Hashtable.Remove("myKey");
Designer
To make your custom task MyTask appear in the dropdown of the designer, simply open the file C:\W4IDataFlow\TasksNames.json and add “MyTask” in it as follows:
[
...
"MyTask"
]
You must also add the settings by opening the file C:\W4IDataFlow\TasksSettings.json and adding your custom settings as follows:
{
...
"MyTask": ["settingName"]
}
That’s it. MyTask will show up in the designer and when selected its settings will show up too.
Debugging
To debug custom tasks, you can use logging.
You can also clone W4IDataFlow’s repository and open W4IDataFlow.sln in Visual Studio and follow these guidelines to run W4IDataFlow server from code. Then, you can create your custom task in the solution and debug it. To debug it you have to proceed as follows:
- Create your custom task
- Reference your custom task in W4IDataFlow.Server
- Create a workflow using your custom task
- Open W4IDataFlow Manager or the backend and trigger your workflow from there
13. Command Line Client
Command Line Client
W4IDataFlow provides a command line client for querying W4IDataFlow server. The command line tool is available for both .NET and .NET Core and works on Windows, Linux and macOS.
Windows (.NET)
The command line tool is located in C:\Program Files\W4IDataFlow\W4IDataFlow.Clients.CommandLine. To run the command line tool, just run the executable C:\Program Files\W4IDataFlow\W4IDataFlow.Clients.CommandLine\W4IDataFlow.Clients.CommandLine.exe
The configuration file C:\Program Files\W4IDataFlow\W4IDataFlow.Clients.CommandLine\W4IDataFlow.Clients.CommandLine.exe.config contains W4IDataFlowWebServiceUri, Username and Password settings.
Windows (.NET Core)
The command line tool is located in .\W4IDataFlow.Clients.CommandLine. To run the command line tool, just run the following command
cd .\W4IDataFlow.Clients.CommandLine
dotnet W4IDataFlow.Clients.CommandLine.dll
The configuration file .\W4IDataFlow.Clients.CommandLine\appsettings.json contains W4IDataFlowWebServiceUri, Username and Password settings.
Linux
After installing W4IDataFlow on Linux, the command line tool is located in /opt/W4IDataFlow/W4IDataFlow.Clients.CommandLine. To run the command line tool, just run the following command
cd /opt/W4IDataFlow/W4IDataFlow.Clients.CommandLine
dotnet W4IDataFlow.Clients.CommandLine.dll
The configuration file /opt/W4IDataFlow/W4IDataFlow.Clients.CommandLine/appsettings.json contains W4IDataFlowWebServiceUri, Username and Password settings.
macOS
After installing W4IDataFlow on macOS, the command line tool is located in /Applications/W4IDataFlow/W4IDataFlow.Clients.CommandLine. To run the command line tool, just run the following command
cd /Applications/W4IDataFlow/W4IDataFlow.Clients.CommandLine
dotnet W4IDataFlow.Clients.CommandLine.dll
The configuration file /Applications/W4IDataFlow/W4IDataFlow.Clients.CommandLine/appsettings.json contains W4IDataFlowWebServiceUri, Username and Password settings.
Options
-o, --operation Required. start|suspend|resume|stop|approve|reject
-i, --workflowId Required. Workflow Id
-j, --jobId Job instance id (Guid)
-w, --wait (Default: false) Wait until workflow finishes
--help Display this help screen.
--version Display version information.
Examples
Fire and wait
The following command starts the workflow 41 and waits until it finishes its jobs:
W4IDataFlow.Clients.CommandLine.exe -o start -i 41 -w
Fire and forget
The following command starts the workflow 41:
W4IDataFlow.Clients.CommandLine.exe -o start -i 41
Stop
The following command stops the workflow 41:
W4IDataFlow.Clients.CommandLine.exe -o stop -i 41 -j 9144e328-dde3-468e-a8ba-913e3d5b7b92
Suspend
The following command suspends the workflow 41:
W4IDataFlow.Clients.CommandLine.exe -o suspend -i 41 -j 9144e328-dde3-468e-a8ba-913e3d5b7b92
Resume
The following command resumes the workflow 41:
W4IDataFlow.Clients.CommandLine.exe -o resume -i 41 -j 9144e328-dde3-468e-a8ba-913e3d5b7b92
Approve
The following command approves the workflow 126:
W4IDataFlow.Clients.CommandLine.exe -o approve -i 126 -j 9144e328-dde3-468e-a8ba-913e3d5b7b92
Reject
The following command rejects the workflow 126:
W4IDataFlow.Clients.CommandLine.exe -o reject -i 126 -j 9144e328-dde3-468e-a8ba-913e3d5b7b92
14. Restful API
RESTful API
W4IDataFlow Server is a standalone language-agnostic solution that can be integrated to an app that runs PHP, NodeJS, Ruby, Python, etc. via a RESTful API.
Basic access authentication is used for all API methods. Thus, Authorization header must be set in every API call.
Passwords must be encrypted in MD5 checksums.
Here is a sample Authorization header for the user admin:
Authorization Basic YWRtaW46ZWUwNWVhYWJhN2I3NmYxNmUyODVkOTgzZDYwNWM5YmY=
which corresponds to username:md5(password) base64 encoded:
Basic base64(admin:ee05eaaba7b76f16e285d983d605c9bf)
Basic base64(admin:md5(W4IDataFlow2018))
The default password of the user admin is W4IDataFlow2018. You can change it from the back end.
You can access Swagger UI from http://localhost:8000.
Dashboard
GET http://localhost:8000/W4IDataFlow/statusCount
Returns status count.
GET http://localhost:8000/W4IDataFlow/entriesCountByDate?s={keyword}&from={date}&to={date}
Returns entries count by keyword and date filter.
GET http://localhost:8000/W4IDataFlow/searchEntriesByPageOrderBy?s={keyword}&from={date}&to={date}&page={page}&entriesCount={entriesCount}&heo={orderBy}
Searches for entries.
GET http://localhost:8000/W4IDataFlow/entryStatusDateMin
Returns entry min date.
GET http://localhost:8000/W4IDataFlow/entryStatusDateMax
Returns entry max date.
Manager
GET http://localhost:8000/W4IDataFlow/search?s={keyword}
Search for workflows.
GET http://localhost:8000/W4IDataFlow/searchApprovalWorkflows?s={keyword}
Search for approval workflows.
GET http://localhost:8000/W4IDataFlow/workflow?w={id}
Returns a workflow from its id.
POST http://localhost:8000/W4IDataFlow/start?w={id}
Starts a workflow.
POST http://localhost:8000/W4IDataFlow/startWithVariables
Starts a workflow with variables.
Here is a sample payload:
{
"WorkflowId":131,
"Variables":[
{
"Name":"restVar1",
"Value":"C:\\W4IDataFlowTesting\\file1.txt"
},
{
"Name":"restVar2",
"Value":"C:\\W4IDataFlowTesting\\file2.txt"
}
]
}
Here is a sample workflow:
<Workflow xmlns="urn:W4IDataFlow-schema" id="138" name="Workflow_RestVariables" description="Workflow_RestVariables">
<Settings>
<Setting name="launchType" value="trigger" />
<Setting name="enabled" value="true" />
</Settings>
<LocalVariables></LocalVariables>
<Tasks>
<Task id="1" name="FilesLoader" description="Loading files" enabled="true">
<Setting name="file" value="$restVar1" />
<Setting name="file" value="$restVar2" />
</Task>
<Task id="2" name="ListFiles" description="Listing files" enabled="true"></Task>
</Tasks>
</Workflow>
POST http://localhost:8000/W4IDataFlow/stop?w={id}
Stops a workflow.
POST http://localhost:8000/W4IDataFlow/suspend?w={id}
Suspends a workflow.
POST http://localhost:8000/W4IDataFlow/resume?w={id}
Resumes a workflow.
POST http://localhost:8000/W4IDataFlow/approve?w={id}
Approves a workflow.
POST http://localhost:8000/W4IDataFlow/disapprove?w={id}
Disapproves a workflow.
Designer
GET http://localhost:8000/W4IDataFlow/tasks/{id}
Returns workflow’s tasks.
GET http://localhost:8000/W4IDataFlow/xml/{id}
Returns a workflow as XML.
GET http://localhost:8000/W4IDataFlow/json/{id}
Returns a workflow as JSON.
GET http://localhost:8000/W4IDataFlow/taskNames
Returns task names.
GET http://localhost:8000/W4IDataFlow/settings/{taskName}
Returns task settings.
POST http://localhost:8000/W4IDataFlow/taskToXml
Returns a task as XML.
GET http://localhost:8000/W4IDataFlow/isWorkflowIdValid/{id}
Checks if a workflow id is valid.
GET http://localhost:8000/W4IDataFlow/isCronExpressionValid?e={cronExpression}
Checks if a cron expression is valid.
GET http://localhost:8000/W4IDataFlow/isPeriodValid/{period}
Checks if a period is valid.
POST http://localhost:8000/W4IDataFlow/isXmlWorkflowValid
Checks if the XML of a workflow is valid.
POST http://localhost:8000/W4IDataFlow/saveXml
Saves a workflow from XML.
POST http://localhost:8000/W4IDataFlow/save
Saves a workflow from JSON.
POST http://localhost:8000/W4IDataFlow/delete?w={id}
Deletes a workflow.
POST http://localhost:8000/W4IDataFlow/deleteWorkflows
Deletes workflows.
GET http://localhost:8000/W4IDataFlow/graph/{id}
Returns the execution graph of the workflow.
History
GET http://localhost:8000/W4IDataFlow/historyEntriesCountByDate?s={keyword}&from={date}&to={date}
Returns history entries count by keyword and date filter.
GET http://localhost:8000/W4IDataFlow/searchHistoryEntriesByPageOrderBy?s={keyword}&from={date}&to={date}&page={page}&entriesCount={entriesCount}&heo={orderBy}
Searches for history entries.
GET http://localhost:8000/W4IDataFlow/historyEntryStatusDateMin
Returns history entry min date.
GET http://localhost:8000/W4IDataFlow/historyEntryStatusDateMax
Returns history entry max date.
Users
GET http://localhost:8000/W4IDataFlow/user?username={username}
Returns a user from his username.
GET http://localhost:8000/W4IDataFlow/searchUsers?keyword={keyword}&uo={orderBy}
Searches for users.
POST http://localhost:8000/W4IDataFlow/insertUser?username={username}&password={password}&up={userProfile}&email={email}
Inserts a user.
POST http://localhost:8000/W4IDataFlow/updateUser?userId={userId}&username={username}&password={password}&up={userProfile}&email={email}
Updates a user.
POST http://localhost:8000/W4IDataFlow/updateUsernameAndEmailAndUserProfile?userId={userId}&username={username}&password={password}&up={userProfile}&email={email}
Updates the username, the email and the user profile of a user.
POST http://localhost:8000/W4IDataFlow/deleteUser?username={username}&password={password}
Deletes a user.
POST http://localhost:8000/W4IDataFlow/resetPassword?username={username}
Resets a password.
Profiles
GET http://localhost:8000/W4IDataFlow/searchAdmins?keyword={keyword}&uo={orderBy}
Searches for administrators.
GET http://localhost:8000/W4IDataFlow/userWorkflows?u={userId}
Returns user workflows.
POST http://localhost:8000/W4IDataFlow/saveUserWorkflows
Saves user workflow relations.
15. Run From Code
Run From Code
To run W4IDataFlow from code on Windows, proceed as follows:
- Clone the source down to your machine:
git clone https://github.com/aelassas/W4IDataFlow.git - Install Visual Studio.
- .NET: Copy the folders “W4IDataFlow” and “W4IDataFlowTesting” in C:\. You can download them from here.
- .NET Core: Copy the folders “W4IDataFlow-dotnet-core” and “W4IDataFlowTesting” in C:\. You can download them from here.
- If you installed W4IDataFlow, make sure that W4IDataFlow Windows Service is stopped and that the port 8000 is available. Othewise, you can change the port from the settings.
- Restore nuget packages through the following command:
nuget restore W4IDataFlow.sln - Open W4IDataFlow.sln in Visual Studio.
- Make sure that W4IDataFlow.Server is set as a startup project.
- Debug the project W4IDataFlow.Server to start W4IDataFlow Server in debug mode.
- Debug the project W4IDataFlow.Clients.Manager to start W4IDataFlow Manager in debug mode or open the backend and trigger your workflows from there.
Web Hooks
Lorem ipsum dolor sit amet, consectetuer adipiscing
1. Introduction
2. Get Started
Nam liber tempor cum soluta nobis eleifend option congue nihil imperdiet doming id quod mazim placerat facer possim assum. Typi non habent claritatem insitam; est usus legentis in iis qui facit eorum claritatem. Investigationes demonstraverunt lectores legere me lius quod ii legunt saepius. Claritas est etiam processus dynamicus, qui sequitur mutationem consuetudium lectorum.
Mirum est notare quam littera gothica, quam nunc putamus parum claram, anteposuerit litterarum formas humanitatis per seacula quarta decima et quinta decima. Eodem modo typi, qui nunc nobis videntur parum clari, fiant sollemnes in futurum.
Request a Feature or Open a Ticket
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
