Web Service Call

Overview

The Web Service Call tool connects to an external web service via HTTP or HTTPS, and constructs a request using mapped or variable URL, body, and header fields. If a response is expected, it maps the resulting headers and body onto output fields.

This tool does not format records into the request message body, nor does it scan records out of the response message body. Input records should be correctly formatted as a text or binary "blob," and they are returned as the web service result on output. Use the JSON Output, XML Output, and XML Output2 tools to format records into blobs. Use the JSON Input, XML Input, and XML Input2 tools to scan records from blobs.

Input

For each input record, the Web Service Call tool generates an HTTP request and reads a response. The request consists of one of several commands:

GET: a GET request is sent, using either a static URL, or a URL from an input field.
POST: a POST request is sent, using either a static URL, or a URL from an input field, and using a request body from either an input field or a file. The Web Service Call tool does not format records into JSON or XML request bodies. This must be done by an upstream tool (JSON Output, XML Output, XML Output2) that formats records into blobs that are read by the Web Service Call tool.
PUT: the client is sending data to the server to a specific URI that corresponds to a specific resource that the client knows. Use PUT for a URI that is updating a new record on the server.
PATCH: this is like PUT, but need only contain the changes to the resource, not the complete resource.
DELETE: this is like GET, in that no request body is expected. Instead, the URI identifies a specific resource to be deleted.
OPTIONS: this method requests information about the communication options available on the request/response chain identified by the Request-URI. This allows the client to determine the options and requirements associated with a resource, or the capabilities of a server, without implying a resource action or initiating a resource retrieval. No request body is sent along with an OPTIONS request.
HEAD: this method is identical to GET except that the server does not return a message-body in the response. The metainformation contained in the HTTP headers in response to a HEAD request should be identical to the information sent in response to a GET request. This method can be used for obtaining metainformation about the entity implied by the request without transferring the entity-body itself. This method is often used for testing hypertext links for validity, accessibility, and recent modification.

HTTP headers for the request may come from two sources:

Constant headers that are sent with every request.
Variable headers that are read from a separate header (H) input and matched to request bodies by ID.

Output

The Web Service Call tool produces one output record for each input record. The output record contains the status code/message, the response body, and (optionally) all of the input fields.

The Web Service Call tool does not scan records from JSON or XML response bodies. This must be done by a downstream tool (JSON Input, XML Input, XML Input2) that scans records from the blobs that are output by this tool.

Optionally, response headers may be read and sent to the Headers output. Output headers are linked to responses by an ID field.

The Web Service Call tool is a transform tool, not an output tool. Unlike an output tool, transform tools need at least one downstream tool requesting input; otherwise, they may stop processing records. Consider using a downstream Filter tool to examine the status code/message returned by the Web Service Call tool before routing the dataflow to a follow-on action.

Authentication

See Client-side OAuth2 support in the Web Service Call tool and SSL/TLS client authentication.

Cookies

Many web services use cookies to save session credentials, requiring multiple web service call to first validate the user, and then perform work using that validation. A typical scenario is:

The caller sends login and password credentials.
A “success” response contains Set-Cookie headers, at least one of which contains a session identifier.
Subsequent requests contain the same cookie or cookies returned by the login request.

The Web Service Call tool automatically stores cookies from web service responses, and sends previously-stored cookies with requests. To enable this, select the Enable cookies option, and specify a Cookie scope. The Cookie scope may be any string. All Web Service Call tools in a project with the same Cookie scope will share the same cookies. These cookie settings can be used to control a heterogeneous collection of Web Service Call tools within a single project that may or may not interact with each other.

Using the Web Service Call tool

The repository folders PublicWebServices and Samples/Web Services contain examples of the Web Service Call tool in action. These demonstrate several usage patterns that can help you achieve correct results and improved performance.

The macro AtTaskProjectQuery (located in the PublicWebServices folder) performs a multi-step cascading query into the AtTask work management system. You need an AtTask log on and password to run it. This macro demonstrates some important techniques:

Login and cookie validation: note that the WebServiceCall1 tool sends a log on and password, and has Use cookies specified. The tool's purpose is to validate the login and capture the resulting credentials.
Parallelism: because this service supports multiple connections, you can process the bottlenecked requests in parallel, using the Splitter and Merge tools just as you would in any other data-parallel processing configuration.

Web Service Call tools configured to run in parallel.

Using a separate JSON Input tool for scanning response bodies and producing output records: this technique is common in web service callouts. Note how the JSON Input tool is set to read from input fields.

Using Copy last response to clipboard. When you construct a project that calls out to a web service, connect and configure the Web Service Call tool and run the project. When it finishes, view the Web Service Call tool's Response tab on the properties pane, and select Copy last response to clipboard. This will give you a sample response that can be analyzed directly by either the JSON Input or XML Input2 tools, making the next step of project creation much easier—just paste the last response into the Sample tab of these tools and select Analyze.

Web Service Call tool configuration parameters

The Web Service Call tool has four sets of configuration parameters in addition to the standard execution options.

Request

Parameter	Description
HTTP command	Type of action to perform on the specified web service URL: `GET`: (default) issues a `GET` command, corresponding to a "Read" operation. No body is sent, and it is assumed that all parameters are encoded in the URL. It is idempotent, which means that making multiple identical requests has the same result as a single request. `POST`: issues a `POST` command, corresponding to a "Create" operation. A body is sent, and the Content-Length header is set. It is non-idempotent; if you execute it multiple times, the server state will change every time. Making two identical `POST` requests will likely result in two resources containing the same information. `PUT`: issues a `PUT` command, corresponding to an "Update/Replace" operation. Sends data to a known resource URI with the request body containing the newly-updated representation of the original resource. It is idempotent and will result in the same server state no matter how many times you execute it. `PATCH`: issues a `PATCH` command, corresponding to an "Update/Modify" operation. This is like `PUT`, but need only contain the changes to the resource, not the complete resource. For example, to change only a couple of columns in a record stored at a specific URI on a server, you could use the `PATCH` method. Like `PUT`, it is idempotent. `DELETE`: issues a `DELETE` command, corresponding to a "Delete" operation. No body is sent. The URI identifies the resource to be deleted. `OPTIONS`: issues an `OPTIONS` request, requesting information about the communication options available on the request/response chain identified by the Request-URI. No request body is sent. `HEAD`: issues a `HEAD` command, identical to `GET` except that the server does not return a message-body in the response. The metainformation contained in the HTTP headers in response to a `HEAD` request should be identical to the information sent in response to a `GET` request.
URL source	Source of the web service URL: Literal: (default) URL is defined by the specified URL parameter. This method is best when the URL of the web service doesn't change, which is usually the case when HTTP Command is POST. Field: URL is contained in the specified URL field. This method is best when HTTP Command is GET because the request parameters will usually be encoded in the URL and thus will be different for each request.
URL/URL field	When URL source is Literal, URL of the web service. When URL source is Field, field containing the URL of the web service.
Input body from	If HTTP Command is `POST`, source that will be read and injected into the HTTP request as the `POST` body: Field: body is read from the specified Body field. File: body is read from the file specified as Body filename field. This option supports very large posts (such as file uploads).
Constant and input headers	If defined, headers that are sent with every request. Name may be any HTTP header except Content-length. Value is the literal string associated with Name. Field is the field from which the Name value will be read at request time. You can define a Value or select a Field, but not both.

Response

Parameter	Description
Include input fields in output	If selected, all input fields from the default input will be copied to the output if they do not conflict with other output field names.
Output body to	Specified how the response body will be written: Field: body is written to the specified Body field. File: body is written to the file specified as Body filename field. This option supports very large large gets (such as file downloads).
Status field	The name of the field to which the status code and/or response message will be written.
Enable header output	If selected, enables writing of headers from a separate Header (H) output connection.
Response ID field	If Enable header output is selected, the name of the field on the default output that contains a unique ID for the requests.
Header ID field	If Enable header output is selected, the name of the field on the Header (H) output that links the header records to the request records. The Header (H) output should already be sorted by this field for best performance.
Header name field	If Enable header output is selected, the name of the field on the Header (H) output containing the header name. Header names may be duplicated.
Header value field	If Enable header output is selected, the name of the field on the Header (H) output containing the header value.
Copy last response to clipboard	After the project containing the Web Service Call tool is run, select this button to copy the last HTTP response to clipboard, providing sample data for analysis.

Options

Parameter	Description
Requests/connection	The number of requests that will be sent on a connection before closing the connection and starting a new connection. Not all web services support multiple requests per connection, but for those that do, configuring this will improve performance by reducing the effects of connection startup/teardown time.
Timeout in seconds	Amount of time to wait for a response before giving up and retrying the outstanding request(s) on a new connection. This option can reduce the impact of buggy or overloaded web services.
Override site-level SSL validation	Select this to override the default SSL/TLS security strategy.
Reject invalid SSL certificates	If Override site-level SSL validation is selected, select this to reject SSL certificates that have expired or are self-signed.
Enable cookies	Select to enable cookies, which are often necessary for sessions.
Cookie scope	If Enable cookies is selected, optional string value indicating which tools in a project share sets of cookies. All tools with the same Cookie scope will share cookies, allowing "session start" web service calls to share session IDs with "data processing" calls and "session end" calls. If not specified, a default cookie scope is used.
Enable logging	If selected, a log file containing detailed, byte-by-byte logging of all requests, responses, and status updates will be created. This has a performance impact and may create large files. Make sure that each Web Service Call tool has a different log file.
Log file	If Enable logging is selected, file to which communications logs will be written.
Connection error retry limit	Maximum number of connection retry attempts. Defaults to 3.
Request error retry strategy	Action to be taken in response to errors: Never retry: any error other than connection closed by server results in the error action. Only retry server errors: (default) retry status codes in the 500 range (which are mostly server-side errors). Always retry: retry any error. Not recommended, because many errors are non-recoverable (for example, invalid credentials or bad request format).
Request error retry limit	Number of times to try a single request before returning an error. Defaults to 2.
Request error limit	Maximum number of error results before the project is aborted. Defaults to 1000.
No port in Host header	If selected, suppresses the port value from the Host HTTP header.
Enable header input	If selected, enables reading of headers from a separate Header (H) input connection.
Request ID field	If Enable header input is selected, the name of the field on the default input that contains a unique ID for the requests. The default input should already be sorted by this field for best performance.
Header ID field	If Enable header input is selected, the name of the field on the Header (H) input that links the header records to the request records. The Header (H) input should already be sorted by this field for best performance.
Header name field	If Enable header input is selected, the name of the field on the Header (H) input containing the header name. Header names may be duplicated. The header name Content-Length is ignored.
Header value field	If Enable header input is selected, the name of the field on the Header (H) input containing the header value.

Authentication

If configured, overrides the defaults specified in Site settings.

Field	Description
Authentication type	None: disables client authentication for this web service. OAuth2: overrides site-level OAuth2 default credentials specified in Site settings. 2-Way SSL: overrides client certificate defaults specified in Site settings.

Authentication type is OAuth2

Field	Description
Grant type	OAuth2 grant type: Credentials: the Web Service Call tool presents a Client ID and Client secret to the OAuth2 authentication server, and receives an access token. See Configure OAuth2 with the Credentials grant type. Password: the Web Service Call tool presents a Username, Password, Client ID, and Client secret to the OAuth2 authentication server, and receives an access token. See Configure OAuth2 with the Passwords grant type. Google JWT: the Web Service Call tool constructs a JSON Web Token from a JSON web service key, signs it, and then either makes a request to the OAuth2 server using the JWT grant type, or places the signed token in the Authorization header of the web service call. See Configure OAuth2 with the Google JWT grant type and Configure OAuth2 with a Google JWT signed token. Generic JWT: the Web Service Call tool constructs a JSON Web Token from components entered into the tool properties, signs it, and then either makes a request to the OAuth2 server using the JWT grant type, or places the signed token in the Authorization header of the web service call. See Configure OAuth2 with the Generic JWT grant type and Configure OAuth2 with a Generic JWT signed token.
Auth server URI	URI of the OAuth2 authentication server.
Username Password	Resource owner's user name and password or Key Vault reference.
Client ID Client secret	Credentials that this web service must present to obtain an access token, which may be a Password or Key Vault reference.
JWT usage	Determines whether the JSON Web Token (JWT) is Authenticated by the OAuth2 server, or Signed and placed in the Authorization header of the web service call.
JWT properties	JSON Web Token Claims that may be needed for this request. Obtain these from the web service API documentation.
Private key source	Source for the private key to be used to sign this request. Choose Enter here and enter a Private key (which may be a Password or Key Vault reference), or a choose File and browse to the file containing your RSA private key.
Lifetime	Grant token lifetime.
JWT signature type	Algorithm used to sign the JWT. Leave this set to the default of RS256 unless your web service requires ES256.
Google service key	JSON format key file typically generated from the Google console.
Allow self-signed certificates	When selected, self-signed SSL certificates will be accepted. Otherwise, the tool will fail with an error if the authentication server is secured using a self-signed SSL certificate.
Access scopes	Defines the level of access granted to a service. These are typically URLs for specific services being called.
Additional fields Additional claims	Name/value pairs to be added to the request form/claims section of the JWT if the OAuth2 server requires them.

Authentication type is 2-Way SSL

Field

Description

Certificate file

Private key file

Specifies the client's trusted CA certificate and private key file to use for this web service.

Configure the Web Service Call tool

Select the Web Service Call tool.
Select the Request tab and configure the input parameters.
Select the Response tab and configure the output parameters.
Select the Options tab and specify the desired file control settings and options.
Optionally, select the Authentication tab, and configure client authentication options for the target web service.
Optionally, select the Execution tab, and then set Report options and Web service options.