API Integration Guide

Manager+Agents allows the integration of third-party products via API on two levels:

  • The Manager’s web service API can be used to allow third-party applications to initiate file transfers
  • Custom workflows may include components that interact with third-party products

Manager+Agents supports API integration using either REST or SOAP web services. Both architecture types are available in Native and FIMS formats. FIMS is a framework of service definitions for implementing media related operations. While native APIs offer greater functionality than FIMS APIs, choose a FIMS API when adherence to a standard is required. The same API type can be used regardless of source and destination storage type.

The use of Native REST API is strongly recommended. While support for native SOAP API will continue, all new feature development will be for REST.

Note: All examples in this document refer to REST API.

API Type and Feature Availability

Type/Feature Minimum Release
Native REST API Percent Complete 13.5
Native REST API 13.1
Native SOAP API 6.1
FIMS 1.1 REST API 11.4
FIMS 1.0 REST API 10.6
FIMS 1.1 SOAP API 11.4
FIMS 1.0 SOAP API 10.1
FIMS Pause & Resume 11.3

Additional Licensing Requirements

Separate licensing is required for cloud object storage and local object storage, as well as for the use of custom workflows.

Storage/Engine Minimum Release Licensing
Amazon S3 Storage 13.4 Requires Flight subscription
Microsoft Azure Blob Storage 13.4 Requires Flight subscription
Local Object Storage 13.4 Requires specific license
Workflow Engine 8.0 Requires specific license

Invoking the API

Connection to the API is only supported over HTTPS (443/TCP). Any requests via HTTP (80/TCP) are redirected to HTTPS.

https://<ManagerHostname>/signiant/spring/admin/v1.0/<EndPoint>

When making requests to the API, you must include your username and password, and specify application/json as the content-type.

Authenticating and Authorizing Users

The Manager supports native, Active Directory and LDAP authentication. Each authenticated user has access to the features and resources they have been given permission to view and/or edit. Create specific users for every third-party application making API calls.

To authorize a user:

  1. Navigate to Administration > Users > List.
  2. Under Roles, select Administration Interface Login.
  3. For improved performance when many API calls are being made to the Manager, enable Improve web service API performance with reduced security auditing on the General tab. This setting will prevent all successful or failed authentications for this user from being logged in the Manager web log.

Creating Jobs

Every newly created job is based on a specified workflow, called a job template. Manager+Agents provides pre-defined job templates which allow file transfers between all supported storage types. Media Mover job templates are used for transfers in file system storage. Object Mover job templates are used for file transfer to or from local object storage or cloud object storage.

While you can run a single job multiple times, updating it via the API as required, Signiant recommends that you create a new job each time. This approach improves trackability and simplifies troubleshooting.

Each job is uniquely identified by its name and the job group to which it belongs. Job names and job groups names must begin with a letter and contain only alphanumeric and underscore characters. To make it easier to identify the same job in both the third-party application and the Manager, include unique identifiers in the job name.

After a job is created via the API, it will be visible in the Manager, and can be edited to verify the supplied parameters.

Setting Job Parameters

To configure a job, you must set its parameters. These are the same parameters that would be displayed if you configured a job in the Manager. At a minimum, the source and target Agents, plus the source file/folder and the target directory, must be specified. Any local or network storage path is supported, including CIFS (Windows shares), Windows drive letters, and NFS.

To determine the configuration parameter names:

  1. In your Manager, navigate to Jobs > Templates.
  2. Select the appropriate job template library and click Open Canvas.
  3. Right-click on the first component in the appropriate job template and select Show Web Services Info…. The resulting dialogue will show the corresponding internal name for each job template prompt.

Note: The parameters are case sensitive.

Configuring a Media Mover Job

Media Mover offers three job templates.

Job Template Type Description of Transfer
MediaDistributor Push One-to-one, one-to-many
MediaAggregator Pull One-to-one, many-to-one
MediaReplicator Push One-to-one, one-to-many - optimized for many small files

Among these templates, MediaReplicator is recommended, unless the specific functionality of the other workflows is required. MediaReplicator allows you to schedule a distribution from a source Agent to one or more target Agent and is designed for dealing with large numbers of files.

MediaReplicator

POST /jobs

Payload

[
     {
        "job":{
            "jobName":                                        "MyJobName",
            "fields":{
                "jobGroupName":                               "MyJobGroupName",
                "jobTemplateLibraryName":                     "Media_Mover_Workflows",
                "jobTemplateName":                            "MediaReplicator",
                "jobArgs":{
                    "MediaReplicator.Source.SourceAgent":     "sourceagent.com",
                    "MediaReplicator.Source.SourceData":      "c:/source",
                    "MediaReplicator.Target.TargetAgents":    "targetagent.com",
                    "MediaReplicator.Target.TargetDirectory": "c:/target",
                    "MediaReplicator.Schedule._sp_frequency": "once"
                }
            }
        }
    }
] 

Note: Both forward slashes and back slashes are supported for all operating systems. Each backslash must be escaped by an additional backslash.

Response

{
    "creator":"admin",
    "jobs":[
       {
          "id":      1680861,
          "jobName": "MyJobName"
       }
    ]
}

Configuring Object Mover Jobs

Object Mover allows transfers between local on-premises object storage and Amazon S3 or Microsoft Azure cloud object storage. An individual Agent can be configured to transfer files to or from local object storage, or to cloud object storage, but not both.

Object Mover offers three job templates:

Job Template Type Source Destination
ObjectUploader Pull File System Object Storage (local or cloud)
ObjectDownloader Push Object Storage (local or cloud) File System
ObjectReplicator Push Object Storage (local or cloud) Object Storage (local or cloud)

In the Object Mover job parameters, you must reference a storage profile.

Note: To check the storage profile name in your Manager, navigate to Administration > Storage Profiles > List.

The storage profile parameter is in a JSON encoded format.

{
\"name\": \"My Storage Profile Name\"
}

If desired, you can specify a sub-folder for the storage profile. This folder will be appended to any folder specified as part of the storage profile itself.

{
\"name\":      \"My Storage Profile Name\",
\”subfolder\”: \”MySubfolder\”
}

Note: Quotation marks must be escaped.

ObjectUploader

POST /jobs

Payload

[
     {
        "job":{
            "jobName":                                        "MyJobName",
            "fields":{
                "jobGroupName":                               "MyJobGroupName",
                "jobTemplateLibraryName":                     "Object_Mover_Workflows",
                "jobTemplateName":                            "ObjectUploader",
                "jobArgs":{
                    "ObjectUploader.Source.SourceAgent":     "sourceagent.com",
                    "ObjectUploader.Source.SourceData":      "/tmp/src",
                    "ObjectUploader.Target.TargetAgents":    "targetagent.com",
                    "ObjectUploader.Target.TargetDirectory": "{\"name\":\"MyStorageProfileName\"}",
                    "ObjectUploader.Schedule._sp_frequency": "once"
                }
            }
        }
    }
] 

ObjectDownloader

POST /jobs

Payload

[
     {
        "job":{
            "jobName":                                        "MyJobName",
            "fields":{
                "jobGroupName":                               "MyJobGroupName",
                "jobTemplateLibraryName":                     "Object_Mover_Workflows",
                "jobTemplateName":                            "ObjectDownloader",
                "jobArgs":{
                    "ObjectDownloader.Source.SourceAgent":     "sourceagent.com",
                    "ObjectDownloader.Source.SourceData":      "{\"name\":\"MyStorageProfileName\"}",
                    "ObjectDownloader.Target.TargetAgents":    "targetagent.com",
                    "ObjectDownloader.Target.TargetDirectory": "/temp/tgt",
                    "ObjectDownloader.Schedule._sp_frequency": "once"
                }
            }
        }
    }
]

ObjectReplicator

POST /jobs

Payload

[
     {
        "job":{
            "jobName":                                               "MyJobName",
            "fields":{
                "jobGroupName":                                     "MyJobGroupName",
                "jobTemplateLibraryName":                           "Object_Mover_Workflows",
                "jobTemplateName":                                  "ObjectReplicator",
                "jobArgs":{
                    "ObjectUploader.Source.SourceAgent":            "sourceagent.com",
                    "ObjectUploader.Target.sourceObjectStorage":    "{\"name\":\"MyStorageProfileName\"}",
                    "ObjectUploader.Source.SourceData":             "/tmp/src",
                    "ObjectUploader.Target.TargetAgents":           "targetagent.com",
                    "ObjectUploader.Target.TargetDirectory":        "{\"name\":\"MyStorageProfileName\"}",
                    "ObjectUploader.Schedule._sp_frequency":        "once"
                }
            }
        }
    }
]

Implementing Custom Workflows

For guaranteed support, Signiant recommends the use of the standard Media Mover or Object Mover job templates. If, however, specific functionality is required, a customized workflow may be created. You can export and import workflows, both via the Manager and using SOAP API(exportJobTemplateLibary,importJobTemplateLibary), allowing you to distribute a custom workflow either manually or programmatically.

Integrating Workflow Components

The Manager includes a full workflow engine. You can, however, create custom workflow components that integrate with third-party products, when specific functions are required. The integration is typically managed at the API level, where files processed within the workflow are directed to a third-party product. The Signiant job monitors the progress and status of the third-party processing, and reports results to the Manager.

Workflow components may be written in any interpreted language (e.g. Perl, Python, Ruby, Windows Shell). The Signiant components included with every Agent installation are written in Perl, as the language provides multi-platform support.

Configuring API Integration for Agent Groups

To improve efficiency when transferring files directly to or from specific Agents, you can configure Agent groups.

There are several benefits to using Agent groups:

  • The third-party application does not need to know precise Agent names. Consistent descriptive labels can be used, i.e. NewYork instead of newyork.signiant.com.
  • Agents can be swapped in and out of a group without any configuration changes to the third-party application.
  • An Agent group can be flagged as load-balanced, to provide redundancy and scalability where a transfer occurs between Agents in the group.

For more information, see Configuring Agent Groups.

Agent groups are always associated with a specific organization. When configuring a transfer to run to or from a given Agent group, the organization’s API Identifier must be referenced.

To obtain the organization’s API Identifier:

  1. In your Manager, navigate to Administration > Manager > Organizations.
  2. Note the API Identifier for the relevant organization.

When creating a job with an Agent group, the following format must be used: <AgentGroupName>!<OrgId>, eg. NewYork!9

POST /jobs

Payload

[
    {
        "job":{
           "jobName":                                           "MyJobName",
           "fields":{
                "jobGroupName":                                 "MyJobGroupName",
                "jobTemplateLibraryName":                       "Media_Mover_Workflows",
                "jobTemplateName":                              "MediaReplicator",
                "jobArgs":{
                    "MediaReplicator.Source.SourceAgent":       "NewYork!9",
                    "MediaReplicator.Source.SourceData":        "c:/source/",
                    "MediaReplicator.Target.TargetAgents":      "targetagent.com",
                    "MediaReplicator.Target.TargetDirectory":   "c:/target",
                    "MediaReplicator.Schedule._sp_frequency":   "once"
                }
           }
        }
    }
]

Checking Job Status

The job status is maintained across two fields:

  • activeState: What the job is currently doing
  • scheduledState: What the job is scheduled to do in the future

For a job scheduled to run once, for example, the activeState status will be RUNNING or IDLE, and the scheduledState will be DORMANT.

To minimize load on the Manager, it is recommended that you check the status of a job no more than every 15 seconds. Statistics are processed by the Manager at this rate, so checking more frequently will not result in updated information.

Returned statistics only reflect workflow components that have started. The order of the statistics is non-deterministic but, for standard job templates, the first component to run is the file transfer component. This is the only component which takes measurable execution time.

As of the Manager 13.5 release, the percent complete progress is returned with job status information.

GET /jobs/MyJobName/MyJobGroupName

Response (success) with percentComplete

[
    {
      "job": {
        "id":                        1680861,
        "jobName":                   "MyJobName",
        "fields": {
          "jobGroupName":            "MyJobGroupName",
          "jobTemplateLibraryName":  "Media_Mover_Workflows",
          "jobTemplateName":         "MediaReplicator",
          "activeState":             "IDLE",
          "scheduledState":          "DORMANT",
          "lastExitCode":            0,
          "percentComplete":         "54%",
          "lastActiveStatusMessage": "",
          "activeFilename":          "",
          "jobArgs": {
            ...
          }
        }
      }
   }
]

In versions of the Manager that precede 13.5, there is no method to request the percent complete via the native REST API. The percentage complete can, however, be calculated from statistics gathered within the Manager.

Job Template Component Type Package Type Total Bytes Portion of Bytes Completed
All Command LOCAL_CMD rmt_cmd_twu rmt_cmd_wuc
MediaDistributor MediaAggregator MediaReplicator SigniantAggregate SigniantDistribute FILE_TRF byte_count effective_bytes
ObjectUploader SigniantObjectUpload OBJECT_PUT byte_count effective_bytes
ObjectDownloader SigniantObjectDownload OBJECT_GET byte_count effective_bytes
ObjectReplicator SigniantObjectTransfer OBJECT_TRF byte_count effective_bytes

For example, the percent complete for a file transfer can be calculated using the following equation :

effective_bytes / byte_count x 100

POST /jobs/stats//

Payload

[
   "status",
   "effective_bytes",
   "byte_count",
   "rmt_cmd_twu",
   "rmt_cmd_wuc"
]

Response (success)

[
    {
       "package_name":       "JobReport4",
       "package_type":       "LOCAL_CMD",
       "source_host":        "mgr1.signiant.com",
       "target_host":        "mgr1.signiant.com",
       "status":             "COMPLETE",
       "effective_bytes":    "0",
       "byte_count":         "0",
       "rmt_cmd_twu":        "1",
       "rmt_cmd_wuc":        "1"
    },
    {
       "package_name":       "SigniantReplicate",
       "package_type":       "FILE_TRF",
       "source_host":        "mgr1.signiant.com",
       "target_host":        "mgr1.signiant.com",
       "status":             "COMPLETE",
       "effective_bytes":    "4100096",
       "byte_count":         "4100096",
       "rmt_cmd_twu":        "0",
       "rmt_cmd_wuc":        "0"
    }
]

Once the job has completed, the lastExitCode field displays the overall exit code for the job. A zero indicates success while all other numbers indicate failure.

Troubleshooting a Job Failure

When a job fails, troubleshooting details are available in the Manager’s job logs.

To view a job log:

  1. In your Manager, navigate to Jobs > Groups.
  2. Select the Job Group and click View Jobs.
  3. Select the failed job and click Details.
  4. Under Job Logs, select Job Log and View. Log entries can be sorted according to the severity of error.

A REST API endpoint is available to retrieve the job log programmatically for display in the third-party application.

Deleting a Job

Once a job is complete, you have the option to delete it. While successful jobs can be deleted immediately to save space, it is recommended that you wait several day before deleting failed jobs. This approach ensures that the job and its associated logs are available, for example, after a long weekend.

There are two options for deleting a job via the API:

  • Soft Delete: A soft delete acts as if the job was deleted via the Manager. The job is marked as deleted in the Manager, but the database records are kept for reporting purposes until they are deleted by a maintenance job.

  • Hard Delete: Immediately deletes the job and all associated database records. This choice prevents database growth that can impact performance.

Soft Delete

DELETE /jobs/MyJobName/MyJobGroupName

Response (success)

{
    "success":"true"
}

Hard Delete

DELETE /jobs

Payload

[
    {
        "job":{
            "fields":{
               "jobGroupName":"MyJobGroupName"
            },
            "jobName":        "MyJobName"
        }
    }
]

Response (success)

{
    "success":"true"
}