External Integrations — ADL v0.3 Reference

35.1 Overview

ADL provides declarative integrations for communicating with external systems. Integrations are event-driven: when a history event fires, a workflow step completes, or a manual trigger executes, the integration dispatches an API call, processes a webhook, or publishes to a message queue.

Each integration declares the protocol (REST, webhook, S3, SMTP, AMQP, MQTT), authentication, data mapping, error handling, retry policy, and resilience patterns. The runtime manages token refresh, circuit breakers, rate limiting, dead letter queues, and logging.

35.2 Integration Types

REST (Outbound HTTP)

Calls external REST APIs:

integrations:
  grantsGovSubmission:
    description: "Submit proposal to Grants.gov"
    type: rest
    baseUrl: "https://api.grants.gov/v2"
    
    auth:
      type: api_key
      header: X-API-Key
      value: "${GRANTS_GOV_API_KEY}"
    
    operations:
      submitProposal:
        trigger:
          resource: Proposal
          event: transition
          transition: submitToSponsor
        
        request:
          method: POST
          path: "/submissions"
          headers:
            Content-Type: "application/json"
          body:
            opportunity_id: $record.fundingOpportunityId
            project:
              title: $record.title
              abstract: $record.abstractText
            pi:
              first_name: $record.piFirstName
              last_name: $record.piLastName
              email: $record.piEmail
        
        onSuccess:
          action: update
          resource: Proposal
          target: $record.id
          input:
            submissionId: $response.submission_id
            submittedAt: $now()
          history:
            event: "proposalSubmitted"
            label: "Proposal submitted — ID {submissionId}"
        
        onError:
          action: transition
          resource: Proposal
          target: $record.id
          transition: submissionFailed

Webhook (Inbound HTTP)

Receives events from external systems:

integrations:
  sponsorNotifications:
    description: "Receive award notifications from sponsor"
    type: webhook
    path: /webhooks/sponsor-notifications
    
    auth:
      type: hmac_signature
      secret: "${WEBHOOK_SECRET}"
      header: X-Webhook-Signature
      algorithm: sha256
    
    events:
      awardNotification:
        filter:
          field: "event_type"
          value: "award_notification"
        
        mapping:
          awardNumber: $payload.award_id
          sponsorAwardId: $payload.sponsor_reference
          awardedAmount: $payload.total_amount
          startDate: "$parseDate($payload.period_start, 'yyyy-MM-dd')"
          endDate: "$parseDate($payload.period_end, 'yyyy-MM-dd')"
        
        action: create
        resource: Award
        input: $mapped
        
        history:
          event: "awardReceived"
          label: "Award received from {sponsor} — {awardNumber}"

S3 (Object Storage)

AWS S3-compatible storage for files:

integrations:
  documentStorage:
    type: s3
    bucket: "${S3_BUCKET}"
    region: "${AWS_REGION}"
    
    auth:
      type: aws_sigv4
      accessKeyId: "${AWS_ACCESS_KEY_ID}"
      secretAccessKey: "${AWS_SECRET_ACCESS_KEY}"

Operations: upload, download, delete, presigned URL generation.

SMTP (Email)

Direct email sending:

integrations:
  emailServer:
    type: smtp
    host: "${SMTP_HOST}"
    port: 587
    
    auth:
      type: basic
      username: "${SMTP_USER}"
      password: "${SMTP_PASS}"

AMQP (Message Queue)

RabbitMQ and AMQP-compatible brokers:

integrations:
  instrumentDataQueue:
    type: amqp
    url: "${AMQP_URL}"
    queue: "instrument.results"
    
    credentials:
      username: "${AMQP_USER}"
      password: "${AMQP_PASS}"
    
    events:
      resultReceived:
        mapping:
          sampleId: $message.sample_accession
          rawResult: $message.result_value
          resultUnit: $message.result_unit
        
        action: update
        resource: Sample
        target: "$findOne(Sample, accessionNumber == $mapped.sampleId).id"
        input:
          rawResult: $mapped.rawResult
          resultUnit: $mapped.resultUnit

MQTT (IoT)

Lightweight messaging for sensors:

integrations:
  sensorNetwork:
    type: mqtt
    broker: "${MQTT_BROKER_URL}"
    topics:
      - "sensors/+/temperature"
      - "sensors/+/pressure"

35.3 Authentication Methods

Type	Use Case
`api_key`	API key in header or query parameter
`bearer`	Static bearer token
`oauth2_client_credentials`	OAuth 2.0 with automatic token refresh
`oauth2_authorization_code`	OAuth 2.0 for user-delegated access
`basic`	Basic HTTP authentication
`hmac_signature`	HMAC signature verification (webhooks)
`mtls`	Mutual TLS with client certificates
`ip_whitelist`	IP-based access control (webhooks)
`aws_sigv4`	AWS Signature V4 for AWS services

OAuth 2.0 Example

auth:
  type: oauth2_client_credentials
  tokenUrl: "https://auth.service.com/token"
  clientId: "${OAUTH_CLIENT_ID}"
  clientSecret: "${OAUTH_CLIENT_SECRET}"
  scopes: ["read", "write"]
  tokenCache: true

The runtime caches the access token, refreshes it before expiry, and retries requests with a new token if the original returns 401.

35.4 Data Mapping

Outbound Mapping (Domain → External)

body:
  opportunity_id: $record.fundingOpportunityId
  applicant:
    duns: "${INSTITUTION_DUNS}"
    ein: "${INSTITUTION_EIN}"
    name: "${INSTITUTION_NAME}"
  project:
    title: $record.title
    abstract: $record.abstractText
    start_date: "$formatDate($record.proposedStartDate, 'yyyy-MM-dd')"
  budget:
    total_requested: $record.requestedAmount

Nested objects are supported. AEL expressions transform values. Environment variables inject deployment-specific values.

Inbound Mapping (External → Domain)

mapping:
  awardNumber: $payload.award_id
  sponsorAwardId: $payload.sponsor_reference
  awardedAmount: $payload.total_amount
  startDate: "$parseDate($payload.period_start, 'yyyy-MM-dd')"
  endDate: "$parseDate($payload.period_end, 'yyyy-MM-dd')"

$payload is the parsed incoming message. $message is used for queue-based integrations. $mapped references the completed mapping for use in actions.

Transform Functions

Function	Description
`$formatDate(value, pattern)`	Format date to string
`$parseDate(string, pattern)`	Parse string to date
`$toUpper(string)`	Uppercase
`$toLower(string)`	Lowercase
`$trim(string)`	Trim whitespace
`$substring(string, start, len)`	Extract substring
`$replace(string, find, replace)`	String replacement
`$split(string, delimiter)`	Split to array
`$join(array, delimiter)`	Join array to string
`$toNumber(string)`	Parse numeric string
`$coalesce(a, b, c)`	First non-null value
`$findOne(Resource, filter)`	Query single record
`$findMany(Resource, filter)`	Query multiple records
`$now()`	Current timestamp
`$uuid()`	Generate UUID

35.5 Resilience Patterns

Retry with Exponential Backoff

retry:
  maxAttempts: 3
  backoff: exponential
  baseDelay: 1s
  maxDelay: 30s
  retryOn: [500, 502, 503, 504, 429]

Retry sequence: 1s → 2s → 4s (exponential) or 1s → 1s → 1s (fixed).

Circuit Breaker

circuitBreaker:
  enabled: true
  failureThreshold: 5
  successThreshold: 2
  timeout: 60s

States: closed (normal) → open (failing) → half_open (testing) → closed

When open, requests fail immediately without hitting the external service.

Rate Limiting

rateLimit:
  requestsPerSecond: 10
  burstSize: 20
  onExceeded: queue

Strategy	Behavior
`queue`	Queue requests until capacity available
`reject`	Return 429 immediately
`backoff`	Delay before retrying

Dead Letter Queue

deadLetter:
  enabled: true
  maxRetries: 3
  notifyOn: [integration_failure]

Failed messages (after all retries) are stored in the dead letter queue with:

Original payload
Error details
Attempt count
Timestamps

Manual retry:

POST /api/v1/{domain}/integrations/{name}/deadletter/{id}/retry

35.6 Webhook Security

HMAC Signature Verification

auth:
  type: hmac_signature
  secret: "${WEBHOOK_SECRET}"
  header: X-Webhook-Signature
  algorithm: sha256
  encoding: hex
  prefix: "sha256="

The runtime computes HMAC-SHA256(secret, request_body) and compares it to the header value. Mismatches are rejected.

Replay Protection

replayProtection:
  enabled: true
  timestampHeader: X-Webhook-Timestamp
  maxAge: 5 minutes
  nonceHeader: X-Webhook-Nonce

Rejects messages older than maxAge and tracks nonces to prevent duplicate delivery.

IP Whitelist

auth:
  type: ip_whitelist
  allowed:
    - "52.1.0.0/16"
    - "203.0.113.0/24"

35.7 Integration Logging

All integration operations are logged to {domain}_integration_log:

CREATE TABLE {domain}_integration_log (
    id              TEXT PRIMARY KEY,
    integration     TEXT NOT NULL,
    operation       TEXT NOT NULL,
    direction       TEXT NOT NULL,     -- outbound, inbound
    method          TEXT,
    url             TEXT,
    requestBody     TEXT,
    responseStatus  INTEGER,
    responseBody    TEXT,
    durationMs      INTEGER,
    success         BOOLEAN NOT NULL,
    errorMessage    TEXT,
    attemptNumber   INTEGER,
    triggeredBy     TEXT,
    createdAt       TEXT NOT NULL
);

Sensitive data (API keys, tokens, passwords) is automatically redacted from logs.

35.8 Integration APIs

List Integrations

GET /api/v1/{domain}/integrations

Get Integration Status

GET /api/v1/{domain}/integrations/{name}/status

Returns circuit breaker state, rate limiter metrics, recent success/failure rate.

Manual Trigger

POST /api/v1/{domain}/integrations/{name}/operations/{operation}
{
  "recordId": "proposal-uuid",
  "dryRun": false
}

Dry run mode shows the request payload without sending it.

Dead Letter Queue

GET /api/v1/{domain}/integrations/{name}/deadletter
    ?status=pending
    
POST /api/v1/{domain}/integrations/{name}/deadletter/{id}/retry

Integration Health

GET /api/v1/{domain}/integrations/{name}/health

Returns:

Circuit breaker state
Recent error rate
Average response time
Rate limiter status
Dead letter queue size

35.9 Use Cases

Scenario	Integration Type
Submit proposal to Grants.gov	REST outbound with OAuth 2.0
Receive award notifications	Webhook inbound with HMAC verification
Pull credit reports	REST outbound with API key
Store documents in S3	S3 with AWS Signature V4
Send email notifications	SMTP with basic auth
Receive lab instrument results	AMQP message queue
Monitor IoT sensors	MQTT with topic subscriptions
File drug safety reports to FDA	REST outbound with mutual TLS
Sync with external database	Database connector

35.10 Integration Points

Feature	Integration
History (§21)	Event triggers fire integration operations; operations produce history events
Workflows (§25)	Workflow steps can trigger integration operations; integrations can start workflows
Notifications (§23)	Integrations use SMTP for email delivery
Templates (§27)	Document generation can trigger S3 upload
Schedules (§24)	Scheduled tasks can trigger integration operations
Rules (§22)	Rule violations can trigger external notifications

End of Chapter 35.

End of Volume IV — Runtime Subsystems.

Next: Volume V — Reference

Chapter 36: AEL (ADL Expression Language)