Skip to main content

Ecosystem Integrations Management: Extending Our Platform's Reach

1. Introduction: A Force Multiplier and a Risk​

Our platform's ability to integrate deeply with the tools our customers use every day—in ecosystems like Microsoft (Teams, PowerBI), ServiceNow, Workday, and Okta—is a powerful force multiplier. It makes our platform stickier and more valuable. However, each integration also represents a dependency on a third-party system whose stability and release schedule we do not control.

This guide details our framework for managing the entire lifecycle of an ecosystem integration, from initial onboarding to proactive maintenance and monitoring.

Core Mission: To maximize the value of our ecosystem integrations while minimizing the associated operational risks through proactive monitoring, automated testing, and a clear maintenance strategy.

2. Guiding Principles for Integration​

  • Never Trust, Always Verify: We treat API calls to external systems as inherently unreliable. Every integration must be wrapped with robust error handling, timeouts, and circuit breakers.
  • Proactive Monitoring: We do not wait for customers to tell us an integration is broken. We proactively monitor vendor changelogs, API health dashboards, and our own integration metrics.
  • Isolate and Insulate: An issue with a third-party integration should never be allowed to cause a cascading failure in our core platform. Integrations are run in isolated processes or sandboxes where possible.
  • Customer-Centric Communication: When a third-party integration has an issue, we must be able to detect it, identify the root cause (internal vs. external), and communicate clearly with our customers. This is a key function of our Service Chain Monitoring.

3. Verbose Workflow: Lifecycle of an Integration​

The lifecycle of an integration is a continuous process, not a one-time setup.

  1. Onboarding & Development:

    • A new integration is built against the vendor's sandbox environment.
    • All API clients must be instrumented with timeouts, retries (with exponential backoff), and a circuit breaker mechanism.
    • The integration is wrapped in a dedicated service or module to isolate it from the core platform.

  2. Certification & Release:

    • Before being released, every new integration must pass a rigorous certification checklist, including security, performance, and resilience testing.
    • A new New Relic dashboard is created specifically to monitor the health of this integration.

  3. Proactive Monitoring (The Daily Grind):

    • Vendor Changelogs: Sparky runs a scheduled job every 6 hours to poll the developer changelogs and RSS feeds for all our key partners. Any new entries are parsed and summarized in the #ecosystem-updates Slack channel.
    • Health Dashboards: We monitor the official status pages and API health dashboards of our partners.
    • Synthetic Monitoring: For critical integrations, we have New Relic synthetic tests that run every 5 minutes, executing a simple, read-only API call to the partner's service to validate connectivity and authentication. A failure immediately creates a P4 incident in Jira Service Management.

  4. Automated Compatibility Testing:

    • If Sparky detects a new "beta" or "upcoming" version of a partner's API, it automatically triggers a GitHub Actions workflow.
    • This workflow deploys our integration into a temporary sandbox environment and runs a full suite of compatibility tests against the new beta API version.

  5. Remediation:

    • If the compatibility tests fail, indicating an upcoming breaking change, a P2 incident is automatically created and assigned to the owning team. This gives us weeks or even months to adapt our code before the breaking change hits production.
    • The goal is to never be surprised by a partner's breaking change.

4. Monitoring Strategy for Key Partners​

While the general principles apply to all, we have specific strategies for our most critical partners.

Microsoft (Teams, PowerBI)​

  • Monitoring: We subscribe to the Microsoft 365 Developer Blog RSS feed and the Microsoft Graph API changelog.
  • Key Metric: For the Teams integration, we monitor the latency of posting adaptive cards. A p95 latency > 2 seconds is a warning sign.
  • NRQL Alert: SELECT percentile(duration, 95) FROM ExternalServiceCall WHERE service = 'msteams' AND action = 'postAdaptiveCard'

ServiceNow​

  • Monitoring: We use synthetic monitoring to test our ability to create and update incidents in our ServiceNow sandbox every 5 minutes.
  • Key Metric: End-to-end success rate of creating an incident via the ServiceNow Table API.
  • NRQL Alert: SELECT percentage(count(*), WHERE result = 'SUCCESS') FROM SyntheticCheck WHERE monitorName = 'servicenow-incident-creation'

5. Sparky's Role in Ecosystem Management​

Sparky is our automated guardian against the chaos of third-party changes.

  • Changelog Watcher: As mentioned, Sparky is responsible for polling all vendor changelogs and alerting the team to upcoming changes.
  • Compatibility Tester: Sparky automates the process of spinning up a sandbox and running our test suite against a new partner API version.
  • Intelligent Triage: When a synthetic monitor for an integration fails, Sparky is the first responder. It immediately checks the partner's public status page.
    • If the partner is reporting an outage, Sparky updates our own public status page to reflect the issue and communicates to our customers that the issue is external, insulating our own support team.
    • If the partner is not reporting an outage, Sparky escalates to the on-call SRE, as the issue is likely with our own code or configuration.