Skip to main content

Split Synchronizer

Last updated on

Split Synchronizer coordinates feature flag and segment data between Harness Feature Management Experimentation (FME) servers and SDKs that do not have native shared caching. It uses Redis as a shared datastore, allowing multiple SDK instances to stay synchronized without each fetching data independently.

Use Split Synchronizer with PHP, Python, .NET, Node.js, Ruby, Go, or Java SDKs in environments where multiple application processes run concurrently and need consistent feature flag evaluations. The Synchronizer also posts impression data (records of flag evaluations) and event data (custom tracking from SDK .track calls) back to Harness servers for analytics and experimentation reporting.

Version 5.0 and proxy mode

As of version 5.0, Split Synchronizer introduced significant improvements to impression and event handling, including a new approach to eviction that replaces periodic fetching and posting with dedicated threads for better throughput. What was once proxy mode is now a separate tool called Split Proxy. Go to Split Proxy to learn about proxy mode configuration.

Currently supported SDK versions with Synchronizer 5.0:

  • Ruby 6.0.0+
  • PHP 6.0.0+
  • Node.js 10.6.0+
  • Go 3.0.0+
  • Python 7.0.0+
  • .NET 4.0.0+
  • Java 4.4.0+
Parameter name changes in version 5.0

If you are upgrading from Split Synchronizer version 4.x or earlier, note that configuration parameter names and values changed in version 5.0. For example, impressionsMaxSize became impressions-fetch-size, and boolean options now use true/false instead of on/off. The configuration reference section uses version 5.0 parameter names.

What you will learn from this topic​

  • What Split Synchronizer is and when to use it: Understand how Split Synchronizer coordinates data across SDK instances and which SDK languages require it for shared caching.
  • How to install Split Synchronizer: Learn installation methods including Docker, command-line (Linux/Mac/Windows), and cloud platforms like Heroku and AWS ECS.
  • How to configure for different Redis topologies: Configure Split Synchronizer for standard Redis (single instance), Redis Sentinel (high availability with automatic failover), or Redis Cluster (horizontal scaling).
  • How to monitor and troubleshoot: Use the admin dashboard, health check endpoints, and troubleshooting guidance to ensure Synchronizer runs reliably in production.

Before you begin​

Before installing Split Synchronizer, ensure you have the following prerequisites:

  • Harness FME account: You need access to Harness Feature Management Experimentation. If FME does not appear in your Harness account, contact your account administrator or Harness Support.

  • Redis instance: Split Synchronizer requires a Redis instance running and accessible. Supported topologies include standard Redis (single instance), Redis Sentinel (high availability with automatic failover), and Redis Cluster (horizontal scaling). Go to Redis installation documentation to install Redis if you do not have it already. For production deployments, Redis 5.0 or later is recommended.

  • SDK API key: You need an SDK-type API key to authenticate Split Synchronizer with Harness FME servers. To find your SDK API key:

    1. In Harness, go to Project Settings > Feature Flags > Environments
    2. Select your environment
    3. Copy the SDK Key (also called Server SDK Key or API Key)

    Note: Use an SDK-type key, not an Admin or Client key. SDK keys allow Synchronizer to fetch feature flag definitions and post impressions (flag evaluation records) and events (custom tracking data from SDK .track calls) back to Harness for analytics.

  • Permissions: If your Harness account uses role-based access control (RBAC), ensure you have permissions to view environment settings and SDK keys. Typically, you need View permissions on Feature Flags. Go to RBAC in Harness to understand permissions and Manage roles to configure role assignments.

Supported SDKs​

Split Synchronizer works with the following Harness FME server-side SDKs:

Synchronizer compatibility matrix​

LanguageSupported Synchronizer VersionsSupported SDK Versions
JavaNot implemented—
JavaScript1.x, 2.xJavaScript SDK 9.x, 10.x
Ruby1.6, 1.7, 1.8, 2.xRuby SDK 4.x, 5.x, 6.x (required for 2.x features)
PHP1.6, 1.7, 1.8, 2.xPHP SDK 5.x
Python1.6, 1.7, 1.8, 2.xPython SDK 5.x
.NET/.NET Core1.6, 1.7, 1.8, 2.x.NET/Core 2.1+, 3.x
Go Lang1.x, 2.xGoLang SDK 1.x
iOSN/A—
AndroidN/A—

Notes:

  • “Not implemented” means no Synchronizer support for that language.
  • “N/A” means not applicable. The Synchronizer is not used for those platforms.
  • SDK versions noted are minimum or recommended versions tested with each Synchronizer version.

If you need Split Synchronizer support for a language not listed here, contact Harness Support to discuss your options.

Architecture​

Split Synchronizer performs five primary actions:

  • Fetch feature flags: Retrieve feature flag definitions from Harness FME servers.
  • Fetch segments: Retrieve segment lists and membership data.
  • Post impressions: Send impression data (records of flag evaluations) generated by SDKs to Harness servers for analytics.
  • Post telemetry: Send SDK performance metrics such as latencies and operational data.
  • Post events: Send custom event data generated by SDK .track method calls to Harness servers.
Pipelined data eviction in version 5.0

Starting with version 5.0, Split Synchronizer uses a new approach to impressions and events eviction that replaces periodic fetching and posting. The new pipeline has three stages: a thread dedicated to fetching data from Redis and placing it in a buffer, N threads (where N is derived from available CPU cores) dedicated to parsing and formatting the data into a second buffer, and N configurable threads that pick formatted data and post it to Harness servers. This results in significantly increased throughput for high-volume deployments.

Synchronizer architecture diagram​

Split Synchronizer architecture diagram showing data flow between SDKs, Redis, and Harness FME servers

Split Synchronizer coordinates data between Redis (shared by SDKs) and Harness FME servers

Setup​

Split Synchronizer can be installed using Docker (recommended), command-line binary, or deployed to cloud platforms like Heroku or AWS ECS.

Docker is the recommended installation method for most deployments. The Split Synchronizer Docker image is available on Docker Hub.

To install using Docker:

  1. Pull the image: docker pull splitsoftware/split-synchronizer
  2. Run the container with your configuration:
docker run --rm --name split-synchronizer \
 -p 3010:3010 \
 -e SPLIT_SYNC_APIKEY="your-sdk-key" \
 -e SPLIT_SYNC_REDIS_HOST=<your-redis-host> \
 -e SPLIT_SYNC_REDIS_PORT=<your_redis_port> \
 splitsoftware/split-synchronizer
Synchronizer mode with local redis instance

Sometimes, when building POCs or testing the synchronizer locally, you might want to launch our docker container image, pointing to a local redis server (or another container with redis, whose port has been mapped to a local one). In such case, you should consider adding the option --network="host" (appending it to the command shown above) when launching the synchronizer. This will allow you to use -e SPLIT_SYNC_REDIS_HOST="localhost", with the split-synchronizer container properly reaching your local redis server.

Docker configuration

The Advanced configuration section includes additional Docker information in the column Docker environment variable.

See Deploy Synchronizer Docker Container in AWS ECS for a full step-by-step AWS deployment guide.

Command line​

To install and run the service from command line, follow the steps below depending of your platform.

Linux​

On Linux systems, the Synchronizer service install script is invoked with this.

curl -L -o install_linux.bin 'https://downloads.split.io/synchronizer/install_split_sync_linux.bin' && chmod 755 install_linux.bin && ./install_linux.bin

OSX​

On OSX systems, the Synchronizer service install script is invoked with this.

curl -L -o install_osx.bin 'https://downloads.split.io/synchronizer/install_split_sync_osx.bin' && chmod 755 install_osx.bin && ./install_osx.bin

Windows​

On Microsoft Windows systems, follow these steps:

  1. Download the app from https://downloads.split.io/synchronizer/split_sync_windows.zip.

  2. Unzip the downloaded file.

  3. Run it!

Download previous versions

The links above point to the latest version. To download a previous version of split-sync, go to https://downloads.split.io/synchronizer/downloads.sync.html.

Run the service​

To run the service, paste the snippet below into your command line terminal and add in your SDK-Key.

Linux/Mac​

split-sync -apikey "your_sdk_key"

Windows​

Open the cmd terminal or the PowerShell terminal, go to (cd) unzipped Split Synchronizer folder, and type:

split-sync.exe -apikey "your_sdk_key"
Redis instance

On the samples above, Redis is running as a local service with default host: localhost and default port: 6379. For further information, see Advanced configuration.

Redis database

To maximize performance and isolation, we recommend connecting to a Redis database dedicated to the Split Synchronizer. For further information, see Advanced configuration.

Redis Sentinel support

Split Synchronizer also supports Redis Sentinel (v2) replication for high availability with automatic failover. Go to Redis Sentinel documentation to learn how to set up replication.

Redis Cluster support

Split Synchronizer supports Redis Cluster (Redis 3.0.0 and later) for horizontal scaling across multiple Redis nodes. Go to Redis Cluster documentation to learn how to configure clustering.

You can run the service with the simple steps above, but the system is more stable in your production environment when you run the job with a scheduling system. We recommend starting the synchronizer via supervisord, a daemon that launches other processes and ensures they stay running.

To use supervisord, make sure that it is installed on your machine. You can get help on the installation at the official Supervisord documentation.

When supervisord is installed into your project, copy and paste the program below anywhere into the supervisord.conf file that should now be in your project.

[program:splitio_sync]
command=/usr/local/bin/split-sync -config /path/to/your/config.file.json
process_name = SplitIO
numprocs = 1
autostart=true
autorestart=true
user = your_user
stderr_logfile=/var/log/splitio.err.log
stderr_logfile_maxbytes = 1MB
stdout_logfile=/var/log/splitio.out.log
stdout_logfile_maxbytes = 1MB

Advanced configuration​

The Synchronizer service has a number of knobs for configuring performance. Each knob is tuned to a reasonable default, however, you can override the default values by changing a splitio.config.json file or by setting your customer values as parameters of -config in the command line option. In this section, we lay out all the different knobs you can configure for performance, Redis, and logging.

The splitio.config.json file provided via the -config option lets you control how often the synchronizer fetches data from Harness servers. You can create a sample JSON file automatically with default values by running this command.

./split-sync -write-default-config "/home/someuser/splitio.config.json"
Configuration path file

Save the JSON config file on your server in your desired folder. For instance, on Linux systems, it could be saved in the etc folder. Remember to set the right path as the -config parameter.

{
  "apikey": "YOUR_SDK_KEY",
  "ipAddressEnabled": true,
  "initialization": {
    "timeoutMS": 10000,
    "forceFreshStartup": false
  },
  "storage": {
    "type": "redis",
    "redis": {
      "host": "localhost",
      "port": 6379,
      "db": 0,
      "username": "",
      "password": "",
      "prefix": "",
      "network": "tcp",
      "maxRetries": 0,
      "dialTimeout": 5,
      "readTimeout": 10,
      "writeTimeout": 5,
      "poolSize": 10,
      "sentinelReplication": false,
      "sentinelAddresses": "",
      "sentinelMaster": "",
      "clusterMode": false,
      "clusterNodes": "",
      "keyHashTag": "",
      "enableTLS": false,
      "tlsServerName": "",
      "caCertificates": null,
      "tlsSkipNameValidation": false,
      "tlsClientCertificate": "",
      "tlsClientKey": ""
    }
  },
  "sync": {
    "splitRefreshRateMs": 60000,
    "segmentRefreshRateMs": 60000,
    "impressionsMode": "optimized",
    "advanced": {
      "streamingEnabled": true,
      "httpTimeoutMs": 30000,
      "internalTelemetryRateMs": 3600000,
      "telemetryPushRateMs": 60000,
      "impressionsFetchSize": 0,
      "impressionsProcessConcurrency": 0,
      "impressionsProcessBatchSize": 0,
      "impressionsPostConcurrency": 0,
      "impressionsPostSize": 0,
      "impressionsAccumWaitMs": 0,
      "eventsFetchSize": 0,
      "eventsProcessConcurrency": 0,
      "eventsProcessBatchSize": 0,
      "eventsPostConcurrency": 0,
      "eventsPostSize": 0,
      "eventsAccumWaitMs": 0
    }
  },
  "admin": {
    "host": "0.0.0.0",
    "port": 3010,
    "username": "",
    "password": "",
    "secureChecks": false
  },
  "integrations": {
    "impressionListener": {
      "endpoint": "",
      "queueSize": 100
    },
    "slack": {
      "webhook": "",
      "channel": ""
    }
  },
  "logging": {
    "level": "info",
    "output": "stdout",
    "rotationMaxFiles": 10,
    "rotationMaxSizeKb": 0
  },
  "healthcheck": {
    "app": {
      "storageCheckRateMs": 3600000
    }
  }
}
Command line parameters

All the options available in the JSON file are also included as command line options. Run the command followed by the -help option to see more details, or just keep reading this documentation page.

Methods to configure the Split Synchronizer​

You can configure the Split Synchronizer service using the command line or by directly editing the above mentioned JSON configuration file.

Config values priority

All config values are set with a default value that you can see in the example JSON file above. You can overwrite the default value from the JSON config file, and you can overwrite the JSON config file from the command line. See a sample below for how to do that via command line.

split-sync -config "/etc/splitio.config.json" -log-level "debug" -redis-pass "somePass"

Handling High Impression Rates with Synchronizer​

When using a server-side SDK with the Split Synchronizer and Redis, it is important to properly configure the Synchronizer to handle a high load of incoming impressions efficiently.

Key Configuration Parameters​

The Split Synchronizer (version 1.6.0 and above) exposes several parameters that control impression processing performance:

  • impressionsMaxSize: Maximum size for the impressions queue.
  • impressionsRefreshRate: How often impressions are processed and sent (in seconds).
  • impressionsThreads: Number of threads handling impressions processing.
  • impressionsPerPost: Number of impressions sent per post request.

Adjust these parameters according to your expected impression volume. Because the Synchronizer uses multithreading, increasing impressionsThreads can reduce latency in posting impressions.

Below is an example JSON configuration designed to handle approximately 100,000 impressions per minute. Update the API Key, Redis host, port, and database number to match your environment before applying.

{
  "apiKey": "YOUR_API_KEY",
  "proxy": {
    "port": 3000,
    "adminPort": 3010,
    "adminUsername": "",
    "adminPassword": "",
    "dashboardTitle": "",
    "persistInFilePath": "",
    "impressionsMaxSize": 10485760,
    "eventsMaxSize": 10485760,
    "auth": {
      "sdkAPIKeys": [
        "SDK_API_KEY"
      ]
    }
  },
  "redis": {
    "host": "localhost",
    "port": 6379,
    "db": 0,
    "password": "",
    "prefix": "",
    "network": "tcp",
    "maxRetries": 0,
    "dialTimeout": 5,
    "readTimeout": 10,
    "writeTimeout": 5,
    "poolSize": 10,
    "sentinelReplication": false,
    "sentinelAddresses": "",
    "sentinelMaster": ""
  },
  "sync": {
    "admin": {
      "adminPort": 3010,
      "adminUsername": "",
      "adminPassword": "",
      "dashboardTitle": ""
    }
  },
  "log": {
    "verbose": false,
    "debug": false,
    "stdout": false,
    "file": "/tmp/split-agent.log",
    "fileMaxSizeBytes": 2000000,
    "fileBackupCount": 3,
    "slackChannel": "",
    "slackWebhookURL": ""
  },
  "impressionListener": {
    "endpoint": ""
  },
  "splitsRefreshRate": 60,
  "segmentsRefreshRate": 60,
  "impressionsRefreshRate": 20,
  "impressionsPerPost": 10000,
  "impressionsThreads": 5,
  "eventsPushRate": 60,
  "eventsConsumerReadSize": 10000,
  "eventsConsumerThreads": 1,
  "metricsRefreshRate": 60,
  "httpTimeout": 60
}

Configuration reference​

Split Synchronizer configuration can be specified using command-line flags, JSON configuration file, or environment variables. The following reference lists all available configuration options.

CLI configuration options and equivalents in JSON and environment variables​

Boolean option values in version 5.0

Since version 5.0, Split Synchronizer only accepts true and false (lowercase) for boolean flags. Values like enabled, on, yes, or tRue will cause startup errors. This applies to JSON configuration, CLI arguments, and environment variables.

Command line optionJSON optionEnvironment variable (container-only)Description
log-levellevelSPLIT_SYNC_LOG_LEVELLog level (error|warning|info|debug|verbose)
log-outputoutputSPLIT_SYNC_LOG_OUTPUTWhere to output logs (defaults to stdout)
log-rotation-max-filesrotationMaxFilesSPLIT_SYNC_LOG_ROTATION_MAX_FILESMax number of files to keep when rotating logs
log-rotation-max-size-kbrotationMaxSizeKbSPLIT_SYNC_LOG_ROTATION_MAX_SIZE_KBMax file size before rotating log files.
admin-hosthostSPLIT_SYNC_ADMIN_HOSTHost where the admin server will listen
admin-portportSPLIT_SYNC_ADMIN_PORTAdmin port where incoming connections will be accepted
admin-usernameusernameSPLIT_SYNC_ADMIN_USERNAMEHTTP basic auth username for admin endpoints
admin-passwordpasswordSPLIT_SYNC_ADMIN_PASSWORDHTTP basic auth password for admin endpoints
admin-secure-hcsecureChecksSPLIT_SYNC_ADMIN_SECURE_HCSecure Healthcheck endpoints as well.
admin-tls-enabledenabledSPLIT_SYNC_ADMIN_TLS_ENABLEDEnable HTTPS on proxy endpoints.
admin-tls-client-validationclientValidationSPLIT_SYNC_ADMIN_TLS_CLIENT_VALIDATIONEnable client cert validation.
admin-tls-server-nameserverNameSPLIT_SYNC_ADMIN_TLS_SERVER_NAMEServer name as it appears in provided server-cert.
admin-tls-cert-chain-fncertChainFnSPLIT_SYNC_ADMIN_TLS_CERT_CHAIN_FNX509 Server certificate chain.
admin-tls-private-key-fnprivateKeyFnSPLIT_SYNC_ADMIN_TLS_PRIVATE_KEY_FNPEM Private key file name.
admin-tls-client-validation-root-certclientValidationRootCertFnSPLIT_SYNC_ADMIN_TLS_CLIENT_VALIDATION_ROOT_CERTX509 root cert for client validation.
admin-tls-min-tls-versionminTlsVersionSPLIT_SYNC_ADMIN_TLS_MIN_TLS_VERSIONMinimum TLS version to allow X.Y.
admin-tls-allowed-cipher-suitesallowedCipherSuitesSPLIT_SYNC_ADMIN_TLS_ALLOWED_CIPHER_SUITESComma-separated list of cipher suites to allow.
impression-listener-endpointendpointSPLIT_SYNC_IMPRESSION_LISTENER_ENDPOINTHTTP endpoint to forward impressions to
impression-listener-queue-sizequeueSizeSPLIT_SYNC_IMPRESSION_LISTENER_QUEUE_SIZEmax number of impressions bulks to queue
slack-webhookwebhookSPLIT_SYNC_SLACK_WEBHOOKslack webhook to post log messages
slack-channelchannelSPLIT_SYNC_SLACK_CHANNELslack channel to post log messages
apikeyapikeySPLIT_SYNC_APIKEYSplit Server-side SDK api-key
ip-address-enabledipAddressEnabledSPLIT_SYNC_IP_ADDRESS_ENABLEDBundle host's ip address when sending data to Harness FME
timeout-mstimeoutMSSPLIT_SYNC_TIMEOUT_MSHow long to wait until the synchronizer is ready
snapshotsnapshotSPLIT_SYNC_SNAPSHOTSnapshot file to use as a starting point
force-fresh-startupforceFreshStartupSPLIT_SYNC_FORCE_FRESH_STARTUPWipe storage before starting the synchronizer
storage-typetypeSPLIT_SYNC_STORAGE_TYPEStorage driver to use for caching feature flags and segments and user-generated data
split-refresh-rate-mssplitRefreshRateMsSPLIT_SYNC_SPLIT_REFRESH_RATE_MSHow often to refresh feature flags
segment-refresh-rate-mssegmentRefreshRateMsSPLIT_SYNC_SEGMENT_REFRESH_RATE_MSHow often to refresh segments
impressions-modeimpressionsModeSPLIT_SYNC_IMPRESSIONS_MODEwhether to send all impressions for debugging
streaming-enabledstreamingEnabledSPLIT_SYNC_STREAMING_ENABLEDEnable/disable streaming functionality
http-timeout-mshttpTimeoutMsSPLIT_SYNC_HTTP_TIMEOUT_MSTotal http request timeout
internal-metrics-rate-msinternalTelemetryRateMsSPLIT_SYNC_INTERNAL_METRICS_RATE_MSHow often to send internal metrics
telemetry-push-rate-mstelemetryPushRateMsSPLIT_SYNC_TELEMETRY_PUSH_RATE_MShow often to flush sdk telemetry
impressions-fetch-sizeimpressionsFetchSizeSPLIT_SYNC_IMPRESSIONS_FETCH_SIZEImpression fetch bulk size
impressions-process-concurrencyimpressionsProcessConcurrencySPLIT_SYNC_IMPRESSIONS_PROCESS_CONCURRENCY#Threads for processing imps
impressions-process-batch-sizeimpressionsProcessBatchSizeSPLIT_SYNC_IMPRESSIONS_PROCESS_BATCH_SIZESize of imp processing batchs
impressions-post-concurrencyimpressionsPostConcurrencySPLIT_SYNC_IMPRESSIONS_POST_CONCURRENCY#concurrent imp post threads
impressions-post-sizeimpressionsPostSizeSPLIT_SYNC_IMPRESSIONS_POST_SIZEMax #impressions to send per POST
impressions-accum-wait-msimpressionsAccumWaitMsSPLIT_SYNC_IMPRESSIONS_ACCUM_WAIT_MSMax ms to wait to close an impressions bulk
events-fetch-sizeeventsFetchSizeSPLIT_SYNC_EVENTS_FETCH_SIZEHow many impressions to pop from storage at once
events-process-concurrencyeventsProcessConcurrencySPLIT_SYNC_EVENTS_PROCESS_CONCURRENCY#Threads for processing imps
events-process-batch-sizeeventsProcessBatchSizeSPLIT_SYNC_EVENTS_PROCESS_BATCH_SIZESize of imp processing batchs
events-post-concurrencyeventsPostConcurrencySPLIT_SYNC_EVENTS_POST_CONCURRENCY#concurrent imp post threads
events-post-sizeeventsPostSizeSPLIT_SYNC_EVENTS_POST_SIZEMax #impressions to send per POST
events-accum-wait-mseventsAccumWaitMsSPLIT_SYNC_EVENTS_ACCUM_WAIT_MSMax ms to wait to close an events bulk
redis-hosthostSPLIT_SYNC_REDIS_HOSTRedis server hostname
redis-portportSPLIT_SYNC_REDIS_PORTRedis Server port
redis-dbdbSPLIT_SYNC_REDIS_DBRedis DB
redis-passpasswordSPLIT_SYNC_REDIS_PASSRedis password
redis-userusernameSPLIT_SYNC_REDIS_USERRedis username
redis-prefixprefixSPLIT_SYNC_REDIS_PREFIXRedis key prefix
redis-networknetworkSPLIT_SYNC_REDIS_NETWORKRedis network protocol
redis-max-retriesmaxRetriesSPLIT_SYNC_REDIS_MAX_RETRIESRedis connection max retries
redis-dial-timeoutdialTimeoutSPLIT_SYNC_REDIS_DIAL_TIMEOUTRedis connection dial timeout
redis-read-timeoutreadTimeoutSPLIT_SYNC_REDIS_READ_TIMEOUTRedis connection read timeout
redis-write-timeoutwriteTimeoutSPLIT_SYNC_REDIS_WRITE_TIMEOUTRedis connection write timeout
redis-poolpoolSizeSPLIT_SYNC_REDIS_POOLRedis connection pool size
redis-sentinel-replicationsentinelReplicationSPLIT_SYNC_REDIS_SENTINEL_REPLICATIONRedis sentinel replication enabled.
redis-sentinel-addressessentinelAddressesSPLIT_SYNC_REDIS_SENTINEL_ADDRESSESList of redis sentinels
redis-sentinel-mastersentinelMasterSPLIT_SYNC_REDIS_SENTINEL_MASTERName of master
redis-cluster-modeclusterModeSPLIT_SYNC_REDIS_CLUSTER_MODERedis cluster enabled.
redis-cluster-nodesclusterNodesSPLIT_SYNC_REDIS_CLUSTER_NODESList of redis cluster nodes.
redis-cluster-key-hashtagkeyHashTagSPLIT_SYNC_REDIS_CLUSTER_KEY_HASHTAGkeyHashTag for redis cluster.
redis-tlsenableTLSSPLIT_SYNC_REDIS_TLSUse SSL/TLS for connecting to redis
redis-tls-server-nametlsServerNameSPLIT_SYNC_REDIS_TLS_SERVER_NAMEServer name to use when validating a server public key
redis-tls-ca-certscaCertificatesSPLIT_SYNC_REDIS_TLS_CA_CERTSRoot CA certificates to connect to a redis server via SSL/TLS
redis-tls-skip-name-validationtlsSkipNameValidationSPLIT_SYNC_REDIS_TLS_SKIP_NAME_VALIDATIONBlindly accept server's public key.
redis-tls-client-certificatetlsClientCertificateSPLIT_SYNC_REDIS_TLS_CLIENT_CERTIFICATEClient certificate signed by a known CA
redis-tls-client-keytlsClientKeySPLIT_SYNC_REDIS_TLS_CLIENT_KEYClient private key matching the certificate.
storage-check-rate-msstorageCheckRateMsSPLIT_SYNC_STORAGE_CHECK_RATE_MSHow often to check storage health
flag-sets-filterflagSetsFilterSPLIT_SYNC_FLAG_SETS_FILTERThis setting allows the Split Synchronizer to only synchronize the feature flags in the specified flag sets, avoiding unused or unwanted flags from being synced on the Split Synchronizer instance, bringing all the benefits from a reduced payload.

Deploying Synchronizer on Heroku​

Follow these steps to deploy the Synchronizer container on Heroku:

  1. Clone the GitHub repository or download the source code and unzip it to a new folder (e.g., mysync).

  2. Open a terminal and cd to that folder.

  3. Run this command to create a Heroku app:

    heroku create

  4. You should see a response like:

    Creating app... done, ⬢ secret-anchorage-16496
    https://secret-anchorage-16496.herokuapp.com/ | https://git.heroku.com/secret-anchorage-16496.git

  5. Set the stack to container and add the Go language buildpack:

    heroku stack:set container
    heroku buildpacks:set heroku/go

  6. Open the existing Dockerfile and replace the last line:

    ENTRYPOINT ["sh", "./entrypoint.sh"]

    with:

    CMD ["sh", "./entrypoint.sh"]

    Heroku only supports CMD for containers.

  7. Add the following content to a heroku.yml file:

    build:
     docker:
       web: Dockerfile
       worker:
         dockerfile: Dockerfile

  8. Add a Procfile with the content:

    worker: sh entrypoint.sh

  9. On the Heroku dashboard, open your app, go to the Settings tab, and add the following environment variables:

    • SPLIT_SYNC_API_KEY
    • SPLIT_SYNC_REDIS_HOST
    • SPLIT_SYNC_REDIS_PORT
    • SPLIT_SYNC_REDIS_DB
    • SPLIT_SYNC_REDIS_PASS

  10. Modify entrypoint.sh to map the admin dashboard to Heroku's $PORT environment variable.

    Replace the last line:

    exec split-sync ${PARAMETERS}

    with:

    exec split-sync ${PARAMETERS} -sync-admin-port $PORT

  11. Run these git commands to deploy to Heroku:

    git init
    git add .
    git commit -m "Deploy Synchronizer to Heroku"
    git push heroku master

  12. Check logs using heroku logs.

  13. You should see output similar to:

    2019-09-11T19:53:06.954882+00:00 app[worker.1]: __      ____        _ _ _
    2019-09-11T19:53:06.954887+00:00 app[worker.1]: / /__   / ___| _ __ | (_) |_
    2019-09-11T19:53:06.954890+00:00 app[worker.1]: / / \ \  \___ \| '_ \| | | __|
    2019-09-11T19:53:06.954891+00:00 app[worker.1]: \ \  \ \  ___) | |_) | | | |_
    2019-09-11T19:53:06.954893+00:00 app[worker.1]: \_\ / / |____/| .__/|_|_|\__|
    2019-09-11T19:53:06.954895+00:00 app[worker.1]: /_/        |_|
    2019-09-11T19:53:06.954897+00:00 app[worker.1]: 
    2019-09-11T19:53:06.954899+00:00 app[worker.1]: 
    2019-09-11T19:53:06.954922+00:00 app[worker.1]: 
    2019-09-11T19:53:06.954924+00:00 app[worker.1]: Split Synchronizer - Version: 2.5.1 (2178c61)
    2019-09-11T19:53:06.955154+00:00 app[worker.1]: Log file: /tmp/split-agent.log

  14. Access the Admin Dashboard by visiting https://[heroku-app-name].herokuapp.com/admin/dashboard.

Using the Synchronizer as a proxy service on Heroku​

Since Heroku maps only one port, you need to assign the proxy listener port to the $PORT environment variable.

  1. Update entrypoint.sh by replacing exec split-sync ${PARAMETERS} with exec split-sync ${PARAMETERS} -proxy-port $PORT.

  2. Modify Procfile by changing its content to web: sh entrypoint.sh.

  3. Add the following environment variables in the Heroku UI:

    • SPLIT_SYNC_PROXY (value: on)
    • SPLIT_SYNC_PROXY_SDK_APIKEYS (value: your customer API key)

  4. Deploy changes:

    git add .
    git commit -m "Enable proxy mode on Heroku"
    git push heroku master

  5. Configure SDKs. An example Java SDK configuration:

    SplitClientConfig config = SplitClientConfig.builder()
     .setBlockUntilReadyTimeout(5000)
     .endpoint("http://[heroku-app-name].herokuapp.com", "http://[heroku-app-name].herokuapp.com")
     .build();

    try {
        splitFactory = SplitFactoryBuilder.build("custom api key", config);
        client = splitFactory.client();
        client.blockUntilReady();
    } catch (Exception e) {
        System.out.print("Exception: " + e.getMessage());
    }

Since Heroku maps only a single port, the Admin Dashboard will not be accessible when running in proxy mode.

Listener​

The Split Synchronizer provides an impression listener that bulks post impressions to a user-defined HTTP endpoint.

The endpoint should expect a POST request, containing a JSON body with the following format.

JSON Impression
{
 "impressions": [
  {
   "testName": "feature1",
   "keyImpressions": [
    {
     "keyName": "user1",
     "treatment": "on",
     "time": 1502754901182,
     "changeNumber": -1,
     "label": ""
    },
    {
     "keyName": "user2",
     "treatment": "off",
     "time": 1502754876144,
     "changeNumber": -1,
     "label": ""
    }
   ]
  }
 ],
 "sdkVersion": "php-5.2.2",
 "machineIP": "208.63.222.7",
 "MachineName": ""
}

Currently, the configuration options are available in the integrations.impressionListener section of the JSON configuration file detailed in Advanced configuration.

Using a network proxy​

If you need to use a network proxy, configure proxies by setting the environment variables HTTP_PROXY and HTTPS_PROXY. The internal HTTP client reads those variables and uses them to perform the server request.

Example: Environment variables
$ export HTTP_PROXY="http://10.10.1.10:3128"
$ export HTTPS_PROXY="http://10.10.1.10:1080"

Inject a certificate into a Synchronizer Docker image​

If the Synchronizer Docker container is running in a network that uses an SSL proxy, the Synchronizer may fail to authenticate the root certificate. This causes errors like the following when trying to connect to Harness FME servers to fetch feature flag definitions:

SPLITIO-AGENT | ERROR: 2020/08/19 14:42:51 fetchdataforproxy.go:209: Error fetching split changes  
Get https://sdk.split.io/api/splitChanges?since=-1: x509: certificate signed by unknown authority

To resolve this, you need to inject the root certificate into the Synchronizer Docker image by rebuilding it with the proxy certificates included.

  1. Clone the Synchronizer public repository:

    git clone https://github.com/splitio/split-synchronizer

  2. Change to the cloned directory:

    cd split-synchronizer

  3. Copy your proxy certificates (root, intermediate, and proxy certs) into this folder. For example:

    cp /path/to/certs/root.crt .
    cp /path/to/certs/intermediate.crt .
    cp /path/to/certs/proxy.pem .

  4. Open the Dockerfile in the split-synchronizer folder with a text editor, and just before the line containing EXPOSE 3000 3010, add:

    COPY root.crt /etc/ssl/certs/root.crt
    COPY intermediate.crt /etc/ssl/certs/intermediate.crt
    COPY proxy.pem /etc/ssl/certs/proxy.pem
    RUN cat /etc/ssl/certs/root.crt >> /etc/ssl/certs/ca-certificates.crt

  5. Save and close the Dockerfile.

  6. Build the new Docker image:

    docker build --tag split-sync:latest .

  7. Run the new image to verify it works (replace environment variables as needed). The http_proxy and https_proxy variables are optional based on your setup:

    docker run --rm --name split-sync -p 3010:3010 --net="host" \
     -e SPLIT_SYNC_API_KEY="SDK API KEY" \
     -e SPLIT_SYNC_LOG_STDOUT="on" \
     -e SPLIT_SYNC_LOG_DEBUG="true" \
     -e SPLIT_SYNC_LOG_VERBOSE="true" \
     -e SPLIT_SYNC_REDIS_HOST="Redis Host" \
     -e SPLIT_SYNC_REDIS_PORT=6379 \
     -e http_proxy="https://[internal proxy host]" \
     -e https_proxy="https://[internal proxy host]" \
     split-sync

Admin dashboard and monitoring​

Split Synchronizer provides admin endpoints and a dashboard for monitoring synchronization status, cached data, and performance metrics in real-time.

Admin endpoints​

The Split Synchronizer service exposes admin endpoints on port 3010 by default. For security, endpoints support HTTP Basic Authentication configured through admin-username and admin-password settings.

/info/ping

A ping endpoint to monitor the service status. If the service is running, it sends the text response pong and the HTTP status code 200.

/info/version

Returns the split-sync version in JSON format.

{
  "version" : "1.1.0"
}

/info/uptime

Returns the uptime string representation in JSON format.

{
  "uptime" : "5d 3h 36m 39s"
}

/info/config Returns a JSON object describing the current configuration of the Synchronizer.

{
  "config": {
    "apikey": "*",
    "ipAddressEnabled": true,
    "initialization": {
      "timeoutMS": 10000,
      "forceFreshStartup": false
    },
    "storage": {
      "type": "redis",
      "redis": {
        "host": "localhost",
        "port": 6379,
        "db": 0,
        "username": "",
        "password": "",
        "prefix": "",
        "network": "tcp",
        "maxRetries": 0,
        "dialTimeout": 5,
        "readTimeout": 10,
        "writeTimeout": 5,
        "poolSize": 10,
        "sentinelReplication": false,
        "sentinelAddresses": "",
        "sentinelMaster": "",
        "clusterMode": false,
        "clusterNodes": "",
        "keyHashTag": "",
        "enableTLS": false,
        "tlsServerName": "",
        "caCertificates": null,
        "tlsSkipNameValidation": false,
        "tlsClientCertificate": "",
        "tlsClientKey": ""
      }
    },
    "sync": {
      "splitRefreshRateMs": 60000,
      "segmentRefreshRateMs": 60000,
      "impressionsMode": "optimized",
      "advanced": {
        "streamingEnabled": true,
        "httpTimeoutMs": 30000,
        "internalTelemetryRateMs": 3600000,
        "telemetryPushRateMs": 60000,
        "impressionsFetchSize": 0,
        "impressionsProcessConcurrency": 0,
        "impressionsProcessBatchSize": 0,
        "impressionsPostConcurrency": 0,
        "impressionsPostSize": 0,
        "impressionsAccumWaitMs": 0,
        "eventsFetchSize": 0,
        "eventsProcessConcurrency": 0,
        "eventsProcessBatchSize": 0,
        "eventsPostConcurrency": 0,
        "eventsPostSize": 0,
        "eventsAccumWaitMs": 0
      }
    },
    "admin": {
      "host": "0.0.0.0",
      "port": 3010,
      "username": "",
      "password": "",
      "secureChecks": false
    },
    "integrations": {
      "impressionListener": {
        "endpoint": "",
        "queueSize": 100
      },
      "slack": {
        "webhook": "",
        "channel": ""
      }
    },
    "logging": {
      "level": "info",
      "output": "stdout",
      "rotationMaxFiles": 10,
      "rotationMaxSizeKb": 0
    },
    "healthcheck": {
      "app": {
        "storageCheckRateMs": 3600000
      }
    }
  }
}

/health/application Returns a JSON object describing whether the synchronizer is healthy or not.

{
  "healthy": true,
  "healthySince": "2021-11-20T19:04:46.528242-03:00",
  "items": [
    {
      "name": "Splits",
      "healthy": true,
      "lastHit": "2021-11-20T19:04:49.079956-03:00"
    },
    {
      "name": "Segments",
      "healthy": true,
      "lastHit": "2021-11-20T19:04:49.268349-03:00"
    },
    {
      "name": "Storage",
      "healthy": true
    }
  ]
}

/health/dependencies Returns a JSON object describing whether the servers the synchronizer depends on are healthy or not.

{
  "serviceStatus": "healthy",
  "dependencies": [
    {
      "service": "https://telemetry.split.io/health",
      "healthy": true,
      "healthySince": "2021-11-20T19:04:46.528262-03:00"
    },
    {
      "service": "https://auth.split.io/health",
      "healthy": true,
      "healthySince": "2021-11-20T19:04:46.528264-03:00"
    },
    {
      "service": "https://sdk.split.io/api/version",
      "healthy": true,
      "healthySince": "2021-11-20T19:04:46.528265-03:00"
    },
    {
      "service": "https://events.split.io/api/version",
      "healthy": true,
      "healthySince": "2021-11-20T19:04:46.528266-03:00"
    },
    {
      "service": "https://streaming.split.io/health",
      "healthy": true,
      "healthySince": "2021-11-20T19:04:46.528266-03:00"
    }
  ]
}

Observability​

The Split Synchronizer exposes an observability endpoint that provides visibility into cached data.

You can use this endpoint in versions 5.0.3 or later.

GET http://<SERVER_HOST>:<SERVER_PORT>/admin/observability

The following metrics are available:

MetricCategoryDescription
Active segmentsCache stateMap of segment names to key counts.
Active splitsCache stateList of feature flags currently loaded.

Example response​

{
  "activeSegments": {
    "employees": 9984,
    "starred_customers": 10122
  },
  "activeSplits": [
    "admin_panel",
    "main_campaign",
    "ab_test"
  ]
}

Use this endpoint to verify that feature flags and segments are synchronized and stored correctly.

Admin Dashboard​

Split-sync has a web admin UI out of the box that exposes all available endpoints. Browse to /admin/dashboard to see it.

The dashboard is organized in four sections for ease of visualization:

  • Dashboard: Tile-sorted summary information, including these metrics:
    • Uptime: Uptime metric
    • Healthy Since: Time passed without errors
    • Logged Errors: Total count of error messages
    • SDKs Total Hits: Total SDKs requests
    • Backend Total Hits: Total backend requests between split-sync and Harness servers
    • Cached Feature flags: Number of feature flags cached in memory
    • Cached Segments: Number of segments cached in memory
    • Impressions Queue Size: shows the total amount of Impressions stored in Redis (only Producer Mode).
    • Impressions Lambda: shows the eviction rate for Impressions (only Producer Mode).
    • Events Queue Size: shows the total amount of Events stored in Redis (only Producer Mode).
    • Events Lambda: shows the eviction rate for Events (only Producer Mode).
    • SDK Server: displays the status of Split server for SDK.
    • Events Server: displays the status of Split server for Events.
    • Streaming Server*: displays the status of Split streaming service
    • Auth Server*: displays the status of Split server for initial streaming authentication
    • Telemetry Server*: displays the status of Split server for telemetry capturing.
    • Storage: (only Sync mode) displays the status of the storage.
    • Sync: displays the status of the Synchronizer.
    • Last Errors Log: List of the last 10 error messages
  • SDK stats: Metrics numbers and a latency graph, measured between SDKs requests integration and proxy.
  • Data inspector: Cached data showing feature flags and segments; filters to find keys and feature flag definitions.
  • Queue Manager: expose sizes of Impressions and Events queues.

Dashboard refresh rate

The dashboard numbers are committed every 60 seconds. The Logged Errors, Last Errors Log tiles, and the Data inspector section are populated each time the dashboard is refreshed. For Impressions and Events Queue size the numbers are refreshed every 10 seconds.

Service shutdown​

The split-sync service can catch a kill sig command and start a graceful shutdown, flushing all cached data progressively. Additionally, you can perform graceful stop and force stop (kill -9) with one click from the admin dashboard.

If you have configured a Slack channel and the Slack Webhook URL, an alert is sent to the channel when and initialization or shutdown is performed.

Troubleshooting​

Use the following troubleshooting guidance to resolve common Split Synchronizer issues.

Synchronizer returns 500 HTTP error when SDK connects in proxy mode

Verify the SDK API key used by Synchronizer matches an environment that has active feature flags configured. Check Synchronizer debug logs for empty feature flag responses from Harness FME servers.

Details: When using Synchronizer in proxy mode, the SDK may never become ready and receive a 500 HTTP error. Synchronizer’s debug logs show successful calls to Harness FME servers, but the JSON response contains an empty list of feature flags:

SPLITIO-AGENT  - DEBUG - 2020/10/12 21:41:51 logger.go:35: GET |500| 285.71µs | 10.10.6.249 | /api/splitChanges
SPLITIO-AGENT  - DEBUG - 2020/10/12 21:41:52 client.go:60: Authorization [ApiKey]:  1c9s...e19o
SPLITIO-AGENT  - DEBUG - 2020/10/12 21:41:52 client.go:56: [GET]  https://sdk.split.io/api/splitChanges?since=-1
SPLITIO-AGENT  - DEBUG - 2020/10/12 21:41:52 client.go:64: Headers: map[Accept-Encoding:[gzip] Content-Type:[application/json]]
SPLITIO-AGENT  - VERBOSE - 2020/10/12 21:41:52 client.go:95: [RESPONSE_BODY] {"splits":[],"since":-1,"till":-1} [END_RESPONSE_BODY]

This indicates that no feature flags are present in the environment associated with the SDK API key used by the Synchronizer. Although the upstream HTTP call succeeds, Synchronizer returns a 500 error to the SDK because it cannot compute any treatments without any feature flags

  • Ensure that the environment linked to the SDK API key used by Synchronizer has active feature flags configured.
  • Alternatively, verify that you are using the correct SDK API key for the intended environment with feature flags.

SDK getTreatment Always Returning 'control' When Using Synchronizer Docker​

After installing and running the Split Synchronizer Docker instance with a Redis instance, and configuring an SDK to use Redis, the getTreatment call always returns 'control'.

By default, the Synchronizer Docker instance uses a prefix for Redis keys. If the SDK does not specify the same prefix in its Redis configuration, it cannot read the data stored by Synchronizer.

  1. Check if the Synchronizer is using a prefix by running:

    redis-cli
    keys *

    Look for text before "SPLITIO" in the keys.

    Example output showing the prefix myprefix:

    127.0.0.1:6379> keys *
    1) "myprefix.SPLITIO.split.Split1"
    2) "myprefix.SPLITIO.splits.till"
    3) "myprefix.SPLITIO.split.Split2"
    4) "myprefix.SPLITIO.split.nico_test"
    5) "myprefix.SPLITIO.split.coach_matching_v1"
    6) "myprefix.SPLITIO.split.clients_on"
    7) "myprefix.SPLITIO.split.Split3"
    8) "myprefix.SPLITIO.split.sample_feature"
    9) "myprefix.SPLITIO.segments.registered"
    10) "myprefix.SPLITIO.split.Demo_split"
    11) "myprefix.SPLITIO.split.clients"

  2. Update your SDK configuration. Specify the redisPrefix parameter in your SDK configuration so it matches the Synchronizer prefix.

    For example, in the Python SDK:

    from splitio import get_factory

    config = {
        'redisHost': 'localhost',
        'redisPort': 6379,
        'redisDb': 0,
        'redisPassword': 'somePassword',
        'redisPrefix': 'myprefix'
    }

No Impressions Sent from Python SDK 7.x and Synchronizer 1.x​

When using Synchronizer 1.x with Python SDK 7.x, the Python SDK processes treatments correctly and Synchronizer does not report any errors. However, no impressions are sent to the Harness FME servers.

Starting in Python SDK 7.0.0, design changes were made to align with the enhancements introduced in Synchronizer 2.0. Therefore, when using Python SDK 7.x, you must upgrade to Synchronizer 2.x.

POST method returns 404 Not Found error in Split Synchronizer logs and admin dashboard

Verify the API key is SDK type (not Admin or Client type). Check Harness FME Admin settings API keys page to confirm key type. SDK keys are required for Synchronizer to post impressions and events.

Details: After starting Split Synchronizer (version 1.6.0 and above), the debug log and admin dashboard show the error POST method: Status Code: 404 - 404 Not Found on all network POST calls.

This error occurs because an incorrect API key is passed to the Synchronizer, causing the Synchronizer to be unable to find the Account in the Harness FME servers.

  1. Verify the API key used by Synchronizer is correct. Synchronizer API key must be of SDK type. API keys can be viewed from Admin settings on the API keys page: https://app.split.io/org/[Your Account ID]/admin/apis.

  2. Ensure the API key is properly passed to Synchronizer.

    Common ways include:

    • Command line argument: -api-key <APIKEY>

    • JSON configuration file: The apiKey property is used to issue requests against Harness FME servers.

    • Proxy mode usage: When Synchronizer is used in proxy mode (not Redis), the "auth" section and "sdkAPIKeys" allow setting custom API keys for internal use, enabling SDKs to use the internal custom API key.

    • Docker environment variable: If running Synchronizer within the packaged Docker image, use:

      -e SPLIT_SYNC_API_KEY=<APIKEY>
Redis connection refused when starting Split Synchronizer

Verify Redis is running and accessible at the configured host and port. Check firewall rules allow connections to Redis port (default 6379). Test connectivity with redis-cli -h <host> -p <port> ping.

Split Synchronizer fails to connect to Harness FME servers with TLS or certificate errors

Verify TLS configuration in Synchronizer settings. If using custom certificates, ensure cert paths are correct and readable. For Docker deployments, mount certificate files into the container and reference mounted paths in configuration.

High memory usage or performance degradation in Split Synchronizer under heavy impression load

Tune impression processing parameters: increase impressions-process-concurrency to match CPU cores, adjust impressions-fetch-size and impressions-post-size based on volume. Monitor admin dashboard queue sizes and latency metrics to identify bottlenecks.

Next steps​

You have successfully installed and configured Split Synchronizer. Your SDKs can now share feature flag data through Redis while Synchronizer coordinates updates with Harness FME servers.

  • Configure FME SDKs: Update SDK configuration to connect to your Redis instance and use Split Synchronizer.
  • Monitor Split Synchronizer health: Use the admin dashboard at http://<host>:3010/admin/dashboard to monitor synchronization status, queue sizes, and performance metrics.