VOLTTRON Federation

Overview

Federation allows multiple VOLTTRON platforms to securely communicate with each other, enabling cross-platform messaging using publish and subscribe. This document explains how to set up federation between VOLTTRON platforms, including the REST API endpoints for registration and discovery.

Federation Components

The federation system consists of several components:

Platform lookup service: An external centralized service that tracks available VOLTTRON platforms
Federation Service: A VOLTTRON platform service that manages connections to other platforms
Routing Service: Routes messages between local and remote platforms

Setting Up Federation

Start the Platform Lookup or Registry Service

The the volttron platform lookup and registry Service is a critical component that enables platform discovery. It must be set up before configuring individual platforms.

Install from pypi:
```
pip install volttron-platform-lookup
```

OR Install from source:

git clone https://github.com/VOLTTRON/platform-lookup.git
cd platform-lookup
poetry install

(Optional) Pre-create the registration password:

The registry service uses a master password stored in registration.key to authenticate platform registrations. You can optionally generate and store your own password in this file before starting the service:
```
# Generate a strong password and store it in registration.key
echo "your-strong-master-password" > registration.key
```
If you don’t create this file, the service will auto-generate a secure password and save it to registration.key on first start.
Start the registry service:
```
# Default port is 8000 you could send a custom port as command line argument or use -h/--help to see usage
volttron-platform-lookup
```
On first run:
- If registration.key doesn’t exist, the service will auto-generate a secure password and save it to registration.key
- If registration.key already exists, the service will use the password from that file
Important: The password in registration.key must be securely communicated to all trusted VOLTTRON instance administrators who need to register and participate in the federated community. Each VOLTTRON platform will need a copy of this password stored in its $VOLTTRON_HOME/platform-lookup.key file.

Securing the Federation Password

Before enabling federation, the master password from the registry service must be stored securely on your VOLTTRON platform:

Optionally setup VOLTTRON_HOME env variable

If a explicit VOLTTRON_HOME env is not set, volttron by default create $USER_HOME/.volttron directory on first start. If using this default ~/.volttron dir please replace $VOLTTRON_HOME with this path in all commands below
If a volttron instance was never started for the configured $VOLTTRON_HOME(or default ~/.volttron), create the directory

Create a secure password file at $VOLTTRON_HOME/platform-lookup.key with restricted permissions and save the password generated by platform lookup service

# Create the secure password file with restricted permissions (read-only for owner)
echo "password generated or configured in registration service" > $VOLTTRON_HOME/platform-lookup.key
chmod 600 $VOLTTRON_HOME/platform-lookup.key

The password file must be readable only by the VOLTTRON process owner (unix permissions 0o600)

Important Security Notes:

Never commit the platform-lookup.key file to version control
Only share the master password through secure channels with trusted administrators
Each VOLTTRON platform instance needs a copy of the same master password
The password is loaded at runtime and is never stored in configuration files

Enabling Federation

To enable federation on a VOLTTRON platform, set the appropriate parameters when starting the platform:

volttron --enable-federation --address tcp://192.168.1.10:22916 --federation-url http://192.168.1.10:8000 --instance-name volttron-instance-1

The parameters are:

--enable-federation: Enables the federation service
--address: Specifies the external address that other platforms will use to connect
--federation-url: URL of the platform lookup service
--instance-name: each volttron instance in a federation should have a unique name

After starting with the above parameters, the parameters gets persisted in the $VOLTTRON_HOME/config file, so the next time you run `volttron` these parameters are read automatically from the config file.

If you have abn existing platform configuration file ($VOLTTRON_HOME/config), instead of starting volttron with the federation parameters as command line arguments you can directly append the following parameters at the end of an existing config file and restart volttron

In an existing volttron config file add the below entries at the end .. code-block:: config

enable_federation = true federation_url = http://registry-service:8000 address = tcp://192.168.1.10:22916 instance-name = volttron-instance-1

Example config file with minimal federation entries

[volttron]
instance-name = fed-test-2
address = tcp://172.17.0.1:22917
agent-isolation-mode = True
messagebus = zmq
auth-enabled = True
server-messagebus-id = vip.server
agent-monitor-frequency = 600
enable-federation = True
federation-url = http://172.17.0.1:8000
enable-federation-cache = True

Once enabled, the federation service will automatically load the password from $VOLTTRON_HOME/platform-lookup.key and register with the federation registry service.

Federation caching

When federation caching is enabled, outbound messages to remote federated platforms are cached while the connection is unavailable. Messages are saved in a SQLite database on the publishing VOLTTRON instance. This ensures no message loss during transient network outages or peer platform restarts. Once the connection is restored and the remote platform becomes reachable again, cached messages are automatically replayed to the peer in order. Message order is preserved on replay from cache, but messages could be interleaved with live messages arriving in real time.

Default Configuration

Federation message caching is enabled by default. The default configuration allows unlimited cache size with messages retained for up to 24 hours. This means:

enable-federation-cache: True (enabled by default)
federation-cache-limit-gb: None (no size limit by default)
federation-cache-limit-hours: 24 (messages retained for 24 hours)

Modifying Cache Configuration

You can modify the caching behavior in two ways:

Option 1: Command-line arguments (changes are persisted to the config file)

Pass the following options when starting the platform:

volttron-ctl start --enable-federation-cache=False --federation-cache-limit-gb=10 --federation-cache-limit-hours=48

Option 2: Direct config file editing

Edit $VOLTTRON_HOME/config and add or modify the following parameters in the [volttron] section:

[volttron]
enable-federation-cache = True
federation-cache-limit-gb = None
federation-cache-limit-hours = 24

After editing the config file, restart the platform for changes to take effect.

Configuration parameters:

enable-federation-cache (bool): Enable or disable message caching for federation. Default: True
federation-cache-limit-gb (float): Maximum cache size in gigabytes. Set to None for unlimited. Default: None
federation-cache-limit-hours (float): Maximum age of cached messages in hours. Default: 24

REST API for Federation Registry

Registering a Platform

When federation is enabled, the federation service of volttron platform automatically registers with the specified registry service using the master password for authentication.

Request:

POST /platform HTTP/1.1
Host: registry-service:8000
Content-Type: application/json
Registration-Password: your-master-password-here

{
  "id": "platform1",
  "address": "tcp://192.168.1.10:22916",
  "public_credentials": "7Dkj4a...vB3Mm==",
  "group": "default"
}

Headers:

Registration-Password (required): Master password for write operations. Must match the password configured on the registry service.

Body Fields:

id: Unique identifier for this platform
address: External VIP address for other platforms to connect to
public_credentials: Platform’s public key for secure communication
group (optional): Group name for categorizing platforms

Response:

HTTP/1.1 201 Created
Content-Type: application/json

{
  "id": "platform1",
  "address": "tcp://192.168.1.10:22916",
  "public_credentials": "7Dkj4a...vB3Mm==",
  "group": "default",
  "created_at": "2023-06-24T15:30:22.123456Z"
}

Discovering Platforms

After registration, the federation service periodically queries the registry for other available platforms. Authentication is optional for read operations.

Request:

GET /platforms HTTP/1.1
Host: registry-service:8000
Accept: application/json
Registration-Password: your-master-password-here

Headers:

Registration-Password (optional): Master password for authentication. Provides full access to all platforms.
Platform-Credentials (optional): A registered platform’s public credentials. Allows reading platforms in the same group.

Response:

HTTP/1.1 200 OK
Content-Type: application/json

[
  {
    "id": "platform1",
    "address": "tcp://192.168.1.10:22916",
    "public_credentials": "7Dkj4a...vB3Mm==",
    "group": "default"
  },
  {
    "id": "platform2",
    "address": "tcp://192.168.1.11:22916",
    "public_credentials": "9Zkr8b...xF5Nn==",
    "group": "default"
  }
]

Federation Connection Process

Registration: Each platform registers with the federation registry service on startup
Discovery: Platforms periodically query the registry to discover other platforms
Authentication Setup: For each discovered platform, credentials are registered with the auth service
Connection Establishment: A secure connection is established between platforms
Subscription Synchronization: Topic subscriptions are synchronized between platforms
Message Routing: Messages are routed between platforms based on subscriptions

Managing Connections

The federation service manages connections to other platforms automatically. When a new platform is discovered, a connection is attempted. If a connection fails, the service will retry periodically.

Federation Configuration

The federation configuration is stored in the VOLTTRON_HOME directory as federation_config.json:

{
  "platforms": [
    {
      "id": "platform2",
      "address": "tcp://192.168.1.11:22916",
      "public_credentials": "9Zkr8b...xF5Nn==",
      "group": "default",
      "connected": true,
      "last_heartbeat": 1687624821.456
    },
    {
      "id": "platform3",
      "address": "tcp://192.168.1.12:22916",
      "public_credentials": "5Qpn3c...tR7Kk==",
      "group": "default",
      "connected": false,
      "last_heartbeat": 1687624450.123
    }
  ]
}

Federation Usage Examples

Publishing to a Remote Platform

Agents can publish messages that will be delivered to subscribers on remote platforms:

def publish_to_all_platforms(self):
    self.vip.pubsub.publish(
        peer='pubsub',
        topic='global/heartbeat',
        message={'status': 'active', 'timestamp': time.time()}
    )

Subscribing to Remote Topics

Agents can subscribe to topics from other platforms:

def setup(self):
    # Subscribe to topic from any platform
    self.vip.pubsub.subscribe(
        peer='pubsub',
        prefix='global/heartbeat',
        callback=self.on_heartbeat,
        all_platforms=True  # Subscribe across all federated platforms
    )

def on_heartbeat(self, peer, sender, bus, topic, headers, message):
    platform_id = headers.get('platform')
    self.vip.pubsub.publish(
        peer='pubsub',
        topic='local/response',
        message={'received_from': platform_id}
    )

Troubleshooting

Common Issues

Connection Failures: - Verify network connectivity between platforms - Check that addresses are externally accessible - Ensure public keys are correctly registered
Message Delivery Issues: - Verify topic subscriptions are synchronized - Check authorization for cross-platform communication - Examine logs for routing errors

Security Considerations

Federation uses multiple layers of security:

Registry Service Authentication: HTTP header-based password authentication for all platform registrations and updates
Platform Communication: ZeroMQ CURVE authentication with public/private keys for inter-platform communication
Encryption: All communications between platforms are encrypted

Password Management:

The master password is stored in $VOLTTRON_HOME/platform-lookup.key with restricted Unix permissions (0o600)
The password file is read-only and never persisted in configuration files
Only administrators with access to the password file can register or modify platforms in the registry
The password must be manually shared with trusted VOLTTRON instance administrators through secure channels

Best Practices:

Use TLS for the federation registry service in production (HTTPS instead of HTTP)
Restrict network access to the federation registry service
Use strong, randomly-generated master passwords
Regularly rotate the master password and update all platform instances
Restrict federation to trusted platforms
Use strict authorization rules for cross-platform messaging
Monitor registry service logs for unauthorized access attempts
Keep the platform-lookup.key file secure and backed up appropriately