How to set up High Availability configuration

Guidelines for setting up HA configuration for extra performance and availability.

In High Availability mode, multiple Agents execute the same Processes with assistance from a load balancer, in order to distribute load from one Agent to multiple Agents, while also making possible to have Agent machines down without immediate to service availability. High Availability is the method to scale your Frends platform horizontally.

You can learn more about High Availability configuration at High Availability page in our Docs.

Requirements

In addition to the requirements for setting up each Agent machine (listed in Installing Frends Agent), a separate SQL Server 2016 (or later) installation, or Azure SQL Database instance is highly recommended for a functioning High Availability set up. You can also use a SQL cluster for this to avoid the database becoming a central point of failure.

While High Availability configuration is possible to set up without a shared database between the Agents, it will limit the functionality of the High Availability mode.

In order to handle REST API and HTTP requests in High Availability mode, a load balancer setup or an API Gateway Agent is needed.

Finally, make sure your Frends license supports setting up High Availability configuration with enough Agents allowed to be installed for each Agent Group.

Connectivity requirements

When using a central database as Shared State Store, each server hosting the Agents must have access to it, requiring outbound connectivity from the Agent machines to the database.

SQL Server and Azure SQL Database instance require at least port 1433 to be open for outbound connectivity from the client. You can learn more about SQL Server connectivity requirements and Azure SQL Database connectivity requirements from the corresponding links.

Chosen load balancer or API Gateway Agent needs to be facing the public internet in case public access to the created REST APIs and HTTP Triggers is required, allowing external connections to come in. If the connections are to be internal only, you can adjust the connectivity more freely within your internal, on premise network based on your requirements.

Load balancing systems, including API Gateway Agent, often act as proxy between outside network and the executing servers, meaning they will use same or similar connection details from the load balancer to the executing server, as what was used to call the load balancing proxy. Therefore the connections between load balancer and agent servers must also be taken into account, by allowing inbound connections from the load balancer to reach your Agents.

High Availability variations

High Availability configuration can work in two different modes, depending on the available database.

Multiple Agents in an Agent Group with a shared SQL database

If a central SQL Server or Azure SQL Database instance is available and an Agent Group has a connection string set for it, the Agents will use a shared SQL database called Shared State Store and all Triggers are available in High Availability mode, meaning all connected Agents can execute any Trigger.

This is the most common mode of installing multiple Agents in one Agent Group and should be used in most cases.

Multiple Agents without a shared SQL database

If there's no central database available or the connection string is not set, all the Agents will use a local database for storing configuration information by themselves.

The first deployed Agent in Agent Group will be the "Primary Agent". This Agent will be able to execute all Triggers. Other Agents will be able to run HTTP, API and Service Bus Triggers in the Agent Group. If this mode is selected, Schedule and File Triggers cannot be run in High Availability mode, meaning only the Primary Agent will run those Triggers.

This mode of installation is useful if you do not have a SQL Server but still want to have redundancy and load balancing for HTTP and API Processes.

What is the shared database used for?

Frends Agents will have always have SQLite database installed with them. It is used to store, for example, configuration data. In addition to that, Frends Agents may use an external, shared database.

In general the central database, called Shared State Store or Agent Store database, is quite lightweight, but Shared State Tasks, Schedule and File Triggers use the database for synchronization. Excessive use of those shapes will increase the load to the database. This synchronization or state sharing is what enables the High Availability capability in Frends.

In general, the SQLite and Shared State Store databases are used as follows:

Cross platform Agent without a configured database:
- All data is stored locally in the SQLite database
Cross platform Agent with a central database:
- Data needed for Triggers and Shared State Tasks is stored in the Shared State Store database
- Rest of the information, such as Processes and Environment Variables, is stored in the local SQLite database for each Agent

Database setup for High Availability

If using a shared database for the High Availability configuration, install or create a SQL Server 2016 or later, Azure SQL Database instance or SQL cluster, and make sure the planned Agent machines can reach this database.

A database user is also necessary to be set up before setting up the database for Frends High Availability usage. The user should have at least db_datareader, db_datawriter and execute permissions.

For running the Database Initializer tool to set up the database for Frends High Availability use you will need a user in the database with database admin role, or a connection string providing the admin capabilities. The tool will perform the following actions:

Create a database
Create a login
Create stored procedures
Alter tables
Create backup jobs
Run backup jobs

In order to create the necessary database tables and content, download the Database Initializer from your Frends Control Panel, available in the Administration > Environments view by selecting the target Agent Group. Database Initializer can be downloaded either directly from the download options for each Agent, or by enabling the Allow new agents to connect to Agent Group option under Advanced settings, and then downloading the Kubernetes configuration package.

After downloading and extracting the archive, run Frends.DatabaseInitializer.exe from the command line with the following parameters:

Frends.DatabaseInitializer.exe -c [SqlServerConnectionString] -n Frends.Agent.Data.Initialization.Migrations.SqlServerAgentStore -u [FRENDSAgentDatabaseUsername] -p [FRENDSAgentDatabasePassword] --doNotCreateBackupJobs

Replace the content in square brackets to fit your environment and setup:

SqlServerConnectionString: Connection string to your database, that has at least alter permissions to the database
FRENDSAgentDatabaseUsername: Username for the user account to use by Frends
FRENDSAgentDatabasePassword: Password for the user account

You can learn more about how to use and alter the tool's actions by running the following command:

> Frends.DatabaseInitializer.exe --help

Frends.DatabaseInitializer 2.0.0.0
Copyright c 2022. All rights reserved.
USAGE:
Deploy local agent database for a user:
Frends.DatabaseInitializer --connectionstring "Data Source=(local);Initial Catalog=FRENDS;Integrated Security=True"
--migrationNamespace Frends.DatabaseInitializer.AgentStore --sqlUserName HFWS460\FrendsUser
Upgrade an existing Configuration store database to Azure:
Frends.DatabaseInitializer --assemblyFile .\Frends.DatabaseMigrations.dll --connectionstring
"Server=tcp:[serverName].database.windows.net;Database=myDataBase;User
ID=[LoginForDb]@[serverName];Password=myPassword;Trusted_Connection=False;Encrypt=True;" --doNotCreate
--migrationNamespace Frends.DatabaseMigrations.ConfigurationStore

  -c, --connectionstring                 Required. Connection string to the FRENDS database, e.g. 'Data
                                         Source=(local);Initial Catalog=FRENDS;Integrated Security=True'

  -l, --logConnectionstring              When migrating configuration store to 5.4, some data is migrated from the
                                         logStore to the configuration store. In that case this parameter is needed.

  -u, --sqlUserName                      Username of the SQL user, e.g. 'DOMAIN\FrendsUser'

  -p, --sqlUserPassword                  Password of the SQL user when using SQL logins

  -m, --giveUserTableManagementRights    (Default: false) Should the SQL user be allowed to manage the tables, used
                                         e.g. for reorganizing indexes

  -a, --assemblyFile                     The assembly with the fluent migrations to run. If not given, the internal
                                         agent database migrations will be run.

  -n, --migrationNamespace               (Default: Frends.DatabaseInitializer.AgentStore) The namespace from which to
                                         run the migrations. If not given, the Agent store migrations will be run.

  --scriptMigrationNamespace             The namespace from which to run embedded Entity framework SQL migration
                                         scripts. If not given, no SQL script migrations will be run.

  --doNotCreate                          (Default: false) Switch for disabling the creation of the database and user

  --doNotMigrate                         (Default: false) Switch for disabling the database migration

  --doNotEnableSnapshotIsolation         (Default: false) Switch for disabling the default enabling of snapshot
                                         isolation (it is e.g. not supported for Azure SQL)

  -t, --timeout                          (Default: 240) The command timeout (in seconds) used by the SQL commands,
                                         default value 240

  --collation                            (Default: Latin1_General_CI_AS) The collation of the database to create,
                                         default value Latin1_General_CI_AS

  --showSql                              (Default: false) Outputs the executed SQL

  --doNotCreateBackupJobs                (Default: false) If flag is included backup jobs for the sql server will not
                                         be created

  --doNotSetSimpleBackupRecoveryModel    (Default: false) If flag is included recovery mode will not be changed to
                                         simple

  --backupRetentionMonths                (Default: 2) Define retention months for backup data, default is 2 months

  --help                                 Display this help screen.

  --version                              Display version information.

Agent Group & Agent configuration for High Availability

Once the database for High Availability is set up, you need to provide Frends the connection string for the database. This can be done either in Frends Control Panel by adding the connection string to the target Agent Group, or through application settings configuration file by setting the field AgentDatabaseConnectionString.

In Frends Control Panel, navigate to Administration > Environments page, and select the Agent Group you are creating the High Availability configuration to, or create a new Agent Group for this purpose to avoid breaking existing configuration.

For the chosen Agent Group, set the connection string to value such as:

Server=tcp:[SqlServerHostName],1433;Database=[FRENDSAgentDatabase];User ID=[FRENDSAgentDatabaseUsername];Password=[FRENDSAgentDatabasePassword];Trusted_Connection=False;Encrypt=True;Connection Timeout=30;

And replacing the values in square brackets by the actual values for your environment.

Deploying the Agents

Once the connection string has been set up, you can create, download and deploy the necessary amount of Agents as per your requirements by following the Installing Frends Agent guide for each Agent.

If you are using Kubernetes to deploy and manage multiple Agents, you can use for example the following settings in Frends-Agent-Deploy.yaml configuration file for the Kubernetes deployment in order to spawn more than one Agent.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: frends-agent-linux
spec:
  replicas: 2

Deploying the Agents without shared database

If you chose to not install and initialize a shared database for the HA configuration, you do not need the connection string to be set up, and can simply deploy the Agents as usual. The first deployed Agent will become the Primary Agent for each of your Agent Groups, and it will be used to run all other Triggers than HTTP, API and Service Bus Triggers.

Set up load balancing

In order for High Availability to work for HTTP and API requests, a load balancer is required. Load balancer is used to distribute the REST API and HTTP requests to the Agents equally, based on incoming and handled load. Any load balancing proxy setup can be used in this, that allows the incoming connections to be distributed and forwarded to each Agent server.

Instead of third part load balancing system, if you do not have one yet, Frends provides the possibility to create and use an API Gateway Agent. While identical to standard Frends Agent, its configuration options specify that it will act as a proxy for HTTP and REST API requests instead. When added to an Agent Group, the Gateway Agent distributes the incoming requests automatically in a round-robin method for the other Agents in the same Agent Group.

PreviousInstalling Frends Agent

Last updated 3 days ago

Was this helpful?