How to replace a Quantum Maestro Orchestrator

Support Center > Search Results > SecureKnowledge Details

Technical Level

Solution ID	sk174202
Technical Level
Product	Quantum Maestro
Version	R80.20SP, R81.10
OS	Gaia
Platform / Model	MHO Maestro
Date Created	28-Jun-2021
Last Modified	30-Jun-2022

Solution

Follow the steps below to replace (RMA) a failed Quantum Maestro Orchestrator.

Important Notes

We recommend to schedule a maintenance window.
In a Dual Site deployment, perform the RMA procedure on the Standby site.

The procedure below uses these Orchestrator IDs:

ID of the Orchestrator that continues to work - 1_1
ID of the Orchestrator that failed - 1_2

Procedure for Orchestrators that run the R80.20SP version - when the Orchestrator that failed is still accessible

Show / Hide this section

Export the configuration files of the Orchestrator that failed:
- If the Orchestrators run R80.20SP with the Jumbo Hotfix Accumulator Take 310 and higher:
  
  Note - Starting from R80.20SP Jumbo Hotfix Accumulator Take 310, each Orchestrator keeps the configuration of the peer Orchestrator as a local backup.
  1. Connect to the command line on the Orchestrator that continues to work.
  2. Log in to Gaia Clish.
  3. Export the configuration files of the Orchestrator that failed:
    
    set maestro export remote orchestrator id <ID of the Orchestrator That Failed> configuration archive-name <Name of Output Archive File> path <Full Path on Orchestrator That Continues to Work>
    
    Example:
    
    set maestro export remote orchestrator id 1_2 configuration archive-name Export_from_failed_Orch_1_2 path /var/log/
- If the Orchestrators run R80.20SP with the Jumbo Hotfix Accumulator Take 309 and lower:
  1. Connect to the command line on the Orchestrator that failed.
  2. Log in to the Expert mode.
  3. Export the configuration files of the Orchestrator that failed:
    
    tar -czf <Full Path on Orchestrator That Failed>/<Name of Output Archive File>.tgz -C /etc maestro.json sgdb.json smodb.json maestro_full.json
    
    Example:
    
    tar -czf /var/log/Export_from_failed_Orch_1_2.tgz -C /etc maestro.json sgdb.json smodb.json maestro_full.json
On the Orchestrator that failed, copy the archive with the exported configuration files to your computer.

Use an SCP client (WinSCP requires the default user shell to be /bin/bash).
On the Orchestrator that failed, stop the Orchestrator service:
1. Connect to the command line on the Orchestrator that failed.
2. Log in to the Expert mode.
3. Stop the service:
  
  orchd stop
On the Orchestrator that failed, write down the port numbers, to which the Uplink cable, the Downlink cables, and the Sync cables are connected.
On the Orchestrator that failed, disconnect all cables.

Important - Do not halt (shut down) the Orchestrator that failed. It might reboot, and the Orchestrator services will start again.
Remove the Orchestrator that failed from the rack.
Install the replacement Orchestrator in the rack.

Do not connect the power or network cables yet. You do it gradually later.
On the replacement Orchestrator, connect only these cables:
1. The network cable from your computer to the Orchestrator's MGMT interface.
2. The console cable from your computer to the Orchestrator's Console Port.
  
  In your console client, configure these settings:
  - Baud Rate - 9600
  - Data bits - 8
  - Stop bits - 1
  - Parity - None
  - Flow Control - None
3. The power cables to the Orchestrator's Power Supply Units.
See the Quantum Maestro Getting Started Guide.
On the replacement Orchestrator, in the console session, log in to Gaia Clish.
On the replacement Orchestrator, configure the Orchestrator's MGMT interface:
1. Configure the IPv4 address and Mask Length:
  
  set interface Mgmt1 ipv4-address <IPv4 Address of MGMT Interface> mask-length <Mask>
2. Enable the interface:
  
  set interface Mgmt1 state on
3. Configure the default gateway:
  
  set static-route default nexthop gateway address <IPv4 Address of Default Gateway> on
4. Save the changes:
  
  save config
See the R80.20SP Maestro Gaia Administration Guide.
On the replacement Orchestrator, connect to the command line through the MGMT interface (over SSH).
On the replacement Orchestrator, log in to Gaia Clish.
On the replacement Orchestrator, configure these Gaia settings:
1. Configure the same date and time settings as configured on all other working Orchestrators (in our example, "1_1").
  
  Note - The date and time settings must be the same on all Orchestrators.
2. Configure the hostname.
3. Change the default password for the 'admin' user.
4. Save the changes:
  
  save config
See the R80.20SP Maestro Gaia Administration Guide.
On the replacement Orchestrator, install the same Take of the R80.20SP Jumbo Hotfix Accumulator as installed on the working Orchestrator (in our example, "1_1").

Wait for the Orchestrator to reboot.
On the replacement Orchestrator, stop the services:
1. Connect to the command line the replacement Orchestrator through the MGMT interface (over SSH).
2. Log in to the Expert mode.
3. Stop the Orchestrator service:
  
  orchd stop
4. Stop the LLDP service:
  
  tellpm process:lldpd
On the replacement Orchestrator - copy the archive with the exported configuration files from your computer to the replacement Orchestrator to some directory (for example, /var/log/).

Use an SCP client (WinSCP requires the default user shell to be /bin/bash).
On the replacement Orchestrator, import the configuration files you collected earlier:
- If the Orchestrator runs R80.20SP with the Jumbo Hotfix Accumulator Take 310 and higher:
  1. Connect to the command line on the replacement Orchestrator.
  2. Log in to Gaia Clish.
  3. Import the configuration files:
    
    set maestro import configuration archive-name <Name of Output Archive File> path <Full Local Path>
    
    Example:
    
    set maestro import configuration archive-name Export_from_failed_Orch_1_2.tgz path /var/log/
- If the Orchestrator runs R80.20SP with the Jumbo Hotfix Accumulator Take 309 and lower:
  1. Connect to the command line on the replacement Orchestrator.
  2. Log in to the Expert mode.
  3. Unpack the configuration files to the /etc/ directory:
    
    tar -xzf <Full Local Path>/<Name of Output Archive File>.tgz -C /etc
    
    Example:
    
    tar -xzf /var/log/Export_from_failed_Orch_1_2.tgz -C /etc
On all Orchestrators, make sure the configuration files are the same.
1. Connect to the command line on each Orchestrator.
2. Log in to the Expert mode.
3. Run:
  
  jsont -f /etc/sgdb.json -P | egrep -v "topology_timestamp|session_id|active_sgms|all_sgm" | md5sum
  
  The MD5 checksums must be the same on the two Orchestrators.
  
  If the MD5 checksums are different:
  1. Copy the /etc/sgdb.json file from the Orchestrator that continues to work ("1_1") to the replacement Orchestrator ("1_2") to the /etc/ directory.
  2. Check the MD5 checksums again.
On the replacement Orchestrator, connect these cables to ports with the same numbers as they were connected on the Orchestrator that failed:
- The Downlink cables
- The Sync cable(s)
Important - Make sure you connected the cables to the correct ports.
On the replacement Orchestrator, start the Orchestrator service:
1. Connect to the command line on each Orchestrator.
2. Log in to the Expert mode.
3. Disable the Link State Propagation (LSP):
  
  jsont -f /etc/smodb.json -s /orch_lsp_state -v off
4. Start the Orchestrator service:
  
  orchd start
5. Enable the Link State Propagation (LSP):
  
  jsont -f /etc/smodb.json -s /orch_lsp_state -v on
6. Restart the Orchestrator service (this also starts the LLDP service):
  
  orchd restart
On all Orchestrators, make sure the traffic can pass traffic between Orchestrators:
- In a Single Site deployment:
  
  On the Orchestrator that continues to work (in our example, "1_1"), send pings to the replacement Orchestrator (in our example, "1_2"):
  
  ping 1_2
- In a Dual Site deployment:
  
  On the replacement Orchestrator (in our example, "1_2"), send pings to each working Orchestrator:
  
  ping 1_1
  
  ping 2_1
  
  ping 2_2
Make sure the Security Group Members can pass traffic to each other:
1. Connect to the command line on the Security Group.
2. Log in to the Expert mode.
3. Identify the Security Group Member that runs as SMO:
  
  asg stat -i tasks
4. Examine the cluster state of the Security Group Members.
  
  On the SMO Security Group Member, run:
  
  cphaprob state
  
  The output must show that all Security Group Members are active.
5. Send pings between Security Group Members:
  1. Connect to one of the Security Group Members
    (in our example, we connect to the first one - "1_1"):
    
    member 1_1
  2. On this Security Group Member, send pings to any other Security Group Member
    (in our example, we send pings to the second one - "1_2" / "2_2"):
    - In a Single Site deployment:
      
      ping 1_2
    - In a Dual Site deployment:
      
      ping 1_2
      
      ping 2_2
On the replacement Orchestrator, connect the Uplink cables to ports with the same numbers as they were connected on the Orchestrator that failed.
On each Security Group, make sure all links are up on the Security Group:
1. Connect to the command line on the Security Group.
2. Examine the state of links:
  
  asg_if

Procedure for Orchestrators that run the R80.20SP version with the Jumbo Hotfix Accumulator Take 309 and lower - when the Orchestrator that failed is not accessible anymore, or after restoring to factory default

Show / Hide this section

Important - Follow the procedure below if the Orchestrator that failed is not accessible, or if its configuration is empty (for example, after restoring to factory default).

On the Orchestrator that continues to work, export the required configuration files:
1. Connect to the command line.
2. Log in to the Expert mode.
3. Copy these files from the Orchestrator that continues to work to your computer:
  - /etc/maestro_remote.json
  - /etc/maestro_full.json
  - /etc/sgdb.json
  - /etc/smodb.json
  Use an SCP client (WinSCP requires the default user shell to be /bin/bash).
On the Orchestrator that failed, write down the port numbers, to which the Uplink cable, the Downlink cables, and the Sync cables are connected.
Remove the Orchestrator that failed from the rack.
Install the replacement Orchestrator in the rack.

Do not connect the power or network cables yet. You do it gradually later.
On the replacement Orchestrator, connect only these cables:
1. The network cable from your computer to the Orchestrator's MGMT interface.
2. The console cable from your computer to the Orchestrator's Console Port.
  
  In your console client, configure these settings:
  - Baud Rate - 9600
  - Data bits - 8
  - Stop bits - 1
  - Parity - None
  - Flow Control - None
3. The power cables to the Orchestrator's Power Supply Units.
See the Quantum Maestro Getting Started Guide.
On the replacement Orchestrator, in the console session, log in to Gaia Clish.
On the replacement Orchestrator, configure the Orchestrator's MGMT interface:
1. Configure the IPv4 address and Mask Length:
  
  set interface Mgmt1 ipv4-address <IPv4 Address of MGMT Interface> mask-length <Mask>
2. Enable the interface:
  
  set interface Mgmt1 state on
3. Configure the default gateway:
  
  set static-route default nexthop gateway address <IPv4 Address of Default Gateway> on
4. Save the changes:
  
  save config
See the R80.20SP Maestro Gaia Administration Guide.
On the replacement Orchestrator, connect to the command line through the MGMT interface (over SSH).
On the replacement Orchestrator, log in to Gaia Clish.
On the replacement Orchestrator, configure these Gaia settings:
1. Configure the same date and time settings as configured on all other working Orchestrators (in our example, "1_1").
  
  Note - The date and time settings must be the same on all Orchestrators.
2. Configure the hostname.
3. Change the default password for the 'admin' user.
4. Save the changes:
  
  save config
See the R80.20SP Maestro Gaia Administration Guide.
On the replacement Orchestrator, install the same Take of the R80.20SP Jumbo Hotfix Accumulator as installed on the working Orchestrator (in our example, "1_1").

Wait for the Orchestrator to reboot.
On the replacement Orchestrator, stop the services:
1. Connect to the command line through the MGMT interface (over SSH).
2. Log in to the Expert mode.
3. Stop the Orchestrator service:
  
  orchd stop
4. Stop the LLDP service:
  
  tellpm process:lldpd
On the replacement Orchestrator - copy these files from your computer to the replacement Orchestrator to the /etc/ directory:
- maestro_remote.json
- maestro_full.json
- sgdb.json
- smodb.json
Use an SCP client (WinSCP requires the default user shell to be /bin/bash).
On the replacement Orchestrator, edit the configuration files (in the Expert mode):
1. Rename the file:
  
  from /etc/maestro_remote.json
  
  to /etc/maestro.json
2. Edit the file /etc/smodb.json and change the value of the member_id parameter to the ID of the Orchestrator that failed (in our example, "2").
On all Orchestrators, make sure the configuration files are the same.
1. Connect to the command line on each Orchestrator.
2. Log in to the Expert mode.
3. Run:
  
  jsont -f /etc/sgdb.json -P | egrep -v "topology_timestamp|session_id|active_sgms|all_sgm" | md5sum
  
  The MD5 checksums must be the same on the two Orchestrators.
  
  If the MD5 checksums are different:
  1. Copy the /etc/sgdb.json file from the Orchestrator that continues to work ("1_1") to the replacement Orchestrator ("1_2") to the /etc/ directory.
  2. Check the MD5 checksums again.
On the replacement Orchestrator, connect these cables to ports with the same numbers as they were connected on the Orchestrator that failed:
- The Downlink cables
- The Sync cable(s)
Important - Make sure you connected the cables to the correct ports.
On the replacement Orchestrator, start the Orchestrator service:
1. Connect to the command line.
2. Log in to the Expert mode.
3. Disable the Link State Propagation (LSP):
  
  jsont -f /etc/smodb.json -s /orch_lsp_state -v off
4. Start the Orchestrator service:
  
  orchd start
5. Enable the Link State Propagation (LSP):
  
  jsont -f /etc/smodb.json -s /orch_lsp_state -v on
6. Restart the Orchestrator service (this also starts the LLDP service):
  
  orchd restart
On all Orchestrators, make sure the traffic can pass between Orchestrators:
- In a Single Site deployment:
  
  On the working Orchestrator (in our example, "1_1"), send pings to the replacement Orchestrator (in our example, "1_2"):
  
  ping 1_2
- In a Dual Site deployment:
  
  On the replacement Orchestrator (in our example, "1_2"), send pings to each working Orchestrator:
  
  ping 1_1
  
  ping 2_1
  
  ping 2_2
Make sure the Security Group Members can pass traffic to each other:
1. Connect to the command line on the Security Group.
2. Log in to the Expert mode.
3. Identify the Security Group Member that runs as SMO:
  
  asg stat -i tasks
4. Examine the cluster state of the Security Group Members.
  
  On the SMO Security Group Member, run:
  
  cphaprob state
  
  The output must show that all Security Group Members are active.
5. Send pings between Security Group Members:
  1. Connect to one of the Security Group Members
    (in our example, we connect to the first one - "1_1"):
    
    member 1_1
  2. On this Security Group Member, send pings to any other Security Group Member
    (in our example, we send pings to the second one - "1_2" / "2_2"):
    - In a Single Site deployment:
      
      ping 1_2
    - In a Dual Site deployment:
      
      ping 1_2
      
      ping 2_2
On the replacement Orchestrator, connect the Uplink cables to ports with the same numbers as they were connected on the Orchestrator that failed.
On each Security Group, make sure all links are up on the Security Group:
1. Connect to the command line on the Security Group.
2. Examine the state of links:
  
  asg_if

Procedure for Orchestrators that run the version R81.10 or higher

Show / Hide this section

On the Orchestrator that continues to work, export the configuration files for the Orchestrator that failed:
1. Connect to the command line on the Orchestrator that continues to work.
2. Log in to Gaia Clish.
3. Export the configuration files for the Orchestrator that failed:
  
  set maestro export remote orchestrator id <ID of the Orchestrator That Failed> configuration archive-name <Name of Output Archive File> path <Full Path on Orchestrator That Continues to Work>
  
  Example:
  
  set maestro export remote orchestrator id 1_2 configuration archive-name Export_from_failed_Orch_1_2 path /var/log/
On the Orchestrator that continues to work, copy the archive with the exported configuration files to your computer.

Use an SCP client (WinSCP requires the default user shell to be /bin/bash).
On the Orchestrator that failed, stop the Orchestrator service:
1. Connect to the command line on the Orchestrator that failed.
2. Log in to the Expert mode.
3. Stop the service:
  
  orchd stop
On the Orchestrator that failed, write down the port numbers, to which the Uplink cable, the Downlink cables, and the Sync cables are connected.
On the Orchestrator that failed, disconnect all cables.

Important - Do not halt (shut down) the Orchestrator that failed. It might reboot, and the Orchestrator services will start again.
Remove the Orchestrator that failed from the rack.
Install the replacement Orchestrator in the rack.

Do not connect the power or network cables yet. You do it gradually later.
On the replacement Orchestrator, connect only these cables:
1. The network cable from your computer to the Orchestrator's MGMT interface.
2. The console cable from your computer to the Orchestrator's Console Port.
  
  In your console client, configure these settings:
  - Baud Rate - 9600
  - Data bits - 8
  - Stop bits - 1
  - Parity - None
  - Flow Control - None
3. The power cables to the Orchestrator's Power Supply Units.
See the Quantum Maestro Getting Started Guide.
On the replacement Orchestrator, in the console session, log in to Gaia Clish.

Important Note - Choose not to activate the Orchestrator. If you activated the Orchestrator, then in the Expert mode, run the "orchd stop" command.
On the replacement Orchestrator, configure the Orchestrator's MGMT interface:
1. Configure the IPv4 address and Mask Length:
  
  set interface Mgmt1 ipv4-address <IPv4 Address of MGMT Interface> mask-length <Mask>
2. Enable the interface:
  
  set interface Mgmt1 state on
3. Configure the default gateway:
  
  set static-route default nexthop gateway address <IPv4 Address of Default Gateway> on
4. Save the changes:
  
  save config
See the R81.10 Gaia Administration Guide.
On the replacement Orchestrator, connect to the command line through the MGMT interface (over SSH).
On the replacement Orchestrator, log in to Gaia Clish.
On the replacement Orchestrator, configure these Gaia settings:
1. Configure the same date and time settings as configured on all other working Orchestrators (in our example, "1_1").
  
  Note - The date and time settings must be the same on all Orchestrators.
2. Configure the hostname.
3. Change the default password for the 'admin' user.
4. Save the changes:
  
  save config
See the R81.10 Gaia Administration Guide.
On the replacement Orchestrator, install the same Take of the R81.10 Jumbo Hotfix Accumulator as installed on the working Orchestrator (in our example, "1_1").

Wait for the Orchestrator to reboot.

Important Note - Choose not to activate the Orchestrator. If you activated the Orchestrator, then in the Expert mode, run the "orchd stop" command.
On the replacement Orchestrator, copy the archive with the exported configuration files from your computer to the replacement Orchestrator to some directory (for example, /var/log/).

Use an SCP client (WinSCP requires the default user shell to be /bin/bash).
On the replacement Orchestrator, import the configuration files you collected earlier:
1. Connect to the command line on the replacement Orchestrator.
2. Log in to Gaia Clish.
3. Import the configuration files:
  
  set maestro import configuration archive-name <Name of Output Archive File> path <Full Local Path>
  
  Example:
  
  set maestro import configuration archive-name Export_from_failed_Orch_1_2.tgz path /var/log/
On all Orchestrators, make sure the configuration files are the same.
1. Connect to the command line on each Orchestrator.
2. Log in to the Expert mode.
3. Run:
  
  jsont -f /etc/sgdb.json -P | egrep -v "topology_timestamp|session_id|active_sgms|all_sgm" | md5sum
  
  The MD5 checksums must be the same on the two Orchestrators.
  
  If the MD5 checksums are different:
  1. Copy the /etc/sgdb.json file from the working Orchestrator (1_1) to the replacement Orchestrator (1_2) to the /etc/ directory.
  2. Check the MD5 checksums again.
On the replacement Orchestrator, connect these cables to ports with the same numbers as they were connected on the Orchestrator that failed:
- The Downlink cables
- The Sync cable(s)
Important - Make sure you connected the cables to the correct ports.
On the replacement Orchestrator, start the Orchestrator service:
1. Log in to the Expert mode.
2. Disable the Link State Propagation (LSP):
  
  jsont -f /etc/smodb.json -s /orch_lsp_state -v off
3. Check the Orchestrator services status:
  
  orchd status
4. If the Orchestrator is not active, then activate it (it will start the Orchestrator services):
  
  orchd activate
  
  (Otherwise, start the Orchestrator services: orchd start)
5. Enable the Link State Propagation (LSP):
  
  jsont -f /etc/smodb.json -s /orch_lsp_state -v on
6. Restart the Orchestrator port monitoring daemon with these commands:
  
  tellpm process:ssm_pmd
  
  tellpm process:ssm_pmd t
Note - It may take a few seconds for the Orchestrator to discover the connected Security Appliances for the first time. In the Expert mode, run the "watch -d -n 1 orch_stat" command and wait for all the LAGs to be in the "up" status.
On all Orchestrators, make sure the traffic can pass between Orchestrators:
- In a Single Site deployment:
  
  On the Orchestrator that continues to work (in our example, "1_1"), send pings to the replacement Orchestrator (in our example, "1_2"):
  
  ping 1_2
- In a Dual Site deployment:
  
  On the replacement Orchestrator (in our example, "1_2"), send pings to each working Orchestrator:
  
  ping 1_1
  
  ping 2_1
  
  ping 2_2
Make sure the Security Group Members can pass traffic to each other:
1. Connect to the command line on the Security Group.
2. Log in to the Expert mode.
3. Identify the Security Group Member that runs as SMO:
  
  asg stat -i tasks
4. Examine the cluster state of the Security Group Members.
  
  On the SMO Security Group Member, run:
  
  cphaprob state
  
  The output must show that all Security Group Members are active.
5. Send pings between Security Group Members:
  1. Connect to one of the Security Group Members
    (in our example, we connect to the first one - "1_1"):
    
    member 1_1
  2. On this Security Group Member, send pings to any other Security Group Member
    (in our example, we send pings to the second one - "1_2" / "2_2"):
    - In a Single Site deployment:
      
      ping 1_2
    - In a Dual Site deployment:
      
      ping 1_2
      
      ping 2_2
On the replacement Orchestrator, connect the Uplink cables to ports with the same numbers as they were connected on the Orchestrator that failed.
On each Security Group, make sure all links are up on the Security Group:
1. Connect to the command line on the Security Group.
2. Examine the state of links:
  
  asg_if

Revision History

Show / Hide this section

Date	Change
15 May 2022	Corrected the Baud rate values - removed "115200"
21 March 2022	Improved the procedures Added the procedure for an Orchestrator that failed and is not accessible anymore, or after restoring to factory default
14 March 2022	Improved the terms used in procedures
10 Aug 2021	Added the procedure for Orchestrators that run the R81.10 version
28 June 2021	Created this article for Orchestrators that run the R80.20SP version

Give us Feedback

Please rate this document

[1=Worst,5=Best]

Comment
	Submitting rating