1. Export the configuration files of the Orchestrator that failed:

   • If the Orchestrators run R80.20SP with the Jumbo Hotfix Accumulator Take 310 and higher:

     Note - Starting from R80.20SP Jumbo Hotfix Accumulator Take 310, each Orchestrator keeps the configuration of the peer Orchestrator as a local backup.

     1. Connect to the command line on the Orchestrator that continues to work.

     2. Log in to Gaia Clish.

     3. Export the configuration files of the Orchestrator that failed:

        set maestro export remote orchestrator id <ID of the Orchestrator That Failed> configuration archive-name <Name of Output Archive File> path <Full Path on Orchestrator That Continues to Work>

        Example:

        set maestro export remote orchestrator id 1_2 configuration archive-name Export_from_failed_Orch_1_2 path /var/log/

   • If the Orchestrators run R80.20SP with the Jumbo Hotfix Accumulator Take 309 and lower:

     1. Connect to the command line on the Orchestrator that failed.

     2. Log in to the Expert mode.

     3. Export the configuration files of the Orchestrator that failed:

        tar -czf <Full Path on Orchestrator That Failed>/<Name of Output Archive File>.tgz -C /etc maestro.json sgdb.json smodb.json maestro_full.json

        Example:

        tar -czf /var/log/Export_from_failed_Orch_1_2.tgz -C /etc maestro.json sgdb.json smodb.json maestro_full.json

2. On the Orchestrator that failed, copy the archive with the exported configuration files to your computer.

   Use an SCP client (WinSCP requires the default user shell to be /bin/bash).

3. On the Orchestrator that failed, stop the Orchestrator service:

   1. Connect to the command line on the Orchestrator that failed.

   2. Log in to the Expert mode.

   3. Stop the service:

      orchd stop

4. On the Orchestrator that failed, write down the port numbers, to which the Uplink cable, the Downlink cables, and the Sync cables are connected.

5. On the Orchestrator that failed, disconnect all cables.

   Important - Do not halt (shut down) the Orchestrator that failed. It might reboot, and the Orchestrator services will start again.

6. Remove the Orchestrator that failed from the rack.

7. Install the replacement Orchestrator in the rack.

   Do not connect the power or network cables yet. You do it gradually later.

8. On the replacement Orchestrator, connect only these cables:

   1. The network cable from your computer to the Orchestrator's MGMT interface.

   2. The console cable from your computer to the Orchestrator's Console Port.

      In your console client, configure these settings:

      • Baud Rate - 9600
      • Data bits - 8
      • Stop bits - 1
      • Parity - None
      • Flow Control - None

   3. The power cables to the Orchestrator's Power Supply Units.

   See the Quantum Maestro Getting Started Guide.

9. On the replacement Orchestrator, in the console session, log in to Gaia Clish.

10. On the replacement Orchestrator, configure the Orchestrator's MGMT interface:

    1. Configure the IPv4 address and Mask Length:

       set interface Mgmt1 ipv4-address <IPv4 Address of MGMT Interface> mask-length <Mask>

    2. Enable the interface:

       set interface Mgmt1 state on

    3. Configure the default gateway:

       set static-route default nexthop gateway address <IPv4 Address of Default Gateway> on

    4. Save the changes:

       save config

    See the R80.20SP Maestro Gaia Administration Guide.

11. On the replacement Orchestrator, connect to the command line through the MGMT interface (over SSH).

12. On the replacement Orchestrator, log in to Gaia Clish.

13. On the replacement Orchestrator, configure these Gaia settings:

    1. Configure the same date and time settings as configured on all other working Orchestrators (in our example, "1_1").

       Note - The date and time settings must be the same on all Orchestrators.

    2. Configure the hostname.

    3. Change the default password for the 'admin' user.

    4. Save the changes:

       save config

    See the R80.20SP Maestro Gaia Administration Guide.

14. On the replacement Orchestrator, install the same Take of the R80.20SP Jumbo Hotfix Accumulator as installed on the working Orchestrator (in our example, "1_1").

    Wait for the Orchestrator to reboot.

15. On the replacement Orchestrator, stop the services:

    1. Connect to the command line the replacement Orchestrator through the MGMT interface (over SSH).

    2. Log in to the Expert mode.

    3. Stop the Orchestrator service:

       orchd stop

    4. Stop the LLDP service:

       tellpm process:lldpd

16. On the replacement Orchestrator - copy the archive with the exported configuration files from your computer to the replacement Orchestrator to some directory (for example, /var/log/).

    Use an SCP client (WinSCP requires the default user shell to be /bin/bash).

17. On the replacement Orchestrator, import the configuration files you collected earlier:

    • If the Orchestrator runs R80.20SP with the Jumbo Hotfix Accumulator Take 310 and higher:

      1. Connect to the command line on the replacement Orchestrator.

      2. Log in to Gaia Clish.

      3. Import the configuration files:

         set maestro import configuration archive-name <Name of Output Archive File> path <Full Local Path>

         Example:

         set maestro import configuration archive-name Export_from_failed_Orch_1_2.tgz path /var/log/

    • If the Orchestrator runs R80.20SP with the Jumbo Hotfix Accumulator Take 309 and lower:

      1. Connect to the command line on the replacement Orchestrator.

      2. Log in to the Expert mode.

      3. Unpack the configuration files to the /etc/ directory:

         tar -xzf <Full Local Path>/<Name of Output Archive File>.tgz -C /etc

         Example:

         tar -xzf /var/log/Export_from_failed_Orch_1_2.tgz -C /etc

18. On all Orchestrators, make sure the configuration files are the same.

    1. Connect to the command line on each Orchestrator.

    2. Log in to the Expert mode.

    3. Run:

       jsont -f /etc/sgdb.json -P | egrep -v "topology_timestamp|session_id|active_sgms|all_sgm" | md5sum

       The MD5 checksums must be the same on the two Orchestrators.

       If the MD5 checksums are different:

       1. Copy the /etc/sgdb.json file from the Orchestrator that continues to work ("1_1") to the replacement Orchestrator ("1_2") to the /etc/ directory.
       2. Check the MD5 checksums again.

19. On the replacement Orchestrator, connect these cables to ports with the same numbers as they were connected on the Orchestrator that failed:

    • The Downlink cables
    • The Sync cable(s)

    Important - Make sure you connected the cables to the correct ports.

20. On the replacement Orchestrator, start the Orchestrator service:

    1. Connect to the command line on each Orchestrator.

    2. Log in to the Expert mode.

    3. Disable the Link State Propagation (LSP):

       jsont -f /etc/smodb.json -s /orch_lsp_state -v off

    4. Start the Orchestrator service:

       orchd start

    5. Enable the Link State Propagation (LSP):

       jsont -f /etc/smodb.json -s /orch_lsp_state -v on

    6. Restart the Orchestrator service (this also starts the LLDP service):

       orchd restart

21. On all Orchestrators, make sure the traffic can pass traffic between Orchestrators:

    • In a Single Site deployment:

      On the Orchestrator that continues to work (in our example, "1_1"), send pings to the replacement Orchestrator (in our example, "1_2"):

      ping 1_2

    • In a Dual Site deployment:

      On the replacement Orchestrator (in our example, "1_2"), send pings to each working Orchestrator:

      ping 1_1

      ping 2_1

      ping 2_2

22. Make sure the Security Group Members can pass traffic to each other:

    1. Connect to the command line on the Security Group.

    2. Log in to the Expert mode.

    3. Identify the Security Group Member that runs as SMO:

       asg stat -i tasks

    4. Examine the cluster state of the Security Group Members.

       On the SMO Security Group Member, run:

       cphaprob state

       The output must show that all Security Group Members are active.

    5. Send pings between Security Group Members:

       1. Connect to one of the Security Group Members
          (in our example, we connect to the first one - "1_1"):

          member 1_1

       2. On this Security Group Member, send pings to any other Security Group Member
          (in our example, we send pings to the second one - "1_2" / "2_2"):

          • In a Single Site deployment:

            ping 1_2

          • In a Dual Site deployment:

            ping 1_2

            ping 2_2

23. On the replacement Orchestrator, connect the Uplink cables to ports with the same numbers as they were connected on the Orchestrator that failed.

24. On each Security Group, make sure all links are up on the Security Group:

    1. Connect to the command line on the Security Group.

    2. Examine the state of links:

       asg_if

Important - Follow the procedure below if the Orchestrator that failed is not accessible, or if its configuration is empty (for example, after restoring to factory default).

1. On the Orchestrator that continues to work, export the required configuration files:

   1. Connect to the command line.

   2. Log in to the Expert mode.

   3. Copy these files from the Orchestrator that continues to work to your computer:

      • /etc/maestro_remote.json
      • /etc/maestro_full.json
      • /etc/sgdb.json
      • /etc/smodb.json

      Use an SCP client (WinSCP requires the default user shell to be /bin/bash).

2. On the Orchestrator that failed, write down the port numbers, to which the Uplink cable, the Downlink cables, and the Sync cables are connected.

3. Remove the Orchestrator that failed from the rack.

4. Install the replacement Orchestrator in the rack.

   Do not connect the power or network cables yet. You do it gradually later.

5. On the replacement Orchestrator, connect only these cables:

   1. The network cable from your computer to the Orchestrator's MGMT interface.

   2. The console cable from your computer to the Orchestrator's Console Port.

      In your console client, configure these settings:

      • Baud Rate - 9600
      • Data bits - 8
      • Stop bits - 1
      • Parity - None
      • Flow Control - None

   3. The power cables to the Orchestrator's Power Supply Units.

   See the Quantum Maestro Getting Started Guide.

6. On the replacement Orchestrator, in the console session, log in to Gaia Clish.

7. On the replacement Orchestrator, configure the Orchestrator's MGMT interface:

   1. Configure the IPv4 address and Mask Length:

      set interface Mgmt1 ipv4-address <IPv4 Address of MGMT Interface> mask-length <Mask>

   2. Enable the interface:

      set interface Mgmt1 state on

   3. Configure the default gateway:

      set static-route default nexthop gateway address <IPv4 Address of Default Gateway> on

   4. Save the changes:

      save config

   See the R80.20SP Maestro Gaia Administration Guide.

8. On the replacement Orchestrator, connect to the command line through the MGMT interface (over SSH).

9. On the replacement Orchestrator, log in to Gaia Clish.

10. On the replacement Orchestrator, configure these Gaia settings:

    1. Configure the same date and time settings as configured on all other working Orchestrators (in our example, "1_1").

       Note - The date and time settings must be the same on all Orchestrators.

    2. Configure the hostname.

    3. Change the default password for the 'admin' user.

    4. Save the changes:

       save config

    See the R80.20SP Maestro Gaia Administration Guide.

11. On the replacement Orchestrator, install the same Take of the R80.20SP Jumbo Hotfix Accumulator as installed on the working Orchestrator (in our example, "1_1").

    Wait for the Orchestrator to reboot.

12. On the replacement Orchestrator, stop the services:

    1. Connect to the command line through the MGMT interface (over SSH).

    2. Log in to the Expert mode.

    3. Stop the Orchestrator service:

       orchd stop

    4. Stop the LLDP service:

       tellpm process:lldpd

13. On the replacement Orchestrator - copy these files from your computer to the replacement Orchestrator to the /etc/ directory:

    • maestro_remote.json
    • maestro_full.json
    • sgdb.json
    • smodb.json

    Use an SCP client (WinSCP requires the default user shell to be /bin/bash).

14. On the replacement Orchestrator, edit the configuration files (in the Expert mode):

    1. Rename the file:

       from /etc/maestro_remote.json

       to /etc/maestro.json

    2. Edit the file /etc/smodb.json and change the value of the member_id parameter to the ID of the Orchestrator that failed (in our example, "2").

15. On all Orchestrators, make sure the configuration files are the same.

    1. Connect to the command line on each Orchestrator.

    2. Log in to the Expert mode.

    3. Run:

       jsont -f /etc/sgdb.json -P | egrep -v "topology_timestamp|session_id|active_sgms|all_sgm" | md5sum

       The MD5 checksums must be the same on the two Orchestrators.

       If the MD5 checksums are different:

       1. Copy the /etc/sgdb.json file from the Orchestrator that continues to work ("1_1") to the replacement Orchestrator ("1_2") to the /etc/ directory.
       2. Check the MD5 checksums again.

16. On the replacement Orchestrator, connect these cables to ports with the same numbers as they were connected on the Orchestrator that failed:

    • The Downlink cables
    • The Sync cable(s)

    Important - Make sure you connected the cables to the correct ports.

17. On the replacement Orchestrator, start the Orchestrator service:

    1. Connect to the command line.

    2. Log in to the Expert mode.

    3. Disable the Link State Propagation (LSP):

       jsont -f /etc/smodb.json -s /orch_lsp_state -v off

    4. Start the Orchestrator service:

       orchd start

    5. Enable the Link State Propagation (LSP):

       jsont -f /etc/smodb.json -s /orch_lsp_state -v on

    6. Restart the Orchestrator service (this also starts the LLDP service):

       orchd restart

18. On all Orchestrators, make sure the traffic can pass between Orchestrators:

    • In a Single Site deployment:

      On the working Orchestrator (in our example, "1_1"), send pings to the replacement Orchestrator (in our example, "1_2"):

      ping 1_2

    • In a Dual Site deployment:

      On the replacement Orchestrator (in our example, "1_2"), send pings to each working Orchestrator:

      ping 1_1

      ping 2_1

      ping 2_2

19. Make sure the Security Group Members can pass traffic to each other:

    1. Connect to the command line on the Security Group.

    2. Log in to the Expert mode.

    3. Identify the Security Group Member that runs as SMO:

       asg stat -i tasks

    4. Examine the cluster state of the Security Group Members.

       On the SMO Security Group Member, run:

       cphaprob state

       The output must show that all Security Group Members are active.

    5. Send pings between Security Group Members:

       1. Connect to one of the Security Group Members
          (in our example, we connect to the first one - "1_1"):

          member 1_1

       2. On this Security Group Member, send pings to any other Security Group Member
          (in our example, we send pings to the second one - "1_2" / "2_2"):

          • In a Single Site deployment:

            ping 1_2

          • In a Dual Site deployment:

            ping 1_2

            ping 2_2

20. On the replacement Orchestrator, connect the Uplink cables to ports with the same numbers as they were connected on the Orchestrator that failed.

21. On each Security Group, make sure all links are up on the Security Group:

    1. Connect to the command line on the Security Group.

    2. Examine the state of links:

       asg_if

1. On the Orchestrator that continues to work, export the configuration files for the Orchestrator that failed:

   1. Connect to the command line on the Orchestrator that continues to work.

   2. Log in to Gaia Clish.

   3. Export the configuration files for the Orchestrator that failed:

      set maestro export remote orchestrator id <ID of the Orchestrator That Failed> configuration archive-name <Name of Output Archive File> path <Full Path on Orchestrator That Continues to Work>

      Example:

      set maestro export remote orchestrator id 1_2 configuration archive-name Export_from_failed_Orch_1_2 path /var/log/

2. On the Orchestrator that continues to work, copy the archive with the exported configuration files to your computer.

   Use an SCP client (WinSCP requires the default user shell to be /bin/bash).

3. On the Orchestrator that failed, stop the Orchestrator service:

   1. Connect to the command line on the Orchestrator that failed.

   2. Log in to the Expert mode.

   3. Stop the service:

      orchd stop

4. On the Orchestrator that failed, write down the port numbers, to which the Uplink cable, the Downlink cables, and the Sync cables are connected.

5. On the Orchestrator that failed, disconnect all cables.

   Important - Do not halt (shut down) the Orchestrator that failed. It might reboot, and the Orchestrator services will start again.

6. Remove the Orchestrator that failed from the rack.

7. Install the replacement Orchestrator in the rack.

   Do not connect the power or network cables yet. You do it gradually later.

8. On the replacement Orchestrator, connect only these cables:

   1. The network cable from your computer to the Orchestrator's MGMT interface.

   2. The console cable from your computer to the Orchestrator's Console Port.

      In your console client, configure these settings:

      • Baud Rate - 9600
      • Data bits - 8
      • Stop bits - 1
      • Parity - None
      • Flow Control - None

   3. The power cables to the Orchestrator's Power Supply Units.

   See the Quantum Maestro Getting Started Guide.

9. On the replacement Orchestrator, in the console session, log in to Gaia Clish.

   Important Note - Choose not to activate the Orchestrator. If you activated the Orchestrator, then in the Expert mode, run the "orchd stop" command.

10. On the replacement Orchestrator, configure the Orchestrator's MGMT interface:

    1. Configure the IPv4 address and Mask Length:

       set interface Mgmt1 ipv4-address <IPv4 Address of MGMT Interface> mask-length <Mask>

    2. Enable the interface:

       set interface Mgmt1 state on

    3. Configure the default gateway:

       set static-route default nexthop gateway address <IPv4 Address of Default Gateway> on

    4. Save the changes:

       save config

    See the R81.10 Gaia Administration Guide.

11. On the replacement Orchestrator, connect to the command line through the MGMT interface (over SSH).

12. On the replacement Orchestrator, log in to Gaia Clish.

13. On the replacement Orchestrator, configure these Gaia settings:

    1. Configure the same date and time settings as configured on all other working Orchestrators (in our example, "1_1").

       Note - The date and time settings must be the same on all Orchestrators.

    2. Configure the hostname.

    3. Change the default password for the 'admin' user.

    4. Save the changes:

       save config

    See the R81.10 Gaia Administration Guide.

14. On the replacement Orchestrator, install the same Take of the R81.10 Jumbo Hotfix Accumulator as installed on the working Orchestrator (in our example, "1_1").

    Wait for the Orchestrator to reboot.

    Important Note - Choose not to activate the Orchestrator. If you activated the Orchestrator, then in the Expert mode, run the "orchd stop" command.

15. On the replacement Orchestrator, copy the archive with the exported configuration files from your computer to the replacement Orchestrator to some directory (for example, /var/log/).

    Use an SCP client (WinSCP requires the default user shell to be /bin/bash).

16. On the replacement Orchestrator, import the configuration files you collected earlier:

    1. Connect to the command line on the replacement Orchestrator.

    2. Log in to Gaia Clish.

    3. Import the configuration files:

       set maestro import configuration archive-name <Name of Output Archive File> path <Full Local Path>

       Example:

       set maestro import configuration archive-name Export_from_failed_Orch_1_2.tgz path /var/log/

17. On all Orchestrators, make sure the configuration files are the same.

    1. Connect to the command line on each Orchestrator.

    2. Log in to the Expert mode.

    3. Run:

       jsont -f /etc/sgdb.json -P | egrep -v "topology_timestamp|session_id|active_sgms|all_sgm" | md5sum

       The MD5 checksums must be the same on the two Orchestrators.

       If the MD5 checksums are different:

       1. Copy the /etc/sgdb.json file from the working Orchestrator (1_1) to the replacement Orchestrator (1_2) to the /etc/ directory.
       2. Check the MD5 checksums again.

18. On the replacement Orchestrator, connect these cables to ports with the same numbers as they were connected on the Orchestrator that failed:

    • The Downlink cables
    • The Sync cable(s)

    Important - Make sure you connected the cables to the correct ports.

19. On the replacement Orchestrator, start the Orchestrator service:

    1. Log in to the Expert mode.

    2. Disable the Link State Propagation (LSP):

       jsont -f /etc/smodb.json -s /orch_lsp_state -v off

    3. Check the Orchestrator services status:

       orchd status

    4. If the Orchestrator is not active, then activate it (it will start the Orchestrator services):

       orchd activate

       (Otherwise, start the Orchestrator services: orchd start)

    5. Enable the Link State Propagation (LSP):

       jsont -f /etc/smodb.json -s /orch_lsp_state -v on

    6. Restart the Orchestrator port monitoring daemon with these commands:

       tellpm process:ssm_pmd

       tellpm process:ssm_pmd t

    Note - It may take a few seconds for the Orchestrator to discover the connected Security Appliances for the first time. In the Expert mode, run the "watch -d -n 1 orch_stat" command and wait for all the LAGs to be in the "up" status.

20. On all Orchestrators, make sure the traffic can pass between Orchestrators:

    • In a Single Site deployment:

      On the Orchestrator that continues to work (in our example, "1_1"), send pings to the replacement Orchestrator (in our example, "1_2"):

      ping 1_2

    • In a Dual Site deployment:

      On the replacement Orchestrator (in our example, "1_2"), send pings to each working Orchestrator:

      ping 1_1

      ping 2_1

      ping 2_2

21. Make sure the Security Group Members can pass traffic to each other:

    1. Connect to the command line on the Security Group.

    2. Log in to the Expert mode.

    3. Identify the Security Group Member that runs as SMO:

       asg stat -i tasks

    4. Examine the cluster state of the Security Group Members.

       On the SMO Security Group Member, run:

       cphaprob state

       The output must show that all Security Group Members are active.

    5. Send pings between Security Group Members:

       1. Connect to one of the Security Group Members
          (in our example, we connect to the first one - "1_1"):

          member 1_1

       2. On this Security Group Member, send pings to any other Security Group Member
          (in our example, we send pings to the second one - "1_2" / "2_2"):

          • In a Single Site deployment:

            ping 1_2

          • In a Dual Site deployment:

            ping 1_2

            ping 2_2

22. On the replacement Orchestrator, connect the Uplink cables to ports with the same numbers as they were connected on the Orchestrator that failed.

23. On each Security Group, make sure all links are up on the Security Group:

    1. Connect to the command line on the Security Group.

    2. Examine the state of links:

       asg_if