Secondary MDS/MLM upgrade stops and eventually rolls back

Support Center > Search Results > SecureKnowledge Details

Technical Level

Solution ID	sk169332
Technical Level
Product	Multi-Domain Management
Version	R80.10, R80.20, R80.30, R80.40, R81
OS	Gaia
Date Created	20-Oct-2020
Last Modified	21-Dec-2020

Symptoms

Inplace upgrade of MLM/secondary MDS takes a very long time (over 2 hours) and the system reverts the upgrade to the previous version.
Virtually no resource consumption occurring on the appliance and "ps -aux | grep upgrade" shows:

admin 3503 0.0 0.0 2644 580 pts/2 S+ 18:26 0:00 grep --color=auto upgrade admin 31302 0.0 0.0 3496 1360 ? S 16:55 0:00 /bin/sh /opt/CPsuite-R80.40/fw1/scripts/migrate_server import -v R80.40 -skip_upgrade_tools_check -n -d /new_upgrade_tools_migrate_ admin 31303 0.0 0.0 3764 1784 ? S 16:55 0:01 /bin/sh /opt/CPupgrade-tools-R80.40/scripts/upgrade_client.sh import -v R80.40 -non-interactive -d /new_upgrade_tools_migrate_dir/m admin 39094 1.1 0.0 4532 1928 ? S 16:57 1:01 /bin/sh /opt/CPupgrade-tools-R80.40/scripts/upgrade_client.sh import -v R80.40 -non-interactive -d /new_upgrade_tools_migrate_dir/m
During the upgrade, the following messages appear in in $MDS_TEMPLATE/log/cpm.elg:
DATE TIME,PID INFO upgrade.common.GenerateUpgradeReportTimerTask [Timer-0]: UpgradeStats: there are 2 reports in $MDS_FWDIR/log DATE TIME,PID INFO upgrade.common.GenerateUpgradeReportTimerTask [Timer-0]: UpgradeStats: delete /opt/CPsuite-R80.40/fw1/log/upgrade_report-2020.09.12_19.57.24.html DATE TIME,PID INFO upgrade.ngmImport.MultiSiteNewUpgradeSvcImpl [taskExecutor-48]: Waiting for FullSync 1430 seconds DATE TIME,PID INFO cpm.commands.Server [main]: <<== Memory Report: Total Memory - 1306 MB; Used - 977 MB, Free - 328 MB, Heap - 977 MB, Non Heap - 207 MB, Percent Used - 74 %==>> DATE TIME,PID6 INFO cpm.commands.Server [main]: <<== Threads Report: number of threads - 198 ==>> DATE TIME,PID INFO upgrade.ngmImport.MultiSiteNewUpgradeSvcImpl [taskExecutor-48]: Waiting for FullSync 1440 seconds DATE TIME,PID INFO upgrade.ngmImport.MultiSiteNewUpgradeSvcImpl [taskExecutor-48]: Waiting for FullSync 1450 seconds DATE TIME,PID INFO upgrade.common.GenerateUpgradeReportTimerTask [Timer-0]: UpgradeStats: generating report: /opt/CPsuite-R80.40/fw1/log/upgrade_report-2020.09.12_19.58.24.html DATE TIME,PID INFO upgrade.common.GenerateUpgradeReportTimerTask [Timer-0]: UpgradeStats: Upgrade Report is now available at /opt/CPsuite-R80.40/fw1/log/upgrade_report-2020.09.12_19.58.24.html
Jumbo Hotfix was installed on the primary MDS server after it was successfully upgraded.

Cause

Full sync does not complete after the first reboot (operating system upgraded) due to different hotfixes installed on the primary Multi-Domain Management server.

Solution

Note: To view this solution you need to Sign In .