Upgrading from ACS 4.2 to 4.8 - VR upgrade issue

[Cloud List]

Hi,

I tried upgrading our CloudStack 4.2.0 on our test environment to 4.8.0,
and I am having issues on VR upgrade. I followed the upgrade instructions
at below URL:

http://cloudstack-release-notes.readthedocs.io/ja/stable/upgrade/upgrade-4.2.html

On the last step to restart all the system VMs and VRs, it seems that the
cloudstack-sysvmadm script managed to restart the system VMs (SSVM and
CPVM) but not the VR.

# nohup cloudstack-sysvmadm -d IPaddress -u cloud -p password -a >
sysvm.log 2>&1 &

Here's the output of the sysvm.log file:

====
nohup: ignoring input
/usr/bin/cloudstack-sysvmadm: line 21: /etc/rc.d/init.d/functions: No such
file or directory

Stopping and starting 1 secondary storage vm(s)...
Done stopping and starting secondary storage vm(s)

Stopping and starting 1 console proxy vm(s)...
Done stopping and starting console proxy vm(s) .

Stopping and starting 1 running routing vm(s)...
*ERROR: Failed to restart domainRouter with id 5*

Done restarting router(s).
====

Output of the cloud.log file:

====
[2016.06.05-22.28.40] Stopping and starting 1 secondary storage vm(s)...
[2016.06.05-22.28.40] INFO: Stopping secondary storage vm with id 3
[2016.06.05-22.28.46] INFO: Starting secondary storage vm with id 3
[2016.06.05-23.29.00] ERROR: Failed to startSystemVm id=3; jobId is
121c5d89-745a-4e8e-9f96-0b348ef42abf
[2016.06.05-23.29.00] ERROR: Failed to start secondary storage vm with id 3
[2016.06.05-23.29.00] ERROR: Failed to start secondary storage vm with id 3
[2016.06.05-23.29.00] Done stopping and starting secondary storage vm(s).
[2016.06.05-23.29.00] Stopping and starting 1 console proxy vm(s)...
[2016.06.05-23.29.00] INFO: Stopping console proxy with id 2
[2016.06.05-23.29.06] INFO: Starting console proxy vm with id 2
[2016.06.05-23.30.49] Done stopping and starting console proxy vm(s) .
[2016.06.05-23.30.49] Stopping and starting 1 running routing vm(s)...
[2016.06.05-23.30.49] INFO: Restarting router with id 5
[2016.06.06-00.30.59] ERROR: Failed to restart domainRouter with id 5;
jobId 4a3f681e-c4a6-4856-9d42-0d3b79bffdae
[2016.06.06-00.31.04] Done restarting router(s).
====

Initially I noted that the VR remains in 4.2.0 version (instead of the
expected 4.8.0 version), tried to destroy the router so it'll be recreated
but it doesn't get created successfully.

Tried to run the cloudstack-sysvmadm manually with the -r option to restart
only the VR, and the problem still persists.

At the moment, the version of the VR is "UNKNOWN" and the "Require Upgrade"
state is "Yes".

Any advice how to make the VR back up and running?

Looking forward to your reply, thank you.

Cheers.

-ip-

[Kirk Kosinski]

Hi, what is the status of the CPVM and SSVM in the UI? Were they really upgraded successfully?

Also, are there any errors in the management-server.log?

***@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HS
@shapeblue



-----Original Message-----
From: Cloud List [mailto:cloud-***@sg.or.id]
Sent: Monday, June 06, 2016 9:51 AM
To: ***@cloudstack.apache.org; ***@cloudstack.apache.org
Subject: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue

Hi,

I tried upgrading our CloudStack 4.2.0 on our test environment to 4.8.0, and I am having issues on VR upgrade. I followed the upgrade instructions at below URL:

http://cloudstack-release-notes.readthedocs.io/ja/stable/upgrade/upgrade-4.2.html

On the last step to restart all the system VMs and VRs, it seems that the cloudstack-sysvmadm script managed to restart the system VMs (SSVM and
CPVM) but not the VR.

# nohup cloudstack-sysvmadm -d IPaddress -u cloud -p password -a > sysvm.log 2>&1 &

Here's the output of the sysvm.log file:

====
nohup: ignoring input
/usr/bin/cloudstack-sysvmadm: line 21: /etc/rc.d/init.d/functions: No such file or directory

Stopping and starting 1 secondary storage vm(s)...
Done stopping and starting secondary storage vm(s)

Stopping and starting 1 console proxy vm(s)...
Done stopping and starting console proxy vm(s) .

Stopping and starting 1 running routing vm(s)...
*ERROR: Failed to restart domainRouter with id 5*

Done restarting router(s).
====

Output of the cloud.log file:

====
[2016.06.05-22.28.40] Stopping and starting 1 secondary storage vm(s)...
[2016.06.05-22.28.40] INFO: Stopping secondary storage vm with id 3 [2016.06.05-22.28.46] INFO: Starting secondary storage vm with id 3 [2016.06.05-23.29.00] ERROR: Failed to startSystemVm id=3; jobId is 121c5d89-745a-4e8e-9f96-0b348ef42abf
[2016.06.05-23.29.00] ERROR: Failed to start secondary storage vm with id 3 [2016.06.05-23.29.00] ERROR: Failed to start secondary storage vm with id 3 [2016.06.05-23.29.00] Done stopping and starting secondary storage vm(s).
[2016.06.05-23.29.00] Stopping and starting 1 console proxy vm(s)...
[2016.06.05-23.29.00] INFO: Stopping console proxy with id 2 [2016.06.05-23.29.06] INFO: Starting console proxy vm with id 2 [2016.06.05-23.30.49] Done stopping and starting console proxy vm(s) .
[2016.06.05-23.30.49] Stopping and starting 1 running routing vm(s)...
[2016.06.05-23.30.49] INFO: Restarting router with id 5 [2016.06.06-00.30.59] ERROR: Failed to restart domainRouter with id 5; jobId 4a3f681e-c4a6-4856-9d42-0d3b79bffdae
[2016.06.06-00.31.04] Done restarting router(s).
====

Initially I noted that the VR remains in 4.2.0 version (instead of the expected 4.8.0 version), tried to destroy the router so it'll be recreated but it doesn't get created successfully.

Tried to run the cloudstack-sysvmadm manually with the -r option to restart only the VR, and the problem still persists.

At the moment, the version of the VR is "UNKNOWN" and the "Require Upgrade"
state is "Yes".

Any advice how to make the VR back up and running?

Looking forward to your reply, thank y

[Cloud List]

Hi Kirk,

Thanks for your reply.

Here's the status of the CPVM and SSVM in the UI:

Name v-2-VM
ID fc6d23bf-58ac-4641-af9b-506d1dc46df3
State Running
Type Console Proxy VM
Zone sc-test-cs42-zone
Public IP Address 192.168.0.162
Private IP Address 192.168.0.160
Link Local IP Address 169.254.3.147
Host sc-test-kvm-03
Gateway 192.168.0.1
Created 31 May 2016 15:24:20
Active Sessions 0

Name s-29-VM
ID d26f54f8-1c1f-4f3d-9862-fc0405159620
State Running
Type Secondary Storage VM
Zone sc-test-cs42-zone
Public IP Address 192.168.0.161
Private IP Address 192.168.0.151
Link Local IP Address 169.254.1.207
Host sc-test-kvm-03
Gateway 192.168.0.1
Created 06 Jun 2016 00:31:04
Active Sessions

Both system VMs seem to be on "Running" state but I can't verify what
template version they are using, whether they are already on 4.6/4.8
template or still using 4.2 template, are you able to advise how can I
tell? The SSVM creation date seem to be after we performed the upgrade but
the CPVM created date is before the upgrade.

How to ensure that all systemVMs using the new version template? Shall I
destroy the system VMs so new ones will be created, or shall I re-run the
/usr/bin/cloudstack-sysvmadm again in full?

Looking forward to your reply, thank you.

Cheers.

-ip-

Post by Kirk Kosinski
Hi, what is the status of the CPVM and SSVM in the UI? Were they really
upgraded successfully?
Also, are there any errors in the management-server.log?
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HS
@shapeblue
-----Original Message-----
Sent: Monday, June 06, 2016 9:51 AM
Subject: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue
Hi,
I tried upgrading our CloudStack 4.2.0 on our test environment to 4.8.0,
and I am having issues on VR upgrade. I followed the upgrade instructions
http://cloudstack-release-notes.readthedocs.io/ja/stable/upgrade/upgrade-4.2.html
On the last step to restart all the system VMs and VRs, it seems that the
cloudstack-sysvmadm script managed to restart the system VMs (SSVM and
CPVM) but not the VR.
# nohup cloudstack-sysvmadm -d IPaddress -u cloud -p password -a > sysvm.log 2>&1 &
====
nohup: ignoring input
/usr/bin/cloudstack-sysvmadm: line 21: /etc/rc.d/init.d/functions: No such
file or directory
Stopping and starting 1 secondary storage vm(s)...
Done stopping and starting secondary storage vm(s)
Stopping and starting 1 console proxy vm(s)...
Done stopping and starting console proxy vm(s) .
Stopping and starting 1 running routing vm(s)...
*ERROR: Failed to restart domainRouter with id 5*
Done restarting router(s).
====
====
[2016.06.05-22.28.40] Stopping and starting 1 secondary storage vm(s)...
[2016.06.05-22.28.40] INFO: Stopping secondary storage vm with id 3
[2016.06.05-22.28.46] INFO: Starting secondary storage vm with id 3
[2016.06.05-23.29.00] ERROR: Failed to startSystemVm id=3; jobId is
121c5d89-745a-4e8e-9f96-0b348ef42abf
[2016.06.05-23.29.00] ERROR: Failed to start secondary storage vm with id
3 [2016.06.05-23.29.00] ERROR: Failed to start secondary storage vm with id
3 [2016.06.05-23.29.00] Done stopping and starting secondary storage vm(s).
[2016.06.05-23.29.00] Stopping and starting 1 console proxy vm(s)...
[2016.06.05-23.29.00] INFO: Stopping console proxy with id 2
[2016.06.05-23.29.06] INFO: Starting console proxy vm with id 2
[2016.06.05-23.30.49] Done stopping and starting console proxy vm(s) .
[2016.06.05-23.30.49] Stopping and starting 1 running routing vm(s)...
[2016.06.05-23.30.49] INFO: Restarting router with id 5
[2016.06.06-00.30.59] ERROR: Failed to restart domainRouter with id 5;
jobId 4a3f681e-c4a6-4856-9d42-0d3b79bffdae
[2016.06.06-00.31.04] Done restarting router(s).
====
Initially I noted that the VR remains in 4.2.0 version (instead of the
expected 4.8.0 version), tried to destroy the router so it'll be recreated
but it doesn't get created successfully.
Tried to run the cloudstack-sysvmadm manually with the -r option to
restart only the VR, and the problem still persists.
At the moment, the version of the VR is "UNKNOWN" and the "Require Upgrade"
state is "Yes".
Any advice how to make the VR back up and running?
Looking forward to your reply, thank you.
Cheers.
-ip-

[Cloud List]

Dear all,

Could it be the VR is not able to use the new systemVM template because we
registered the template as Routing = NO? Will it resolve the problem if
during upgrade, we register the template as Routing = YES?

Based on the upgrade instruction at
http://cloudstack-release-notes.readthedocs.io/ja/stable/upgrade/upgrade-4.2.html
, it says that routing option should be "no", is this correct? Since the
systemVM template will also be used by VR?

Any advice is appreciated.

Looking forward to your reply, thank you.

Cheers.

-ip-

Post by Cloud List
Hi Kirk,
Thanks for your reply.
Name v-2-VM
ID fc6d23bf-58ac-4641-af9b-506d1dc46df3
State Running
Type Console Proxy VM
Zone sc-test-cs42-zone
Public IP Address 192.168.0.162
Private IP Address 192.168.0.160
Link Local IP Address 169.254.3.147
Host sc-test-kvm-03
Gateway 192.168.0.1
Created 31 May 2016 15:24:20
Active Sessions 0
Name s-29-VM
ID d26f54f8-1c1f-4f3d-9862-fc0405159620
State Running
Type Secondary Storage VM
Zone sc-test-cs42-zone
Public IP Address 192.168.0.161
Private IP Address 192.168.0.151
Link Local IP Address 169.254.1.207
Host sc-test-kvm-03
Gateway 192.168.0.1
Created 06 Jun 2016 00:31:04
Active Sessions
Both system VMs seem to be on "Running" state but I can't verify what
template version they are using, whether they are already on 4.6/4.8
template or still using 4.2 template, are you able to advise how can I
tell? The SSVM creation date seem to be after we performed the upgrade but
the CPVM created date is before the upgrade.
How to ensure that all systemVMs using the new version template? Shall I
destroy the system VMs so new ones will be created, or shall I re-run the
/usr/bin/cloudstack-sysvmadm again in full?
Looking forward to your reply, thank you.
Cheers.
-ip-

Post by Kirk Kosinski
Hi, what is the status of the CPVM and SSVM in the UI? Were they really
upgraded successfully?
Also, are there any errors in the management-server.log?
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HS
@shapeblue
-----Original Message-----
Sent: Monday, June 06, 2016 9:51 AM
Subject: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue
Hi,
I tried upgrading our CloudStack 4.2.0 on our test environment to 4.8.0,
and I am having issues on VR upgrade. I followed the upgrade instructions
http://cloudstack-release-notes.readthedocs.io/ja/stable/upgrade/upgrade-4.2.html
On the last step to restart all the system VMs and VRs, it seems that the
cloudstack-sysvmadm script managed to restart the system VMs (SSVM and
CPVM) but not the VR.
# nohup cloudstack-sysvmadm -d IPaddress -u cloud -p password -a > sysvm.log 2>&1 &
====
nohup: ignoring input
/usr/bin/cloudstack-sysvmadm: line 21: /etc/rc.d/init.d/functions: No
such file or directory
Stopping and starting 1 secondary storage vm(s)...
Done stopping and starting secondary storage vm(s)
Stopping and starting 1 console proxy vm(s)...
Done stopping and starting console proxy vm(s) .
Stopping and starting 1 running routing vm(s)...
*ERROR: Failed to restart domainRouter with id 5*
Done restarting router(s).
====
====
[2016.06.05-22.28.40] Stopping and starting 1 secondary storage vm(s)...
[2016.06.05-22.28.40] INFO: Stopping secondary storage vm with id 3
[2016.06.05-22.28.46] INFO: Starting secondary storage vm with id 3
[2016.06.05-23.29.00] ERROR: Failed to startSystemVm id=3; jobId is
121c5d89-745a-4e8e-9f96-0b348ef42abf
[2016.06.05-23.29.00] ERROR: Failed to start secondary storage vm with id
3 [2016.06.05-23.29.00] ERROR: Failed to start secondary storage vm with id
3 [2016.06.05-23.29.00] Done stopping and starting secondary storage vm(s).
[2016.06.05-23.29.00] Stopping and starting 1 console proxy vm(s)...
[2016.06.05-23.29.00] INFO: Stopping console proxy with id 2
[2016.06.05-23.29.06] INFO: Starting console proxy vm with id 2
[2016.06.05-23.30.49] Done stopping and starting console proxy vm(s) .
[2016.06.05-23.30.49] Stopping and starting 1 running routing vm(s)...
[2016.06.05-23.30.49] INFO: Restarting router with id 5
[2016.06.06-00.30.59] ERROR: Failed to restart domainRouter with id 5;
jobId 4a3f681e-c4a6-4856-9d42-0d3b79bffdae
[2016.06.06-00.31.04] Done restarting router(s).
====
Initially I noted that the VR remains in 4.2.0 version (instead of the
expected 4.8.0 version), tried to destroy the router so it'll be recreated
but it doesn't get created successfully.
Tried to run the cloudstack-sysvmadm manually with the -r option to
restart only the VR, and the problem still persists.
At the moment, the version of the VR is "UNKNOWN" and the "Require Upgrade"
state is "Yes".
Any advice how to make the VR back up and running?
Looking forward to your reply, thank you.
Cheers.
-ip-

[Sam Ceylani]

Same thing happened to me, upgrading from 4.5.2 to 4.8 and I had to restart network with clean-up option checked for each network which rebuilds the router VM. If your SSVM works
And if you can test it then I don’t see any problem rebuilding network(router vm) with new template which took about 30 minutes for about 20 networks in total...may not be an option
for large installations...how many routers you are running?

-----Original Message-----
From: Cloud List [mailto:cloud-***@sg.or.id]
Sent: Wednesday, June 8, 2016 12:23 AM
To: ***@cloudstack.apache.org; ***@cloudstack.apache.org
Subject: Re: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue

Dear all,

Could it be the VR is not able to use the new systemVM template because we registered the template as Routing = NO? Will it resolve the problem if during upgrade, we register the template as Routing = YES?

Based on the upgrade instruction at
http://cloudstack-release-notes.readthedocs.io/ja/stable/upgrade/upgrade-4.2.html
, it says that routing option should be "no", is this correct? Since the systemVM template will also be used by VR?

Any advice is appreciated.

Looking forward to your reply, thank you.

Cheers.

-ip-

Post by Cloud List
Hi Kirk,
Thanks for your reply.
Name v-2-VM
ID fc6d23bf-58ac-4641-af9b-506d1dc46df3
State Running
Type Console Proxy VM
Zone sc-test-cs42-zone
Public IP Address 192.168.0.162
Private IP Address 192.168.0.160
Link Local IP Address 169.254.3.147
Host sc-test-kvm-03
Gateway 192.168.0.1
Created 31 May 2016 15:24:20
Active Sessions 0
Name s-29-VM
ID d26f54f8-1c1f-4f3d-9862-fc0405159620
State Running
Type Secondary Storage VM
Zone sc-test-cs42-zone
Public IP Address 192.168.0.161
Private IP Address 192.168.0.151
Link Local IP Address 169.254.1.207
Host sc-test-kvm-03
Gateway 192.168.0.1
Created 06 Jun 2016 00:31:04
Active Sessions
Both system VMs seem to be on "Running" state but I can't verify what
template version they are using, whether they are already on 4.6/4.8
template or still using 4.2 template, are you able to advise how can I
tell? The SSVM creation date seem to be after we performed the upgrade
but the CPVM created date is before the upgrade.
How to ensure that all systemVMs using the new version template? Shall
I destroy the system VMs so new ones will be created, or shall I
re-run the /usr/bin/cloudstack-sysvmadm again in full?
Looking forward to your reply, thank you.
Cheers.
-ip-
On Tue, Jun 7, 2016 at 7:22 AM, Kirk Kosinski

Post by Kirk Kosinski
Hi, what is the status of the CPVM and SSVM in the UI? Were they
really upgraded successfully?
Also, are there any errors in the management-server.log?
www.shapeblue.com
-----Original Message-----
Sent: Monday, June 06, 2016 9:51 AM
Subject: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue
Hi,
I tried upgrading our CloudStack 4.2.0 on our test environment to 4.8.0,
and I am having issues on VR upgrade. I followed the upgrade instructions
http://cloudstack-release-notes.readthedocs.io/ja/stable/upgrade/upgrade-4.2.html
On the last step to restart all the system VMs and VRs, it seems that the
cloudstack-sysvmadm script managed to restart the system VMs (SSVM and
CPVM) but not the VR.
# nohup cloudstack-sysvmadm -d IPaddress -u cloud -p password -a > sysvm.log 2>&1 &
====
nohup: ignoring input
/usr/bin/cloudstack-sysvmadm: line 21: /etc/rc.d/init.d/functions: No
such file or directory
Stopping and starting 1 secondary storage vm(s)...
Done stopping and starting secondary storage vm(s)
Stopping and starting 1 console proxy vm(s)...
Done stopping and starting console proxy vm(s) .
Stopping and starting 1 running routing vm(s)...
*ERROR: Failed to restart domainRouter with id 5*
Done restarting router(s).
====
====
[2016.06.05-22.28.40] Stopping and starting 1 secondary storage vm(s)...
[2016.06.05-22.28.40] INFO: Stopping secondary storage vm with id 3
[2016.06.05-22.28.46] INFO: Starting secondary storage vm with id 3
[2016.06.05-23.29.00] ERROR: Failed to startSystemVm id=3; jobId is
121c5d89-745a-4e8e-9f96-0b348ef42abf
[2016.06.05-23.29.00] ERROR: Failed to start secondary storage vm with id
3 [2016.06.05-23.29.00] ERROR: Failed to start secondary storage vm with id
3 [2016.06.05-23.29.00] Done stopping and starting secondary storage vm(s).
[2016.06.05-23.29.00] Stopping and starting 1 console proxy vm(s)...
[2016.06.05-23.29.00] INFO: Stopping console proxy with id 2
[2016.06.05-23.29.06] INFO: Starting console proxy vm with id 2
[2016.06.05-23.30.49] Done stopping and starting console proxy vm(s) .
[2016.06.05-23.30.49] Stopping and starting 1 running routing vm(s)...
[2016.06.05-23.30.49] INFO: Restarting router with id 5
[2016.06.06-00.30.59] ERROR: Failed to restart domainRouter with id 5;
jobId 4a3f681e-c4a6-4856-9d42-0d3b79bffdae
[2016.06.06-00.31.04] Done restarting router(s).
====
Initially I noted that the VR remains in 4.2.0 version (instead of the
expected 4.8.0 version), tried to destroy the router so it'll be recreated
but it doesn't get created successfully.
Tried to run the cloudstack-sysvmadm manually with the -r option to
restart only the VR, and the problem still persists.
At the moment, the version of the VR is "UNKNOWN" and the "Require Upgrade"
state is "Yes".
Any advice how to make the VR back up and running?
Looking forward to your reply,

[Cloud List]

Hi Sam,

Thanks for your reply.

My SSVM and CPVM seems to work fine -- although I can't verify if they are
really using the new template or not, but my VR doesn't work even after I
destroy the VR and restart the network with clean-up option. So the VR is
never able to be started up after the upgrade. It kept on "Starting" state
and agent.log file shows it keeps on trying to connect to the link local IP
but never able to connect.

I tested the upgrade on a staging environment with just 1 VR, I have
reverted back the setup to 4.2 and now re-doing the upgrade again for the
second attempt, will see if it works. My production has around 10+ networks
so I need to ensure this works before I initiate the upgrade on my
production server.

Thank you.

Post by Sam Ceylani
Same thing happened to me, upgrading from 4.5.2 to 4.8 and I had to
restart network with clean-up option checked for each network which
rebuilds the router VM. If your SSVM works
And if you can test it then I don’t see any problem rebuilding
network(router vm) with new template which took about 30 minutes for about
20 networks in total...may not be an option
for large installations...how many routers you are running?
-----Original Message-----
Sent: Wednesday, June 8, 2016 12:23 AM
Subject: Re: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue
Dear all,
Could it be the VR is not able to use the new systemVM template because we
registered the template as Routing = NO? Will it resolve the problem if
during upgrade, we register the template as Routing = YES?
Based on the upgrade instruction at
http://cloudstack-release-notes.readthedocs.io/ja/stable/upgrade/upgrade-4.2.html
, it says that routing option should be "no", is this correct? Since the
systemVM template will also be used by VR?
Any advice is appreciated.
Looking forward to your reply, thank you.
Cheers.
-ip-

Post by Cloud List
Hi Kirk,
Thanks for your reply.
Name v-2-VM
ID fc6d23bf-58ac-4641-af9b-506d1dc46df3
State Running
Type Console Proxy VM
Zone sc-test-cs42-zone
Public IP Address 192.168.0.162
Private IP Address 192.168.0.160
Link Local IP Address 169.254.3.147
Host sc-test-kvm-03
Gateway 192.168.0.1
Created 31 May 2016 15:24:20
Active Sessions 0
Name s-29-VM
ID d26f54f8-1c1f-4f3d-9862-fc0405159620
State Running
Type Secondary Storage VM
Zone sc-test-cs42-zone
Public IP Address 192.168.0.161
Private IP Address 192.168.0.151
Link Local IP Address 169.254.1.207
Host sc-test-kvm-03
Gateway 192.168.0.1
Created 06 Jun 2016 00:31:04
Active Sessions
Both system VMs seem to be on "Running" state but I can't verify what
template version they are using, whether they are already on 4.6/4.8
template or still using 4.2 template, are you able to advise how can I
tell? The SSVM creation date seem to be after we performed the upgrade
but the CPVM created date is before the upgrade.
How to ensure that all systemVMs using the new version template? Shall
I destroy the system VMs so new ones will be created, or shall I
re-run the /usr/bin/cloudstack-sysvmadm again in full?
Looking forward to your reply, thank you.
Cheers.
-ip-
On Tue, Jun 7, 2016 at 7:22 AM, Kirk Kosinski

Post by Kirk Kosinski
Hi, what is the status of the CPVM and SSVM in the UI? Were they
really upgraded successfully?
Also, are there any errors in the management-server.log?
www.shapeblue.com
-----Original Message-----
Sent: Monday, June 06, 2016 9:51 AM
Subject: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue
Hi,
I tried upgrading our CloudStack 4.2.0 on our test environment to 4.8.0,
and I am having issues on VR upgrade. I followed the upgrade

instructions
http://cloudstack-release-notes.readthedocs.io/ja/stable/upgrade/upgrade-4.2.html

Post by Cloud List

Post by Kirk Kosinski
On the last step to restart all the system VMs and VRs, it seems that

the

Post by Cloud List

Post by Kirk Kosinski
cloudstack-sysvmadm script managed to restart the system VMs (SSVM and
CPVM) but not the VR.
# nohup cloudstack-sysvmadm -d IPaddress -u cloud -p password -a > sysvm.log 2>&1 &
====
nohup: ignoring input
/usr/bin/cloudstack-sysvmadm: line 21: /etc/rc.d/init.d/functions: No
such file or directory
Stopping and starting 1 secondary storage vm(s)...
Done stopping and starting secondary storage vm(s)
Stopping and starting 1 console proxy vm(s)...
Done stopping and starting console proxy vm(s) .
Stopping and starting 1 running routing vm(s)...
*ERROR: Failed to restart domainRouter with id 5*
Done restarting router(s).
====
====
[2016.06.05-22.28.40] Stopping and starting 1 secondary storage vm(s)...
[2016.06.05-22.28.40] INFO: Stopping secondary storage vm with id 3
[2016.06.05-22.28.46] INFO: Starting secondary storage vm with id 3
[2016.06.05-23.29.00] ERROR: Failed to startSystemVm id=3; jobId is
121c5d89-745a-4e8e-9f96-0b348ef42abf
[2016.06.05-23.29.00] ERROR: Failed to start secondary storage vm with

id

Post by Cloud List

Post by Kirk Kosinski
3 [2016.06.05-23.29.00] ERROR: Failed to start secondary storage vm

with id

Post by Cloud List

Post by Kirk Kosinski
3 [2016.06.05-23.29.00] Done stopping and starting secondary storage

vm(s).

Post by Cloud List

Post by Kirk Kosinski
[2016.06.05-23.29.00] Stopping and starting 1 console proxy vm(s)...
[2016.06.05-23.29.00] INFO: Stopping console proxy with id 2
[2016.06.05-23.29.06] INFO: Starting console proxy vm with id 2
[2016.06.05-23.30.49] Done stopping and starting console proxy vm(s) .
[2016.06.05-23.30.49] Stopping and starting 1 running routing vm(s)...
[2016.06.05-23.30.49] INFO: Restarting router with id 5
[2016.06.06-00.30.59] ERROR: Failed to restart domainRouter with id 5;
jobId 4a3f681e-c4a6-4856-9d42-0d3b79bffdae
[2016.06.06-00.31.04] Done restarting router(s).
====
Initially I noted that the VR remains in 4.2.0 version (instead of the
expected 4.8.0 version), tried to destroy the router so it'll be

recreated

Post by Cloud List

Post by Kirk Kosinski
but it doesn't get created successfully.
Tried to run the cloudstack-sysvmadm manually with the -r option to
restart only the VR, and the problem still persists.
At the moment, the version of the VR is "UNKNOWN" and the "Require Upgrade"
state is "Yes".
Any advice how to make the VR back up and running?
Looking forward to your reply, thank you.
Cheers.
-ip-

[Sam Ceylani]

So check your firewall on your management server, test that connection. I believe there are some new stuff introduced
that changed how routers and system vms connect to MS. But you said SSVM connects fine right? What Hypervisor are you using?
Because I believe they changed system vm connections to ssh on xen for system vms etc. and there was even a port number that I know
I put in to my firewall which wasn’t there before...

-----Original Message-----
From: Cloud List [mailto:cloud-***@sg.or.id]
Sent: Wednesday, June 8, 2016 12:37 AM
To: ***@cloudstack.apache.org
Subject: Re: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue

Hi Sam,

Thanks for your reply.

My SSVM and CPVM seems to work fine -- although I can't verify if they are really using the new template or not, but my VR doesn't work even after I destroy the VR and restart the network with clean-up option. So the VR is never able to be started up after the upgrade. It kept on "Starting" state and agent.log file shows it keeps on trying to connect to the link local IP but never able to connect.

I tested the upgrade on a staging environment with just 1 VR, I have reverted back the setup to 4.2 and now re-doing the upgrade again for the second attempt, will see if it works. My production has around 10+ networks so I need to ensure this works before I initiate the upgrade on my production server.

Thank you.

Post by Sam Ceylani
Same thing happened to me, upgrading from 4.5.2 to 4.8 and I had to
restart network with clean-up option checked for each network which
rebuilds the router VM. If your SSVM works And if you can test it then
I don’t see any problem rebuilding network(router vm) with new
template which took about 30 minutes for about
20 networks in total...may not be an option for large
installations...how many routers you are running?
-----Original Message-----
Sent: Wednesday, June 8, 2016 12:23 AM
Subject: Re: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue
Dear all,
Could it be the VR is not able to use the new systemVM template
because we registered the template as Routing = NO? Will it resolve
the problem if during upgrade, we register the template as Routing = YES?
Based on the upgrade instruction at
http://cloudstack-release-notes.readthedocs.io/ja/stable/upgrade/upgra
de-4.2.html , it says that routing option should be "no", is this
correct? Since the systemVM template will also be used by VR?
Any advice is appreciated.
Looking forward to your reply, thank you.
Cheers.
-ip-

Post by Cloud List
Hi Kirk,
Thanks for your reply.
Name v-2-VM
ID fc6d23bf-58ac-4641-af9b-506d1dc46df3
State Running
Type Console Proxy VM
Zone sc-test-cs42-zone
Public IP Address 192.168.0.162
Private IP Address 192.168.0.160
Link Local IP Address 169.254.3.147
Host sc-test-kvm-03
Gateway 192.168.0.1
Created 31 May 2016 15:24:20
Active Sessions 0
Name s-29-VM
ID d26f54f8-1c1f-4f3d-9862-fc0405159620
State Running
Type Secondary Storage VM
Zone sc-test-cs42-zone
Public IP Address 192.168.0.161
Private IP Address 192.168.0.151
Link Local IP Address 169.254.1.207
Host sc-test-kvm-03
Gateway 192.168.0.1
Created 06 Jun 2016 00:31:04
Active Sessions
Both system VMs seem to be on "Running" state but I can't verify
what template version they are using, whether they are already on
4.6/4.8 template or still using 4.2 template, are you able to advise
how can I tell? The SSVM creation date seem to be after we performed
the upgrade but the CPVM created date is before the upgrade.
How to ensure that all systemVMs using the new version template?
Shall I destroy the system VMs so new ones will be created, or shall
I re-run the /usr/bin/cloudstack-sysvmadm again in full?
Looking forward to your reply, thank you.
Cheers.
-ip-
On Tue, Jun 7, 2016 at 7:22 AM, Kirk Kosinski

Post by Kirk Kosinski
Hi, what is the status of the CPVM and SSVM in the UI? Were they
really upgraded successfully?
Also, are there any errors in the management-server.log?
www.shapeblue.com
-----Original Message-----
Sent: Monday, June 06, 2016 9:51 AM
Subject: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue
Hi,
I tried upgrading our CloudStack 4.2.0 on our test environment to
4.8.0, and I am having issues on VR upgrade. I followed the upgrade

instructions
http://cloudstack-release-notes.readthedocs.io/ja/stable/upgrade/upgra
de-4.2.html

Post by Cloud List

Post by Kirk Kosinski
On the last step to restart all the system VMs and VRs, it seems that

the

Post by Cloud List

Post by Kirk Kosinski
cloudstack-sysvmadm script managed to restart the system VMs (SSVM and
CPVM) but not the VR.
# nohup cloudstack-sysvmadm -d IPaddress -u cloud -p password -a > sysvm.log 2>&1 &
====
nohup: ignoring input
No such file or directory
Stopping and starting 1 secondary storage vm(s)...
Done stopping and starting secondary storage vm(s)
Stopping and starting 1 console proxy vm(s)...
Done stopping and starting console proxy vm(s) .
Stopping and starting 1 running routing vm(s)...
*ERROR: Failed to restart domainRouter with id 5*
Done restarting router(s).
====
====
[2016.06.05-22.28.40] Stopping and starting 1 secondary storage vm(s)...
[2016.06.05-22.28.40] INFO: Stopping secondary storage vm with id 3
[2016.06.05-22.28.46] INFO: Starting secondary storage vm with id 3
[2016.06.05-23.29.00] ERROR: Failed to startSystemVm id=3; jobId is
121c5d89-745a-4e8e-9f96-0b348ef42abf
[2016.06.05-23.29.00] ERROR: Failed to start secondary storage vm with

id

Post by Cloud List

Post by Kirk Kosinski
3 [2016.06.05-23.29.00] ERROR: Failed to start secondary storage vm

with id

Post by Cloud List

Post by Kirk Kosinski
3 [2016.06.05-23.29.00] Done stopping and starting secondary storage

vm(s).

Post by Cloud List

Post by Kirk Kosinski
[2016.06.05-23.29.00] Stopping and starting 1 console proxy vm(s)...
[2016.06.05-23.29.00] INFO: Stopping console proxy with id 2
[2016.06.05-23.29.06] INFO: Starting console proxy vm with id 2
[2016.06.05-23.30.49] Done stopping and starting console proxy vm(s) .
[2016.06.05-23.30.49] Stopping and starting 1 running routing vm(s)...
[2016.06.05-23.30.49] INFO: Restarting router with id 5
[2016.06.06-00.30.59] ERROR: Failed to restart domainRouter with id
5; jobId 4a3f681e-c4a6-4856-9d42-0d3b79bffdae
[2016.06.06-00.31.04] Done restarting router(s).
====
Initially I noted that the VR remains in 4.2.0 version (instead of
the expected 4.8.0 version), tried to destroy the router so it'll
be

recreated

Post by Cloud List

Post by Kirk Kosinski
but it doesn't get created successfully.
Tried to run the cloudstack-sysvmadm manually with the -r option to
restart only the VR, and the problem still persists.
At the moment, the version of the VR is "UNKNOWN" and the "Require Upgrade"
state is "Yes".
Any advice how to make the VR back up and running?
Looking forward to your reply, thank you.
Cheers.
-ip-

[Cloud List]

Hi Sam,

Thanks for your reply.

Connection to MS is OK, since SSVM works fine. Problem seems to be isolated
to the VR. I am using KVM hypervisor. Testing the upgrade again now and
will update the result.

Cheers.

-ip-

Post by Sam Ceylani
So check your firewall on your management server, test that connection. I
believe there are some new stuff introduced
that changed how routers and system vms connect to MS. But you said SSVM
connects fine right? What Hypervisor are you using?
Because I believe they changed system vm connections to ssh on xen for
system vms etc. and there was even a port number that I know
I put in to my firewall which wasn’t there before...
-----Original Message-----
Sent: Wednesday, June 8, 2016 12:37 AM
Subject: Re: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue
Hi Sam,
Thanks for your reply.
My SSVM and CPVM seems to work fine -- although I can't verify if they are
really using the new template or not, but my VR doesn't work even after I
destroy the VR and restart the network with clean-up option. So the VR is
never able to be started up after the upgrade. It kept on "Starting" state
and agent.log file shows it keeps on trying to connect to the link local IP
but never able to connect.
I tested the upgrade on a staging environment with just 1 VR, I have
reverted back the setup to 4.2 and now re-doing the upgrade again for the
second attempt, will see if it works. My production has around 10+ networks
so I need to ensure this works before I initiate the upgrade on my
production server.
Thank you.

Post by Sam Ceylani
Same thing happened to me, upgrading from 4.5.2 to 4.8 and I had to
restart network with clean-up option checked for each network which
rebuilds the router VM. If your SSVM works And if you can test it then
I don’t see any problem rebuilding network(router vm) with new
template which took about 30 minutes for about
20 networks in total...may not be an option for large
installations...how many routers you are running?
-----Original Message-----
Sent: Wednesday, June 8, 2016 12:23 AM
Subject: Re: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue
Dear all,
Could it be the VR is not able to use the new systemVM template
because we registered the template as Routing = NO? Will it resolve
the problem if during upgrade, we register the template as Routing = YES?
Based on the upgrade instruction at
http://cloudstack-release-notes.readthedocs.io/ja/stable/upgrade/upgra
de-4.2.html , it says that routing option should be "no", is this
correct? Since the systemVM template will also be used by VR?
Any advice is appreciated.
Looking forward to your reply, thank you.
Cheers.
-ip-

Post by Cloud List
Hi Kirk,
Thanks for your reply.
Name v-2-VM
ID fc6d23bf-58ac-4641-af9b-506d1dc46df3
State Running
Type Console Proxy VM
Zone sc-test-cs42-zone
Public IP Address 192.168.0.162
Private IP Address 192.168.0.160
Link Local IP Address 169.254.3.147
Host sc-test-kvm-03
Gateway 192.168.0.1
Created 31 May 2016 15:24:20
Active Sessions 0
Name s-29-VM
ID d26f54f8-1c1f-4f3d-9862-fc0405159620
State Running
Type Secondary Storage VM
Zone sc-test-cs42-zone
Public IP Address 192.168.0.161
Private IP Address 192.168.0.151
Link Local IP Address 169.254.1.207
Host sc-test-kvm-03
Gateway 192.168.0.1
Created 06 Jun 2016 00:31:04
Active Sessions
Both system VMs seem to be on "Running" state but I can't verify
what template version they are using, whether they are already on
4.6/4.8 template or still using 4.2 template, are you able to advise
how can I tell? The SSVM creation date seem to be after we performed
the upgrade but the CPVM created date is before the upgrade.
How to ensure that all systemVMs using the new version template?
Shall I destroy the system VMs so new ones will be created, or shall
I re-run the /usr/bin/cloudstack-sysvmadm again in full?
Looking forward to your reply, thank you.
Cheers.
-ip-
On Tue, Jun 7, 2016 at 7:22 AM, Kirk Kosinski

Post by Kirk Kosinski
Hi, what is the status of the CPVM and SSVM in the UI? Were they
really upgraded successfully?
Also, are there any errors in the management-server.log?
www.shapeblue.com
-----Original Message-----
Sent: Monday, June 06, 2016 9:51 AM
Subject: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue
Hi,
I tried upgrading our CloudStack 4.2.0 on our test environment to
4.8.0, and I am having issues on VR upgrade. I followed the upgrade

instructions
http://cloudstack-release-notes.readthedocs.io/ja/stable/upgrade/upgra
de-4.2.html

Post by Cloud List

Post by Kirk Kosinski
On the last step to restart all the system VMs and VRs, it seems that

the

Post by Cloud List

Post by Kirk Kosinski
cloudstack-sysvmadm script managed to restart the system VMs (SSVM and
CPVM) but not the VR.
# nohup cloudstack-sysvmadm -d IPaddress -u cloud -p password -a >
sysvm.log 2>&1 &
====
nohup: ignoring input
No such file or directory
Stopping and starting 1 secondary storage vm(s)...
Done stopping and starting secondary storage vm(s)
Stopping and starting 1 console proxy vm(s)...
Done stopping and starting console proxy vm(s) .
Stopping and starting 1 running routing vm(s)...
*ERROR: Failed to restart domainRouter with id 5*
Done restarting router(s).
====
====
[2016.06.05-22.28.40] Stopping and starting 1 secondary storage

vm(s)...

Post by Sam Ceylani

Post by Cloud List

Post by Kirk Kosinski
[2016.06.05-22.28.40] INFO: Stopping secondary storage vm with id 3
[2016.06.05-22.28.46] INFO: Starting secondary storage vm with id 3
[2016.06.05-23.29.00] ERROR: Failed to startSystemVm id=3; jobId is
121c5d89-745a-4e8e-9f96-0b348ef42abf
[2016.06.05-23.29.00] ERROR: Failed to start secondary storage vm with

id

Post by Cloud List

Post by Kirk Kosinski
3 [2016.06.05-23.29.00] ERROR: Failed to start secondary storage vm

with id

Post by Cloud List

Post by Kirk Kosinski
3 [2016.06.05-23.29.00] Done stopping and starting secondary storage

vm(s).

Post by Cloud List

Post by Kirk Kosinski
[2016.06.05-23.29.00] Stopping and starting 1 console proxy vm(s)...
[2016.06.05-23.29.00] INFO: Stopping console proxy with id 2
[2016.06.05-23.29.06] INFO: Starting console proxy vm with id 2
[2016.06.05-23.30.49] Done stopping and starting console proxy vm(s) .
[2016.06.05-23.30.49] Stopping and starting 1 running routing vm(s)...
[2016.06.05-23.30.49] INFO: Restarting router with id 5
[2016.06.06-00.30.59] ERROR: Failed to restart domainRouter with id
5; jobId 4a3f681e-c4a6-4856-9d42-0d3b79bffdae
[2016.06.06-00.31.04] Done restarting router(s).
====
Initially I noted that the VR remains in 4.2.0 version (instead of
the expected 4.8.0 version), tried to destroy the router so it'll
be

recreated

Post by Cloud List

Post by Kirk Kosinski
but it doesn't get created successfully.
Tried to run the cloudstack-sysvmadm manually with the -r option to
restart only the VR, and the problem still persists.
At the moment, the version of the VR is "UNKNOWN" and the "Require Upgrade"
state is "Yes".
Any advice how to make the VR back up and running?
Looking forward to your reply, thank you.
Cheers.
-ip-

[Sanjeev Neelarapu]

Check /etc/cloudstack-releases file inside system vms to know the system template version after system vm upgrades.
Check MS log to know the reasons for VR upgrade failure.

Best Regards,
Sanjeev N
Chief Product Engineer, Accelerite
Off: [phone] | EMail: ***@accelerite.com


-----Original Message-----
From: Cloud List [mailto:cloud-***@sg.or.id]
Sent: Wednesday, June 08, 2016 10:19 AM
To: ***@cloudstack.apache.org
Subject: Re: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue

Hi Sam,

Thanks for your reply.

Connection to MS is OK, since SSVM works fine. Problem seems to be isolated to the VR. I am using KVM hypervisor. Testing the upgrade again now and will update the result.

Cheers.

-ip-

Post by Sam Ceylani
So check your firewall on your management server, test that
connection. I believe there are some new stuff introduced that changed
how routers and system vms connect to MS. But you said SSVM connects
fine right? What Hypervisor are you using?
Because I believe they changed system vm connections to ssh on xen for
system vms etc. and there was even a port number that I know I put in
to my firewall which wasn’t there before...
-----Original Message-----
Sent: Wednesday, June 8, 2016 12:37 AM
Subject: Re: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue
Hi Sam,
Thanks for your reply.
My SSVM and CPVM seems to work fine -- although I can't verify if they
are really using the new template or not, but my VR doesn't work even
after I destroy the VR and restart the network with clean-up option.
So the VR is never able to be started up after the upgrade. It kept on
"Starting" state and agent.log file shows it keeps on trying to
connect to the link local IP but never able to connect.
I tested the upgrade on a staging environment with just 1 VR, I have
reverted back the setup to 4.2 and now re-doing the upgrade again for
the second attempt, will see if it works. My production has around 10+
networks so I need to ensure this works before I initiate the upgrade
on my production server.
Thank you.

Post by Sam Ceylani
Same thing happened to me, upgrading from 4.5.2 to 4.8 and I had to
restart network with clean-up option checked for each network which
rebuilds the router VM. If your SSVM works And if you can test it
then I don’t see any problem rebuilding network(router vm) with new
template which took about 30 minutes for about
20 networks in total...may not be an option for large
installations...how many routers you are running?
-----Original Message-----
Sent: Wednesday, June 8, 2016 12:23 AM
Subject: Re: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue
Dear all,
Could it be the VR is not able to use the new systemVM template
because we registered the template as Routing = NO? Will it resolve
the problem if during upgrade, we register the template as Routing = YES?
Based on the upgrade instruction at
http://cloudstack-release-notes.readthedocs.io/ja/stable/upgrade/upg
ra de-4.2.html , it says that routing option should be "no", is this
correct? Since the systemVM template will also be used by VR?
Any advice is appreciated.
Looking forward to your reply, thank you.
Cheers.
-ip-

Post by Cloud List
Hi Kirk,
Thanks for your reply.
Name v-2-VM
ID fc6d23bf-58ac-4641-af9b-506d1dc46df3
State Running
Type Console Proxy VM
Zone sc-test-cs42-zone
Public IP Address 192.168.0.162
Private IP Address 192.168.0.160
Link Local IP Address 169.254.3.147
Host sc-test-kvm-03
Gateway 192.168.0.1
Created 31 May 2016 15:24:20
Active Sessions 0
Name s-29-VM
ID d26f54f8-1c1f-4f3d-9862-fc0405159620
State Running
Type Secondary Storage VM
Zone sc-test-cs42-zone
Public IP Address 192.168.0.161
Private IP Address 192.168.0.151
Link Local IP Address 169.254.1.207
Host sc-test-kvm-03
Gateway 192.168.0.1
Created 06 Jun 2016 00:31:04
Active Sessions
Both system VMs seem to be on "Running" state but I can't verify
what template version they are using, whether they are already on
4.6/4.8 template or still using 4.2 template, are you able to
advise how can I tell? The SSVM creation date seem to be after we
performed the upgrade but the CPVM created date is before the upgrade.
How to ensure that all systemVMs using the new version template?
Shall I destroy the system VMs so new ones will be created, or
shall I re-run the /usr/bin/cloudstack-sysvmadm again in full?
Looking forward to your reply, thank you.
Cheers.
-ip-
On Tue, Jun 7, 2016 at 7:22 AM, Kirk Kosinski

Post by Kirk Kosinski
Hi, what is the status of the CPVM and SSVM in the UI? Were they
really upgraded successfully?
Also, are there any errors in the management-server.log?
www.shapeblue.com
-----Original Message-----
Sent: Monday, June 06, 2016 9:51 AM
Subject: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue
Hi,
I tried upgrading our CloudStack 4.2.0 on our test environment to
4.8.0, and I am having issues on VR upgrade. I followed the upgrade

instructions
http://cloudstack-release-notes.readthedocs.io/ja/stable/upgrade/upg
ra
de-4.2.html

Post by Cloud List

Post by Kirk Kosinski
On the last step to restart all the system VMs and VRs, it seems that

the

Post by Cloud List

Post by Kirk Kosinski
cloudstack-sysvmadm script managed to restart the system VMs (SSVM and
CPVM) but not the VR.
# nohup cloudstack-sysvmadm -d IPaddress -u cloud -p password -a

Post by Cloud List
sysvm.log 2>&1 &

====
nohup: ignoring input
No such file or directory
Stopping and starting 1 secondary storage vm(s)...
Done stopping and starting secondary storage vm(s)
Stopping and starting 1 console proxy vm(s)...
Done stopping and starting console proxy vm(s) .
Stopping and starting 1 running routing vm(s)...
*ERROR: Failed to restart domainRouter with id 5*
Done restarting router(s).
====
====
[2016.06.05-22.28.40] Stopping and starting 1 secondary storage

vm(s)...

Post by Sam Ceylani

Post by Cloud List

Post by Kirk Kosinski
[2016.06.05-22.28.40] INFO: Stopping secondary storage vm with id
3 [2016.06.05-22.28.46] INFO: Starting secondary storage vm with
id 3 [2016.06.05-23.29.00] ERROR: Failed to startSystemVm id=3;
jobId is 121c5d89-745a-4e8e-9f96-0b348ef42abf
[2016.06.05-23.29.00] ERROR: Failed to start secondary storage vm with

id

Post by Cloud List

Post by Kirk Kosinski
3 [2016.06.05-23.29.00] ERROR: Failed to start secondary storage vm

with id

Post by Cloud List

Post by Kirk Kosinski
3 [2016.06.05-23.29.00] Done stopping and starting secondary storage

vm(s).

Post by Cloud List

Post by Kirk Kosinski
[2016.06.05-23.29.00] Stopping and starting 1 console proxy vm(s)...
[2016.06.05-23.29.00] INFO: Stopping console proxy with id 2
[2016.06.05-23.29.06] INFO: Starting console proxy vm with id 2
[2016.06.05-23.30.49] Done stopping and starting console proxy vm(s) .
[2016.06.05-23.30.49] Stopping and starting 1 running routing vm(s)...
[2016.06.05-23.30.49] INFO: Restarting router with id 5
[2016.06.06-00.30.59] ERROR: Failed to restart domainRouter with
id 5; jobId 4a3f681e-c4a6-4856-9d42-0d3b79bffdae
[2016.06.06-00.31.04] Done restarting router(s).
====
Initially I noted that the VR remains in 4.2.0 version (instead
of the expected 4.8.0 version), tried to destroy the router so
it'll be

recreated

Post by Cloud List

Post by Kirk Kosinski
but it doesn't get created successfully.
Tried to run the cloudstack-sysvmadm manually with the -r option
to restart only the VR, and the problem still persists.
At the moment, the version of the VR is "UNKNOWN" and the
"Require Upgrade"
state is "Yes".
Any advice how to make the VR back up and running?
Looking forward to your reply, thank you.
Cheers.
-ip-

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected

[Indra Pramana]

Thanks Sanjeev,

I have re-done the upgrade again and after some other hiccups along the
way, finally managed to perform the upgrade and all the system VMs are now
updated successfully, including the VR. Not too sure what caused the VR
upgrade failed on the first upgrade attempt, though. And since I couldn't
find the solution, the only way was to revert back to version 4.2
(re-install CloudStack version 4.2, restore the DB) and restart the upgrade
process again.

Cheers.


On Wed, Jun 8, 2016 at 1:19 PM, Sanjeev Neelarapu <

Post by Sanjeev Neelarapu
Check /etc/cloudstack-releases file inside system vms to know the system
template version after system vm upgrades.
Check MS log to know the reasons for VR upgrade failure.
Best Regards,
Sanjeev N
Chief Product Engineer, Accelerite
-----Original Message-----
Sent: Wednesday, June 08, 2016 10:19 AM
Subject: Re: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue
Hi Sam,
Thanks for your reply.
Connection to MS is OK, since SSVM works fine. Problem seems to be
isolated to the VR. I am using KVM hypervisor. Testing the upgrade again
now and will update the result.
Cheers.
-ip-

Post by Sam Ceylani
So check your firewall on your management server, test that
connection. I believe there are some new stuff introduced that changed
how routers and system vms connect to MS. But you said SSVM connects
fine right? What Hypervisor are you using?
Because I believe they changed system vm connections to ssh on xen for
system vms etc. and there was even a port number that I know I put in
to my firewall which wasn’t there before...
-----Original Message-----
Sent: Wednesday, June 8, 2016 12:37 AM
Subject: Re: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue
Hi Sam,
Thanks for your reply.
My SSVM and CPVM seems to work fine -- although I can't verify if they
are really using the new template or not, but my VR doesn't work even
after I destroy the VR and restart the network with clean-up option.
So the VR is never able to be started up after the upgrade. It kept on
"Starting" state and agent.log file shows it keeps on trying to
connect to the link local IP but never able to connect.
I tested the upgrade on a staging environment with just 1 VR, I have
reverted back the setup to 4.2 and now re-doing the upgrade again for
the second attempt, will see if it works. My production has around 10+
networks so I need to ensure this works before I initiate the upgrade
on my production server.
Thank you.

Post by Sam Ceylani
Same thing happened to me, upgrading from 4.5.2 to 4.8 and I had to
restart network with clean-up option checked for each network which
rebuilds the router VM. If your SSVM works And if you can test it
then I don’t see any problem rebuilding network(router vm) with new
template which took about 30 minutes for about
20 networks in total...may not be an option for large
installations...how many routers you are running?
-----Original Message-----
Sent: Wednesday, June 8, 2016 12:23 AM
Subject: Re: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue
Dear all,
Could it be the VR is not able to use the new systemVM template
because we registered the template as Routing = NO? Will it resolve
the problem if during upgrade, we register the template as Routing =

YES?

Post by Sam Ceylani

Post by Sam Ceylani
Based on the upgrade instruction at
http://cloudstack-release-notes.readthedocs.io/ja/stable/upgrade/upg
ra de-4.2.html , it says that routing option should be "no", is this
correct? Since the systemVM template will also be used by VR?
Any advice is appreciated.
Looking forward to your reply, thank you.
Cheers.
-ip-

Post by Cloud List
Hi Kirk,
Thanks for your reply.
Name v-2-VM
ID fc6d23bf-58ac-4641-af9b-506d1dc46df3
State Running
Type Console Proxy VM
Zone sc-test-cs42-zone
Public IP Address 192.168.0.162
Private IP Address 192.168.0.160
Link Local IP Address 169.254.3.147
Host sc-test-kvm-03
Gateway 192.168.0.1
Created 31 May 2016 15:24:20
Active Sessions 0
Name s-29-VM
ID d26f54f8-1c1f-4f3d-9862-fc0405159620
State Running
Type Secondary Storage VM
Zone sc-test-cs42-zone
Public IP Address 192.168.0.161
Private IP Address 192.168.0.151
Link Local IP Address 169.254.1.207
Host sc-test-kvm-03
Gateway 192.168.0.1
Created 06 Jun 2016 00:31:04
Active Sessions
Both system VMs seem to be on "Running" state but I can't verify
what template version they are using, whether they are already on
4.6/4.8 template or still using 4.2 template, are you able to
advise how can I tell? The SSVM creation date seem to be after we
performed the upgrade but the CPVM created date is before the

upgrade.

Post by Sam Ceylani

Post by Sam Ceylani

Post by Cloud List
How to ensure that all systemVMs using the new version template?
Shall I destroy the system VMs so new ones will be created, or
shall I re-run the /usr/bin/cloudstack-sysvmadm again in full?
Looking forward to your reply, thank you.
Cheers.
-ip-
On Tue, Jun 7, 2016 at 7:22 AM, Kirk Kosinski

Post by Kirk Kosinski
Hi, what is the status of the CPVM and SSVM in the UI? Were they
really upgraded successfully?
Also, are there any errors in the management-server.log?
www.shapeblue.com
-----Original Message-----
Sent: Monday, June 06, 2016 9:51 AM
Subject: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue
Hi,
I tried upgrading our CloudStack 4.2.0 on our test environment to
4.8.0, and I am having issues on VR upgrade. I followed the upgrade

instructions
http://cloudstack-release-notes.readthedocs.io/ja/stable/upgrade/upg
ra
de-4.2.html

Post by Cloud List

Post by Kirk Kosinski
On the last step to restart all the system VMs and VRs, it seems that

the

Post by Cloud List

Post by Kirk Kosinski
cloudstack-sysvmadm script managed to restart the system VMs (SSVM and
CPVM) but not the VR.
# nohup cloudstack-sysvmadm -d IPaddress -u cloud -p password -a

Post by Cloud List
sysvm.log 2>&1 &

====
nohup: ignoring input
No such file or directory
Stopping and starting 1 secondary storage vm(s)...
Done stopping and starting secondary storage vm(s)
Stopping and starting 1 console proxy vm(s)...
Done stopping and starting console proxy vm(s) .
Stopping and starting 1 running routing vm(s)...
*ERROR: Failed to restart domainRouter with id 5*
Done restarting router(s).
====
====
[2016.06.05-22.28.40] Stopping and starting 1 secondary storage

vm(s)...

Post by Sam Ceylani

Post by Cloud List

Post by Kirk Kosinski
[2016.06.05-22.28.40] INFO: Stopping secondary storage vm with id
3 [2016.06.05-22.28.46] INFO: Starting secondary storage vm with
id 3 [2016.06.05-23.29.00] ERROR: Failed to startSystemVm id=3;
jobId is 121c5d89-745a-4e8e-9f96-0b348ef42abf
[2016.06.05-23.29.00] ERROR: Failed to start secondary storage vm with

id

Post by Cloud List

Post by Kirk Kosinski
3 [2016.06.05-23.29.00] ERROR: Failed to start secondary storage vm

with id

Post by Cloud List

Post by Kirk Kosinski
3 [2016.06.05-23.29.00] Done stopping and starting secondary storage

vm(s).

Post by Cloud List

Post by Kirk Kosinski
[2016.06.05-23.29.00] Stopping and starting 1 console proxy vm(s)...
[2016.06.05-23.29.00] INFO: Stopping console proxy with id 2
[2016.06.05-23.29.06] INFO: Starting console proxy vm with id 2
[2016.06.05-23.30.49] Done stopping and starting console proxy

vm(s) .

Post by Sam Ceylani

Post by Sam Ceylani

Post by Cloud List

Post by Kirk Kosinski
[2016.06.05-23.30.49] Stopping and starting 1 running routing

vm(s)...

Post by Sam Ceylani

Post by Sam Ceylani

Post by Cloud List

Post by Kirk Kosinski
[2016.06.05-23.30.49] INFO: Restarting router with id 5
[2016.06.06-00.30.59] ERROR: Failed to restart domainRouter with
id 5; jobId 4a3f681e-c4a6-4856-9d42-0d3b79bffdae
[2016.06.06-00.31.04] Done restarting router(s).
====
Initially I noted that the VR remains in 4.2.0 version (instead
of the expected 4.8.0 version), tried to destroy the router so
it'll be

recreated

Post by Cloud List

Post by Kirk Kosinski
but it doesn't get created successfully.
Tried to run the cloudstack-sysvmadm manually with the -r option
to restart only the VR, and the problem still persists.
At the moment, the version of the VR is "UNKNOWN" and the "Require Upgrade"
state is "Yes".
Any advice how to make the VR back up and running?
Looking forward to your reply, thank you.
Cheers.
-ip-

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is
the property of Accelerite, a Persistent Systems business. It is intended
only for the use of the individual or entity to which it is addressed. If
you are not the intended recipient, you are not authorized to read, retain,
copy, print, distribute or use this message. If you have received this
communication in error, please notify the sender and delete all copies of
this message. Accelerite, a Persistent Systems business does not accept any
liability for virus infected mails.