IMO. this would take modifying the XenCenter code to trigger running the SrRepairAction routine if it sees multiple, comma-separated entries in the target input field to make a new iSCSi conenction. That would be an ugly way to fix things, but might at least work.
If you try this and get one host conencted via XenCEnter and then do an SR Repair operation via XenCenter does that still work to get all the hosts conencted?
________________________________
From: xs-devel-***@lists.xenserver.org [xs-devel-***@lists.xenserver.org] on behalf of Vinícius Ferrão [***@ferrao.eti.br]
Sent: Friday, February 20, 2015 8:49 PM
To: xs-***@lists.xenserver.org
Subject: Re: [xs-devel] Switchless shared iSCSI Network with /30 links
Any ideia on how to test this? I tried to create the pool for the first time using the string with all the six IP addresses, but XenCenter just fails without and error message, just an single X appears. The only thing I can provide at this moment is the /var/log/xensource.log; /var/log/audit.log; /var/log/messages and /var/log/SMlog file during the IQN/LUN phase, it's on the end of this message. The /var/log/SMlog appears to be the most valuable one.
I'm really dedicated on helping solving this issue, since I'm delaying the implementation of this farm just to keep the test scenario live!
Thanks,
PS: The logs...
*****/var/log/xensource.log*****
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|Async.SR.create R:b933cd75031b|dispatcher] spawning a new thread to handle the current task (trackid=8c8347a3f363fd40a9bd41eb1705050f)
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|audit] SR.create: name label = '__gui__'
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi] SR.create name_label=__gui__ sm_config=[ ]
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|Async.SR.create R:b933cd75031b|storage_access] SR d403d0fc-4a14-7f99-fe92-dfc6301ed764 will be implemented by /services/SM/lvmoiscsi in VM OpaqueRef:47f2c383-de3e-730e-9836-d652d5eae646
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|mux] register SR d403d0fc-4a14-7f99-fe92-dfc6301ed764 (currently-registered = [ f31651b7-4219-dfe5-d871-4748c05e900c, ee47042b-76bf-5f62-2879-0457db8e892d, 12945887-3593-6c24-0b77-2efd62949415, d403d0fc-4a14-7f99-fe92-dfc6301ed764, 551b026c-6b94-5e08-ae67-3c76c15b8b44 ])
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|dummytaskhelper] task SR.create D:ebc6f0ef8713 created by task R:b933cd75031b
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|sm] SM lvmoiscsi sr_create sr=OpaqueRef:383eb943-faa8-1946-f169-6626b1996fe7 size=0
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Session.create trackid=70acc8c1d9618c45c2f07860af213730 pool=false uname= originator= is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|mscgen] xapi=>xapi [label="(XML)"];
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14339 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid D:7e7d99be1ccd created by task D:208b3f98f095
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|mscgen] smapiv2=>smapiv1 [label="sr_create"];
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|13421 INET 0.0.0.0:80|dispatch:task.add_to_other_config D:e234520429b7|api_effect] task.add_to_other_config
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|xenops] Event on VM 6dd2a08f-43a3-4f0c-b6c4-7e040829dc6a; resident_here = true
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|mscgen] xapi=>xenops [label="VM.exists"];
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|xenops] kernel is not in the whitelist: ignoring
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|mscgen] xapi=>xapi [label="(XML)"];
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14343 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:event.from D:ff5a707a4074 created by task D:4b65ef16de2c
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|13421 INET 0.0.0.0:80|dispatch:task.add_to_other_config D:7d79581bbf17|api_effect] task.add_to_other_config
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14344 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config D:27e54f4f8e11 created by task D:ebc6f0ef8713
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14345 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config D:fac3f1373b6e created by task D:ebc6f0ef8713
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14346 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config D:1713380916eb created by task D:ebc6f0ef8713
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|13421 INET 0.0.0.0:80|dispatch:task.add_to_other_config D:538ce8692691|api_effect] task.add_to_other_config
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Raised at sm_exec.ml:212.7-69 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Session.destroy trackid=70acc8c1d9618c45c2f07860af213730
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|backtrace] Raised at pervasiveext.ml:26.22-25 -> server_helpers.ml:76.11-23
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|dispatcher] Server_helpers.exec exception_handler: Got exception SR_BACKEND_FAILURE_96: [ ; The request is missing or has an incorrect target IQN parameter; <?xml version="1.0" ?> <iscsi-target-iqns> <TGT> <Index> 0 </Index> <IPAddress> 192.168.10.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 1 </Index> <IPAddress> 192.168.11.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 2 </Index> <IPAddress> 192.168.20.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 3 </Index> <IPAddress> 192.168.21.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 4 </Index> <IPAddress> 192.168.30.1 </IPAddress>
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|dispatcher] Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|backtrace] Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Raised at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|backtrace] Raised at storage_access.ml:161.15-44 -> server_helpers.ml:76.11-23
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|dispatcher] Server_helpers.exec exception_handler: Got exception INTERNAL_ERROR: [ Storage_interface.Backend_error(_) ]
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|dispatcher] Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|backtrace] Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|xapi] Raised at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|SR.create D:ebc6f0ef8713|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi] Raised at pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [error|xenserver1|14336|Async.SR.create R:b933cd75031b|storage_access] Re-raising as SR_BACKEND_FAILURE_96 [ ; The request is missing or has an incorrect target IQN parameter; <?xml version="1.0" ?> <iscsi-target-iqns> <TGT> <Index> 0 </Index> <IPAddress> 192.168.10.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 1 </Index> <IPAddress> 192.168.11.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 2 </Index> <IPAddress> 192.168.20.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 3 </Index> <IPAddress> 192.168.21.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 4 </Index> <IPAddress> 192.168.30.1 </IPAddress> <TargetIQN>iqn.2015-01.b
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|backtrace] Raised at xapi_sr.ml:225.9-10 -> server.ml:21898.82-267 -> rbac.ml:229.16-23
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|backtrace] Raised at rbac.ml:238.10-15 -> server_helpers.ml:79.11-41
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|dispatcher] Server_helpers.exec exception_handler: Got exception SR_BACKEND_FAILURE_96: [ ; The request is missing or has an incorrect target IQN parameter; <?xml version="1.0" ?> <iscsi-target-iqns> <TGT> <Index> 0 </Index> <IPAddress> 192.168.10.1 </IPAddress> <TargetIQN>iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 1 </Index> <IPAddress> 192.168.11.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 2 </Index> <IPAddress>192.168.20.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 3 </Index> <IPAddress> 192.168.21.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 4 </Index> <IPAddress> 192.168.30.1 </IPAdd
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|dispatcher] Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|backtrace] Raised at hashtbl.ml:97.23-32 -> debug.ml:144.36-65
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi] Raised at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336|Async.SR.create R:b933cd75031b|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336||xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:01 xenserver1 xapi: [debug|xenserver1|14336||xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|Async.SR.create R:202cf9100203|dispatcher] spawning a new thread to handle the current task (trackid=8c8347a3f363fd40a9bd41eb1705050f)
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|audit] SR.create: name label = '__gui__'
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi] SR.create name_label=__gui__ sm_config=[ ]
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|Async.SR.create R:202cf9100203|storage_access] SR 207535ee-c14c-7b48-3869-96318c1c7877 will be implemented by /services/SM/lvmoiscsi in VM OpaqueRef:47f2c383-de3e-730e-9836-d652d5eae646
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|mux] register SR 207535ee-c14c-7b48-3869-96318c1c7877 (currently-registered = [ f31651b7-4219-dfe5-d871-4748c05e900c, ee47042b-76bf-5f62-2879-0457db8e892d, 207535ee-c14c-7b48-3869-96318c1c7877, 12945887-3593-6c24-0b77-2efd62949415, d403d0fc-4a14-7f99-fe92-dfc6301ed764, 551b026c-6b94-5e08-ae67-3c76c15b8b44 ])
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|dummytaskhelper] task SR.create D:7e6802e5fed5 created by task R:202cf9100203
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|sm] SM lvmoiscsi sr_create sr=OpaqueRef:5591eb47-1a35-9321-6aa4-bd8beb748cbf size=0
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Session.create trackid=836c7784c6fd5a42124fb4bd64aa2cd5 pool=false uname= originator= is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|mscgen] xapi=>xapi [label="(XML)"];
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14357 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid D:c14a1f2ccf30 created by task D:4a4af7d945d9
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|mscgen] smapiv2=>smapiv1 [label="sr_create"];
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|13421 INET 0.0.0.0:80|dispatch:task.add_to_other_config D:6f63db465b85|api_effect] task.add_to_other_config
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|xenops] Event on VM 6dd2a08f-43a3-4f0c-b6c4-7e040829dc6a; resident_here = true
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|mscgen] xapi=>xenops [label="VM.exists"];
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|xenops] kernel is not in the whitelist: ignoring
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|429|xapi events D:4b65ef16de2c|mscgen] xapi=>xapi [label="(XML)"];
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14359 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:event.from D:dd4a82303df3 created by task D:4b65ef16de2c
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|13421 INET 0.0.0.0:80|dispatch:task.add_to_other_config D:454990b8d01f|api_effect] task.add_to_other_config
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14360 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config D:513472992344 created by task D:7e6802e5fed5
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14361 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config D:4cb6c60153d2 created by task D:7e6802e5fed5
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|14362 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:host.get_other_config D:901d21392528 created by task D:7e6802e5fed5
Feb 21 01:41:06 xenserver1 xapi: [debug|xenserver1|13421 INET 0.0.0.0:80|dispatch:task.add_to_other_config D:f7f97c5c1dec|api_effect] task.add_to_other_config
Feb 21 01:41:17 xenserver1 xapi: [debug|xenserver1|14365 INET 0.0.0.0:80|Get RRD updates. D:8f822f0bc4c4|xapi] hand_over_connection GET /rrd_updates to /var/xapi/xcp-rrdd.forwarded
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Raised at forkhelpers.ml:187.30-76 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Raised at sm_exec.ml:190.10-100 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Raised at pervasiveext.ml:26.22-25 -> sm_exec.ml:173.23-1023 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [ info|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Session.destroy trackid=836c7784c6fd5a42124fb4bd64aa2cd5
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|backtrace] Raised at pervasiveext.ml:26.22-25 -> server_helpers.ml:76.11-23
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|dispatcher] Server_helpers.exec exception_handler: Got exception SR_BACKEND_FAILURE: [ non-zero exit; ; Traceback (most recent call last): File "/opt/xensource/sm/LVMoISCSISR", line 582, in ? SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File "/opt/xensource/sm/SRCommand.py", line 343, in run sr = driver(cmd, cmd.sr_uuid) File "/opt/xensource/sm/SR.py", line 142, in __init__ self.load(sr_uuid) File "/opt/xensource/sm/LVMoISCSISR", line 156, in load self.iscsi = self.iscsiSRs[0] IndexError: list index out of range ]
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|dispatcher] Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|backtrace] Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Raised at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|backtrace] Raised at storage_access.ml:161.15-44 -> server_helpers.ml:76.11-23
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|dispatcher] Server_helpers.exec exception_handler: Got exception INTERNAL_ERROR: [ Storage_interface.Backend_error(_) ]
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|dispatcher] Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|backtrace] Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|xapi] Raised at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|SR.create D:7e6802e5fed5|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi] Raised at pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [error|xenserver1|14352|Async.SR.create R:202cf9100203|storage_access] Re-raising as SR_BACKEND_FAILURE [ non-zero exit; ; Traceback (most recent call last): File "/opt/xensource/sm/LVMoISCSISR", line 582, in ? SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File "/opt/xensource/sm/SRCommand.py", line 343, in run sr = driver(cmd, cmd.sr_uuid) File "/opt/xensource/sm/SR.py", line 142, in __init__ self.load(sr_uuid) File "/opt/xensource/sm/LVMoISCSISR", line 156, in load self.iscsi = self.iscsiSRs[0] IndexError: list index out of range ]
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|backtrace] Raised at xapi_sr.ml:225.9-10 -> server.ml:21898.82-267 -> rbac.ml:229.16-23
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|backtrace] Raised at rbac.ml:238.10-15 -> server_helpers.ml:79.11-41
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|dispatcher] Server_helpers.exec exception_handler: Got exception SR_BACKEND_FAILURE: [ non-zero exit; ; Traceback (most recent call last): File "/opt/xensource/sm/LVMoISCSISR", line 582, in ? SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File "/opt/xensource/sm/SRCommand.py", line 343, in run sr = driver(cmd, cmd.sr_uuid) File "/opt/xensource/sm/SR.py", line 142, in __init__ self.load(sr_uuid) File "/opt/xensource/sm/LVMoISCSISR", line 156, in load self.iscsi = self.iscsiSRs[0] IndexError: list index out of range ]
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|dispatcher] Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|backtrace] Raised at hashtbl.ml:93.19-28 -> debug.ml:144.36-65
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi] Raised at server_helpers.ml:94.14-15 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352|Async.SR.create R:202cf9100203|xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352||xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:35 xenserver1 xapi: [debug|xenserver1|14352||xapi] Raised at pervasiveext.ml:26.22-25 -> pervasiveext.ml:22.2-9
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0 monitor|main|rrdd_stats] system stats: MemTotal: 3959600 KiB; MemFree: 3262140 KiB; Buffered: 33832 KiB; Cached: 291048 KiB; SwapTotal: 524284 KiB; SwapFree: 524284 KiB
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0 monitor|main|rrdd_stats] Clock drift: -0
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0 monitor|main|rrdd_stats] xcp-rrdd stats (n = 1): size: 155052 KiB; rss: 8368 KiB; data: 71620 KiB; stack: 136 KiB
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0 monitor|main|rrdd_stats] xapi stats (n = 2): size: 604328 KiB; rss: 101844 KiB; data: 451340 KiB; stack: 272 KiB
Feb 21 01:41:36 xenserver1 xcp-rrdd: [debug|xenserver1|0 monitor|main|rrdd_stats] xenopsd stats (n = 1): size: 278248 KiB; rss: 8308 KiB; data: 193692 KiB; stack: 136 KiB
***** /var/log/audit.log *****
Feb 21 01:41:01 xenserver1 xapi: [20150221T03:41:01.707Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.add_to_other_config D:ee0292d7107f|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.add_to_other_config' (('self' 'Async.SR.create' '577acdd7-2b8c-b177-bf45-f8422c4ed7ca' 'OpaqueRef:b933cd75-031b-6ff0-5a0a-2341507af648')))
Feb 21 01:41:01 xenserver1 xapi: [20150221T03:41:01.747Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.add_to_other_config D:ef6d521270fb|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.add_to_other_config' (('self' 'Async.SR.create' '577acdd7-2b8c-b177-bf45-f8422c4ed7ca' 'OpaqueRef:b933cd75-031b-6ff0-5a0a-2341507af648') ('value' '' 'e1aaff51-0da2-2745-cf25-ab034aea8a25' 'OpaqueRef:85f25443-719e-105d-35aa-a1c1043fdd8e')))
Feb 21 01:41:01 xenserver1 xapi: [20150221T03:41:01.787Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.add_to_other_config D:43071305947a|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.add_to_other_config' (('self' 'Async.SR.create' '577acdd7-2b8c-b177-bf45-f8422c4ed7ca' 'OpaqueRef:b933cd75-031b-6ff0-5a0a-2341507af648')))
Feb 21 01:41:01 xenserver1 xapi: [20150221T03:41:01.885Z|audit|xenserver1|14336|Async.SR.create R:b933cd75031b|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'ERROR:SR_BACKEND_FAILURE_96: [ ; The request is missing or has an incorrect target IQN parameter; <?xml version=\"1.0\" ?> <iscsi-target-iqns> <TGT> <Index> 0 </Index> <IPAddress> 192.168.10.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 1 </Index> <IPAddress> 192.168.11.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 2 </Index> <IPAddress> 192.168.20.1 </IPAddress> <TargetIQN>iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 3 </Index> <IPAddress> 192.168.21.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index>
Feb 21 01:41:02 xenserver1 xapi: [20150221T03:41:02.883Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.destroy D:5804968b2ed6|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.destroy' (('self' 'Async.SR.create' '577acdd7-2b8c-b177-bf45-f8422c4ed7ca' 'OpaqueRef:b933cd75-031b-6ff0-5a0a-2341507af648')))
Feb 21 01:41:06 xenserver1 xapi: [20150221T03:41:06.598Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.add_to_other_config D:a09ea039fcd0|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.add_to_other_config' (('self' 'Async.SR.create' 'a0d8e973-9995-fb15-532e-f36ac5184c28' 'OpaqueRef:202cf910-0203-a943-9054-8d0e25a2b4a7')))
Feb 21 01:41:06 xenserver1 xapi: [20150221T03:41:06.638Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.add_to_other_config D:b4dd3f0d946a|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.add_to_other_config' (('self' 'Async.SR.create' 'a0d8e973-9995-fb15-532e-f36ac5184c28' 'OpaqueRef:202cf910-0203-a943-9054-8d0e25a2b4a7') ('value' '' 'e1aaff51-0da2-2745-cf25-ab034aea8a25' 'OpaqueRef:85f25443-719e-105d-35aa-a1c1043fdd8e')))
Feb 21 01:41:06 xenserver1 xapi: [20150221T03:41:06.679Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.add_to_other_config D:9b0a16a91401|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.add_to_other_config' (('self' 'Async.SR.create' 'a0d8e973-9995-fb15-532e-f36ac5184c28' 'OpaqueRef:202cf910-0203-a943-9054-8d0e25a2b4a7')))
Feb 21 01:41:17 xenserver1 xapi: [20150221T03:41:18.001Z|audit|xenserver1|14365 INET 0.0.0.0:80|handler:http/rrd_updates D:3d1bf1a8b00c|audit] ('trackid=bf92de30818eb264b5673ffe734ad369' 'LOCAL_SUPERUSER' '' 'ALLOWED' 'OK' 'HTTP' 'http/rrd_updates' ())
Feb 21 01:41:35 xenserver1 xapi: [20150221T03:41:35.155Z|audit|xenserver1|14352|Async.SR.create R:202cf9100203|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'ERROR:SR_BACKEND_FAILURE: [ non-zero exit; ; Traceback (most recent call last): File \"/opt/xensource/sm/LVMoISCSISR\", line 582, in ? SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File \"/opt/xensource/sm/SRCommand.py\", line 343, in run sr = driver(cmd, cmd.sr_uuid) File \"/opt/xensource/sm/SR.py\", line 142, in __init__ self.load(sr_uuid) File \"/opt/xensource/sm/LVMoISCSISR\", line 156, in load self.iscsi = self.iscsiSRs[0] IndexError: list index out of range ]' 'API' 'SR.create' (('host' 'xenserver1.local.iq.ufrj.br<http://xenserver1.local.iq.ufrj.br>' 'c8927969-b7bf-4a0f-9725-035550381d9c' 'OpaqueRef:03239cf1-76cf-9c86-b927-56ea27cb6b94') ('name_label' '__gui__' '' '') ('name_description' 'SHOULD NEVER BE CREATED' '' '')))
Feb 21 01:41:35 xenserver1 xapi: [20150221T03:41:35.707Z|audit|xenserver1|13421 INET 0.0.0.0:80|task.destroy D:7ff3ff7db08a|audit] ('trackid=8c8347a3f363fd40a9bd41eb1705050f' 'LOCAL_SUPERUSER' 'root' 'ALLOWED' 'OK' 'API' 'task.destroy' (('self' 'Async.SR.create' 'a0d8e973-9995-fb15-532e-f36ac5184c28' 'OpaqueRef:202cf910-0203-a943-9054-8d0e25a2b4a7')))
***** /var/log/messages *****
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|Async.SR.create R:b933cd75031b|dispatcher] spawning a new thread to handle the current task (trackid=8c8347a3f363fd40a9bd41eb1705050f)
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|Async.SR.create R:b933cd75031b|storage_access] SR d403d0fc-4a14-7f99-fe92-dfc6301ed764 will be implemented by /services/SM/lvmoiscsi in VM OpaqueRef:47f2c383-de3e-730e-9836-d652d5eae646
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Session.create trackid=70acc8c1d9618c45c2f07860af213730 pool=false uname= originator= is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Feb 21 01:41:01 xenserver1 xapi: [ info|xenserver1|14336|sm_exec D:208b3f98f095|xapi] Session.destroy trackid=70acc8c1d9618c45c2f07860af213730
Feb 21 01:41:01 xenserver1 xapi: [error|xenserver1|14336|Async.SR.create R:b933cd75031b|storage_access] Re-raising as SR_BACKEND_FAILURE_96 [ ; The request is missing or has an incorrect target IQN parameter; <?xml version="1.0" ?> <iscsi-target-iqns> <TGT> <Index> 0 </Index> <IPAddress> 192.168.10.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 1 </Index> <IPAddress> 192.168.11.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 2 </Index> <IPAddress> 192.168.20.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 3 </Index> <IPAddress> 192.168.21.1 </IPAddress> <TargetIQN> iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 </TargetIQN> </TGT> <TGT> <Index> 4 </Index> <IPAddress> 192.168.30.1 </IPAddress> <TargetIQN>iqn.2015-01.b
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|Async.SR.create R:202cf9100203|dispatcher] spawning a new thread to handle the current task (trackid=8c8347a3f363fd40a9bd41eb1705050f)
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|Async.SR.create R:202cf9100203|storage_access] SR 207535ee-c14c-7b48-3869-96318c1c7877 will be implemented by /services/SM/lvmoiscsi in VM OpaqueRef:47f2c383-de3e-730e-9836-d652d5eae646
Feb 21 01:41:06 xenserver1 xapi: [ info|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Session.create trackid=836c7784c6fd5a42124fb4bd64aa2cd5 pool=false uname= originator= is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05155|bond|INFO|bond bond0: shift 21kB of load (with hash 35) from eth1 to eth0 (now carrying 233kB and 94kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05156|bond|INFO|bond bond0: shift 224kB of load (with hash 39) from eth1 to eth0 (now carrying 9kB and 318kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05157|bond|INFO|bond bond0: shift 0kB of load (with hash 9) from eth0 to eth1 (now carrying 318kB and 9kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05158|bond|INFO|bond bond0: shift 1kB of load (with hash 12) from eth0 to eth1 (now carrying 317kB and 10kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05159|bond|INFO|bond bond0: shift 1kB of load (with hash 14) from eth0 to eth1 (now carrying 316kB and 11kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05160|bond|INFO|bond bond0: shift 0kB of load (with hash 20) from eth0 to eth1 (now carrying 315kB and 11kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05161|bond|INFO|bond bond0: shift 0kB of load (with hash 21) from eth0 to eth1 (now carrying 315kB and 12kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05162|bond|INFO|bond bond0: shift 0kB of load (with hash 25) from eth0 to eth1 (now carrying 315kB and 12kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05163|bond|INFO|bond bond0: shift 15kB of load (with hash 46) from eth0 to eth1 (now carrying 300kB and 27kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05164|bond|INFO|bond bond0: shift 2kB of load (with hash 70) from eth0 to eth1 (now carrying 297kB and 30kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05165|bond|INFO|bond bond0: shift 1kB of load (with hash 84) from eth0 to eth1 (now carrying 295kB and 32kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05166|bond|INFO|bond bond0: shift 0kB of load (with hash 94) from eth0 to eth1 (now carrying 295kB and 32kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05167|bond|INFO|bond bond0: shift 2kB of load (with hash 112) from eth0 to eth1 (now carrying 293kB and 34kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05168|bond|INFO|bond bond0: shift 28kB of load (with hash 118) from eth0 to eth1 (now carrying 265kB and 62kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05169|bond|INFO|bond bond0: shift 5kB of load (with hash 160) from eth0 to eth1 (now carrying 259kB and 68kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05170|bond|INFO|bond bond0: shift 5kB of load (with hash 171) from eth0 to eth1 (now carrying 254kB and 73kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05171|bond|INFO|bond bond0: shift 3kB of load (with hash 206) from eth0 to eth1 (now carrying 251kB and 76kB load, respectively)
Feb 21 01:41:10 xenserver1 ovs-vswitchd: ovs|05172|bond|INFO|bond bond0: shift 2kB of load (with hash 254) from eth0 to eth1 (now carrying 249kB and 78kB load, respectively)
Feb 21 01:41:28 xenserver1 ovs-vswitchd: ovs|05173|ofproto|INFO|xapi0: 29 flow_mods in the last 59 s (29 adds)
Feb 21 01:41:35 xenserver1 fe: 2498 (/opt/xensource/sm/LVMoISCSISR <methodCall><methodName>sr_create</methodName><...) exited with code 1
Feb 21 01:41:35 xenserver1 xapi: [ info|xenserver1|14352|sm_exec D:4a4af7d945d9|xapi] Session.destroy trackid=836c7784c6fd5a42124fb4bd64aa2cd5
Feb 21 01:41:35 xenserver1 xapi: [error|xenserver1|14352|Async.SR.create R:202cf9100203|storage_access] Re-raising as SR_BACKEND_FAILURE [ non-zero exit; ; Traceback (most recent call last): File "/opt/xensource/sm/LVMoISCSISR", line 582, in ? SRCommand.run(LVHDoISCSISR, DRIVER_INFO) File "/opt/xensource/sm/SRCommand.py", line 343, in run sr = driver(cmd, cmd.sr_uuid) File "/opt/xensource/sm/SR.py", line 142, in __init__ self.load(sr_uuid) File "/opt/xensource/sm/LVMoISCSISR", line 156, in load self.iscsi = self.iscsiSRs[0] IndexError: list index out of range ]
Feb 21 01:41:56 xenserver1 xenstored: D3.14 rm attr/eth0
Feb 21 01:41:56 xenserver1 xenstored: D3 write attr/eth0/ip 10.7.0.243
Feb 21 01:41:56 xenserver1 xenstored: D3 write attr/eth0/ipv6/0/addr fe80::484b:acff:fe91:5b19
***** /var/log/SMlog *****
[***@xenserver1 log]# grep "Feb 21 01:41" /var/log/SMlog
Feb 21 01:41:01 xenserver1 SM: [2444] _testHost: Testing host/port: 192.168.10.1,3260
Feb 21 01:41:01 xenserver1 SM: [2444] lock: opening lock file /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: acquired /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: released /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: closed /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: opening lock file /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: acquired /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: released /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] lock: closed /var/lock/sm/iscsiadm/running
Feb 21 01:41:01 xenserver1 SM: [2444] Raising exception [96, The request is missing or has an incorrect target IQN parameter]
Feb 21 01:41:06 xenserver1 SM: [2498] lock: opening lock file /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: acquired /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: released /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] Raising exception [202, General backend error [opterr=rc: 21, stdout: , stderr: iscsiadm: No active sessions.
Feb 21 01:41:06 xenserver1 SM: [2498] ]]
Feb 21 01:41:06 xenserver1 SM: [2498] ['iscsiadm', '-m', 'session'] failed with (u'General backend error [opterr=rc: 21, stdout: , stderr: iscsiadm: No active sessions.\n]',)
Feb 21 01:41:06 xenserver1 SM: [2498] lock: closed /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] ['ls', '/sys/class/scsi_host', '-1', '--color=never']
Feb 21 01:41:06 xenserver1 SM: [2498] pread SUCCESS
Feb 21 01:41:06 xenserver1 SM: [2498] []
Feb 21 01:41:06 xenserver1 SM: [2498] PATHDICT: key 192.168.10.1:3260: {'path': '/dev/iscsi/*/192.168.10.1:3260', 'ipaddr': '192.168.10.1', 'port': 3260L}
Feb 21 01:41:06 xenserver1 SM: [2498] lock: opening lock file /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: acquired /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: released /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] lock: closed /var/lock/sm/iscsiadm/running
Feb 21 01:41:06 xenserver1 SM: [2498] Discovery for IP 192.168.10.1 returned [('192.168.10.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.11.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.20.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.21.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.30.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.31.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1')]
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.10.1,3260
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.11.1,3260
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.20.1,3260
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.20.1) - (113, 'No route to host')
Feb 21 01:41:06 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:06 xenserver1 SM: [2498] Target Not reachable: (192.168.20.1:3260)
Feb 21 01:41:06 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.21.1,3260
Feb 21 01:41:07 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.21.1) - (113, 'No route to host')
Feb 21 01:41:07 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:07 xenserver1 SM: [2498] Target Not reachable: (192.168.21.1:3260)
Feb 21 01:41:07 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.30.1,3260
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.30.1) - timed out
Feb 21 01:41:09 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:09 xenserver1 SM: [2498] Target Not reachable: (192.168.30.1:3260)
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.31.1,3260
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.31.1) - (113, 'No route to host')
Feb 21 01:41:09 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:09 xenserver1 SM: [2498] Target Not reachable: (192.168.31.1:3260)
Feb 21 01:41:09 xenserver1 SM: [2498] lock: opening lock file /var/lock/sm/iscsiadm/running
Feb 21 01:41:09 xenserver1 SM: [2498] lock: acquired /var/lock/sm/iscsiadm/running
Feb 21 01:41:09 xenserver1 SM: [2498] lock: released /var/lock/sm/iscsiadm/running
Feb 21 01:41:09 xenserver1 SM: [2498] lock: closed /var/lock/sm/iscsiadm/running
Feb 21 01:41:09 xenserver1 SM: [2498] Discovery for IP 192.168.11.1 returned [('192.168.10.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.11.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.20.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.21.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.30.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1'), ('192.168.31.1:3260', '2', 'iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1')]
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.10.1,3260
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.11.1,3260
Feb 21 01:41:09 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.20.1,3260
Feb 21 01:41:11 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.20.1) - timed out
Feb 21 01:41:11 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:11 xenserver1 SM: [2498] Target Not reachable: (192.168.20.1:3260)
Feb 21 01:41:11 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.21.1,3260
Feb 21 01:41:11 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.21.1) - (113, 'No route to host')
Feb 21 01:41:11 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:11 xenserver1 SM: [2498] Target Not reachable: (192.168.21.1:3260)
Feb 21 01:41:11 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.30.1,3260
Feb 21 01:41:12 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.30.1) - (113, 'No route to host')
Feb 21 01:41:12 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:12 xenserver1 SM: [2498] Target Not reachable: (192.168.30.1:3260)
Feb 21 01:41:12 xenserver1 SM: [2498] _testHost: Testing host/port: 192.168.31.1,3260
Feb 21 01:41:14 xenserver1 SM: [2498] _testHost: Connect failed after 2 seconds (192.168.31.1) - timed out
Feb 21 01:41:14 xenserver1 SM: [2498] Raising exception [141, Unable to connect to ISCSI service on target]
Feb 21 01:41:14 xenserver1 SM: [2498] Target Not reachable: (192.168.31.1:3260)
Feb 21 01:41:15 xenserver1 SM: [2498] lock: opening lock file /var/lock/sm/iscsiadm/running
Feb 21 01:41:15 xenserver1 SM: [2498] lock: acquired /var/lock/sm/iscsiadm/running
Feb 21 01:41:35 xenserver1 SM: [2498] lock: released /var/lock/sm/iscsiadm/running
Feb 21 01:41:35 xenserver1 SM: [2498] Raising exception [202, General backend error [opterr=rc: 21, stdout: , stderr: iscsiadm: cannot make connection to 192.168.20.1: No route to host
Feb 21 01:41:35 xenserver1 SM: [2498] iscsiadm: cannot make connection to 192.168.20.1: No route to host
Feb 21 01:41:35 xenserver1 last message repeated 3 times
Feb 21 01:41:35 xenserver1 SM: [2498] iscsiadm: connect to 192.168.20.1 timed out
Feb 21 01:41:35 xenserver1 SM: [2498] iscsiadm: connection login retries (reopen_max) 5 exceeded
Feb 21 01:41:35 xenserver1 SM: [2498] iscsiadm: No portals found
Feb 21 01:41:35 xenserver1 SM: [2498] ]]
Feb 21 01:41:35 xenserver1 SM: [2498] Raising exception [68, ISCSI login failed, verify CHAP credentials]
Feb 21 01:41:35 xenserver1 SM: [2498] lock: closed /var/lock/sm/iscsiadm/running
Feb 21 01:41:35 xenserver1 SM: [2498] ***** LVHDoISCSISR.load: EXCEPTION SR.SROSError, ISCSI login failed, verify CHAP credentials
Feb 21 01:41:35 xenserver1 SM: [2498] File "/opt/xensource/sm/LVMoISCSISR", line 119, in load
Feb 21 01:41:35 xenserver1 SM: [2498] map = iscsilib.discovery(tgt_ip,iscsi.port,iscsi.chapuser,iscsi.chappassword,targetIQN=IQN)
Feb 21 01:41:35 xenserver1 SM: [2498] File "/opt/xensource/sm/iscsilib.py", line 156, in discovery
Feb 21 01:41:35 xenserver1 SM: [2498] raise xs_errors.XenError('ISCSILogin')
Feb 21 01:41:35 xenserver1 SM: [2498] File "/opt/xensource/sm/xs_errors.py", line 52, in __init__
Feb 21 01:41:35 xenserver1 SM: [2498] raise SR.SROSError(errorcode, errormessage)
Feb 21 01:41:35 xenserver1 SM: [2498]
Feb 21 01:41:54 xenserver1 updatempppathd: [7617] The garbage collection routine returned: 0
On Feb 20, 2015, at 9:14 PM, Tobias Kreidl <***@nau.edu<mailto:***@nau.edu>> wrote:
When I mentioned running the SrRepairAction procedure, I meant in conjunction with what is not happening in XenCenter as a follow-up to fix things.
So then, it would seem the piece remaining to resolve is what is missing in XenCenter that is not triggered. My guess is that it can perhaps only perform a connection using one IP address (or wildcarded address) and hence apparently doesn't understand what to do with anything involving a multiSession/multihome list?
It would be a useful diagnostic to see what is parsed out and processed if you pass in the whole multihome list of
192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260
to XenCenter and see what it ends up trying to do with it.
-=Tobias
On 2/20/2015 3:53 PM, Vinícius Ferrão wrote:
Guys, another important thing that I've observed.
All the iscsiadm and multipath commands are useless. I've attached the same storage just issuing a specific PBD command from the Pool Master. So this simplifies the problem a lot.
I just used this command:
xe pbd-create \
sr-uuid=f8d481b8-be10-922e-b875-91c4d73d341d \
host-uuid=44955863-f4a3-4770-bd7a-f4342dd7925a \
device-config:multiSession="192.168.30.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.31.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" \
device-config:target=192.168.30.1,192.168.31.1 \
device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 \
device-config:targetIQN=* \
device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 \
device-config:port=3260
The device-config attributes are sufficient to create all the connection.
Thanks,
Vinicius.
On Feb 20, 2015, at 8:37 PM, Tobias Kreidl <***@nau.edu><mailto:***@nau.edu> wrote:
It may be possible to at least temporarily get this work by forcing the SrRepairAction procedure to be run.
I didn't have time yet, but wanted to compare what it does in sequence vs. what is done by default in the xapi_sr
program to see what is different. As far as XenCenter is concerned, perhaps if multiple IP target addresses are entered in the dialog box (as opposed to a single one or a wildcard entry), then a different means could be triggered to ensure that each host does a proper PBD creation and plug-in for that new SR.
-=Tobias
On 2/20/2015 3:20 PM, Vinícius Ferrão wrote:
Dave,
I've added the "future" server to our pool. And happened exactly what you said. The configurations were cloned. But obviously it failed to clone the PDB's and the Network configuration for the iSCSI network.
IMHO there are two things that needs to be addressed:
1. The network configuration on added host should be "confirmed" to the user, and not done automatically, so the user can adapt is to specific needs.
2. After this the PDB creation should be done accordingly. If the network topology has changed by the user, so the PDBs must be created with this new topology.
In resume, I've three host at this moment on the pool. So I can do the HA tests once again, and I will report to the list the results. At this moment the only thing that failed is cloning the iSCSI network configurations and plugging the PDBs. Only the bounded NIC0+NIC1 was created on the added host.
Thanks in advance,
Vinícius.
On Feb 18, 2015, at 4:50 PM, Dave Scott <***@citrix.com><mailto:***@citrix.com> wrote:
On 18 Feb 2015, at 18:03, Tobias Kreidl <***@nau.edu><mailto:***@nau.edu> wrote:
Vinícius:
Regarding Point 3) I believe what is meant is that you should test having more than one SR that you are connecting to using the same IP addresses for the target array. You know it works fine to create one SR, so can you add additional ones the same way and have them all live together in harmony? :-)
As to the sr-create command, the host-uuid is an optional parameter and hence does not need to be named for the pool master or any other. The only required parameters are the name-label and the SR type. Have you tried that to see if the behavior is any different compared to when you do supply a host name?
I believe the underlying Xapi data model can express what you want, but there are 1 or 2 places where Xapi will helpfully create PBDs for you by cloning the master, which you just need to be aware of.
For example you should be able to use something like
$ xe sr-create .. host-uuid=<any host> shared=false
Setting "shared=false" initially should avoid Xapi 'helpfully' pre-creating identical PBDs for every host with the same configuration[1]. You should end up with an SR attached to a single host.
If you then set the SR to shared=true you should be able to create and plug the exact PBDs you want:
$ xe sr-param-set uuid= shared=true
$ xe pbd-create sr-uuid= host-uuid=
$ xe pbd-plug ...
$ xe pbd-create sr-uuid= host-uuid=
$ xe pbd-plug ...
[ as an aside, it's obviously not great that the PBDs you get depend on exactly when you declare the SR is shared. This has happened because the original design had all PBDs being distinct, and the notion of 'copying the master' was added relatively late ]
Whenever a new host is added to the pool, Xapi will clone the master's PBD for you- you would have to destroy it and re-create it. I suppose cloning the master's configuration is as good a guess as any in this situation :-)
Dave
[1] https://github.com/xapi-project/xen-api/blob/b6f28c52fd9726553968b410e6b8cced5401f385/ocaml/xapi/xapi_sr.ml#L209
-=Tobias
On 2/18/2015 10:47 AM, Vinícius Ferrão wrote:
Hello guys,
Answering about all the questions and issues. For those who aren't understanding the network topology, I've made a drawing of the solution. The image is here: https://www.dropbox.com/s/6zwerk6igcyqou3/UFRJ-IQ-VirtualPool.jpg?dl=0
In the drawing you can see the iSCSI Network made with point-to-point links. The IP's on the storage box are:
192.168.10.1
192.168.11.1
192.168.20.1
192.168.21.1
The convention to this numbering scheme was: 192.168.{XenServerNumber}{InterfaceNumber}.{Storage(1)/Host(2)}.
So on the host #1 I've two addresses and more two on host #2:
192.168.10.2 (XenServer #1 - Interface #0)
192.168.11.2 (XenServer #1 - Interface #1)
192.168.20.2 (XenServer #2 - Interface #0)
192.168.21.2 (XenServer #2 - Interface #1)
Hope this completely clarifies the architecture/topology of the iSCSI network.
Talking about the tests raised by Tim:
1. Already done this test. Tried three different tests:
Restarting both hosts on the same time. [OK]
Restarting the second host. [OK]
Restarting the pool master. [OK]
2. HA is working too. It fences only the damaged host. I even tried a VM Failover. HA happened without any problem and the lost VM has been restarted on other server. To do the HA test, I've power cycled the host, simulated a hardware crash scenario. A cool picture: https://www.dropbox.com/s/g7h7rljk8vh97rg/Screenshot%202015-02-18%2015.45.41.png?dl=0
3. I haven't understood this test. You are asking to create a new LUN and add a new Shared Repository? It's not clear what you're saying with "independent SR".
4. Storage XenMotion works too. I've moved a running VM from local storage (on the Host #2) to the iSCSI pool without any problem. The reverse process works too.
5. I will test this tomorrow, when I will got physical access to the servers (carnival holiday here in BR). But about this specific test, I can't see any problem with plain Xen Motion, since the data is copied over the management network and not the iSCSI network. But I can see a good point when dealing with Storage XenMotion.
Finally about the changes on XenCenter or XenServer/XAPI itself. One thing to prove this point is to create the SR via the CLI. If I would be able to do this, the problem should definitely be on XenCenter. As I said before, I've created a broken Software iSCSI SR via XenCenter and them mapped the iSCSI interfaces, enabled multipath and created the PBD over the CLI.
The iSCSI and Multipath should be done in the physical host. But the PBD creation can be done without any problem on the Pool Master.
To test this scenario I do require some help crafting the xe sr-create command, because I don't know exactly how this command is formed by XenCenter, and it requires some doubtful inputs:
[***@xenserver1 ~]# xe sr-create
content-type= host-uuid= physical-size= sm-config:
device-config: name-label= shared= type=
In this case sm-config and device-config are the problematic ones. Assuming that the host-uuid is always the pool master.
Thanks in advance,
Vinícius.
On Feb 18, 2015, at 2:36 PM, Tim Mackey <***@citrix.com><mailto:***@citrix.com> wrote:
I'll add in a few more tests....
1. Host restart. Does the normal PBD plug mechanism work with such a configuration
2. HA. While HA with two hosts isn't a supported configuration, in this model it might work properly. If you remove the network cables for a single host, does it fence, or do both hosts fence?
3. Multiple SRs. If you have multiple LUNs on the FreeNAS exported and then used as independent SRs, does everything work as it should?
4. Storage XenMotion. With a vdi on one SR presented with the FreeNAS, copy it to any another storage option
5. Motion during path failure. With one path down on a given host, does XenMotion and Storage XenMotion still function correctly?
For those who are questioning the value of such a configuration, this is in essence a "cluster in a box" solution. These configurations were very popular 10-15 years ago as highly available compute infrastructure was quite expensive back then. Today we can accomplish the same things we did back then using virtualization, but "cluster in a box" can always be valuable for those with limited IT skills or for smaller organizations wanting least cost with fewest points of failure.
-tim
From: xs-devel-***@lists.xenserver.org<mailto:xs-devel-***@lists.xenserver.org> [mailto:xs-devel-***@lists.xenserver.org] On Behalf Of Vinícius Ferrão
Sent: Monday, February 16, 2015 7:24 PM
To: Tobias Kreidl
Cc: xs-***@lists.xenserver.org<mailto:xs-***@lists.xenserver.org>
Subject: Re: [xs-devel] Switchless shared iSCSI Network with /30 links
Tobias,
I've done more tests about the SR creation over XenCenter.
All the process definitely occurs only on the pool master. Even if the specified networks are not part of the networks of the host. The connection is tried on the default route.
I requested a new Software iSCSI SR with the following address:
192.168.10.1,192.168.11.1,192.168.20.1,192.168.21.1
The target IQN is discovered without any problem, since the iSCSI Storage reports the LUNs on any interface.
But when trying to discover the LUN, the process only is done on the pool master. During the discovery process, I left a watch command on both XS to with "netstat -an | grep 192.168" and the results are:
On the pool master, there's a lot of things going on:
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 192.168.10.2:55870 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52654 192.168.11.1:3260 TIME_WAIT
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.11.2:52656 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
During the discovery, look at the 192.168.21.1 address being searched through the default route:
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
******************
tcp 0 1 10.7.0.101:56481 192.168.21.1:3260 SYN_SENT
******************
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
Even addresses that does not exist on the pool (since the IQN discovery process reports them):
tcp 0 0 10.7.0.101:443 192.168.172.1:58053 ESTABLISHED
tcp 0 0 192.168.11.2:52686 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:53266 192.168.11.2:36365 ESTABLISHED
tcp 0 0 192.168.10.2:57930 192.168.10.2:36365 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58058 ESTABLISHED
tcp 0 0 192.168.10.2:55886 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55900 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52684 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.10.2:55892 192.168.10.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:52679 192.168.11.1:3260 TIME_WAIT
tcp 0 0 192.168.11.2:36365 192.168.11.2:53266 ESTABLISHED
******************
tcp 0 1 10.7.0.101:52787 192.168.30.1:3260 SYN_SENT
******************
tcp 0 0 192.168.10.2:36365 192.168.10.2:57930 ESTABLISHED
tcp 0 0 192.168.10.2:55893 192.168.10.1:3260 TIME_WAIT
tcp 0 48 10.7.0.101:22 192.168.172.1:58064 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58055 ESTABLISHED
tcp 0 0 10.7.0.101:443 192.168.172.1:58054 ESTABLISHED
tcp 0 0 192.168.10.2:55887 192.168.10.1:3260 TIME_WAIT
udp 0 0 192.168.10.2:123 0.0.0.0:*
udp 0 0 192.168.11.2:123 0.0.0.0:*
If you're wondering the 192.168.30.1 is reported because in the future we will add another XS host to the pool. The point here is that XenCenter should not lookup for this network since it isn't specified during the Software iSCSI creation. On my understanding it should only lookup for the specified addresses.
And to finish the tests, the second XS host remains equally during all the SR creation phase:
[***@xenserver2 ~]# netstat -an | grep -i 192.168
tcp 0 48 10.7.0.102:22 192.168.172.1:58046 ESTABLISHED
udp 0 0 192.168.21.2:123 0.0.0.0:*
udp 0 0 192.168.20.2:123 0.0.0.0:*
As you said, perhaps propagating the initial connection through the host will solve the problem without modifying the XenCenter interface. A more sophisticated approach would be query the network-list over XAPI, retrieve the PIFs associated to those networks and them just check the IP addresses on the SR creation request and match them with the equivalent PIFs.
For example here is the output of the xe network-param-list to retrieve the associated PIF's and the xe pif-param-lst with the UUID's of the retrieved PIFs:
[***@xenserver1 ~]# xe network-param-list uuid=4dc936d7-e806-c69a-a292-78e0ed2d5faa
uuid ( RO) : 4dc936d7-e806-c69a-a292-78e0ed2d5faa
name-label ( RW): iSCSI #0
name-description ( RW):
VIF-uuids (SRO):
PIF-uuids (SRO): 3baa8436-feca-cd04-3173-5002d9ffc66b; 362d63cb-647a-465f-6b81-1b59cd3b1f5d
MTU ( RW): 9000
bridge ( RO): xenbr2
other-config (MRW): automatic: true
blobs ( RO):
tags (SRW):
default-locking-mode ( RW): unlocked
[***@xenserver1 ~]# xe pif-param-list uuid=3baa8436-feca-cd04-3173-5002d9ffc66b
uuid ( RO) : 3baa8436-feca-cd04-3173-5002d9ffc66b
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5c:64
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-master-of ( RO):
bond-slave-of ( RO): <not in database>
tunnel-access-PIF-of ( RO):
tunnel-transport-PIF-of ( RO):
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e
host-name-label ( RO): xenserver2.local.iq.ufrj.br<http://xenserver2.local.iq.ufrj.br>
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.20.2
netmask ( RO): 255.255.255.252
gateway ( RO):
IPv6-configuration-mode ( RO): None
IPv6 ( RO):
IPv6-gateway ( RO):
primary-address-type ( RO): IPv4
DNS ( RO):
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
[***@xenserver1 ~]# xe pif-param-list uuid=362d63cb-647a-465f-6b81-1b59cd3b1f5d
uuid ( RO) : 362d63cb-647a-465f-6b81-1b59cd3b1f5d
device ( RO): eth2
MAC ( RO): 40:f2:e9:f3:5a:1c
physical ( RO): true
managed ( RO): true
currently-attached ( RO): true
MTU ( RO): 9000
VLAN ( RO): -1
bond-master-of ( RO):
bond-slave-of ( RO): <not in database>
tunnel-access-PIF-of ( RO):
tunnel-transport-PIF-of ( RO):
management ( RO): false
network-uuid ( RO): 4dc936d7-e806-c69a-a292-78e0ed2d5faa
network-name-label ( RO): iSCSI #0
host-uuid ( RO): c8927969-b7bf-4a0f-9725-035550381d9c
host-name-label ( RO): xenserver1.local.iq.ufrj.br<http://xenserver1.local.iq.ufrj.br>
IP-configuration-mode ( RO): Static
IP ( RO): 192.168.10.2
netmask ( RO): 255.255.255.252
gateway ( RO):
IPv6-configuration-mode ( RO): None
IPv6 ( RO):
IPv6-gateway ( RO):
primary-address-type ( RO): IPv4
DNS ( RO):
properties (MRO): gro: on
io_read_kbs ( RO): 0.000
io_write_kbs ( RO): 0.000
carrier ( RO): true
vendor-id ( RO): 8086
vendor-name ( RO): Intel Corporation
device-id ( RO): 1521
device-name ( RO): I350 Gigabit Network Connection
speed ( RO): 1000 Mbit/s
duplex ( RO): full
disallow-unplug ( RW): true
pci-bus-path ( RO): 0000:06:00.2
other-config (MRW): management_purpose: iSCSI MPIO #0
So as you can see, we have the IP addresses and the network mask. Which is sufficient to an efficient and precise algorithm of SR creation.
I thinks this clarifies everything we discussed until now. A patch on XenCenter appears to be really viable.
Many thanks,
Vinicius.
On Feb 16, 2015, at 9:02 PM, Tobias Kreidl <***@nau.edu><mailto:***@nau.edu> wrote:
Yes, correct. The MD36xx looks like two separate controller units, each acting as a separate device. For a standalone server or some storage devices, that's right that you need a separate subnet for each connection.
Because a single connection isn't seeing the broadcast from the other connectors at the time of the initial connection from one host, the pool is unaware of the others. The "repair" operation causes this to be conducted from each host, and as I guessed, then fixes the SR connectivity for the entire pool. I am speculating that the request for a new connection of this type via XenCenter could perhaps be solved by propagating that initial connection process to the rest of the hosts in the pool and then rescan the SR.
-=Tobias
On 2/16/2015 2:20 PM, Vinícius Ferrão wrote:
Hi Tobias,
As for I know, the Dell MD36xx is a dual controller based storage. So it's basically "two computers". Considering this with the added knowledge that I've about TCP networking there's a golden rule that says: only one IP address on a given subnet on a single computer. So I can't use the same network on my case. That's why I use point-to-point links too.
Since I've only one machine acting as a storage server (It's a Supermicro Machine with 32 gigs of RAM running FreeNAS 9.3), I do need four separate networks. FreeNAS is FreeBSD based, and BSD enforces this rule by default. I can't have two IP addresses of the same subnet on same machine, even if I manually configure it.
If this wasn't bad TCP network practice, I would be able to configure the SR via XenCenter, since it will "find" the same network on the others hosts without problem.
About the XenCenter, it really appears to be only a missing function on the software. If I was a good programmer I would try to implement this. But this is beyond my knowledge... :(
Thanks once again,
Vinícius.
On Feb 16, 2015, at 6:53 PM, Tobias Kreidl <***@nau.edu><mailto:***@nau.edu> wrote:
No reason to necessarily use four separate subnets. You could have 192.168.10.1. and 20.1 on one iSCSI controller and 192.168.10.2 and 2.2 on the other controller (on, for example, an MD36xx that is standard). If however it is something like an MD32xx which recommends four separate subets, then of course, use four. One should follow the vendor's specifications.
I will respond in more detail later when I get a chance, Vinícius, but after a quick read, I agree with your findings from your other email. It would appear that the initial connection is indeed evidently a missing function in XenCenter and perhaps all that is needed to make this work "out of the box". As to the wildcard discovery, it's generally preferred as it allows the host to figure out the path on its own. The only reason I could see not using a wildcard discovery would be if there were any chance for overlaps and ambiguous connections.
Regards,
--Tobias
On 2/16/2015 1:38 PM, Vinícius Ferrão wrote:
Hello Dave,
In total I have four paths. Two in each host. I've made a drawning using ASCII characters, hope this can clarify the architecture:
+----------------+
| iSCSI Storage |
+--|1||2||3||4|--+
| | | |
| | | \--------------\ iSCSI Multipathed /30
| | \--------------\ | Network with two links
| | | |
+--|1||2|--------+ +--|1||2|--------+
| XenServer Host | | XenServer Host |
+----------------+ +----------------+
The Storage has four gigabit ethernet adapters with the following IP addresses:
192.168.10.1/30
192.168.11.1/30
192.168.20.1/30
192.168.21.1/30
On the first XenServer there are two interfaces with those IP's:
192.168.10.2/30
192.168.11.2/30
And on the second host the equivalent configuration:
192.168.20.2/30
192.168.21.2/30
That's why I'm using multipath. Because I do need MPIO to sum the two network interfaces per host.
To exemplify a little more heres a screenshot of the network configuration made on XenCenter:
https://www.dropbox.com/s/olfuxrh8d8nyheu/Screenshot%202015-02-16%2018.38.43.png?dl=0
Cheers,
Vinícius.
On Feb 15, 2015, at 12:22 PM, Dave Scott <***@citrix.com><mailto:***@citrix.com> wrote:
Hi,
On 15 Feb 2015, at 05:44, Vinícius Ferrão <***@ferrao.eti.br><mailto:***@ferrao.eti.br> wrote:
Hello Tobias,
Thanks for the reply, even to discourage me :)
I really can't understand why it isn't a good idea. VMware do it, they recommend, and they even sell a solution using DELL hardware with two machines and a shared storage running with 10GbE links. Point-ot-point networks, dual controllers on the storage side and two different network for iSCSI. They sell it at least here in Brazil. What I'm trying to do is achieve the same topology with XenServer and commodity hardware.
Why I want to do this? Because I do believe that XenServer is a better product than VMware vSphere.
Perhaps we aren't agreeing at some points due to different markets. For example, I've used Fibre Channel only one time in my life. And, believe it or not, it was a switchless configuration. And it worked... Wasn't for VM workload but was in a GRID Cluster with TORQUE/OpenPBS software. I do have the pictures of the topology, but I need to search for it. At that time, I wasn't the guy who created the solution, but I understood that it was a huge money saving, because the FC switches are a extremely expensive.
Leaving all those "political" points aside, let's talk technically :)
I've managed to achieve what I want. A lot of PBD hacking was necessary in the CLI, but I was comfortable with this. The following steps were necessary to reproduce this: https://www.dropbox.com/s/laigs26ic49omqv/Screenshot%202015-02-15%2003.02.05.png?dl=0
I'm really excited because it worked. So let's me explain what I've did.
First I started creating the SR via XenCenter, it failed as expected. But to bypass I just created the SR via XenCenter with the two Storage IP's of the Pool Master. So the process went OK, but it started broken, since the second XS host cannot see the IP addresses.
Using the xe sr-list and xe sr-params-list commands, I got the SR UUID created by XenCenter and the referenced PBDs. According to what I know about XS, the PBD's are the physical connect from separate XS hosts to the shared storage. So it's not related to other hosts, it's exclusively to the host machine that I'm using. Understood that, the ideia is to destroy the created PBD on the second XS host and recreate it with the proper settings and IP addresses.
That's when things got interesting. To create the PBD, I do need the iSCSI working in the host, and consequently the multipath connection. So I done the process in the terminal with the following commands:
#Discovery of iSCSI Service
iscsiadm -m discovery -t sendtargets -p 192.168.20.1
iscsiadm -m discovery -t sendtargets -p 192.168.21.1
#Login in the iSCSI Targets
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.20.1:3260 --login
iscsiadm -m node --targetname iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1 -p 192.168.21.1:3260 --login
#Enable Multipath
multipath
multipath -ll
After all I was able to create the PBD with those new settings. The tricky part was to pass the required "device-config" values via the xe pbd-create command. I wasn't aware of the method, but after 4 tries I understood the process and typed the following command:
#Create the appropriate PBD
xe pbd-create sr-uuid=4e8351f6-ef83-815d-b40d-1e332b47863f host-uuid=c2ce0cf6-4bdf-4ee7-b387-b8eb47db4f7e device-config:multiSession="192.168.20.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|192.168.21.1,3260,iqn.2015-01.br.ufrj.iq.local.istgt:iscsi0target1|" device-config:target=192.168.20.1,192.168.21.1 device-config:multihomelist=192.168.10.1:3260,192.168.30.1:3260,192.168.20.1:3260,192.168.21.1:3260,192.168.11.1:3260,192.168.31.1:3260 device-config:targetIQN=* device-config:SCSIid=1FreeBSD_iSCSI_Disk_002590946b34000 device-config:port=3260
I'd like to understand your configuration better- have you effectively enabled 2 storage paths for the SR, knowing that each host will only be able to use one path? Are your PBDs identical or are there differences?
Thanks,
Dave
Basically everything necessary is in there. The IP's, the name, the multipath link, the LUN values, everything. After this, I just plugged the PDB via the CLI:
#Plug the PBD
xe pbd-plug uuid=029fbbf0-9f06-c1ee-ef83-a8b147d51342
And BOOM! It worked. As you can see in the screenshot.
To prove that's everything is working fine, I've created a new VM, installed FreeBSD 10 on it, done a pkg install xe-guest-utilities and requested a XenMotion operation. Oh boy, it worked:
Vinicius-Ferraos-MacBook-Pro:~ ferrao$ ping 10.7.0.248
PING 10.7.0.248 (10.7.0.248): 56 data bytes
64 bytes from 10.7.0.248: icmp_seq=0 ttl=63 time=14.185 ms
64 bytes from 10.7.0.248: icmp_seq=1 ttl=63 time=12.771 ms
64 bytes from 10.7.0.248: icmp_seq=2 ttl=63 time=16.773 ms
64 bytes from 10.7.0.248: icmp_seq=3 ttl=63 time=13.328 ms
64 bytes from 10.7.0.248: icmp_seq=4 ttl=63 time=13.108 ms
64 bytes from 10.7.0.248: icmp_seq=5 ttl=63 time=14.775 ms
64 bytes from 10.7.0.248: icmp_seq=6 ttl=63 time=317.368 ms
64 bytes from 10.7.0.248: icmp_seq=7 ttl=63 time=14.362 ms
64 bytes from 10.7.0.248: icmp_seq=8 ttl=63 time=12.574 ms
64 bytes from 10.7.0.248: icmp_seq=9 ttl=63 time=16.565 ms
64 bytes from 10.7.0.248: icmp_seq=10 ttl=63 time=12.273 ms
64 bytes from 10.7.0.248: icmp_seq=11 ttl=63 time=18.787 ms
64 bytes from 10.7.0.248: icmp_seq=12 ttl=63 time=14.131 ms
64 bytes from 10.7.0.248: icmp_seq=13 ttl=63 time=11.731 ms
64 bytes from 10.7.0.248: icmp_seq=14 ttl=63 time=14.585 ms
64 bytes from 10.7.0.248: icmp_seq=15 ttl=63 time=12.206 ms
64 bytes from 10.7.0.248: icmp_seq=16 ttl=63 time=13.873 ms
64 bytes from 10.7.0.248: icmp_seq=17 ttl=63 time=13.692 ms
64 bytes from 10.7.0.248: icmp_seq=18 ttl=63 time=62.291 ms
64 bytes from 10.7.0.248: icmp_seq=19 ttl=63 time=11.968 ms
Request timeout for icmp_seq 20
Request timeout for icmp_seq 21
Request timeout for icmp_seq 22
Request timeout for icmp_seq 23
Request timeout for icmp_seq 24
Request timeout for icmp_seq 25
64 bytes from 10.7.0.248: icmp_seq=26 ttl=63 time=14.007 ms
64 bytes from 10.7.0.248: icmp_seq=27 ttl=63 time=17.851 ms
64 bytes from 10.7.0.248: icmp_seq=28 ttl=63 time=13.637 ms
64 bytes from 10.7.0.248: icmp_seq=29 ttl=63 time=12.817 ms
64 bytes from 10.7.0.248: icmp_seq=30 ttl=63 time=13.096 ms
64 bytes from 10.7.0.248: icmp_seq=31 ttl=63 time=13.136 ms
64 bytes from 10.7.0.248: icmp_seq=32 ttl=63 time=16.278 ms
64 bytes from 10.7.0.248: icmp_seq=33 ttl=63 time=12.930 ms
64 bytes from 10.7.0.248: icmp_seq=34 ttl=63 time=11.506 ms
64 bytes from 10.7.0.248: icmp_seq=35 ttl=63 time=16.592 ms
64 bytes from 10.7.0.248: icmp_seq=36 ttl=63 time=13.329 ms
^C
--- 10.7.0.248 ping statistics ---
37 packets transmitted, 31 packets received, 16.2% packet loss
round-trip min/avg/max/stddev = 11.506/25.372/317.368/54.017 ms
Pics: https://www.dropbox.com/s/auhv542t3ew3agq/Screenshot%202015-02-15%2003.38.45.png?dl=0
I thinks this is sufficient for now, to prove that the concept works and XenServer can create this kind of topology. It's just not available on the GUI. If XenCenter supports this kind of SR on the next patches, it will be breakthrough.
What I ask now: a feedback, if possible.
Thanks in advance,
Vinícius.
PS: Sorry any english mistakes. It's almost 4AM here, and I was really anxious to report this in the list.
On Feb 14, 2015, at 8:14 PM, Tobias Kreidl <***@nau.edu><mailto:***@nau.edu> wrote:
Frankly speaking, it doesn't seem that it's a good idea to try to try to squeeze something out of a technology that is not built-in or future-safe. Would you want to try to connect a fiber-channel storage device without a fiber channel switch (a very similar situation)? There are other alternatives, for example, to connect iSCSi storage to another, XenServer-independent server and then serve storage from it via NFS or use NFS in the first place and not make use at all of iSCSI technology. NFS has come a long way and in many cases, can be equivalent in I/O performance to iSCSI.
The intercommunications and ability to share SRs within a pool are a significant consideration and cannot be ignored as part of the storage support protocols within XenServer. XenServer provides a number of options for storage connectivity to cover a wide range of needs, preferences and costs. Often, trying to save money in the wrong area results in more headaches later on.
That all being said, it is possible to bypass the SR mechanism altogether (obviously, not a supported configuration) and connect at least Linux VMs directly to iSCSI storage using open iSCSI, but I have not explored this without still involving a network switch. As you indicated, it's also not clear if XenServer will remain 100% open iSCSI compatible or not in the future. It also takes aways some of the things an SR can offer, like VM cloning, storage migration, etc.
Finally, I am not sure I see where the lack of a switch factors in in improving performance. Networks switches are much more capable and much faster than any storage device or host in routing packets. The amount of overhead they add is tiny.
Regards,
-=Tobias
On 2/14/2015 2:26 PM, Vinícius Ferrão wrote:
Hello guys,
I was talking with Tim Mackey on Twitter about the viability of using a iSCSI SR without a Switch. The main goal is to reduce the TCO of the solution, get some extra performance removing the overhead that a managed switch puts on the network and reduce the complexity of the network.
At a first glance I thought that I will only achieve those kind of setup with a distributed switch, so DVSC should be an option. But as Tim said it's only a interface to control some policies.
I've looked for help on ServerFault and FreeNAS Forums, since I'm running FreeNAS as iSCSI service. The discussion is going on those links:
http://serverfault.com/questions/667267/xenserver-6-5-iscsi-mpio-with-point-to-point-links-without-a-switch
https://forums.freenas.org/index.php?threads/iscsi-mpio-without-a-switch-30-links.27579/
It appears that XenServer cannot support switchless iSCSI storages without a switch, because XS needs the same network block on all the XS hosts to create a shared storage between the hosts. So the ideia of point-to-point networks doesn't apply.
If I understood correctly the SR is created with the appropriate PBD's. Perhaps one way to solve this issue is ignoring XenCenter on SR creation and try to create an SR with all the PBD's from the XS hosts. But I was unable to test this setting due to lack of my knowledge when trying to create and manipulate the SR via the CLI.
Anyway, I'm opening the discussion here to get some feedback of the developers, to see if this method of building the iSCSI network is viable, since as Tim said, XS is not 100% open-iscsi upstream and if it could be integrated on next versions of XenServer via the XenCenter GUI. I think that's a very common architecture and VMware is doing this on small pools! So let's do it equally and of course surpass them. :)
Thanks in advance,
Vinicius.