All pastes #981715 Raw Edit

Miscellany

public text v1 · immutable
·published 2008-04-12 00:01 UTC
rendered paste body
=========================================================
[ INFO: possible irq lock inversion dependency detected ]
2.6.20-2925.9.fc7xen #1
---------------------------------------------------------
loader/210 just changed the state of lock:
 (&np->rx_lock){-...}, at: [<d115da6c>] network_open+0x28/0x83 [xennet]
but this lock was taken by another, hard-irq-safe lock in the past:
 (&np->tx_lock){+...}
  <Tab>/<Alt-Tab> between elements  | <Space> selects | <F12> next screen
and interrupts could create inverse lock ordering between them.


other info that might help us debug this:
1 lock held by loader/210:
 #0:  (rtnl_mutex){--..}, at: [<c11f631c>] mutex_lock+0x21/0x24

the first lock's dependencies:
-> (&np->rx_lock){-...} ops: 0 {
   initial-use  at:
                        [<c1037ce3>] __lock_acquire+0x438/0xa13
                        [<c10385bb>] lock_acquire+0x61/0x80
                        [<c11f7627>] _spin_lock+0x2b/0x38
                        [<d115cf71>] backend_changed+0x65a/0x7b1 [xennet]
                        [<c114d508>] otherend_changed+0x84/0x8a
                        [<c114bdc2>] xenwatch_handle_callback+0x15/0x45
                        [<c114c886>] xenwatch_thread+0xfd/0x114
                        [<c102ff23>] kthread+0xb3/0xdc
                        [<c10058df>] kernel_thread_helper+0x7/0x10
                        [<ffffffff>] 0xffffffff
   hardirq-on-W at:
                        [<c1037c9e>] __lock_acquire+0x3f3/0xa13
                        [<c10385bb>] lock_acquire+0x61/0x80
                        [<c11f7627>] _spin_lock+0x2b/0x38
                        [<d115da6c>] network_open+0x28/0x83 [xennet]
                        [<c119791a>] dev_open+0x2e/0x66
                        [<c1195feb>] dev_change_flags+0x51/0xf1
                        [<c11d4de6>] devinet_ioctl+0x235/0x53b
                        [<c11d53a8>] inet_ioctl+0x73/0x91
                        [<c118d08a>] sock_ioctl+0x1ac/0x1c9
                        [<c10780c6>] do_ioctl+0x22/0x67
                        [<c107835d>] vfs_ioctl+0x252/0x265
                        [<c10783b9>] sys_ioctl+0x49/0x64
                        [<c1005568>] syscall_call+0x7/0xb
                        [<ffffffff>] 0xffffffff
 }
 ... key      at: [<d1163708>] __key.29552+0x0/0xffffb476 [xennet]
 -> (lweventlist_lock){....} ops: 0 {
    initial-use  at:
                          [<c1037ce3>] __lock_acquire+0x438/0xa13
                          [<c10385bb>] lock_acquire+0x61/0x80
                          [<c11f76a9>] _spin_lock_irqsave+0x38/0x48
                          [<c119ecbb>] linkwatch_fire_event+0x5c/0xb1
                          [<c11a59bf>] netif_carrier_off+0x18/0x1a
                          [<d115d275>] netfront_probe+0x51/0x2e3 [xennet]
                          [<c114d55f>] xenbus_dev_probe+0x51/0xb4
                          [<c1142e0f>] really_probe+0x80/0x104
                          [<c1142f28>] driver_probe_device+0x95/0xa1
                          [<c1143046>] __driver_attach+0x76/0xaf
                          [<c1142429>] bus_for_each_dev+0x3a/0x5f
                          [<c1142cc0>] driver_attach+0x19/0x1b
                          [<c1142719>] bus_add_driver+0x6a/0x170
                          [<c1143269>] driver_register+0x79/0x7e
                          [<c114cf85>] xenbus_register_driver_common+0x2e/0x3f
                          [<c114cfb1>] xenbus_register_frontend+0x1b/0x2e
                          [<d1023070>] 0xd1023070
                          [<c103ea5b>] sys_init_module+0x173e/0x1899
                          [<c1005568>] syscall_call+0x7/0xb
                          [<ffffffff>] 0xffffffff
  }
  ... key      at: [<c12e7950>] lweventlist_lock+0x10/0x1c
 ... acquired at:
   [<c103819a>] __lock_acquire+0x8ef/0xa13
   [<c10385bb>] lock_acquire+0x61/0x80
   [<c11f76a9>] _spin_lock_irqsave+0x38/0x48
   [<c119ecbb>] linkwatch_fire_event+0x5c/0xb1
   [<c11a59da>] netif_carrier_on+0x19/0x2a
   [<d115d076>] backend_changed+0x75f/0x7b1 [xennet]
   [<c114d508>] otherend_changed+0x84/0x8a
   [<c114bdc2>] xenwatch_handle_callback+0x15/0x45
   [<c114c886>] xenwatch_thread+0xfd/0x114
   [<c102ff23>] kthread+0xb3/0xdc
   [<c10058df>] kernel_thread_helper+0x7/0x10
   [<ffffffff>] 0xffffffff

 -> (base_lock_keys + cpu){.+..} ops: 0 {
    initial-use  at:
                          [<c1037ce3>] __lock_acquire+0x438/0xa13
                          [<c10385bb>] lock_acquire+0x61/0x80
                          [<c11f76a9>] _spin_lock_irqsave+0x38/0x48
                          [<c1026fdd>] lock_timer_base+0x18/0x33
                          [<c102707f>] __mod_timer+0x20/0xa3
                          [<c102750e>] mod_timer+0x21/0x27
                          [<c1323bc5>] con_init+0xb9/0x203
                          [<c132363b>] console_init+0x1e/0x2c
                          [<c130e68d>] start_kernel+0x1e5/0x3ea
                          [<c1000079>] 0xc1000079
                          [<ffffffff>] 0xffffffff
    in-softirq-W at:
                          [<c1037c39>] __lock_acquire+0x38e/0xa13
                          [<c10385bb>] lock_acquire+0x61/0x80
                          [<c11f77ed>] _spin_lock_irq+0x35/0x42
                          [<c10267d3>] run_timer_softirq+0x39/0x168
                          [<c1023ae5>] __do_softirq+0x73/0xea
                          [<c1006f76>] do_softirq+0x65/0xd1
                          [<ffffffff>] 0xffffffff
  }
  ... key      at: [<c13b6660>] base_lock_keys+0x0/0x100
 ... acquired at:
   [<c103819a>] __lock_acquire+0x8ef/0xa13
   [<c10385bb>] lock_acquire+0x61/0x80
   [<c11f76a9>] _spin_lock_irqsave+0x38/0x48
   [<c1026fdd>] lock_timer_base+0x18/0x33
   [<c102707f>] __mod_timer+0x20/0xa3
   [<c102d384>] queue_delayed_work+0x72/0x7e
   [<c102d3a1>] schedule_delayed_work+0x11/0x13
   [<c119ed0a>] linkwatch_fire_event+0xab/0xb1
   [<c11a59da>] netif_carrier_on+0x19/0x2a
   [<d115d076>] backend_changed+0x75f/0x7b1 [xennet]
   [<c114d508>] otherend_changed+0x84/0x8a
   [<c114bdc2>] xenwatch_handle_callback+0x15/0x45
   [<c114c886>] xenwatch_thread+0xfd/0x114
   [<c102ff23>] kthread+0xb3/0xdc
   [<c10058df>] kernel_thread_helper+0x7/0x10
   [<ffffffff>] 0xffffffff

 -> (&parent->list_lock){.+..} ops: 0 {
    initial-use  at:
                          [<c1037ce3>] __lock_acquire+0x438/0xa13
                          [<c10385bb>] lock_acquire+0x61/0x80
                          [<c11f7627>] _spin_lock+0x2b/0x38
                          [<c1069ea9>] cache_alloc_refill+0x5c/0x47d
                          [<c106a3fa>] kmem_cache_zalloc+0x6b/0xbc
                          [<c106abbd>] kmem_cache_create+0x178/0x4b2
                          [<c131d614>] kmem_cache_init+0x16a/0x41e
                          [<c130e6fe>] start_kernel+0x256/0x3ea
                          [<c1000079>] 0xc1000079
                          [<ffffffff>] 0xffffffff
    in-softirq-W at:
                          [<c1037c39>] __lock_acquire+0x38e/0xa13
                          [<c10385bb>] lock_acquire+0x61/0x80
                          [<c11f7627>] _spin_lock+0x2b/0x38
                          [<c1069821>] cache_flusharray+0x25/0xa8
                          [<c10699a5>] kmem_cache_free+0x62/0x92
                          [<c101cea8>] free_task+0x21/0x24
                          [<c101e374>] __put_task_struct+0x104/0x10a
                          [<c102070d>] delayed_put_task_struct+0x1e/0x20
                          [<c102dfd3>] __rcu_process_callbacks+0x107/0x17f
                          [<c102e06a>] rcu_process_callbacks+0x1f/0x3d
                          [<c1023bbf>] tasklet_action+0x63/0xd3
                          [<c1023ae5>] __do_softirq+0x73/0xea
                          [<c1006f76>] do_softirq+0x65/0xd1
                          [<ffffffff>] 0xffffffff
  }
  ... key      at: [<c15cb0a8>] __key.17782+0x0/0x8
  -> (&zone->lock){.+..} ops: 0 {
     initial-use  at:
                            [<c1037ce3>] __lock_acquire+0x438/0xa13
                            [<c10385bb>] lock_acquire+0x61/0x80
                            [<c11f7627>] _spin_lock+0x2b/0x38
                            [<c1052a35>] __free_pages_ok+0x12a/0x288
                            [<c1052bc1>] __free_pages+0x2e/0x30
                            [<c131d14e>] __free_pages_bootmem+0x9c/0xa0
                            [<c131bbf4>] free_all_bootmem_core+0xaf/0x147
                            [<c131bc99>] free_all_bootmem+0xd/0xf
                            [<c1318f10>] mem_init+0x14e/0x4d8
                            [<c130e6f9>] start_kernel+0x251/0x3ea
                            [<c1000079>] 0xc1000079
                            [<ffffffff>] 0xffffffff
     in-softirq-W at:
                            [<c1037c39>] __lock_acquire+0x38e/0xa13
                            [<c10385bb>] lock_acquire+0x61/0x80
                            [<c11f7627>] _spin_lock+0x2b/0x38
                            [<c1052a35>] __free_pages_ok+0x12a/0x288
                            [<c1052bc1>] __free_pages+0x2e/0x30
                            [<c1052bec>] free_pages+0x29/0x2b
                            [<c10696da>] kmem_freepages+0xaa/0xb0
                            [<c1069aa5>] slab_destroy+0x70/0x8c
                            [<c1069b74>] free_block+0xb3/0xe8
                            [<c1069876>] cache_flusharray+0x7a/0xa8
                            [<c10699a5>] kmem_cache_free+0x62/0x92
                            [<c101cea8>] free_task+0x21/0x24
                            [<c101e374>] __put_task_struct+0x104/0x10a
                            [<c102070d>] delayed_put_task_struct+0x1e/0x20
                            [<c102dfd3>] __rcu_process_callbacks+0x107/0x17f
                            [<c102e06a>] rcu_process_callbacks+0x1f/0x3d
                            [<c1023bbf>] tasklet_action+0x63/0xd3
                            [<c1023ae5>] __do_softirq+0x73/0xea
                            [<c1006f76>] do_softirq+0x65/0xd1
                            [<ffffffff>] 0xffffffff
   }
   ... key      at: [<c15c3e34>] __key.22301+0x0/0xc
  ... acquired at:
   [<c103819a>] __lock_acquire+0x8ef/0xa13
   [<c10385bb>] lock_acquire+0x61/0x80
   [<c11f7627>] _spin_lock+0x2b/0x38
   [<c1052a35>] __free_pages_ok+0x12a/0x288
   [<c1052bc1>] __free_pages+0x2e/0x30
   [<c1052bec>] free_pages+0x29/0x2b
   [<c10696da>] kmem_freepages+0xaa/0xb0
   [<c1069aa5>] slab_destroy+0x70/0x8c
   [<c1069b74>] free_block+0xb3/0xe8
   [<c1069876>] cache_flusharray+0x7a/0xa8
   [<c10699a5>] kmem_cache_free+0x62/0x92
   [<c101cea8>] free_task+0x21/0x24
   [<c101e374>] __put_task_struct+0x104/0x10a
   [<c102070d>] delayed_put_task_struct+0x1e/0x20
   [<c102dfd3>] __rcu_process_callbacks+0x107/0x17f
   [<c102e06a>] rcu_process_callbacks+0x1f/0x3d
   [<c1023bbf>] tasklet_action+0x63/0xd3
   [<c1023ae5>] __do_softirq+0x73/0xea
   [<c1006f76>] do_softirq+0x65/0xd1
   [<ffffffff>] 0xffffffff

  -> (&on_slab_l3_key){.+..} ops: 0 {
     initial-use  at:
                            [<c1037ce3>] __lock_acquire+0x438/0xa13
                            [<c10385bb>] lock_acquire+0x61/0x80
                            [<c11f7627>] _spin_lock+0x2b/0x38
                            [<c1069ea9>] cache_alloc_refill+0x5c/0x47d
                            [<c1069e1e>] kmem_cache_alloc+0x6b/0x9a
                            [<c1049686>] request_irq+0x52/0xa1
                            [<c11471f1>] bind_virq_to_irqhandler+0x118/0x135
                            [<c10087b5>] setup_cpu0_timer_irq+0x3a/0x52
                            [<c130e709>] start_kernel+0x261/0x3ea
                            [<c1000079>] 0xc1000079
                            [<ffffffff>] 0xffffffff
     in-softirq-W at:
                            [<c1037c39>] __lock_acquire+0x38e/0xa13
                            [<c10385bb>] lock_acquire+0x61/0x80
                            [<c11f7627>] _spin_lock+0x2b/0x38
                            [<c1069821>] cache_flusharray+0x25/0xa8
                            [<c1069913>] kfree+0x6f/0x9f
                            [<c10b92fa>] selinux_task_free_security+0x1a/0x1c
                            [<c101e31b>] __put_task_struct+0xab/0x10a
                            [<c102070d>] delayed_put_task_struct+0x1e/0x20
                            [<c102dfd3>] __rcu_process_callbacks+0x107/0x17f
                            [<c102e06a>] rcu_process_callbacks+0x1f/0x3d
                            [<c1023bbf>] tasklet_action+0x63/0xd3
                            [<c1023ae5>] __do_softirq+0x73/0xea
                            [<c1006f76>] do_softirq+0x65/0xd1
                            [<ffffffff>] 0xffffffff
   }
   ... key      at: [<c15cb0cc>] on_slab_l3_key+0x0/0x8
  ... acquired at:
   [<c103819a>] __lock_acquire+0x8ef/0xa13
   [<c10385bb>] lock_acquire+0x61/0x80
   [<c11f7627>] _spin_lock+0x2b/0x38
   [<c1069821>] cache_flusharray+0x25/0xa8
   [<c10699a5>] kmem_cache_free+0x62/0x92
   [<c1069abb>] slab_destroy+0x86/0x8c
   [<c1069b74>] free_block+0xb3/0xe8
   [<c1069876>] cache_flusharray+0x7a/0xa8
   [<c1069913>] kfree+0x6f/0x9f
   [<c101ce95>] free_task+0xe/0x24
   [<c101e374>] __put_task_struct+0x104/0x10a
   [<c102070d>] delayed_put_task_struct+0x1e/0x20
   [<c102dfd3>] __rcu_process_callbacks+0x107/0x17f
   [<c102e06a>] rcu_process_callbacks+0x1f/0x3d
   [<c1023bbf>] tasklet_action+0x63/0xd3
   [<c1023ae5>] __do_softirq+0x73/0xea
   [<c1006f76>] do_softirq+0x65/0xd1
   [<ffffffff>] 0xffffffff

 ... acquired at:
   [<c103819a>] __lock_acquire+0x8ef/0xa13
   [<c10385bb>] lock_acquire+0x61/0x80
   [<c11f7627>] _spin_lock+0x2b/0x38
   [<c1069ea9>] cache_alloc_refill+0x5c/0x47d
   [<c1069e1e>] kmem_cache_alloc+0x6b/0x9a
   [<c1192b3f>] alloc_skb_from_cache+0x46/0x114
   [<c11489f5>] __alloc_skb+0x77/0x7e
   [<d115c1c6>] network_alloc_rx_buffers+0x5f/0x3e1 [xennet]
   [<d115d091>] backend_changed+0x77a/0x7b1 [xennet]
   [<c114d508>] otherend_changed+0x84/0x8a
   [<c114bdc2>] xenwatch_handle_callback+0x15/0x45
   [<c114c886>] xenwatch_thread+0xfd/0x114
   [<c102ff23>] kthread+0xb3/0xdc
   [<c10058df>] kernel_thread_helper+0x7/0x10
   [<ffffffff>] 0xffffffff

 -> (&zone->lock){.+..} ops: 0 {
    initial-use  at:
                          [<c1037ce3>] __lock_acquire+0x438/0xa13
                          [<c10385bb>] lock_acquire+0x61/0x80
                          [<c11f7627>] _spin_lock+0x2b/0x38
                          [<c1052a35>] __free_pages_ok+0x12a/0x288
                          [<c1052bc1>] __free_pages+0x2e/0x30
                          [<c131d14e>] __free_pages_bootmem+0x9c/0xa0
                          [<c131bbf4>] free_all_bootmem_core+0xaf/0x147
                          [<c131bc99>] free_all_bootmem+0xd/0xf
                          [<c1318f10>] mem_init+0x14e/0x4d8
                          [<c130e6f9>] start_kernel+0x251/0x3ea
                          [<c1000079>] 0xc1000079
                          [<ffffffff>] 0xffffffff
    in-softirq-W at:
                          [<c1037c39>] __lock_acquire+0x38e/0xa13
                          [<c10385bb>] lock_acquire+0x61/0x80
                          [<c11f7627>] _spin_lock+0x2b/0x38
                          [<c1052a35>] __free_pages_ok+0x12a/0x288
                          [<c1052bc1>] __free_pages+0x2e/0x30
                          [<c1052bec>] free_pages+0x29/0x2b
                          [<c10696da>] kmem_freepages+0xaa/0xb0
                          [<c1069aa5>] slab_destroy+0x70/0x8c
                          [<c1069b74>] free_block+0xb3/0xe8
                          [<c1069876>] cache_flusharray+0x7a/0xa8
                          [<c10699a5>] kmem_cache_free+0x62/0x92
                          [<c101cea8>] free_task+0x21/0x24
                          [<c101e374>] __put_task_struct+0x104/0x10a
                          [<c102070d>] delayed_put_task_struct+0x1e/0x20
                          [<c102dfd3>] __rcu_process_callbacks+0x107/0x17f
                          [<c102e06a>] rcu_process_callbacks+0x1f/0x3d
                          [<c1023bbf>] tasklet_action+0x63/0xd3
                          [<c1023ae5>] __do_softirq+0x73/0xea
                          [<c1006f76>] do_softirq+0x65/0xd1
                          [<ffffffff>] 0xffffffff
  }
  ... key      at: [<c15c3e34>] __key.22301+0x0/0xc
 ... acquired at:
   [<c103819a>] __lock_acquire+0x8ef/0xa13
   [<c10385bb>] lock_acquire+0x61/0x80
   [<c11f7627>] _spin_lock+0x2b/0x38
   [<c1052ce6>] get_page_from_freelist+0xeb/0x333
   [<c1052fdb>] __alloc_pages+0xad/0x2e4
   [<c106a0c2>] cache_alloc_refill+0x275/0x47d
   [<c1069e1e>] kmem_cache_alloc+0x6b/0x9a
   [<c1192b3f>] alloc_skb_from_cache+0x46/0x114
   [<c11489f5>] __alloc_skb+0x77/0x7e
   [<d115c1c6>] network_alloc_rx_buffers+0x5f/0x3e1 [xennet]
   [<d115d091>] backend_changed+0x77a/0x7b1 [xennet]
   [<c114d508>] otherend_changed+0x84/0x8a
   [<c114bdc2>] xenwatch_handle_callback+0x15/0x45
   [<c114c886>] xenwatch_thread+0xfd/0x114
   [<c102ff23>] kthread+0xb3/0xdc
   [<c10058df>] kernel_thread_helper+0x7/0x10
   [<ffffffff>] 0xffffffff

 -> (balloon_lock){....} ops: 0 {
    initial-use  at:
                          [<c1037ce3>] __lock_acquire+0x438/0xa13
                          [<c10385bb>] lock_acquire+0x61/0x80
                          [<c11f76a9>] _spin_lock_irqsave+0x38/0x48
                          [<c114e70a>] balloon_update_driver_allowance+0x10/0x25
                          [<d115c46b>] network_alloc_rx_buffers+0x304/0x3e1 [xennet]
                          [<d115d091>] backend_changed+0x77a/0x7b1 [xennet]
                          [<c114d508>] otherend_changed+0x84/0x8a
                          [<c114bdc2>] xenwatch_handle_callback+0x15/0x45
                          [<c114c886>] xenwatch_thread+0xfd/0x114
                          [<c102ff23>] kthread+0xb3/0xdc
                          [<c10058df>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
  }
  ... key      at: [<c12e04b0>] balloon_lock+0x10/0x1c
 ... acquired at:
   [<c103819a>] __lock_acquire+0x8ef/0xa13
   [<c10385bb>] lock_acquire+0x61/0x80
   [<c11f76a9>] _spin_lock_irqsave+0x38/0x48
   [<c114e70a>] balloon_update_driver_allowance+0x10/0x25
   [<d115c46b>] network_alloc_rx_buffers+0x304/0x3e1 [xennet]
   [<d115d091>] backend_changed+0x77a/0x7b1 [xennet]
   [<c114d508>] otherend_changed+0x84/0x8a
   [<c114bdc2>] xenwatch_handle_callback+0x15/0x45
   [<c114c886>] xenwatch_thread+0xfd/0x114
   [<c102ff23>] kthread+0xb3/0xdc
   [<c10058df>] kernel_thread_helper+0x7/0x10
   [<ffffffff>] 0xffffffff


the second lock's dependencies:
-> (&np->tx_lock){+...} ops: 0 {
   initial-use  at:
                        [<c1037ce3>] __lock_acquire+0x438/0xa13
                        [<c10385bb>] lock_acquire+0x61/0x80
                        [<c11f77ed>] _spin_lock_irq+0x35/0x42
                        [<d115cf63>] backend_changed+0x64c/0x7b1 [xennet]
                        [<c114d508>] otherend_changed+0x84/0x8a
                        [<c114bdc2>] xenwatch_handle_callback+0x15/0x45
                        [<c114c886>] xenwatch_thread+0xfd/0x114
                        [<c102ff23>] kthread+0xb3/0xdc
                        [<c10058df>] kernel_thread_helper+0x7/0x10
                        [<ffffffff>] 0xffffffff
   in-hardirq-W at:
                        [<c1037c0f>] __lock_acquire+0x364/0xa13
                        [<c10385bb>] lock_acquire+0x61/0x80
                        [<c11f76a9>] _spin_lock_irqsave+0x38/0x48
                        [<d115d9f5>] netif_int+0x19/0x68 [xennet]
                        [<c1048ee9>] handle_IRQ_event+0x1a/0x4a
                        [<c104a2e4>] handle_level_irq+0x9a/0xea
                        [<c10070a8>] do_IRQ+0xc6/0xee
                        [<ffffffff>] 0xffffffff
 }
 ... key      at: [<d1163710>] __key.29551+0x0/0xffffb46e [xennet]
 -> (&np->rx_lock){-...} ops: 0 {
    initial-use  at:
                          [<c1037ce3>] __lock_acquire+0x438/0xa13
                          [<c10385bb>] lock_acquire+0x61/0x80
                          [<c11f7627>] _spin_lock+0x2b/0x38
                          [<d115cf71>] backend_changed+0x65a/0x7b1 [xennet]
                          [<c114d508>] otherend_changed+0x84/0x8a
                          [<c114bdc2>] xenwatch_handle_callback+0x15/0x45
                          [<c114c886>] xenwatch_thread+0xfd/0x114
                          [<c102ff23>] kthread+0xb3/0xdc
                          [<c10058df>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
    hardirq-on-W at:
                          [<c1037c9e>] __lock_acquire+0x3f3/0xa13
                          [<c10385bb>] lock_acquire+0x61/0x80
                          [<c11f7627>] _spin_lock+0x2b/0x38
                          [<d115da6c>] network_open+0x28/0x83 [xennet]
                          [<c119791a>] dev_open+0x2e/0x66
                          [<c1195feb>] dev_change_flags+0x51/0xf1
                          [<c11d4de6>] devinet_ioctl+0x235/0x53b
                          [<c11d53a8>] inet_ioctl+0x73/0x91
                          [<c118d08a>] sock_ioctl+0x1ac/0x1c9
                          [<c10780c6>] do_ioctl+0x22/0x67
                          [<c107835d>] vfs_ioctl+0x252/0x265
                          [<c10783b9>] sys_ioctl+0x49/0x64
                          [<c1005568>] syscall_call+0x7/0xb
                          [<ffffffff>] 0xffffffff
  }
  ... key      at: [<d1163708>] __key.29552+0x0/0xffffb476 [xennet]
  -> (lweventlist_lock){....} ops: 0 {
     initial-use  at:
                            [<c1037ce3>] __lock_acquire+0x438/0xa13
                            [<c10385bb>] lock_acquire+0x61/0x80
                            [<c11f76a9>] _spin_lock_irqsave+0x38/0x48
                            [<c119ecbb>] linkwatch_fire_event+0x5c/0xb1
                            [<c11a59bf>] netif_carrier_off+0x18/0x1a
                            [<d115d275>] netfront_probe+0x51/0x2e3 [xennet]
                            [<c114d55f>] xenbus_dev_probe+0x51/0xb4
                            [<c1142e0f>] really_probe+0x80/0x104
                            [<c1142f28>] driver_probe_device+0x95/0xa1
                            [<c1143046>] __driver_attach+0x76/0xaf
                            [<c1142429>] bus_for_each_dev+0x3a/0x5f
                            [<c1142cc0>] driver_attach+0x19/0x1b
                            [<c1142719>] bus_add_driver+0x6a/0x170
                            [<c1143269>] driver_register+0x79/0x7e
                            [<c114cf85>] xenbus_register_driver_common+0x2e/0x3f
                            [<c114cfb1>] xenbus_register_frontend+0x1b/0x2e
                            [<d1023070>] 0xd1023070
                            [<c103ea5b>] sys_init_module+0x173e/0x1899
                            [<c1005568>] syscall_call+0x7/0xb
                            [<ffffffff>] 0xffffffff
   }
   ... key      at: [<c12e7950>] lweventlist_lock+0x10/0x1c
  ... acquired at:
   [<c103819a>] __lock_acquire+0x8ef/0xa13
   [<c10385bb>] lock_acquire+0x61/0x80
   [<c11f76a9>] _spin_lock_irqsave+0x38/0x48
   [<c119ecbb>] linkwatch_fire_event+0x5c/0xb1
   [<c11a59da>] netif_carrier_on+0x19/0x2a
   [<d115d076>] backend_changed+0x75f/0x7b1 [xennet]
   [<c114d508>] otherend_changed+0x84/0x8a
   [<c114bdc2>] xenwatch_handle_callback+0x15/0x45
   [<c114c886>] xenwatch_thread+0xfd/0x114
   [<c102ff23>] kthread+0xb3/0xdc
   [<c10058df>] kernel_thread_helper+0x7/0x10
   [<ffffffff>] 0xffffffff

  -> (base_lock_keys + cpu){.+..} ops: 0 {
     initial-use  at:
                            [<c1037ce3>] __lock_acquire+0x438/0xa13
                            [<c10385bb>] lock_acquire+0x61/0x80
                            [<c11f76a9>] _spin_lock_irqsave+0x38/0x48
                            [<c1026fdd>] lock_timer_base+0x18/0x33
                            [<c102707f>] __mod_timer+0x20/0xa3
                            [<c102750e>] mod_timer+0x21/0x27
                            [<c1323bc5>] con_init+0xb9/0x203
                            [<c132363b>] console_init+0x1e/0x2c
                            [<c130e68d>] start_kernel+0x1e5/0x3ea
                            [<c1000079>] 0xc1000079
                            [<ffffffff>] 0xffffffff
     in-softirq-W at:
                            [<c1037c39>] __lock_acquire+0x38e/0xa13
                            [<c10385bb>] lock_acquire+0x61/0x80
                            [<c11f77ed>] _spin_lock_irq+0x35/0x42
                            [<c10267d3>] run_timer_softirq+0x39/0x168
                            [<c1023ae5>] __do_softirq+0x73/0xea
                            [<c1006f76>] do_softirq+0x65/0xd1
                            [<ffffffff>] 0xffffffff
   }
   ... key      at: [<c13b6660>] base_lock_keys+0x0/0x100
  ... acquired at:
   [<c103819a>] __lock_acquire+0x8ef/0xa13
   [<c10385bb>] lock_acquire+0x61/0x80
   [<c11f76a9>] _spin_lock_irqsave+0x38/0x48
   [<c1026fdd>] lock_timer_base+0x18/0x33
   [<c102707f>] __mod_timer+0x20/0xa3
   [<c102d384>] queue_delayed_work+0x72/0x7e
   [<c102d3a1>] schedule_delayed_work+0x11/0x13
   [<c119ed0a>] linkwatch_fire_event+0xab/0xb1
   [<c11a59da>] netif_carrier_on+0x19/0x2a
   [<d115d076>] backend_changed+0x75f/0x7b1 [xennet]
   [<c114d508>] otherend_changed+0x84/0x8a
   [<c114bdc2>] xenwatch_handle_callback+0x15/0x45
   [<c114c886>] xenwatch_thread+0xfd/0x114
   [<c102ff23>] kthread+0xb3/0xdc
   [<c10058df>] kernel_thread_helper+0x7/0x10
   [<ffffffff>] 0xffffffff

  -> (&parent->list_lock){.+..} ops: 0 {
     initial-use  at:
                            [<c1037ce3>] __lock_acquire+0x438/0xa13
                            [<c10385bb>] lock_acquire+0x61/0x80
                            [<c11f7627>] _spin_lock+0x2b/0x38
                            [<c1069ea9>] cache_alloc_refill+0x5c/0x47d
                            [<c106a3fa>] kmem_cache_zalloc+0x6b/0xbc
                            [<c106abbd>] kmem_cache_create+0x178/0x4b2
                            [<c131d614>] kmem_cache_init+0x16a/0x41e
                            [<c130e6fe>] start_kernel+0x256/0x3ea
                            [<c1000079>] 0xc1000079
                            [<ffffffff>] 0xffffffff
     in-softirq-W at:
                            [<c1037c39>] __lock_acquire+0x38e/0xa13
                            [<c10385bb>] lock_acquire+0x61/0x80
                            [<c11f7627>] _spin_lock+0x2b/0x38
                            [<c1069821>] cache_flusharray+0x25/0xa8
                            [<c10699a5>] kmem_cache_free+0x62/0x92
                            [<c101cea8>] free_task+0x21/0x24
                            [<c101e374>] __put_task_struct+0x104/0x10a
                            [<c102070d>] delayed_put_task_struct+0x1e/0x20
                            [<c102dfd3>] __rcu_process_callbacks+0x107/0x17f
                            [<c102e06a>] rcu_process_callbacks+0x1f/0x3d
                            [<c1023bbf>] tasklet_action+0x63/0xd3
                            [<c1023ae5>] __do_softirq+0x73/0xea
                            [<c1006f76>] do_softirq+0x65/0xd1
                            [<ffffffff>] 0xffffffff
   }
   ... key      at: [<c15cb0a8>] __key.17782+0x0/0x8
   -> (&zone->lock){.+..} ops: 0 {
      initial-use  at:
                              [<c1037ce3>] __lock_acquire+0x438/0xa13
                              [<c10385bb>] lock_acquire+0x61/0x80
                              [<c11f7627>] _spin_lock+0x2b/0x38
                              [<c1052a35>] __free_pages_ok+0x12a/0x288
                              [<c1052bc1>] __free_pages+0x2e/0x30
                              [<c131d14e>] __free_pages_bootmem+0x9c/0xa0
                              [<c131bbf4>] free_all_bootmem_core+0xaf/0x147
                              [<c131bc99>] free_all_bootmem+0xd/0xf
                              [<c1318f10>] mem_init+0x14e/0x4d8
                              [<c130e6f9>] start_kernel+0x251/0x3ea
                              [<c1000079>] 0xc1000079
                              [<ffffffff>] 0xffffffff
      in-softirq-W at:
                              [<c1037c39>] __lock_acquire+0x38e/0xa13
                              [<c10385bb>] lock_acquire+0x61/0x80
                              [<c11f7627>] _spin_lock+0x2b/0x38
                              [<c1052a35>] __free_pages_ok+0x12a/0x288
                              [<c1052bc1>] __free_pages+0x2e/0x30
                              [<c1052bec>] free_pages+0x29/0x2b
                              [<c10696da>] kmem_freepages+0xaa/0xb0
                              [<c1069aa5>] slab_destroy+0x70/0x8c
                              [<c1069b74>] free_block+0xb3/0xe8
                              [<c1069876>] cache_flusharray+0x7a/0xa8
                              [<c10699a5>] kmem_cache_free+0x62/0x92
                              [<c101cea8>] free_task+0x21/0x24
                              [<c101e374>] __put_task_struct+0x104/0x10a
                              [<c102070d>] delayed_put_task_struct+0x1e/0x20
                              [<c102dfd3>] __rcu_process_callbacks+0x107/0x17f
                              [<c102e06a>] rcu_process_callbacks+0x1f/0x3d
                              [<c1023bbf>] tasklet_action+0x63/0xd3
                              [<c1023ae5>] __do_softirq+0x73/0xea
                              [<c1006f76>] do_softirq+0x65/0xd1
                              [<ffffffff>] 0xffffffff
    }
    ... key      at: [<c15c3e34>] __key.22301+0x0/0xc
   ... acquired at:
   [<c103819a>] __lock_acquire+0x8ef/0xa13
   [<c10385bb>] lock_acquire+0x61/0x80
   [<c11f7627>] _spin_lock+0x2b/0x38
   [<c1052a35>] __free_pages_ok+0x12a/0x288
   [<c1052bc1>] __free_pages+0x2e/0x30
   [<c1052bec>] free_pages+0x29/0x2b
   [<c10696da>] kmem_freepages+0xaa/0xb0
   [<c1069aa5>] slab_destroy+0x70/0x8c
   [<c1069b74>] free_block+0xb3/0xe8
   [<c1069876>] cache_flusharray+0x7a/0xa8
   [<c10699a5>] kmem_cache_free+0x62/0x92
   [<c101cea8>] free_task+0x21/0x24
   [<c101e374>] __put_task_struct+0x104/0x10a
   [<c102070d>] delayed_put_task_struct+0x1e/0x20
   [<c102dfd3>] __rcu_process_callbacks+0x107/0x17f
   [<c102e06a>] rcu_process_callbacks+0x1f/0x3d
   [<c1023bbf>] tasklet_action+0x63/0xd3
   [<c1023ae5>] __do_softirq+0x73/0xea
   [<c1006f76>] do_softirq+0x65/0xd1
   [<ffffffff>] 0xffffffff

   -> (&on_slab_l3_key){.+..} ops: 0 {
      initial-use  at:
                              [<c1037ce3>] __lock_acquire+0x438/0xa13
                              [<c10385bb>] lock_acquire+0x61/0x80
                              [<c11f7627>] _spin_lock+0x2b/0x38
                              [<c1069ea9>] cache_alloc_refill+0x5c/0x47d
                              [<c1069e1e>] kmem_cache_alloc+0x6b/0x9a
                              [<c1049686>] request_irq+0x52/0xa1
                              [<c11471f1>] bind_virq_to_irqhandler+0x118/0x135
                              [<c10087b5>] setup_cpu0_timer_irq+0x3a/0x52
                              [<c130e709>] start_kernel+0x261/0x3ea
                              [<c1000079>] 0xc1000079
                              [<ffffffff>] 0xffffffff
      in-softirq-W at:
                              [<c1037c39>] __lock_acquire+0x38e/0xa13
                              [<c10385bb>] lock_acquire+0x61/0x80
                              [<c11f7627>] _spin_lock+0x2b/0x38
                              [<c1069821>] cache_flusharray+0x25/0xa8
                              [<c1069913>] kfree+0x6f/0x9f
                              [<c10b92fa>] selinux_task_free_security+0x1a/0x1c
                              [<c101e31b>] __put_task_struct+0xab/0x10a
                              [<c102070d>] delayed_put_task_struct+0x1e/0x20
                              [<c102dfd3>] __rcu_process_callbacks+0x107/0x17f
                              [<c102e06a>] rcu_process_callbacks+0x1f/0x3d
                              [<c1023bbf>] tasklet_action+0x63/0xd3
                              [<c1023ae5>] __do_softirq+0x73/0xea
                              [<c1006f76>] do_softirq+0x65/0xd1
                              [<ffffffff>] 0xffffffff
    }
    ... key      at: [<c15cb0cc>] on_slab_l3_key+0x0/0x8
   ... acquired at:
   [<c103819a>] __lock_acquire+0x8ef/0xa13
   [<c10385bb>] lock_acquire+0x61/0x80
   [<c11f7627>] _spin_lock+0x2b/0x38
   [<c1069821>] cache_flusharray+0x25/0xa8
   [<c10699a5>] kmem_cache_free+0x62/0x92
   [<c1069abb>] slab_destroy+0x86/0x8c
   [<c1069b74>] free_block+0xb3/0xe8
   [<c1069876>] cache_flusharray+0x7a/0xa8
   [<c1069913>] kfree+0x6f/0x9f
   [<c101ce95>] free_task+0xe/0x24
   [<c101e374>] __put_task_struct+0x104/0x10a
   [<c102070d>] delayed_put_task_struct+0x1e/0x20
   [<c102dfd3>] __rcu_process_callbacks+0x107/0x17f
   [<c102e06a>] rcu_process_callbacks+0x1f/0x3d
   [<c1023bbf>] tasklet_action+0x63/0xd3
   [<c1023ae5>] __do_softirq+0x73/0xea
   [<c1006f76>] do_softirq+0x65/0xd1
   [<ffffffff>] 0xffffffff

  ... acquired at:
   [<c103819a>] __lock_acquire+0x8ef/0xa13
   [<c10385bb>] lock_acquire+0x61/0x80
   [<c11f7627>] _spin_lock+0x2b/0x38
   [<c1069ea9>] cache_alloc_refill+0x5c/0x47d
   [<c1069e1e>] kmem_cache_alloc+0x6b/0x9a
   [<c1192b3f>] alloc_skb_from_cache+0x46/0x114
   [<c11489f5>] __alloc_skb+0x77/0x7e
   [<d115c1c6>] network_alloc_rx_buffers+0x5f/0x3e1 [xennet]
   [<d115d091>] backend_changed+0x77a/0x7b1 [xennet]
   [<c114d508>] otherend_changed+0x84/0x8a
   [<c114bdc2>] xenwatch_handle_callback+0x15/0x45
   [<c114c886>] xenwatch_thread+0xfd/0x114
   [<c102ff23>] kthread+0xb3/0xdc
   [<c10058df>] kernel_thread_helper+0x7/0x10
   [<ffffffff>] 0xffffffff

  -> (&zone->lock){.+..} ops: 0 {
     initial-use  at:
                            [<c1037ce3>] __lock_acquire+0x438/0xa13
                            [<c10385bb>] lock_acquire+0x61/0x80
                            [<c11f7627>] _spin_lock+0x2b/0x38
                            [<c1052a35>] __free_pages_ok+0x12a/0x288
                            [<c1052bc1>] __free_pages+0x2e/0x30
                            [<c131d14e>] __free_pages_bootmem+0x9c/0xa0
                            [<c131bbf4>] free_all_bootmem_core+0xaf/0x147
                            [<c131bc99>] free_all_bootmem+0xd/0xf
                            [<c1318f10>] mem_init+0x14e/0x4d8
                            [<c130e6f9>] start_kernel+0x251/0x3ea
                            [<c1000079>] 0xc1000079
                            [<ffffffff>] 0xffffffff
     in-softirq-W at:
                            [<c1037c39>] __lock_acquire+0x38e/0xa13
                            [<c10385bb>] lock_acquire+0x61/0x80
                            [<c11f7627>] _spin_lock+0x2b/0x38
                            [<c1052a35>] __free_pages_ok+0x12a/0x288
                            [<c1052bc1>] __free_pages+0x2e/0x30
                            [<c1052bec>] free_pages+0x29/0x2b
                            [<c10696da>] kmem_freepages+0xaa/0xb0
                            [<c1069aa5>] slab_destroy+0x70/0x8c
                            [<c1069b74>] free_block+0xb3/0xe8
                            [<c1069876>] cache_flusharray+0x7a/0xa8
                            [<c10699a5>] kmem_cache_free+0x62/0x92
                            [<c101cea8>] free_task+0x21/0x24
                            [<c101e374>] __put_task_struct+0x104/0x10a
                            [<c102070d>] delayed_put_task_struct+0x1e/0x20
                            [<c102dfd3>] __rcu_process_callbacks+0x107/0x17f
                            [<c102e06a>] rcu_process_callbacks+0x1f/0x3d
                            [<c1023bbf>] tasklet_action+0x63/0xd3
                            [<c1023ae5>] __do_softirq+0x73/0xea
                            [<c1006f76>] do_softirq+0x65/0xd1
                            [<ffffffff>] 0xffffffff
   }
   ... key      at: [<c15c3e34>] __key.22301+0x0/0xc
  ... acquired at:
   [<c103819a>] __lock_acquire+0x8ef/0xa13
   [<c10385bb>] lock_acquire+0x61/0x80
   [<c11f7627>] _spin_lock+0x2b/0x38
   [<c1052ce6>] get_page_from_freelist+0xeb/0x333
   [<c1052fdb>] __alloc_pages+0xad/0x2e4
   [<c106a0c2>] cache_alloc_refill+0x275/0x47d
   [<c1069e1e>] kmem_cache_alloc+0x6b/0x9a
   [<c1192b3f>] alloc_skb_from_cache+0x46/0x114
   [<c11489f5>] __alloc_skb+0x77/0x7e
   [<d115c1c6>] network_alloc_rx_buffers+0x5f/0x3e1 [xennet]
   [<d115d091>] backend_changed+0x77a/0x7b1 [xennet]
   [<c114d508>] otherend_changed+0x84/0x8a
   [<c114bdc2>] xenwatch_handle_callback+0x15/0x45
   [<c114c886>] xenwatch_thread+0xfd/0x114
   [<c102ff23>] kthread+0xb3/0xdc
   [<c10058df>] kernel_thread_helper+0x7/0x10
   [<ffffffff>] 0xffffffff

  -> (balloon_lock){....} ops: 0 {
     initial-use  at:
                            [<c1037ce3>] __lock_acquire+0x438/0xa13
                            [<c10385bb>] lock_acquire+0x61/0x80
                            [<c11f76a9>] _spin_lock_irqsave+0x38/0x48
                            [<c114e70a>] balloon_update_driver_allowance+0x10/0x25
                            [<d115c46b>] network_alloc_rx_buffers+0x304/0x3e1 [xennet]
                            [<d115d091>] backend_changed+0x77a/0x7b1 [xennet]
                            [<c114d508>] otherend_changed+0x84/0x8a
                            [<c114bdc2>] xenwatch_handle_callback+0x15/0x45
                            [<c114c886>] xenwatch_thread+0xfd/0x114
                            [<c102ff23>] kthread+0xb3/0xdc
                            [<c10058df>] kernel_thread_helper+0x7/0x10
                            [<ffffffff>] 0xffffffff
   }
   ... key      at: [<c12e04b0>] balloon_lock+0x10/0x1c
  ... acquired at:
   [<c103819a>] __lock_acquire+0x8ef/0xa13
   [<c10385bb>] lock_acquire+0x61/0x80
   [<c11f76a9>] _spin_lock_irqsave+0x38/0x48
   [<c114e70a>] balloon_update_driver_allowance+0x10/0x25
   [<d115c46b>] network_alloc_rx_buffers+0x304/0x3e1 [xennet]
   [<d115d091>] backend_changed+0x77a/0x7b1 [xennet]
   [<c114d508>] otherend_changed+0x84/0x8a
   [<c114bdc2>] xenwatch_handle_callback+0x15/0x45
   [<c114c886>] xenwatch_thread+0xfd/0x114
   [<c102ff23>] kthread+0xb3/0xdc
   [<c10058df>] kernel_thread_helper+0x7/0x10
   [<ffffffff>] 0xffffffff

 ... acquired at:
   [<c103819a>] __lock_acquire+0x8ef/0xa13
   [<c10385bb>] lock_acquire+0x61/0x80
   [<c11f7627>] _spin_lock+0x2b/0x38
   [<d115cf71>] backend_changed+0x65a/0x7b1 [xennet]
   [<c114d508>] otherend_changed+0x84/0x8a
   [<c114bdc2>] xenwatch_handle_callback+0x15/0x45
   [<c114c886>] xenwatch_thread+0xfd/0x114
   [<c102ff23>] kthread+0xb3/0xdc
   [<c10058df>] kernel_thread_helper+0x7/0x10
   [<ffffffff>] 0xffffffff


stack backtrace:
 [<c1005d9e>] show_trace_log_lvl+0x1a/0x2f
 [<c1006347>] show_trace+0x12/0x14
 [<c10063c2>] dump_stack+0x16/0x18
 [<c1036916>] print_irq_inversion_bug+0x10c/0x116
 [<c1036db7>] check_usage_backwards+0x3e/0x44
 [<c10370d2>] mark_lock+0x251/0x3de
 [<c                                                                            
 [<c                                                                            
 [<c                                                                            
VFS:                                                                            
 [<c                                                                            
 [<security:  3 users, 6 roles, 1816 types, 78 bools, 1 sens, 1024 cats
security:  60 classes, 65638 rules0x53b
SELinux:  Completing initialization.
SELinux:  Setting up existing superblocks.
SELinux: initialized (dev loop0, type squashfs), not configured for labeling
SELinux: initialized (dev ramfs, type ramfs), uses genfs_contexts
SELinux: initialized (dev ramfs, type ramfs), uses genfs_contexts
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev debugfs, type debugfs), uses genfs_contexts
SELinux: initialized (dev selinuxfs, type selinuxfs), uses genfs_contexts
SELinux: initialized (dev mqueue, type mqueue), uses transition SIDs
SELinux: initialized (dev devpts, type devpts), uses transition SIDs
SELinux: initialized (dev eventpollfs, type eventpollfs), uses task SIDs
SELinux: initialized (dev inotifyfs, type inotifyfs), uses genfs_contexts
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev futexfs, type futexfs), uses genfs_contexts
SELinux: initialized (dev pipefs, type pipefs), uses task SIDs
SELinux: initialized (dev sockfs, type sockfs), uses task SIDs
SELinux: initialized (dev cpuset, type cpuset), uses genfs_contexts
SELinux: initialized (dev proc, type proc), uses genfs_contexts
SELinux: initialized (dev bdev, type bdev), uses genfs_contexts
SELinux: initialized (dev rootfs, type rootfs), uses genfs_contexts
SELinux: initialized (dev sysfs, type sysfs), uses genfs_contexts
au