Out of Memory

今日の夕方ごろサーバでメモリ不足が発生していろいろなプロセスがダウンした様子。

# cat /var/log/kernel.log
Apr 21 15:41:12 bruna kernel: Out of Memory: Killed process 18377 (httpd).
Apr 21 15:44:24 bruna kernel: oom-killer: gfp_mask=0x1d2
Apr 21 15:47:22 bruna kernel: Mem-info:
Apr 21 15:50:02 bruna kernel: DMA per-cpu:
Apr 21 15:51:45 bruna kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 21 15:55:51 bruna kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 21 15:58:38 bruna kernel: Normal per-cpu:
Apr 21 16:01:57 bruna kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 21 16:05:33 bruna kernel: cpu 0 cold: low 0, high 32, batch 16
Apr 21 16:27:28 bruna kernel: HighMem per-cpu: empty
Apr 21 16:27:40 bruna kernel:
Apr 21 16:28:13 bruna kernel: Free pages:         716kB (0kB HighMem)
Apr 21 16:28:23 bruna kernel: Active:78523 inactive:41040 dirty:0 writeback:0 unstable:0 free:179 slab:3440 mapped:120491 pagetables:2484
Apr 21 16:28:24 bruna kernel: DMA free:20kB min:20kB low:40kB high:60kB active:10944kB inactive:176kB present:16384kB pages_scanned:1043258
all_unreclaimable? yes
Apr 21 16:28:24 bruna kernel: protections[]: 0 0 0
Apr 21 16:28:24 bruna kernel: Normal free:696kB min:696kB low:1392kB high:2088kB active:303148kB inactive:163984kB present:507776kB pages_s
canned:20751918 all_unreclaimable? yes
Apr 21 16:28:24 bruna kernel: protections[]: 0 0 0
Apr 21 16:28:24 bruna kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unrec
laimable? no
Apr 21 16:28:24 bruna kernel: protections[]: 0 0 0
Apr 21 16:31:22 bruna kernel: DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
Apr 21 16:31:23 bruna kernel: Normal: 10*4kB 2*8kB 0*16kB 4*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 696kB
Apr 21 16:31:23 bruna kernel: HighMem: empty
Apr 21 16:31:23 bruna kernel: Swap cache: add 322507, delete 322506, find 568501/595776, race 0+46
Apr 21 16:31:23 bruna kernel: 0 bounce buffer pages
Apr 21 16:31:23 bruna kernel: Free swap:            0kB
Apr 21 16:31:23 bruna kernel: 131040 pages of RAM
Apr 21 16:31:23 bruna kernel: 0 pages of HIGHMEM
Apr 21 16:31:23 bruna kernel: 2182 reserved pages
Apr 21 16:31:23 bruna kernel: 24199 pages shared
Apr 21 16:32:32 bruna kernel: 1 pages swap cached

こんな感じのログがわらわら出ていました。
死んだプロセス一覧。

# cat /var/log/kernel.log |grep "Out of Memory"
Apr 21 15:41:12 bruna kernel: Out of Memory: Killed process 18377 (httpd).
Apr 21 16:32:32 bruna kernel: Out of Memory: Killed process 18378 (httpd).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18379 (httpd).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18380 (httpd).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18381 (httpd).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18388 (httpd).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18454 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18347 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18407 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18455 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18387 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18342 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18451 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18431 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18418 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18489 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18504 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18416 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18487 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18372 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18417 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18386 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18474 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18490 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18498 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18469 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18470 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18466 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18443 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18364 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18511 (mt-tb.cgi).
Apr 21 16:48:41 bruna kernel: Out of Memory: Killed process 18421 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18420 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18368 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18419 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18398 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18425 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18429 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18453 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18522 (mobile.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18432 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18452 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18515 (mobile.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18568 (mobile.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18520 (mobile.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18499 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18537 (mobile.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18433 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18403 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18389 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18370 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18401 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18565 (mobile.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18546 (mobile.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18435 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18391 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18480 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18461 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18510 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18392 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18479 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18442 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18531 (mobile.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18467 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18400 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18458 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18468 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18519 (mobile.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18488 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18523 (mobile.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18444 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 11822 (httpd).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18501 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18496 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18502 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18521 (mobile.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18497 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18558 (mobile.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18587 (mobile.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18574 (mobile.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18573 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18383 (mt-tb.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 8949 (spamd).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18665 (mobile.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18663 (mobile.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18664 (mobile.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18667 (mobile.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18666 (mobile.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18662 (mobile.cgi).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 3670 (httpd).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18527 (smtpd).
Apr 21 16:48:42 bruna kernel: Out of Memory: Killed process 18529 (anvil).
Apr 21 16:48:43 bruna kernel: Out of Memory: Killed process 18692 (mobile.cgi).
Apr 21 16:48:43 bruna kernel: Out of Memory: Killed process 18685 (httpd).
Apr 21 16:48:43 bruna kernel: Out of Memory: Killed process 18686 (httpd).
Apr 21 16:48:43 bruna kernel: Out of Memory: Killed process 18687 (httpd).
Apr 21 16:48:43 bruna kernel: Out of Memory: Killed process 18550 (pipe).
Apr 21 16:48:43 bruna kernel: Out of Memory: Killed process 18514 (mrtg).
Apr 21 16:48:43 bruna kernel: Out of Memory: Killed process 18512 (crond).

コメントを残す

メールアドレスが公開されることはありません。

日本語が含まれない投稿は無視されますのでご注意ください。(スパム対策)