ClownWiki

あるとき、rockmongoで以下のようなエラーが発生。

Execute failed:exception: Can't take a write lock while out of disk space
function (){ return db.getCollectionNames(); }

サーバにsshしてみると、レスポンスがやたら悪い。
topコマンド実行してみると、LAがやたら高い。

td-agent（fluentd）のログと見てみると、空き容量がないとのこと。

#less /var/log/td-agent/td-agent.log

～省略～
2013-04-19 10:53:59 +0900 [warn]: temporarily failed to flush the buffer, next retry will be at 2013-04-19 10:53:59 +0900. error="14031: Can't take a write lock while out of disk space" instance=23909922547640
～省略～

でも、dfでみるとまだ800MB弱の空き容量。

# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       3904168   2911456    791192  79% /
/dev/xvda1              101086     22430     73437  24% /boot
tmpfs                   524288         0    524288   0% /dev/shm

とりあえず不要なdbを削除して空き容量を確保。。。と思ったら、その操作もエラー。

# mongo --port 27017
MongoDB shell version: 2.4.1
connecting to: 127.0.0.1:27017/test
> show dbs
admin   (empty)
apache  0.203125GB
debug      0.953125GB
local   0.078125GB
> use apache
switched to db apache
> db.dropDatabase()
{
        "assertion" : "Can't take a write lock while out of disk space",
        "assertionCode" : 14031,
        "errmsg" : "db assertion failure",
        "ok" : 0
}
> 

仕方がないので、先人の知恵を借りるべく徘徊してみる。

公式マニュアル - FAQ（英語）：http://docs.mongodb.org/manual/faq/storage/#faq-disk-size

Preallocated data files.
In the data directory, MongoDB preallocates data files to a particular size, in part to prevent file system fragmentation. MongoDB names first data file <databasename>.0, the next <databasename>.1, etc. The first file mongod allocates is 64 megabytes, the next 128 megabytes, and so on, up to 2 gigabytes, at which point all subsequent files are 2 gigabytes. The data files include files with allocated space but that hold no data. mongod may allocate a 1 gigabyte data file that may be 90% empty. For most larger databases, unused allocated space is small compared to the database.

On Unix-like systems, mongod preallocates an additional data file and initializes the disk space to 0. Preallocating data files in the background prevents significant delays when a new database file is next allocated.

You can disable preallocation with noprealloc run time option. However noprealloc is not intended for use in production environments: only use noprealloc for testing and with small data sets where you frequently drop databases.

On Linux systems you can use hdparm to get an idea of how costly allocation might be:

 time hdparm --fallocate $((1024*1024)) testfile

10gen（日本語訳）：https://wiki.10gen.com/pages/viewpage.action?pageId=17596968

db用のスペースを先取りするために、大きな空き領域が必要です。。。とのこと。

現状を確認してみると、次に先取りしたい容量は1024MB。。。空き容量800MBでは無理ですね。

# ls -hl /var/lib/mongo
total 1.3G
-rw------- 1 mongod mongod  64M Apr 17 19:08 apache.0
-rw------- 1 mongod mongod 128M Apr 17 19:08 apache.1
-rw------- 1 mongod mongod  16M Apr 17 19:08 apache.ns
drwxr-xr-x 2 mongod mongod 4.0K Apr 17 18:50 journal
-rw------- 1 mongod mongod  64M Apr 19 10:52 debug.0
-rw------- 1 mongod mongod 128M Apr 19 10:52 debug.1
-rw------- 1 mongod mongod 256M Apr 19 10:52 debug.2
-rw------- 1 mongod mongod 512M Apr 19 10:52 debug.3
-rw------- 1 mongod mongod  16M Apr 19 10:52 debug.ns
-rw------- 1 mongod mongod  64M Apr 17 20:23 local.0
-rw------- 1 mongod mongod  16M Apr 17 20:23 local.ns
-rwxr-xr-x 1 mongod mongod    6 Apr 17 20:23 mongod.lock
drwxr-xr-x 2 mongod mongod 4.0K Apr 19 11:10 _tmp

今回は、VMのディスク容量を増やして対処したけど、それができない場合の緊急対応ってどうするんだろう。。。