train_gnmt.py met memory runing issue #694

momo1986 · 2019-05-05T04:01:22Z

momo1986
May 5, 2019

Hello, thanks for sharing.
I run the script for google neural translation machine training.
Here is command:
python train_gnmt.py --src_lang en --tgt_lang vi --batch_size 128 --optimizer adam --lr 0.001 --lr_update_factor 0.5 --beam_size 10 --bucket_scheme exp --num_hidden 512 --save_dir gnmt_en_vi_l2_h512_beam10 --epochs 12 --gpu 0

Here is error-log:

Namespace(batch_size=128, beam_size=10, bucket_ratio=0.0, bucket_scheme='exp', clip=5.0, dataset='IWSLT2015', dropout=0.2, epochs=12, gpu=0, log_interval=100, lp_alpha=1.0, lp_k=5, lr=0.001, lr_update_factor=0.5, num_bi_layers=1, num_buckets=5, num_hidden=512, num_layers=2, optimizer='adam', save_dir='gnmt_en_vi_l2_h512_beam10', src_lang='en', src_max_len=50, test_batch_size=32, tgt_lang='vi', tgt_max_len=50)
All Logs will be saved to gnmt_en_vi_l2_h512_beam10/train_gnmt.log
Loading dataset...
Loading dataset...
Loading dataset...
[10:06:50] src/storage/storage.cc:135: Using GPUPooledRoundedStorageManager.
2019-05-05 10:06:50,663 - root - NMTModel(
(tgt_embed): HybridSequential(
(0): Embedding(7709 -> 512, float32)
(1): Dropout(p = 0.0, axes=())
)
(decoder): GNMTDecoder(
(rnn_cells): HybridSequential(
(0): LSTMCell(None -> 2048)
(1): LSTMCell(None -> 2048)
)
(dropout_layer): Dropout(p = 0.2, axes=())
(attention_cell): DotProductAttentionCell(
(_dropout_layer): Dropout(p = 0.0, axes=())
(_proj_query): Dense(None -> 512, linear)
)
)
(tgt_proj): Dense(None -> 7709, linear)
(src_embed): HybridSequential(
(0): Embedding(17191 -> 512, float32)
(1): Dropout(p = 0.0, axes=())
)
(encoder): GNMTEncoder(
(rnn_cells): HybridSequential(
(0): BidirectionalCell(forward=LSTMCell(None -> 2048), backward=LSTMCell(None -> 2048))
(1): LSTMCell(None -> 2048)
)
(dropout_layer): Dropout(p = 0.2, axes=())
)
)
2019-05-05 10:06:50,664 - root - Use beam_size=10, alpha=1.0, K=5
2019-05-05 10:06:50,964 - root - Train Batch Sampler:
FixedBucketSampler:
sample_num=133166, batch_num=1043
key=[(9, 10), (16, 17), (26, 27), (37, 38), (51, 52)]
cnt=[11414, 34897, 37760, 23480, 25615]
batch_size=[128, 128, 128, 128, 128]
2019-05-05 10:06:51,120 - root - Valid Batch Sampler:
FixedBucketSampler:
sample_num=1553, batch_num=51
key=[20, 39, 63, 91, 124]
cnt=[755, 608, 164, 19, 7]
batch_size=[32, 32, 32, 32, 32]
2019-05-05 10:06:51,279 - root - Test Batch Sampler:
FixedBucketSampler:
sample_num=1268, batch_num=42
key=[20, 40, 63, 91, 125]
cnt=[482, 535, 188, 53, 10]
batch_size=[32, 32, 32, 32, 32]
Traceback (most recent call last):
File "train_gnmt.py", line 280, in
train()
File "train_gnmt.py", line 219, in train
gnorm = gluon.utils.clip_global_norm(grads, args.clip)
File "/usr/local/lib/python3.5/dist-packages/mxnet/gluon/utils.py", line 148, in clip_global_norm
if not np.isfinite(total_norm.asscalar()):
File "/usr/local/lib/python3.5/dist-packages/mxnet/ndarray/ndarray.py", line 1998, in asscalar
return self.asnumpy()[0]
File "/usr/local/lib/python3.5/dist-packages/mxnet/ndarray/ndarray.py", line 1980, in asnumpy
ctypes.c_size_t(data.size)))
File "/usr/local/lib/python3.5/dist-packages/mxnet/base.py", line 252, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [10:06:51] src/storage/./pooled_storage_manager.h:299: cudaMalloc failed: out of memory

Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x40123a) [0x7fa98ba2423a]
[bt] (1) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x401851) [0x7fa98ba24851]
[bt] (2) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x340d25b) [0x7fa98ea3025b]
[bt] (3) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x341390e) [0x7fa98ea3690e]
[bt] (4) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(mxnet::NDArray::CheckAndAlloc() const+0x24b) [0x7fa98ba73c2b]
[bt] (5) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2d561ad) [0x7fa98e3791ad]
[bt] (6) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(mxnet::imperative::PushFCompute(std::function<void (nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&, std::vector<mxnet::OpReqType, std::allocatormxnet::OpReqType > const&, std::vector<mxnet::TBlob, std::allocatormxnet::TBlob > const&)> const&, nnvm::Op const*, nnvm::NodeAttrs const&, mxnet::Context const&, std::vector<mxnet::engine::Var*, std::allocatormxnet::engine::Var* > const&, std::vector<mxnet::engine::Var*, std::allocatormxnet::engine::Var* > const&, std::vector<mxnet::Resource, std::allocatormxnet::Resource > const&, std::vector<mxnet::NDArray*, std::allocatormxnet::NDArray* > const&, std::vector<mxnet::NDArray*, std::allocatormxnet::NDArray* > const&, std::vector<unsigned int, std::allocator > const&, std::vector<mxnet::OpReqType, std::allocatormxnet::OpReqType > const&)::{lambda(mxnet::RunContext)#1}::operator()(mxnet::RunContext) const+0x213) [0x7fa98e379683]
[bt] (7) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2ca7ab7) [0x7fa98e2caab7]
[bt] (8) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2ca7ab7) [0x7fa98e2caab7]
[bt] (9) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2ca7ab7) [0x7fa98e2caab7]

Only one GPU:

How can I fix the python command to adapt my hardware system?
Thanks & Regards!
Momo1986

Answered by szha

May 6, 2019

@momo1986 the missing shared object file libnvrtc.so.9.0 is for NVRTC, which should be part of cuda 9.0 as shown in the doc. Also, mxnet-cu90 only works with cuda 9.0 and mxnet-cu92 only with cuda 9.2, so let's try not to mix them on the same system.

I know installing cuda can be tricky on some systems, and personally I find installing from the local runfile to be most reliable. Given that the symptom points to a problem in cuda installation, I'd recommend the following:

If possible, use CUDA 9.2 and mxnet-cu92.
If you don't have sudo access, make sure you work with you system admin on setting the right LD_LIBRARY_PATH. If you're sure the right path is included and it's still not working…

View full answer

szha · 2019-05-05T05:04:03Z

szha
May 5, 2019
Maintainer

Hi @momo1986. Thanks for reporting the issue. What version of mxnet are you using? (you can find by using pip show $package_name)

0 replies

momo1986 · 2019-05-05T05:16:49Z

momo1986
May 5, 2019
Author

Hi @momo1986. Thanks for reporting the issue. What version of mxnet are you using? (you can find by using pip show $package_name)

Hello, @szha ， I use MXNET 1.4.0.post0.

Thanks for your concern!
Regards!

0 replies

szha · 2019-05-05T05:33:22Z

szha
May 5, 2019
Maintainer

@momo1986 there have been some improvements in memory management and dropout state sharing that may be helpful to you. You can try it out by installing the version 1.5.0b20190411 to try it out. If it still doesn't fit, you may need to reduce the batch size or use gradient accumulation to simulate large batch size. (an example can be found in the train_transformer.py)

0 replies

momo1986 · 2019-05-05T07:42:58Z

momo1986
May 5, 2019
Author

@momo1986 there have been some improvements in memory management and dropout state sharing that may be helpful to you. You can try it out by installing the version 1.5.0b20190411 to try it out. If it still doesn't fit, you may need to reduce the batch size or use gradient accumulation to simulate large batch size. (an example can be found in the train_transformer.py)

Hello, @szha .

I try to build with mxnet1.5.

However, the problem is reported:
OSError: libcudart.so.9.2: cannot open shared object file: No such file or directory

I roll back to 1.4.

I tried with the command:
python train_transformer.py --dataset WMT2014BPE --src_lang en --tgt_lang de --batch_size 128 --optimizer adam --num_accumulated 16 --lr 0.01 --warmup_steps 4000 --save_dir transformer_en_de_u512 --epochs 250 --gpus 0 --scaled --average_start 5 --num_buckets 20 --bucket_scheme exp --bleu 13a --log_interval 10 --num_accumulated 1

However, it failed with such stacks:

All Logs will be saved to transformer_en_de_u512/train_transformer.log
2019-05-05 15:39:53,193 - root - Namespace(average_checkpoint=False, average_start=5, batch_size=128, beam_size=4, bleu='13a', bucket_ratio=0.0, bucket_scheme='exp', dataset='WMT2014BPE', dropout=0.1, epochs=250, epsilon=0.1, full=False, gpus='0', hidden_size=2048, log_interval=10, lp_alpha=0.6, lp_k=5, lr=0.01, magnitude=3.0, num_accumulated=1, num_averages=5, num_buckets=20, num_heads=8, num_layers=6, num_units=512, optimizer='adam', save_dir='transformer_en_de_u512', scaled=True, src_lang='en', src_max_len=-1, test_batch_size=256, tgt_lang='de', tgt_max_len=-1, warmup_steps=4000.0)
Loading dataset...
Loading dataset...
Loading dataset...
terminate called after throwing an instance of 'dmlc::Error'
what(): [15:40:35] /home/travis/build/dmlc/mxnet-distro/mxnet-build/3rdparty/mshadow/mshadow/./stream_gpu-inl.h:182: Check failed: e == cudaSuccess CUDA: out of memory

Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x40123a) [0x7f8e1307123a]
[bt] (1) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x401851) [0x7f8e13071851]
[bt] (2) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2c9f8d1) [0x7f8e1590f8d1]
[bt] (3) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2ca0325) [0x7f8e15910325]
[bt] (4) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2cb84e7) [0x7f8e159284e7]
[bt] (5) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2cb8786) [0x7f8e15928786]
[bt] (6) /usr/local/lib/python3.5/dist-packages/mxnet/libmxnet.so(+0x2cb1bc4) [0x7f8e15921bc4]
[bt] (7) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7f8e63863c80]
[bt] (8) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f8e6876c6ba]
[bt] (9) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f8e684a241d]

Aborted (core dumped)

Thanks & regards!

0 replies

momo1986 · 2019-05-05T08:09:06Z

momo1986
May 5, 2019
Author

Set both MXNET_GPU_MEM_POOL_TYPE=Round and MXNET_GPU_MEM_POOL_RESERVE=10

0 replies

szha · 2019-05-05T19:23:39Z

szha
May 5, 2019
Maintainer

@momo1986

OSError: libcudart.so.9.2: cannot open shared object file: No such file or directory

This means the library didn't find cuda 9.2 in your path. Are you using cuda 9.2?

0 replies

momo1986 · 2019-05-06T02:49:21Z

momo1986
May 6, 2019
Author

Hello, @szha
I affirm that my CUDA version is 9.0.

However, problem exists both for mxnet-cu92 and mxnet-cu90.

Traceback (most recent call last):
File "train_transformer.py", line 44, in
import mxnet as mx
File "/usr/local/lib/python3.5/dist-packages/mxnet/init.py", line 24, in
from .context import Context, current_context, cpu, gpu, cpu_pinned
File "/usr/local/lib/python3.5/dist-packages/mxnet/context.py", line 24, in
from .base import classproperty, with_metaclass, _MXClassPropertyMetaClass
File "/usr/local/lib/python3.5/dist-packages/mxnet/base.py", line 214, in
_LIB = _load_lib()
File "/usr/local/lib/python3.5/dist-packages/mxnet/base.py", line 205, in _load_lib
lib = ctypes.CDLL(lib_path[0], ctypes.RTLD_LOCAL)
File "/usr/lib/python3.5/ctypes/init.py", line 347, in init
self._handle = _dlopen(self._name, mode)
OSError: libnvrtc.so.9.0: cannot open shared object file: No such file or directory

0 replies

szha · 2019-05-06T04:59:50Z

szha
May 6, 2019
Maintainer

@momo1986 the missing shared object file libnvrtc.so.9.0 is for NVRTC, which should be part of cuda 9.0 as shown in the doc. Also, mxnet-cu90 only works with cuda 9.0 and mxnet-cu92 only with cuda 9.2, so let's try not to mix them on the same system.

I know installing cuda can be tricky on some systems, and personally I find installing from the local runfile to be most reliable. Given that the symptom points to a problem in cuda installation, I'd recommend the following:

If possible, use CUDA 9.2 and mxnet-cu92.
If you don't have sudo access, make sure you work with you system admin on setting the right LD_LIBRARY_PATH. If you're sure the right path is included and it's still not working, notify the admin of the missing NVRTC problem to fix the CUDA installation.
If you have sudo access to the system, you can choose to reinstall CUDA. The CUDA 9.2 installation files can be found here
In case you follow the suggestion to use a local runfile, you will have the chance to specify the installation path. Be sure to note it down. Once you finish installation, you need to make sure that the lib64 directory in your installation path is specified in LD_LIBRARY_PATH variable.

Hope it helps. Feel free to let us know if you need more help. You can also join our slack channel (registration link) so that you may request help there..

0 replies

szha · 2019-05-18T01:20:20Z

szha
May 18, 2019
Maintainer

@momo1986 did you get a chance to try out the above?

0 replies

leezu · 2019-09-24T06:53:07Z

leezu
Sep 24, 2019
Maintainer

Closing due to lack of activity, feel free to reopen if you have more questions

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

train_gnmt.py met memory runing issue #694

{{title}}

Replies: 10 comments

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

train_gnmt.py met memory runing issue #694

momo1986 May 5, 2019

Replies: 10 comments

szha May 5, 2019 Maintainer

momo1986 May 5, 2019 Author

szha May 5, 2019 Maintainer

momo1986 May 5, 2019 Author

momo1986 May 5, 2019 Author

szha May 5, 2019 Maintainer

momo1986 May 6, 2019 Author

szha May 6, 2019 Maintainer

szha May 18, 2019 Maintainer

leezu Sep 24, 2019 Maintainer

momo1986
May 5, 2019

szha
May 5, 2019
Maintainer

momo1986
May 5, 2019
Author

szha
May 5, 2019
Maintainer

momo1986
May 5, 2019
Author

momo1986
May 5, 2019
Author

szha
May 5, 2019
Maintainer

momo1986
May 6, 2019
Author

szha
May 6, 2019
Maintainer

szha
May 18, 2019
Maintainer

leezu
Sep 24, 2019
Maintainer