You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Have you reproduced the bug with TensorFlow Nightly?
Yes
Source
source
TensorFlow version
2.15.0
Custom code
Yes
OS platform and distribution
colab
Mobile device
No response
Python version
3.10.12
Bazel version
No response
GCC/compiler version
11.4.0
CUDA/cuDNN version
No response
GPU model and memory
No response
Current behavior?
When creating a tf.data.Dataset object using from_generator a naming collision with an iterable used in the generator parameter causes a crash.
On Ubuntu I got the following error (not in colab):
date: F external/local_tsl/tsl/platform/default/env.cc:74] Check failed: ret == 0 (11 vs. 0)Thread tf_data_private_threadpool creation via pthread_create() failed.
Standalone code to reproduce the issue
import sys
print(f"Python version: {sys.version}")
import tensorflow as tf
print(f"{tf.__version__ = }")
# This runs as expected:
ds_no_name_collision = list(range(100))
def gen():
print("gen called")
forxin ds_no_name_collision:
print(f"{x = }")
yield x
# Notice there is no name collision:
ds_without_name_collision = tf.data.Dataset.from_generator(gen, output_signature=tf.TensorSpec(shape=(), dtype=tf.int32))
print("All good so far")
forxin ds_without_name_collision.take(10):
print(x)
# This fails:
ds_name_collision = list(range(100))
def gen():
print("gen called")
forxin ds_name_collision:
print(f"{x = }")
yield x
# Notice the name collision:
ds_name_collision = tf.data.Dataset.from_generator(gen, output_signature=tf.TensorSpec(shape=(), dtype=tf.int32))
print("All good so far")
forxin ds_name_collision.take(10):
print(x)
Relevant log output
Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
tf.__version__ = '2.15.0'
All good so far
gen called
x = 0
x = 1
tf.Tensor(0, shape=(), dtype=int32)
tf.Tensor(1, shape=(), dtype=int32)
x = 2
x = 3
x = 4
tf.Tensor(2, shape=(), dtype=int32)
x = 5
tf.Tensor(3, shape=(), dtype=int32)
tf.Tensor(4, shape=(), dtype=int32)
tf.Tensor(5, shape=(), dtype=int32)
x = 6
x = 7
x = 8
x = 9
tf.Tensor(6, shape=(), dtype=int32)
tf.Tensor(7, shape=(), dtype=int32)
tf.Tensor(8, shape=(), dtype=int32)
tf.Tensor(9, shape=(), dtype=int32)
All good so far
gen called
gen called
gen called
gen called
gen called
gen called
gen called
[log truncated]
The text was updated successfully, but these errors were encountered:
I have replicated the issue with TF2.15v and tf-nightly as well. On Colab the code executes indefinitely due to naming collision of generator and finally runtime will crash without any logs. Attaching gist for reference.
@SuryanarayanaY thank you for your response. One can get a crash (as opposed to indefinite execution without errors) in an official docker (docker pull tensorflow/tensorflow:latest-gpu). I will provide concrete steps if needed.
Hi @SuryanarayanaY is there some plan to resolve this issue? Do you know what causes it? In case resolving would require too large changes do you think it is worth putting a note in the documentation since it is rather hard to figure out what causes this behavior?
Personally I would be interested in the cause of this since some magic needs to be happening here :-)
Issue type
Documentation Bug
Have you reproduced the bug with TensorFlow Nightly?
Yes
Source
source
TensorFlow version
2.15.0
Custom code
Yes
OS platform and distribution
colab
Mobile device
No response
Python version
3.10.12
Bazel version
No response
GCC/compiler version
11.4.0
CUDA/cuDNN version
No response
GPU model and memory
No response
Current behavior?
When creating a
tf.data.Dataset
object usingfrom_generator
a naming collision with an iterable used in thegenerator
parameter causes a crash.On Ubuntu I got the following error (not in colab):
date: F external/local_tsl/tsl/platform/default/env.cc:74] Check failed: ret == 0 (11 vs. 0)Thread tf_data_private_threadpool creation via pthread_create() failed.
Standalone code to reproduce the issue
Relevant log output
Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] tf.__version__ = '2.15.0' All good so far gen called x = 0 x = 1 tf.Tensor(0, shape=(), dtype=int32) tf.Tensor(1, shape=(), dtype=int32) x = 2 x = 3 x = 4 tf.Tensor(2, shape=(), dtype=int32) x = 5 tf.Tensor(3, shape=(), dtype=int32) tf.Tensor(4, shape=(), dtype=int32) tf.Tensor(5, shape=(), dtype=int32) x = 6 x = 7 x = 8 x = 9 tf.Tensor(6, shape=(), dtype=int32) tf.Tensor(7, shape=(), dtype=int32) tf.Tensor(8, shape=(), dtype=int32) tf.Tensor(9, shape=(), dtype=int32) All good so far gen called gen called gen called gen called gen called gen called gen called [log truncated]
The text was updated successfully, but these errors were encountered: