Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

georeferencing of seven A1 sketch maps failed due to result size being too large to store #399

Open
matthiasschaub opened this issue Mar 25, 2024 · 0 comments
Open

georeferencing of seven A1 sketch maps failed due to result size being too large to store #399

matthiasschaub opened this issue Mar 25, 2024 · 0 comments
Labels
bug Something isn't working component:upload-processing priority:high Should be addressed as soon as possible (next release)

Comments

@matthiasschaub
Copy link
Collaborator

Not the georeferencing itself is failing but the storage of the result zip is failing. The vector result are provided perfectly.

psycopg2.errors.InternalError_: invalid memory alloc request size 1073741824

celery-1    | 2024-03-25 08:41:36,417 - WARNING - __init__.py - _inner - Failed operation _store_result.  Retrying 0 more times.
celery-1    | Traceback (most recent call last):
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1970, in _exec_single_context
celery-1    |     self.dialect.do_execute(
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 924, in do_execute
celery-1    |     cursor.execute(statement, parameters)
celery-1    | psycopg2.errors.InternalError_: invalid memory alloc request size 1073741824
celery-1    |
celery-1    |
celery-1    | The above exception was the direct cause of the following exception:
celery-1    |
celery-1    | Traceback (most recent call last):
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/celery/backends/database/__init__.py", line 47, in _inner
celery-1    |     return fun(*args, **kwargs)
celery-1    |            ^^^^^^^^^^^^^^^^^^^^
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/celery/backends/database/__init__.py", line 126, in _store_result
celery-1    |     session.commit()
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1972, in commit
celery-1    |     trans.commit(_to_root=True)
celery-1    |   File "<string>", line 2, in commit
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
celery-1    |     ret_value = fn(self, *arg, **kw)
celery-1    |                 ^^^^^^^^^^^^^^^^^^^^
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1257, in commit
celery-1    |     self._prepare_impl()
celery-1    |   File "<string>", line 2, in _prepare_impl
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
celery-1    |     ret_value = fn(self, *arg, **kw)
celery-1    |                 ^^^^^^^^^^^^^^^^^^^^
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1232, in _prepare_impl
celery-1    |     self.session.flush()
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4296, in flush
celery-1    |     self._flush(objects)
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4431, in _flush
celery-1    |     with util.safe_reraise():
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
celery-1    |     raise exc_value.with_traceback(exc_tb)
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4392, in _flush
celery-1    |     flush_context.execute()
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
celery-1    |     rec.execute(self)
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute
celery-1    |     util.preloaded.orm_persistence.save_obj(
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py", line 85, in save_obj
celery-1    |     _emit_update_statements(
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py", line 912, in _emit_update_statements
celery-1    |     c = connection.execute(
celery-1    |         ^^^^^^^^^^^^^^^^^^^
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1421, in execute
celery-1    |     return meth(
celery-1    |            ^^^^^
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 514, in _execute_on_connection
celery-1    |     return connection._execute_clauseelement(
celery-1    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1643, in _execute_clauseelement
celery-1    |     ret = self._execute_context(
celery-1    |           ^^^^^^^^^^^^^^^^^^^^^^
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1849, in _execute_context
celery-1    |     return self._exec_single_context(
celery-1    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1989, in _exec_single_context
celery-1    |     self._handle_dbapi_exception(
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2356, in _handle_dbapi_exception
celery-1    |     raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1970, in _exec_single_context
celery-1    |     self.dialect.do_execute(
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 924, in do_execute
celery-1    |     cursor.execute(statement, parameters)
celery-1    | sqlalchemy.exc.InternalError: (psycopg2.errors.InternalError_) invalid memory alloc request size 1073741824
celery-1    |
celery-1    | [SQL: UPDATE celery_taskmeta SET status=%(status)s, result=%(result)s, date_done=%(date_done)s WHERE celery_taskmeta.id = %(celery_taskmeta_id)s]
celery-1    | [parameters: {'status': 'SUCCESS', 'result': <psycopg2.extensions.Binary object at 0x7ffadc3dc5a0>, 'date_done': datetime.datetime(2024, 3, 25, 8, 39, 40, 693247), 'celery_taskmeta_id': 947}]
celery-1    | (Background on this error at: https://sqlalche.me/e/20/2j85)
celery-1    | 2024-03-25 08:41:36,434 - WARNING - warnings.py - _showwarnmsg - /home/smt/.conda/envs/smt/lib/python3.11/site-packages/celery/app/trace.py:686: RuntimeWarning: Exception raised outside body: InternalError('(psycopg2.errors.InternalError_) invalid memory alloc request size 1073741824\n'):
celery-1    | Traceback (most recent call last):
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/celery/app/trace.py", line 544, in trace_task
celery-1    |     task.backend.mark_as_done(
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/celery/backends/base.py", line 157, in mark_as_done
celery-1    |     self.store_result(task_id, result, state, request=request)
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/celery/backends/base.py", line 526, in store_result
celery-1    |     self._store_result(task_id, result, state, traceback,
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/celery/backends/database/__init__.py", line 47, in _inner
celery-1    |     return fun(*args, **kwargs)
celery-1    |            ^^^^^^^^^^^^^^^^^^^^
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/celery/backends/database/__init__.py", line 126, in _store_result
celery-1    |     session.commit()
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1972, in commit
celery-1    |     trans.commit(_to_root=True)
celery-1    |   File "<string>", line 2, in commit
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
celery-1    |     ret_value = fn(self, *arg, **kw)
celery-1    |                 ^^^^^^^^^^^^^^^^^^^^
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1257, in commit
celery-1    |     self._prepare_impl()
celery-1    |   File "<string>", line 2, in _prepare_impl
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
celery-1    |     ret_value = fn(self, *arg, **kw)
celery-1    |                 ^^^^^^^^^^^^^^^^^^^^
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1232, in _prepare_impl
celery-1    |     self.session.flush()
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4296, in flush
celery-1    |     self._flush(objects)
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4431, in _flush
celery-1    |     with util.safe_reraise():
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
celery-1    |     raise exc_value.with_traceback(exc_tb)
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4392, in _flush
celery-1    |     flush_context.execute()
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
celery-1    |     rec.execute(self)
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute
celery-1    |     util.preloaded.orm_persistence.save_obj(
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py", line 85, in save_obj
celery-1    |     _emit_update_statements(
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py", line 912, in _emit_update_statements
celery-1    |     c = connection.execute(
celery-1    |         ^^^^^^^^^^^^^^^^^^^
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1421, in execute
celery-1    |     return meth(
celery-1    |            ^^^^^
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 514, in _execute_on_connection
celery-1    |     return connection._execute_clauseelement(
celery-1    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1643, in _execute_clauseelement
celery-1    |     ret = self._execute_context(
celery-1    |           ^^^^^^^^^^^^^^^^^^^^^^
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1849, in _execute_context
celery-1    |     return self._exec_single_context(
celery-1    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1989, in _exec_single_context
celery-1    |     self._handle_dbapi_exception(
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2356, in _handle_dbapi_exception
celery-1    |     raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1970, in _exec_single_context
celery-1    |     self.dialect.do_execute(
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 924, in do_execute
celery-1    |     cursor.execute(statement, parameters)
celery-1    | sqlalchemy.exc.SQLAlchemyError: (psycopg2.errors.InternalError_) invalid memory alloc request size 1073741824
celery-1    |
celery-1    |
celery-1    |   warn(RuntimeWarning(
celery-1    |
celery-1    | 2024-03-25 08:41:37,245 - ERROR - trace.py - _log_error - Task sketch_map_tool.tasks.georeference_sketch_maps[6909e3eb-3cd6-448b-a389-3fa3b084e6e2] raised unexpected: SQLAlchemyError('(psycopg2.errors.InternalError_) invalid memory alloc request size 1073741824\n')
celery-1    | Traceback (most recent call last):
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/celery/app/trace.py", line 544, in trace_task
celery-1    |     task.backend.mark_as_done(
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/celery/backends/base.py", line 157, in mark_as_done
celery-1    |     self.store_result(task_id, result, state, request=request)
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/celery/backends/base.py", line 526, in store_result
celery-1    |     self._store_result(task_id, result, state, traceback,
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/celery/backends/database/__init__.py", line 47, in _inner
celery-1    |     return fun(*args, **kwargs)
celery-1    |            ^^^^^^^^^^^^^^^^^^^^
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/celery/backends/database/__init__.py", line 126, in _store_result
celery-1    |     session.commit()
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1972, in commit
celery-1    |     trans.commit(_to_root=True)
celery-1    |   File "<string>", line 2, in commit
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
celery-1    |     ret_value = fn(self, *arg, **kw)
celery-1    |                 ^^^^^^^^^^^^^^^^^^^^
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1257, in commit
celery-1    |     self._prepare_impl()
celery-1    |   File "<string>", line 2, in _prepare_impl
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
celery-1    |     ret_value = fn(self, *arg, **kw)
celery-1    |                 ^^^^^^^^^^^^^^^^^^^^
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1232, in _prepare_impl
celery-1    |     self.session.flush()
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4296, in flush
celery-1    |     self._flush(objects)
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4431, in _flush
celery-1    |     with util.safe_reraise():
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
celery-1    |     raise exc_value.with_traceback(exc_tb)
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4392, in _flush
celery-1    |     flush_context.execute()
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
celery-1    |     rec.execute(self)
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute
celery-1    |     util.preloaded.orm_persistence.save_obj(
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py", line 85, in save_obj
celery-1    |     _emit_update_statements(
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py", line 912, in _emit_update_statements
celery-1    |     c = connection.execute(
celery-1    |         ^^^^^^^^^^^^^^^^^^^
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1421, in execute
celery-1    |     return meth(
celery-1    |            ^^^^^
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 514, in _execute_on_connection
celery-1    |     return connection._execute_clauseelement(
celery-1    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1643, in _execute_clauseelement
celery-1    |     ret = self._execute_context(
celery-1    |           ^^^^^^^^^^^^^^^^^^^^^^
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1849, in _execute_context
celery-1    |     return self._exec_single_context(
celery-1    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1989, in _exec_single_context
celery-1    |     self._handle_dbapi_exception(
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2356, in _handle_dbapi_exception
celery-1    |     raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1970, in _exec_single_context
celery-1    |     self.dialect.do_execute(
celery-1    |   File "/home/smt/.conda/envs/smt/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 924, in do_execute
celery-1    |     cursor.execute(statement, parameters)
celery-1    | sqlalchemy.exc.SQLAlchemyError: (psycopg2.errors.InternalError_) invalid memory alloc request size 1073741824
celery-1    |
@matthiasschaub matthiasschaub added bug Something isn't working priority:high Should be addressed as soon as possible (next release) component:ml-marking-detection component:upload-processing and removed component:ml-marking-detection labels Mar 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working component:upload-processing priority:high Should be addressed as soon as possible (next release)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@matthiasschaub