Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GPKG] "must specify native_crs" #233

Closed
etj opened this issue Apr 9, 2024 · 2 comments · Fixed by #234
Closed

[GPKG] "must specify native_crs" #233

etj opened this issue Apr 9, 2024 · 2 comments · Fixed by #234
Assignees
@mattiagiupponi

Comments

@etj
Copy link
Contributor

etj commented Apr 9, 2024

The importer finds 2 layers:

  • aree, which is properly imported
  • layer_styles which fails

The GUI reports an error:
image

Celery log 
[2024-04-09 15:49:29,459: INFO/ForkPoolWorker-8] STARTING NEXT STEP importer.import_resource
[2024-04-09 15:49:35,113: INFO/ForkPoolWorker-8] Total number of layers available: 2
[2024-04-09 15:49:36,808: INFO/ForkPoolWorker-2] STARTING NEXT STEP importer.publish_resource
[2024-04-09 15:49:36,808: INFO/ForkPoolWorker-2] STARTING NEXT STEP importer.publish_resource for resource: layer_styles, alternate layer_styles
[2024-04-09 15:49:37,058: ERROR/ForkPoolWorker-12] error during publishing: must specify native_crs
[2024-04-09 15:49:37,146: INFO/ForkPoolWorker-2] STARTING NEXT STEP importer.rollback
[2024-04-09 15:49:37,146: INFO/ForkPoolWorker-2] STARTING NEXT STEP importer.rollback for resource: layer_styles, alternate layer_styles
[2024-04-09 15:49:37,150: INFO/ForkPoolWorker-2] Calling rollback for execution_id b98fc0b8-cd6d-481e-90ec-398bee9ed5ad in progress
[2024-04-09 15:49:37,264: WARNING/ForkPoolWorker-2] Starting rollback for execid: b98fc0b8-cd6d-481e-90ec-398bee9ed5ad resource published was: layer_styles
[2024-04-09 15:49:37,264: INFO/ForkPoolWorker-2] Rollback publishing step in progress for execid: b98fc0b8-cd6d-481e-90ec-398bee9ed5ad resource published was: layer_styles
[2024-04-09 15:49:37,320: ERROR/ForkPoolWorker-12] Task FAILED with ID: b98fc0b8-cd6d-481e-90ec-398bee9ed5ad, reason: must specify native_crs. Request: b98fc0b8-cd6d-481e-90ec-398bee9ed5ad
[2024-04-09 15:49:37,361: INFO/ForkPoolWorker-2] Rollback dynamic model & ogr2ogr step in progress for execid: b98fc0b8-cd6d-481e-90ec-398bee9ed5ad resource published was: layer_styles
[2024-04-09 15:49:37,361: INFO/ForkPoolWorker-2] Dynamic model does not exists, removing ogr2ogr table in progress
[2024-04-09 15:49:38,101: WARNING/ForkPoolWorker-2] Rollback for execid: b98fc0b8-cd6d-481e-90ec-398bee9ed5ad resource published was: layer_styles completed
[2024-04-09 15:49:38,135: ERROR/ForkPoolWorker-12] Task importer.publish_resource[1362c5ff-2480-41aa-9418-d6b704b976f4] raised unexpected: PublishResourceException()
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/importer/celery_tasks.py", line 236, in publish_resource
    _publisher.publish_resources(data)
  File "/usr/local/lib/python3.10/dist-packages/importer/publisher.py", line 67, in publish_resources
    result = self.handler.publish_resources(
  File "/usr/local/lib/python3.10/dist-packages/importer/handlers/common/vector.py", line 138, in publish_resources
    raise e
  File "/usr/local/lib/python3.10/dist-packages/importer/handlers/common/vector.py", line 126, in publish_resources
    catalog.publish_featuretype(
  File "/usr/local/lib/python3.10/dist-packages/geoserver/catalog.py", line 898, in publish_featuretype
    raise ValueError("must specify native_crs")
ValueError: must specify native_crs

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/importer/celery_tasks.py", line 282, in publish_resource
    raise PublishResourceException(detail=error_handler(e, execution_id))
importer.api.exception.PublishResourceException: must specify native_crs. Request: b98fc0b8-cd6d-481e-90ec-398bee9ed5ad
[2024-04-09 15:49:41,878: INFO/ForkPoolWorker-12] STARTING NEXT STEP importer.publish_resource
[2024-04-09 15:49:41,878: INFO/ForkPoolWorker-12] STARTING NEXT STEP importer.publish_resource for resource: aree, alternate aree
[2024-04-09 15:49:44,140: INFO/ForkPoolWorker-9] STARTING NEXT STEP importer.create_geonode_resource
[2024-04-09 15:49:44,140: INFO/ForkPoolWorker-9] STARTING NEXT STEP importer.create_geonode_resource for resource: aree, alternate aree
[2024-04-09 15:49:44,750: INFO/ForkPoolWorker-9] 
[2024-04-09 15:49:44,750: INFO/ForkPoolWorker-9] 0 batches, 0 sent
[2024-04-09 15:49:44,750: INFO/ForkPoolWorker-9] done in 0.05 seconds
[2024-04-09 15:49:50,120: DEBUG/ForkPoolWorker-12] Checking for permissions.
[2024-04-09 15:49:50,138: DEBUG/ForkPoolWorker-12] There are no permissions for this object, setting default perms.
[2024-04-09 15:49:50,206: DEBUG/ForkPoolWorker-12] Finalizing (permissions and notifications) on resource geonode:aree
[2024-04-09 15:49:50,207: DEBUG/ForkPoolWorker-12] Setting permissions {'users': {}, 'groups': {<Group: anonymous>: ['view_resourcebase', 'download_resourcebase']}} on geonode:aree
[2024-04-09 15:49:50,524: DEBUG/ForkPoolWorker-12] Removing all permissions on geonode:aree
[2024-04-09 15:49:52,463: DEBUG/ForkPoolWorker-12] Adding 0 rule deletion operations for 'geonode:aree
[2024-04-09 15:49:52,463: DEBUG/ForkPoolWorker-12] Going to remove 0 rules for layer geonode:aree
[2024-04-09 15:49:52,463: DEBUG/ForkPoolWorker-12] Skipping batch execution "Delete geonode:aree"
[2024-04-09 15:49:52,467: DEBUG/ForkPoolWorker-12]  perm: change_resourcebase_permissions - resource_type: dataset --> assignable: True
[2024-04-09 15:49:52,556: DEBUG/ForkPoolWorker-12]  perm: view_resourcebase - resource_type: dataset --> assignable: True
[2024-04-09 15:49:52,567: DEBUG/ForkPoolWorker-12]  perm: change_resourcebase - resource_type: dataset --> assignable: True
[2024-04-09 15:49:52,577: DEBUG/ForkPoolWorker-12]  perm: change_resourcebase_metadata - resource_type: dataset --> assignable: True
[2024-04-09 15:49:52,587: DEBUG/ForkPoolWorker-12]  perm: publish_resourcebase - resource_type: dataset --> assignable: True
[2024-04-09 15:49:52,598: DEBUG/ForkPoolWorker-12]  perm: delete_resourcebase - resource_type: dataset --> assignable: True
[2024-04-09 15:49:52,608: DEBUG/ForkPoolWorker-12]  perm: download_resourcebase - resource_type: dataset --> assignable: True
[2024-04-09 15:49:52,616: DEBUG/ForkPoolWorker-12]  perm: view_resourcebase - resource_type: dataset --> assignable: True
[2024-04-09 15:49:52,624: DEBUG/ForkPoolWorker-12]  perm: download_resourcebase - resource_type: dataset --> assignable: True
[2024-04-09 15:49:52,654: INFO/ForkPoolWorker-12] Requesting GeoFence rules on resource "geonode:aree" :: Dataset
[2024-04-09 15:49:52,824: DEBUG/ForkPoolWorker-12] Adding 0 rule deletion operations for 'geonode:aree
[2024-04-09 15:49:52,862: DEBUG/ForkPoolWorker-12] Adding GeoFence USER rules: U:admin S:WMS L:aree 
[2024-04-09 15:49:52,862: DEBUG/ForkPoolWorker-12] Creating Rule object: {'userName': 'admin', 'service': 'WMS', 'workspace': 'geonode', 'layer': 'aree', 'access': 'ALLOW', 'priority': 14}
[2024-04-09 15:49:52,862: DEBUG/ForkPoolWorker-12] Adding GeoFence USER rules: U:admin S:GWC L:aree 
[2024-04-09 15:49:52,862: DEBUG/ForkPoolWorker-12] Creating Rule object: {'userName': 'admin', 'service': 'GWC', 'workspace': 'geonode', 'layer': 'aree', 'access': 'ALLOW', 'priority': 15}
[2024-04-09 15:49:52,862: DEBUG/ForkPoolWorker-12] Adding GeoFence USER rules: U:admin S:WPS L:aree 
[2024-04-09 15:49:52,862: DEBUG/ForkPoolWorker-12] Creating Rule object: {'userName': 'admin', 'service': 'WPS', 'workspace': 'geonode', 'layer': 'aree', 'access': 'ALLOW', 'priority': 16}
[2024-04-09 15:49:52,862: DEBUG/ForkPoolWorker-12] Adding GeoFence USER rules: U:admin S:WFS L:aree 
[2024-04-09 15:49:52,862: DEBUG/ForkPoolWorker-12] Creating Rule object: {'userName': 'admin', 'service': 'WFS', 'workspace': 'geonode', 'layer': 'aree', 'access': 'ALLOW', 'priority': 17}
[2024-04-09 15:49:52,862: DEBUG/ForkPoolWorker-12] Adding GeoFence USER rules: U:admin S:* L:aree 
[2024-04-09 15:49:52,862: DEBUG/ForkPoolWorker-12] Creating Rule object: {'userName': 'admin', 'workspace': 'geonode', 'layer': 'aree', 'access': 'ALLOW', 'priority': 18}
[2024-04-09 15:49:52,871: DEBUG/ForkPoolWorker-12] Adding GeoFence ANON rules: S:WMS L:aree 
[2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Creating Rule object: {'service': 'WMS', 'workspace': 'geonode', 'layer': 'aree', 'access': 'ALLOW', 'priority': 19}
[2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Adding GeoFence ANON rules: S:GWC L:aree 
[2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Creating Rule object: {'service': 'GWC', 'workspace': 'geonode', 'layer': 'aree', 'access': 'ALLOW', 'priority': 20}
[2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Adding GeoFence ANON rules: S:WPS L:aree 
[2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Creating Rule object: {'service': 'WPS', 'workspace': 'geonode', 'layer': 'aree', 'access': 'ALLOW', 'priority': 21}
[2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Adding GeoFence ANON rules: S:WFS L:aree 
[2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Creating Rule object: {'service': 'WFS', 'request': 'TRANSACTION', 'workspace': 'geonode', 'layer': 'aree', 'access': 'DENY', 'priority': 22}
[2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Adding GeoFence ANON rules: S:WFS L:aree 
[2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Creating Rule object: {'service': 'WFS', 'request': 'LOCKFEATURE', 'workspace': 'geonode', 'layer': 'aree', 'access': 'DENY', 'priority': 23}
[2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Adding GeoFence ANON rules: S:WFS L:aree 
[2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Creating Rule object: {'service': 'WFS', 'request': 'GETFEATUREWITHLOCK', 'workspace': 'geonode', 'layer': 'aree', 'access': 'DENY', 'priority': 24}
[2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Adding GeoFence ANON rules: S:WFS L:aree 
[2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Creating Rule object: {'service': 'WFS', 'workspace': 'geonode', 'layer': 'aree', 'access': 'ALLOW', 'priority': 25}
[2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Adding GeoFence ANON rules: S:* L:aree 
[2024-04-09 15:49:52,872: DEBUG/ForkPoolWorker-12] Creating Rule object: {'workspace': 'geonode', 'layer': 'aree', 'access': 'ALLOW', 'priority': 26}
[2024-04-09 15:49:52,950: INFO/ForkPoolWorker-12] Pushing 13 changes into GeoFence for resource aree
[2024-04-09 15:49:52,950: DEBUG/ForkPoolWorker-12] Running batch "Set permission for resource geonode:aree" with 13 operations
[2024-04-09 15:49:53,191: DEBUG/ForkPoolWorker-12] ... Refresh GeoServer attributes list for Dataset aree
[2024-04-09 15:49:53,624: DEBUG/ForkPoolWorker-9] +++++ harvesting_dispatcher starting... +++++
[2024-04-09 15:49:53,640: DEBUG/ForkPoolWorker-9] +++++ harvesting_dispatcher ending... +++++
[2024-04-09 15:49:54,016: DEBUG/ForkPoolWorker-12] Found geoserver resource for this dataset: aree
[2024-04-09 15:49:54,609: DEBUG/ForkPoolWorker-12] ... Refresh Legend links for Dataset aree
[2024-04-09 15:49:55,236: DEBUG/ForkPoolWorker-12] save_style: Copying style "polygon" to "geonode:aree
[2024-04-09 15:49:55,519: DEBUG/ForkPoolWorker-12] set_style: Setting default style "aree" for layer "aree
[2024-04-09 15:49:56,083: DEBUG/ForkPoolWorker-12] clean_styles: Retrieving style "aree" for cleanup
[2024-04-09 15:49:56,106: DEBUG/ForkPoolWorker-12] clean_styles: Style does not exist: aree
[2024-04-09 15:49:56,111: DEBUG/ForkPoolWorker-12]  -- Resource Links[Legend link] for layer aree...
[2024-04-09 15:49:56,123: DEBUG/ForkPoolWorker-12]  -- Resource Links[Legend link]...done!
[2024-04-09 15:49:56,204: ERROR/ForkPoolWorker-12] Error 401 reading Style Filter Params GeoWebCache at http://10.10.100.5/geoserver/gwc/rest/layers/geonode:aree.xml
[2024-04-09 15:49:56,235: DEBUG/ForkPoolWorker-12] Error 401 invalidating GeoWebCache at http://10.10.100.5/geoserver/gwc/rest/masstruncate
[2024-04-09 15:49:56,235: DEBUG/ForkPoolWorker-12] ... Creating Default Resource Links for Dataset aree
[2024-04-09 15:49:56,238: DEBUG/ForkPoolWorker-12]  -- Resource Links[Compute parameters for the new links]...
[2024-04-09 15:49:56,238: DEBUG/ForkPoolWorker-12]  -- Resource Links[Set download links for WMS, WCS or WFS and KML]...
[2024-04-09 15:49:56,311: DEBUG/ForkPoolWorker-12]  -- Resource Links[Set download links for WMS, WCS or WFS and KML]...done!
[2024-04-09 15:49:56,311: DEBUG/ForkPoolWorker-12]  -- Resource Links[Legend link]...
[2024-04-09 15:49:56,320: DEBUG/ForkPoolWorker-12]  -- Resource Links[Legend link]...done!
[2024-04-09 15:49:56,321: DEBUG/ForkPoolWorker-12]  -- Resource Links[OWS Links]...
[2024-04-09 15:49:56,334: DEBUG/ForkPoolWorker-12]  -- Resource Links[OWS Links]...done!
[2024-04-09 15:49:56,335: DEBUG/ForkPoolWorker-12] ... Updating the Catalogue entries for Dataset aree
[2024-04-09 15:49:56,582: DEBUG/ForkPoolWorker-12] handling UUID In pre_save_dataset
[2024-04-09 15:49:56,589: DEBUG/ForkPoolWorker-12] Checking for permissions.
[2024-04-09 15:49:56,787: INFO/ForkPoolWorker-12] Creating style for Dataset geonode:aree / {'sld_file': '', 'sld_uploaded': False, 'vals': {'dirty_state': True}}
[2024-04-09 15:49:56,799: INFO/ForkPoolWorker-12] geonode.geoserver.tasks.geoserver_create_style[None]: Using 'MEMCACHED' lock type.
[2024-04-09 15:49:56,800: WARNING/ForkPoolWorker-12] geonode.geoserver.tasks.geoserver_create_style[None]: error 3 from memcached_add: CONNECTION FAILURE
[2024-04-09 15:49:56,956: DEBUG/ForkPoolWorker-12] 'NoneType' object has no attribute 'href'
[2024-04-09 15:49:56,956: DEBUG/ForkPoolWorker-12] 'NoneType' object has no attribute 'href'
[2024-04-09 15:49:56,956: DEBUG/ForkPoolWorker-12] 'NoneType' object has no attribute 'href'
[2024-04-09 15:49:56,978: DEBUG/ForkPoolWorker-12] Failed to make DELETE request: 403, <!doctype html><html lang="en"><head><title>HTTP Status 403 \u2013 Forbidden</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 403 \u2013 Forbidden</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> Can&#39;t delete style referenced by existing layers.</p><p><b>Description</b> The server understood the request but refuses to authorize it.</p><hr class="line" /><h3>Apache Tomcat/9.0.85</h3></body></html>
[2024-04-09 15:49:57,065: DEBUG/ForkPoolWorker-12] save_style: Retrieving style "geonode:aree" for layer "geonode:aree
[2024-04-09 15:49:57,222: DEBUG/ForkPoolWorker-12] clean_styles: Retrieving style "aree" for cleanup
[2024-04-09 15:49:57,248: DEBUG/ForkPoolWorker-12] clean_styles: Style does not exist: aree
[2024-04-09 15:49:57,252: DEBUG/ForkPoolWorker-12]  -- Resource Links[Legend link] for layer aree...
[2024-04-09 15:49:57,262: DEBUG/ForkPoolWorker-12]  -- Resource Links[Legend link]...done!
[2024-04-09 15:50:01,410: DEBUG/ForkPoolWorker-12] ...Checking if image is a blank image
[2024-04-09 15:50:01,684: ERROR/ForkPoolWorker-9] ['import']

Table gpkg_contents:

table_name  |data_type |identifier  |description|last_change             |min_x            |min_y           |max_x            |max_y     |srs_id|
------------+----------+------------+-----------+------------------------+-----------------+----------------+-----------------+----------+------+
aree        |features  |aree        |           |2024-03-14T13:57:15.624Z|1712986.270199999|5047030.14877623|1786866.833708391|5108676.87|  3003|
layer_styles|attributes|layer_styles|           |2024-03-14T14:00:34.922Z|                 |                |                 |          |     0|

Table layer_styles:

id|f_table_catalog|f_table_schema|f_table_name|f_geometry_column|styleName                                |styleQML  
--+---------------+--------------+------------+-----------------+-----------------------------------------+----------
 1|               |              |aree        |geom             |3classi+urbano+natura+comuni_sempl_140324|<!DOCTYPE  [...]

ogrinfo

ogrinfo ~aap.gpkg 
INFO: Open of `aap.gpkg'
      using driver `GPKG' successful.
1: aree (Multi Polygon)
2: layer_styles (None)
@etj
Copy link
Contributor Author

etj commented Apr 9, 2024

An heuristic check to find out if a layer is valid was already in pale, but commented out:

geonode-importer/importer/handlers/common/vector.py

Line 337 in 269ec25

# and layer.GetGeometryColumn() is not None

@mattiagiupponi reports that some valid layers were rejected by such check.

We may add a further heuristic check, and avoid ingestion if the geom type is null and the crs is not defined.

@mattiagiupponi
Copy link
Contributor

@etj it was also removed due this PR: #159
To allow Allow nonspatial layer import. We still make it more smart (at least for the gpkg) and remove the layers we dont want to import

mattiagiupponi added a commit that referenced this issue Apr 17, 2024
@mattiagiupponi
[Fixes #233] [GPKG] must specify native_crs
9b80331
@mattiagiupponi mattiagiupponi mentioned this issue Apr 17, 2024
Merged
@mattiagiupponi mattiagiupponi linked a pull request Apr 17, 2024 that will close this issue
[Fixes #233] [GPKG] must specify native_crs #234
Merged
mattiagiupponi added a commit that referenced this issue Apr 22, 2024
@mattiagiupponi
[Fixes #233] add test coverage
7491390
mattiagiupponi added a commit that referenced this issue May 2, 2024
@mattiagiupponi
[Fixes #233] add test coverage
16cd24c
mattiagiupponi added a commit that referenced this issue May 2, 2024
@mattiagiupponi
[Fixes #233] fix test
c2aa3f1
mattiagiupponi added a commit that referenced this issue May 3, 2024
@mattiagiupponi
[Fixes #233] add test coverage
c817be9
mattiagiupponi added a commit that referenced this issue May 3, 2024
@mattiagiupponi
[Fixes #233] add test coverage
3941363
mattiagiupponi added a commit that referenced this issue May 3, 2024
@mattiagiupponi
[Fixes #233] [GPKG] must specify native_crs (#234)
658a825 
* [Fixes #233] [GPKG] must specify native_crs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mattiagiupponi mattiagiupponi

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Fixes #233] [GPKG] must specify native_crs GeoNode/geonode-importer
2 participants
@etj @mattiagiupponi