Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update the documentation to recommend xloader #8045

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Update the documentation to recommend xloader #8045

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d59ecb3
Update the documentation to recommend xloader
Labuschagne-Miro Jan 31, 2024
b037c74
Update install-from-source.rst
Labuschagne-Miro Jan 31, 2024
bd04d1b
Create 8044.bugfix
Labuschagne-Miro Feb 4, 2024
28b0649
Update datastore.rst
Labuschagne-Miro Feb 4, 2024
187e470
Update install-from-source.rst
Labuschagne-Miro Feb 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions changes/8044.bugfix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
xloader in ckan-docker
21 changes: 6 additions & 15 deletions doc/maintaining/datastore.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,7 @@ The DataStore is integrated into the :doc:`CKAN API </api/index>` and
authorization system.

The DataStore is generally used alongside the
`DataPusher <https://github.com/ckan/datapusher>`_, which will
automatically upload data to the DataStore from suitable files, whether
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line shouldn't have been dropped.

`XLoader <https://github.com/ckan/ckanext-xloader/>`_, which will
uploaded to CKAN's FileStore or externally linked.

.. contents::
Expand Down Expand Up @@ -196,22 +195,14 @@ To find out more about the Data API, see `The Data API`_.


---------------------------------------------------
DataPusher: Automatically Add Data to the DataStore
XLoader(or Express Loader)
---------------------------------------------------

Often, one wants data that is added to CKAN (whether it is linked to or
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This paragraph shouldn't have been dropped.

uploaded to the :doc:`FileStore <filestore>`) to be automatically added to the
DataStore. This requires some processing, to extract the data from your files
and to add it to the DataStore in the format the DataStore can handle.
Express Loader(or XLoader) is designed as a replacement for DataPusher because it offers ten
times the speed and more robustness. You can load CSV (and similar) data into CKAN's DataStore.
XLoader pipes the CSV file directly into PostgreSQL using COPY.

This task of automatically parsing and then adding data to the DataStore is
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This paragraph shouldn't have been dropped, just updated to refer to XLoader.

performed by the `DataPusher`_, a service that runs asynchronously and can be installed
alongside CKAN.

To install this please look at the docs here: https://github.com/ckan/datapusher

.. note:: The DataPusher only imports the first worksheet of a spreadsheet. It also does
not support duplicate column headers. That includes blank column headings.
To install this please look at the docs here: https://github.com/ckan/ckanext-xloader

.. _data_dictionary:

Expand Down
5 changes: 2 additions & 3 deletions doc/maintaining/installing/install-from-source.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -254,9 +254,8 @@ Follow the instructions in :doc:`/maintaining/datastore` to create the required
databases and users, set the right permissions and set the appropriate values
in your CKAN config file.

Once you have set up the DataStore, you may then wish to configure either the DataPusher or XLoader
extensions to add data to the DataStore. To install DataPusher refer to this link:
https://github.com/ckan/datapusher and to install XLoader refer to this link:
Once you have set up the DataStore, you may then wish to configure the XLoader
extension to add data to the DataStore. To install XLoader refer to this link:
https://github.com/ckan/ckanext-xloader

-------------------
Expand Down