Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove Selenium dependency #2

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Remove Selenium dependency #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1f3816e
Remove Selenium dependency
ssiegel Nov 19, 2015
83db7c0
Update README to use --force-new for collection migration
ssiegel Nov 20, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
28 changes: 10 additions & 18 deletions README.rst
View file
Open in desktop
Expand Up @@ -44,15 +44,12 @@ Python 2.7 is required.

Usage
=====
You'll need **Firefox** to be installed; Firefox will be automatically controlled by ``bggcli`` to perform operations
(through Selenium library).

Type ``bggcli`` to get the full help about available commands.

Here is an example to export a collection from an account *account1* and import it to another account *account2*::

$ bggcli -l mylogin1 -p mypassword1 collection-export mycollection.csv
$ bggcli -l mylogin2 -p mypassword2 collection-import mycollection.csv
$ bggcli -l mylogin2 -p mypassword2 collection-import --force-new mycollection.csv

Update a collection
-------------------
Expand All @@ -75,8 +72,9 @@ Notes:
* Column names are those exported by BGG. Any column not recognized will just be ignored
* When a game already exists in your collection, game is updated with provided CSV values only, other fields are not
impacted. You could only update one field for all your game.
* Games are identified by their internal ID, named ``objectid`` in CSV file (name used by BGG). Having the
``objectname`` field (name of the game) is also recommended for logging.
* Games are identified by their internal ID, named ``objectid`` in CSV file (name used by BGG). Collection items are
identified by the ID named ``collid`` (for modifying or deleting existing items). Having the ``objectname`` field
(name of the game) is also recommended for logging.


Remove games from a collection
Expand All @@ -89,7 +87,7 @@ Example:::

Notes:

* Only the ``objectid`` column will be used for this operation: this is the internal ID managed by BGG. All other
* Only the ``collid`` column will be used for this operation: this is the internal ID managed by BGG. All other
columns will just be ignored.

Export a collection
Expand All @@ -102,22 +100,16 @@ Example:::

Notes:

* Only the ``objectid`` column will be used for this operation: this is the internal ID managed by BGG. All other
* Only the ``collid`` column will be used for this operation: this is the internal ID managed by BGG. All other
columns will just be ignored.


Limitations
===========

* Only *Firefox* is supported. This tools relies on Selenium to control browser, and only Firefox is supported
out of the box by Selenium (i.e. without additional requirements). Support of additional browsers could be introduced,
but I'm not sure it's worth it.
* Performance: Selenium+Firefox association is not the fastest way to automate operations, but it's
probably the best regarding stability (no Javascript emulation, Firefox does the work) and simplicity (no need to
install anything else), which is the most important in the current context. On my laptop, I see the import taking
1 min for 5 games.
* Some fields related to the game's version are not exported by BGG: the ``barcode`` and the `language``. Note
although this only applies to custom version you define yourself, which should be quite rare.
* Some fields are not exported by BGG: the ``language`` field related to the game's version and the ``Inventory Date``
and ``Inventory Location`` in the private info. The missing ``language`` field only applies to custom version you
define yourself, which should be quite rare.


Ideas for future versions
Expand All @@ -140,4 +132,4 @@ Links
Final note
==========

Does it really deserve such a development? Probably not, but my second goal was to discover the Python ecosystem!
Does it really deserve such a development? Probably not, but my second goal was to discover the Python ecosystem!
6 changes: 4 additions & 2 deletions bggcli/__init__.py
View file
Open in desktop
Expand Up @@ -10,11 +10,13 @@
"changed and bggcli must be updated, or the site is down for " \
"maintenance."

BGG_SUPPORTED_FIELDS = ['objectname', 'objectid', 'rating', 'own',
BGG_SUPPORTED_FIELDS = ['objectname', 'objectid', 'rating', 'numplays', 'own',
'fortrade', 'want', 'wanttobuy', 'wanttoplay', 'prevowned',
'preordered', 'wishlist', 'wishlistpriority', 'wishlistcomment',
'comment', 'conditiontext', 'haspartslist', 'wantpartslist',
'publisherid', 'imageid', 'year', 'language', 'other', 'pricepaid',
'collid', 'baverage', 'average', 'numowned', 'objecttype', 'minplayers',
'maxplayers', 'playingtime', 'maxplaytime', 'minplaytime', 'yearpublished',
'publisherid', 'imageid', 'year', 'language', 'other', 'barcode', 'pricepaid',
'pp_currency', 'currvalue', 'cv_currency', 'acquisitiondate',
'acquiredfrom', 'quantity', 'privatecomment',
'_versionid']
48 changes: 30 additions & 18 deletions bggcli/commands/collection_delete.py
View file
Open in desktop
Expand Up @@ -2,35 +2,41 @@
Delete games in your collection from a CSV file. BE CAREFUL, this action is irreversible!

Usage: bggcli [-v] -l <login> -p <password>
[-c <name>=<value>]...
collection-delete [--force] <file>

Options:
-v Activate verbose logging
-l, --login <login> Your login on BGG
-p, --password <password> Your password on BGG
--force Force deletion, without any confirmation
-c <name=value> To specify advanced options, see below

Advanced options:
browser-keep=<true|false> If you want to keep your web browser opened at the end of the
operation
browser-profile-dir=<dir> Path or your browser profile if you want to use an existing

Arguments:
<file> The CSV file with games to delete
"""
from urllib import urlencode
import cookielib
import urllib2
import sys

from bggcli import BGG_BASE_URL
from bggcli.commands import check_file
from bggcli.util.csvreader import CsvReader
from bggcli.ui.gamepage import GamePage
from bggcli.ui.loginpage import LoginPage
from bggcli.util.logger import Logger
from bggcli.util.webdriver import WebDriver


def execute(args, options):
def game_deleter(opener, row):
collid = row['collid']
if not collid:
return

response = opener.open(BGG_BASE_URL + '/geekcollection.php', urlencode({
'ajax': 1, 'action': 'delete', 'collid': collid}))

if response.code != 200:
Logger.error("Failed to delete 'collid'=%s!" % collid, sysexit=True)


def execute(args):
login = args['--login']

file_path = check_file(args)
Expand All @@ -53,11 +59,17 @@ def execute(args, options):

Logger.info("Deleting games for '%s' account..." % login)

with WebDriver('collection-delete', args, options) as web_driver:
if not LoginPage(web_driver.driver).authenticate(login, args['--password']):
sys.exit(1)
cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
Logger.info("Authenticating...", break_line=False)
opener.open(BGG_BASE_URL + '/login', urlencode({
'action': 'login', 'username': login, 'password': args['--password']}))
if not any(cookie.name == "bggusername" for cookie in cj):
Logger.info(" [error]", append=True)
Logger.error("Authentication failed for user '%s'!" % login, sysexit=True)
Logger.info(" [done]", append=True)


Logger.info("Deleting %s games..." % game_count)
game_page = GamePage(web_driver.driver)
csv_reader.iterate(lambda row: game_page.delete(row))
Logger.info("Deletion has finished.")
Logger.info("Deleting %s games..." % game_count)
csv_reader.iterate(lambda row: game_deleter(opener, row))
Logger.info("Deletion has finished.")
55 changes: 22 additions & 33 deletions bggcli/commands/collection_export.py
View file
Open in desktop
Expand Up @@ -2,77 +2,67 @@
Export a game collection as a CSV file.

Usage: bggcli [-v] -l <login> -p <password>
[-c <name>=<value>]...
collection-export <file>
collection-export [--save-xml-file] <file>

Options:
-v Activate verbose logging
-l, --login <login> Your login on BGG
-p, --password <password> Your password on BGG
-c <name=value> To specify advanced options, see below

Advanced options:
save-xml-file=<true|false> To store the exported raw XML file in addition (will be
--save-xml-file To store the exported raw XML file in addition (will be
save aside the CSV file, with '.xml' extension
browser-keep=<true|false> If you want to keep your web browser opened at the end of the
operation
browser-profile-dir=<dir> Path or your browser profile if you want to use an existing

Arguments:
<file> The CSV file to generate
"""
import csv
from urllib import urlencode
import cookielib
import urllib2
import time
import sys
import xml.etree.ElementTree as ET

from bggcli import BGG_BASE_URL, BGG_SUPPORTED_FIELDS
from bggcli.ui.loginpage import LoginPage
from bggcli.util.logger import Logger
from bggcli.util.webdriver import WebDriver
from bggcli.util.xmltocsv import XmlToCsv

BGG_SESSION_COOKIE_NAME = 'SessionID'
EXPORT_QUERY_INTERVAL = 5
ERROR_FILE_PATH = 'error.txt'


def execute(args, options):
def execute(args):
login = args['--login']
dest_path = args['<file>']

Logger.info("Exporting collection for '%s' account..." % login)

# 1. Authentication
with WebDriver('collection-export', args, options) as web_driver:
if not LoginPage(web_driver.driver).authenticate(login, args['--password']):
sys.exit(1)
auth_cookie = web_driver.driver.get_cookie(BGG_SESSION_COOKIE_NAME)['value']
cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
Logger.info("Authenticating...", break_line=False)
opener.open(BGG_BASE_URL + '/login', urlencode({
'action': 'login', 'username': login, 'password': args['--password']}))
if not any(cookie.name == "bggusername" for cookie in cj):
Logger.info(" [error]", append=True)
Logger.error("Authentication failed for user '%s'!" % login, sysexit=True)
Logger.info(" [done]", append=True)

# 2. Export
# Easier to rely on a client HTTP call rather than Selenium to download a file
# Just need to pass the session cookie to get the full export with private information

# Use XML2 API, see https://www.boardgamegeek.com/wiki/page/BGG_XML_API2#Collection
# Default CSV export doesn't provide version info!
url = '%s/xmlapi2/collection?username=%s&version=1&showprivate=1&stats=1' \
% (BGG_BASE_URL, login)
req = urllib2.Request(url, None, {'Cookie': '%s=%s' % (BGG_SESSION_COOKIE_NAME, auth_cookie)})
url = BGG_BASE_URL + '/xmlapi2/collection?' + urlencode({
'username': login, 'version': 1, 'showprivate': 1, 'stats': 1})
req = urllib2.Request(url)

# Get a BadStatusLine error most of times without this delay!
# Related to Selenium, but in some conditions that I have not identified
time.sleep(8)
try:
Logger.info('Launching export...')
response = default_export(req)
response = default_export(opener, req)
except Exception as e:
Logger.error('Error while fetching export file!', e, sysexit=True)
return

# 3. Store XML file if requested
xml_file = options.get('save-xml-file')
if xml_file == 'true':
if args['--save-xml-file']:
xml_file_path = write_xml_file(response, dest_path)
Logger.info("XML file save as %s" % xml_file_path)
source = open(xml_file_path, 'rU')
Expand All @@ -92,13 +82,13 @@ def execute(args, options):
Logger.info("Collection has been exported as %s" % dest_path)


def default_export(req):
response = urllib2.urlopen(req)
def default_export(opener, req):
response = opener.open(req)

if response.code == 202:
Logger.info('Export is queued, will retry in %ss' % EXPORT_QUERY_INTERVAL)
time.sleep(EXPORT_QUERY_INTERVAL)
return default_export(req)
return default_export(opener, req)

if response.code == 200:
return response
Expand Down Expand Up @@ -135,4 +125,3 @@ def write_csv(source, dest_path):
if elem.tag == 'item' and elem.attrib.get('subtype') == 'boardgame':
row = XmlToCsv.convert_item(elem)
csv_writer.writerow(row)