Skip to content

A Web Application to collect data from pairwise image comparisons via crowdsourcing

License

Notifications You must be signed in to change notification settings

calista-ai/crowdsourcing-app

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

42 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Crowdsourcing App - Calista

DOI

A Web Application to collect data from pairwise image comparisons via crowdsourcing. Built with MERN stack.

Calista team developed and used this app to build a comparison-based dataset on website aesthetics.

Example from the evaluation process:

Paper: "Calista: A deep learning-based system for understanding and evaluating website aesthetics"

Cite as:
@article{DELITZAS2023,
	title = {Calista: A deep learning-based system for understanding and evaluating website aesthetics},
	journal = {International Journal of Human-Computer Studies},
	volume = {175},
	pages = {103019},
	year = {2023},
	issn = {1071-5819},
	doi = {https://doi.org/10.1016/j.ijhcs.2023.103019},
	url = {https://www.sciencedirect.com/science/article/pii/S1071581923000253},
	author = {Alexandros Delitzas and Kyriakos C. Chatzidimitriou and Andreas L. Symeonidis}
}

Prerequisites

  • Docker
  • Docker-compose

Deployment

Step 1 - Images

  • Define the images that you want to collect data for
  • Add these images in the folder front-end/public/images/

Step 2 - Environment variables

Add a .env file in the root folder of the project and set the following variables:

Environment variable Description
MONGO_URI MongoDB connection string
PORT Server port
MONGO_PORT MongoDB port
ME_CONFIG_MONGODB_ADMINUSERNAME MongoDB username
ME_CONFIG_MONGODB_ADMINPASSWORD MongoDB password
ME_CONFIG_BASICAUTH_USERNAME Mongo-express username
ME_CONFIG_BASICAUTH_PASSWORD Mongo-express password
NODE_ENV Node environment variable
VOTING_ROUNDS Number of voting rounds of each vote session
VOTING_TIME Voting time in seconds of each comparison
NUMBER_OF_IMAGES Number of images in front-end/public/images/
BASEURL Base URL that is used for the requests

Step 3 - Setup

This step is required only for the first time that you will run the application. This may take a few minutes as it will download the images required. If you have already completed this step once successfully, then you can jump directy to Step 4.

TL;DR:

Run the following command:

  ./setup.sh --create --sortimages

More detailed options:

  • The images in front-end/public/images/ must be named with numbers starting from 0 (eg. 0.png, 1.png, 2.png, ...). If the images aren't already in this format, then you can use the option --sortimages to get them automatically renamed during the setup:

    ./setup.sh --sortimages
    
  • If you want to set a new database, then it is required to have the file db/utils/comparisons_data.json. This file contains all possible pairwise comparisons that can be made between the images with random order. It is used for the database initialization. If you want to use a pre-existing comparisons_data.json file, then make sure that it is in db/utils/ directory. Otherwise, you can use the option --create to get this file automatically created during the setup:

    ./setup.sh --create
    
  • If you want to restore an existing database, first make sure to move your backup files in db/backup/ directory. Then you can use the option --restore=<PATH_TO_DATABASE_DUMP_IN_BACKUP_FOLDER>.

    For example, if you had the following structure,

    ├── db
    │   ├── backup
    │   │   ├── 14_04_2020
    │   │   │   └── votes
    │   │   │       ├── comparisons.bson
    │   │   │       ├── comparisons.metadata.json
    │   │   │       ├── votesessions.bson
    │   │   │       └── votesessions.metadata.json
    .   .   .
    .   .   .
    .   .   .

    you would run:

      ./setup.sh --restore=14_04_2020/votes
    

    Note: In this case, a comparisons_data.json file isn't needed.

  • You can also combine multiple options or use none.

Step 4 - Run

Start:

docker-compose -f docker-compose.yml up --build

Stop:

Ctrl-C

For detached mode:

Start:

docker-compose -f docker-compose.yml up -d --build

Stop:

docker-compose down