Twitch Data Exporter

Table of Contents

• Twitch Data Exporter
  • About The Project
    • Built With
  • Getting Started
    • Prerequisites
    • Installation
  • Usage
    • Customizing refresh intervals
  • Roadmap
  • Contributing
  • Authors
  • Acknowledgements
  • License

About The Project

Twitch data Exporter is a small app that allows to export stats from certain channels and store them on your preferred storage method and analyze them later with other tools such as Grafana.

I came up with the idea of this project while streaming and contributing to the galician language streaming community on Twitch. We wanted to show how the community was growing and some major stats for all the channels so I decided to make a small app that would allow to export this data from Twitch API to any storage (PostgreSQL at first), and later use the data on other tools as Grafana to show the relevant stats.

It currently allows the storage of:

• Channels information and channel views & followers evolution.
• Channels clips with total views for each one.
• Channels streamings, with the viewers recorded each minute(or the time range you set)

The implemented storages as of now are the following:

• PostgreSQL

Miss the one you wanted? Request a feature or consider contributing to the project!

Built With

• NodeJS
• Docker

Getting Started

This is an example of how you may give instructions on setting up your project locally. To get a local copy up and running follow these simple example steps.

Prerequisites

Project is build on node and uses many npm packages, You need to set up:

• NodeJS: Follow the instructions on the webpage.
• npm

  npm install npm@latest -g

Installation

1. Setup your Twitch App and get your App ClientId & Client secret from Twitch Developer Console.
2. (Optional) Setup a Google Sheets App if you want to retrieve channels from a spreadsheed. Follow the Node Quickstart Guide to download credentials.json.
3. Clone the repository

   git clone https://github.com/pvillaverde/twitch-data-exporter

4. Install NPM packages dependencies

   npm install

5. Copy config_example.js as config.js and enter your twitch API clientId & clientSecret, as well as the content of credentials.json on google_credentials if you want to use spreadsheets.

    twitch: {
    	clientId: 'REQUIRED', // Twitch App ClientID
    	clientSecret: 'REQUIRED', // Twitch APP ClientSecret
    	channels: null, // null if using google spreadsheet
    },
    google_credentials: {/*credentials.json*/},

6. Setup the storage you want to use, by default postgres, and change the connectionString to match your database connection info.

    storage: 'postgres', // Which storage will be used
    storageConfig: {
    	postgres: {
    		connectionString: 'postgresql://username:password@server:port/database',
    	}
    },

7. Start the app by runing npm start or node app/index.js and follow the steps for google Api auth if you are using spreadsheets.
8. If you want to launch a docker, once you have all setup and ready, build the image and run it with:

   docker build -t pvillaverde/twitch-data-exporter .
   docker run --name twitch-data-exporter  -d pvillaverde/twitch-data-exporter

Usage

As you have seen previously, you can choose how the app learns which channels to check from 2 options:

1. Defining the twitch.channels variable on config.js with an array of channels name, ex:// [clankirfed,twitch_en_galego]
2. Setting up Google Sheets API and specifying a Google Spreadsheet on config.jswhich shall have the channels names on the first column:

	google_spreadsheet: {
		id: '1AFbvk9SLOpOyST4VWG6IOkiMdclzExUPQrKUBuEUHKY', // Google Spreadsheet ID
		range: 'Canles!A2:A', // Sheet & Range
		headers: 'name', // Column Headers (Optional, only first one will be used)
	},

Customizing refresh intervals

Lastly, you can choose when to retrieve each data on the cron option of config.js, by default it is setup like this:

// Minutes Hours DayOfMonth Month DayOfWeek
fetchUsers: '0 1 * * *', // User Channels and total views will be refreshed every day at 1AM
fetchClips: '0 2 * * *', // Channel Clips will be refreshed every day at 2AM
fetchGames: '0 3 * * *', // Missing Games from the database will be fetched every day at 3AM
fetchFollows: '0 4 * * *', // Channel followers will be refreshed every dat at 4AM
fetchStreams: '* * * * *', // Every minute will who's someone on stream and viewers count.

These intervals can be customized using the cron syntax:

  * * * * * *
  | | | | | |
  | | | | | day of week
  | | | | month
  | | | day of month
  | | hour
  | minute
  second ( optional )

Roadmap

Check out the open issues page for a list of proposed features (and known issues).

Contributing

If you want to add any missing feature or storage that feed your needs, go ahead! That's what make the open source community shines, by allowing us to grow and learn from each other creating amazing tools! Any contribution you make is greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

Authors

Pablo Villaverde Castro - @clankirfed

Acknowledgements

• Twitch API
• Google Sheets API
• How to use Node Cron
• TOC Generator
• Dockerizing NodeJs WebApp
• Building Efficient NodeJS DockerFiles

License

Distributed under the GNU GPL-v3 License. See LICENSE for more information.