diff --git a/README.md b/README.md index ab46758..9a8dc39 100644 --- a/README.md +++ b/README.md @@ -1,82 +1,110 @@ # NyaaV2 -## Setup +## Setting up for development +This project uses Python 3.6. There are features used that do not exist in 3.5, so make sure to use Python 3.6. +This guide also assumes you 1) are using Linux and 2) are somewhat capable with the commandline. +It's not impossible to run Nyaa on Windows, but this guide doesn't focus on that. +### Code Quality: +- Before we get any deeper, remember to follow PEP8 style guidelines and run `./lint.sh` before committing. + - You may also use `pycodestyle nyaa/ --show-source --max-line-length=100´ to see a list of warnings/problems instead of having `lint.sh` making modifications for you +- Other than PEP8, try to keep your code clean and easy to understand, as well. It's only polite! + +### Setting up Pyenv +pyenv eases the use of different Python versions, and as not all Linux distros offer 3.6 packages, it's right up our alley. - Install dependencies https://github.com/pyenv/pyenv/wiki/Common-build-problems - Install `pyenv` https://github.com/pyenv/pyenv/blob/master/README.md#installation - Install `pyenv-virtualenv` https://github.com/pyenv/pyenv-virtualenv/blob/master/README.md -- `pyenv install 3.6.1` -- `pyenv virtualenv 3.6.1 nyaa` -- `pyenv activate nyaa` +- Install Python 3.6.1 with `pyenv` and create a virtualenv for the project: + - `pyenv install 3.6.1` + - `pyenv virtualenv 3.6.1 nyaa` + - `pyenv activate nyaa` - Install dependencies with `pip install -r requirements.txt` - Copy `config.example.py` into `config.py` -- Change TABLE_PREFIX to `nyaa_` or `sukebei_` depending on the site + - Change `SITE_FLAVOR` in your `config.py` depending on which instance you want to host -### Setting up MySQL/MariaDB database for advanced functionality +### Setting up MySQL/MariaDB database +You *may* use SQLite but the current support for it in this project is outdated and rather unsupported. - Enable `USE_MYSQL` flag in config.py - Install latest mariadb by following instructions here https://downloads.mariadb.org/mariadb/repositories/ - Tested versions: `mysql Ver 15.1 Distrib 10.0.30-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2` -- Run the following commands logged in as your root db user: +- Run the following commands logged in as your root db user (substitute for your own `config.py` values if desired): - `CREATE USER 'test'@'localhost' IDENTIFIED BY 'test123';` - - `GRANT ALL PRIVILEGES ON * . * TO 'test'@'localhost';` + - `GRANT ALL PRIVILEGES ON *.* TO 'test'@'localhost';` - `FLUSH PRIVILEGES;` - `CREATE DATABASE nyaav2 DEFAULT CHARACTER SET utf8 COLLATE utf8_bin;` -- To setup and import nyaa_maria_vx.sql: - - `mysql -u -p nyaav2` - - `DROP DATABASE nyaav2;` - - `CREATE DATABASE nyaav2 DEFAULT CHARACTER SET utf8 COLLATE utf8_bin;` - - `SOURCE ~/path/to/database/nyaa_maria_vx.sql` ### Finishing up -- Run `python db_create.py` to create the database -- Load the .sql file - - `mysql -u user -p nyaav2` - - `SOURCE cocks.sql` - - Remember to change the default user password to an empty string to disable logging in +- Run `python db_create.py` to create the database and import categories + - Follow the advice of `db_create.py` and run `./db_migrate.py stamp head` to mark the database version for Alembic - Start the dev server with `python run.py` -- When you are finished developing, deactivate your virtualenv with `source deactivate` +- When you are finished developing, deactivate your virtualenv with `pyenv deactivate` or `source deactivate` (or just close your shell session) -## Enabling ElasticSearch +You're now ready for simple testing and development! +Continue below to learn about database migrations and enabling the advanced search engine, Elasticsearch. -### Basics -- Install jdk `sudo apt-get install openjdk-8-jdk` -- Install elasticsearch https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html -- `sudo systemctl enable elasticsearch.service` -- `sudo systemctl start elasticsearch.service` -- Run `curl -XGET 'localhost:9200'` and make sure ES is running -- Optional: install Kabana as a search frontend for ES - -### Enable MySQL Binlogging -- Add the `[mariadb]` bin-log section to my.cnf and reload mysql server -- Connect to mysql -- `SHOW VARIABLES LIKE 'binlog_format';` - - Make sure it shows ROW -- Connect to root user -- `GRANT REPLICATION SLAVE ON *.* TO 'test'@'localhost';` where test is the user you will be running `sync_es.py` with - -### Setting up ES -- Run `./create_es.sh` and this creates two indicies: `nyaa` and `sukebei` -- The output should show `acknowledged: true` twice -- The safest bet is to disable the webapp here to ensure there's no database writes -- Run `python import_to_es.py` with `SITE_FLAVOR` set to `nyaa` -- Run `python import_to_es.py` with `SITE_FLAVOR` set to `sukebei` -- These will take some time to run as it's indexing - -### Setting up sync_es.py -- Sync_es.py keeps the ElasticSearch index updated by reading the BinLog -- Configure the MySQL options with the user where you granted the REPLICATION permissions -- Connect to MySQL, run `SHOW MASTER STATUS;`. -- Copy the output to `/var/lib/sync_es_position.json` with the contents `{"log_file": "FILE", "log_pos": POSITION}` and replace FILENAME with File (something like master1-bin.000002) in the SQL output and POSITION (something like 892528513) with Position -- Set up `sync_es.py` as a service and run it, preferably as the system/root -- Make sure `sync_es.py` runs within venv with the right dependencies - -Enable the `USE_ELASTIC_SEARCH` flag in `config.py`, restart the application, and you're good to go. ## Database migrations -- Uses [flask-Migrate](https://flask-migrate.readthedocs.io/) -- Run `./db_migrate.py db migrate` to generate the migration script after database model changes. -- Take a look at the result in `migrations/versions/...` to make sure nothing went wrong. -- Run `./db_migrate.py db upgrade` to upgrade your database. +- Database migrations are done with [flask-Migrate](https://flask-migrate.readthedocs.io/), a wrapper around [Alembic](http://alembic.zzzcomputing.com/en/latest/). +- If someone has made changes in the database schema and included a new migration script: + - If your database has never been marked by Alembic (you're on a database from before the migrations), run `./db_migrate.py stamp head` before pulling the new migration script(s). + - If you already have the new scripts, check the output of `./db_migrate.py history` instead and choose a hash that matches your current database state, then run `./db_migrate.py stamp `. + - Update your branch (eg. `git fetch && git rebase origin/master`) + - Run `./db_migrate.py upgrade head` to run the migration. Done! +- If *you* have made a change in the database schema: + - Save your changes in `models.py` and ensure the database schema matches the previous version (ie. your new tables/columns are not added to the live database) + - Run `./db_migrate.py migrate -m "Short description of changes"` to automatically generate a migration script for the changes + - Check the script (`migrations/versions/...`) and make sure it works! Alembic may not able to notice all changes. + - Run `./db_migrate.py upgrade` to run the migration and verify the upgrade works. + - (Run `./db_migrate.py downgrade` to verify the downgrade works as well, then upgrade again) -## Code Quality: -- Remember to follow PEP8 style guidelines and run `./lint.sh` before committing. + +## Setting up and enabling Elasticsearch + +### Installing Elasticsearch +- Install JDK with `sudo apt-get install openjdk-8-jdk` +- Install [Elasticsearch](https://www.elastic.co/downloads/elasticsearch) + - [From packages...](https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html) + - Enable the service: + - `sudo systemctl enable elasticsearch.service` + - `sudo systemctl start elasticsearch.service` + - or [simply extracting the archives and running the files](https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html), if you don't feel like permantently installing ES +- Run `curl -XGET 'localhost:9200'` and make sure ES is running + - Optional: install [Kibana](https://www.elastic.co/products/kibana) as a search debug frontend for ES + +### Setting up ES +- Run `./create_es.sh` to create the indices for the torrents: `nyaa` and `sukebei` + - The output should show `acknowledged: true` twice +- Stop the Nyaa app if you haven't already +- Run `python import_to_es.py` to import all the torrents (on nyaa and sukebei) into the ES indices. + - This may take some time to run if you have plenty of torrents in your database. + +Enable the `USE_ELASTIC_SEARCH` flag in `config.py` and (re)start the application. +Elasticsearch should now be functional! The ES indices won't be updated "live" with the current setup, continue below for instructions on how to hook Elasticsearch up to MySQL binlog. + +However, take note that binglog is not necessary for simple ES testing and development; you can simply run `import_to_es.py` from time to time to reindex all the torrents. + +### Enabling MySQL Binlogging +- Edit your MariaDB/MySQL server configuration and add the following under `[mariadb]`: + ``` + log-bin + server_id=1 + log-basename=master1 + binlog-format=row + ``` +- Restart MariaDB/MySQL (`sudo service mysql restart`) +- Copy the example configuration (`es_sync_config.example.json`) as `es_sync_config.json` and adjust options in it to your liking (verify the connection options!) +- Connect to mysql as root + - Verify that the result of `SHOW VARIABLES LIKE 'binlog_format';` is `ROW` + - Execute `GRANT REPLICATION SLAVE ON *.* TO 'username'@'localhost';` to allow your configured user access to the binlog + + +### Setting up sync_es.py +`sync_es.py` keeps the Elasticsearch indices updated by reading the binlog and pushing the changes to the ES indices. +- Make sure `es_sync_config.json` is configured with the user you grated the `REPLICATION` permissions +- Run `import_to_es.py` and copy the outputted JSON into the file specified by `save_loc` in your `es_sync_config.json` +- Run `sync_es.py` as-is *or*, for actual deployment, set it up as a service and run it, preferably as the system/root + - Make sure `sync_es.py` runs within the venv with the right dependencies! + +You're done! The script should now be feeding updates from the database to Elasticsearch. +Take note, however, that the specified ES index refresh interval is 30 seconds, which may feel like a long time on local development. Feel free to adjust it or [poke Elasticsearch yourself!](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-refresh.html) diff --git a/config.example.py b/config.example.py index 3b07da0..aa50cd8 100644 --- a/config.example.py +++ b/config.example.py @@ -7,7 +7,7 @@ USE_EMAIL_VERIFICATION = False USE_MYSQL = True # Enable this once stat integration is done -ENABLE_SHOW_STATS = False +ENABLE_SHOW_STATS = True BASE_DIR = os.path.abspath(os.path.dirname(__file__)) if USE_MYSQL: @@ -21,7 +21,6 @@ SECRET_KEY = '***' # Prefix for running multiple sites, user table will not be prefixed. SITE_FLAVOR = 'nyaa' # 'nyaa' or 'sukebei' -TABLE_PREFIX = SITE_FLAVOR + '_' # for recaptcha and email verification: # keys for localhost. Change as appropriate when actual domain is registered. @@ -33,16 +32,16 @@ MAIL_FROM_ADDRESS = '***' SMTP_USERNAME = '***' SMTP_PASSWORD = '***' -# What the site identifies itself as. +# What the site identifies itself as. This affects templates, not database stuff. SITE_NAME = 'Nyaa' # The maximum number of files a torrent can contain # until the site says "Too many files to display." MAX_FILES_VIEW = 1000 -# """ +# # Setting to make sure main announce url is present in torrent -# """ +# ENFORCE_MAIN_ANNOUNCE_URL = False MAIN_ANNOUNCE_URL = '' @@ -51,10 +50,11 @@ BACKUP_TORRENT_FOLDER = 'torrents' # # Search Options # -# Max ES search results, do not set over 10000 RESULTS_PER_PAGE = 75 +# See README.MD on Elasticsearch setup USE_ELASTIC_SEARCH = False ENABLE_ELASTIC_SEARCH_HIGHLIGHT = False +# Max ES search results, do not set over 10000 ES_MAX_SEARCH_RESULT = 1000 ES_INDEX_NAME = SITE_FLAVOR # we create indicies named nyaa or sukebei \ No newline at end of file diff --git a/config_es_sync.json b/config_es_sync.json deleted file mode 100644 index d2cb889..0000000 --- a/config_es_sync.json +++ /dev/null @@ -1,11 +0,0 @@ -{ -"save_loc": "/tmp/pos.json", -"mysql_host": "127.0.0.1", -"mysql_port": 13306, -"mysql_user": "root", -"mysql_password": "dunnolol", -"database": "nyaav2", -"internal_queue_depth": 10000, -"es_chunk_size": 10000, -"flush_interval": 5 -} diff --git a/db_create.py b/db_create.py index d881377..8d2eed8 100644 --- a/db_create.py +++ b/db_create.py @@ -1,35 +1,60 @@ +#!/usr/bin/env python3 import sys +import sqlalchemy from nyaa import app, db, models -# Create tables +NYAA_CATEGORIES = [ + ('Anime', ['Anime Music Video', 'English-translated', 'Non-English-translated', 'Raw']), + ('Audio', ['Lossless', 'Lossy']), + ('Literature', ['English-translated', 'Non-English-translated', 'Raw']), + ('Live Action', ['English-translated', 'Idol/Promotional Video', 'Non-English-translated', 'Raw']), + ('Pictures', ['Graphics', 'Photos']), + ('Software', ['Applications', 'Games']), +] -db.create_all() -# Insert categories and insert if it doesn't eixst -existing_cats = models.MainCategory.query.all() -if not existing_cats: - if app.config['SITE_FLAVOR'] == 'nyaa': - CATEGORIES = [ - ('Anime', ['Anime Music Video', 'English-translated', 'Non-English-translated', 'Raw']), - ('Audio', ['Lossless', 'Lossy']), - ('Literature', ['English-translated', 'Non-English-translated', 'Raw']), - ('Live Action', ['English-translated', 'Idol/Promotional Video', 'Non-English-translated', 'Raw']), - ('Pictures', ['Graphics', 'Photos']), - ('Software', ['Applications', 'Games']), - ] - elif app.config['SITE_FLAVOR'] == 'sukebei': - CATEGORIES = [ - ('Art', ['Anime', 'Doujinshi', 'Games', 'Manga', 'Pictures']), - ('Real Life', ['Photobooks / Pictures', 'Videos']), - ] - else: - CATEGORIES = [] +SUKEBEI_CATEGORIES = [ + ('Art', ['Anime', 'Doujinshi', 'Games', 'Manga', 'Pictures']), + ('Real Life', ['Photobooks / Pictures', 'Videos']), +] - for main_cat_name, sub_cat_names in CATEGORIES: - main_cat = models.MainCategory(name=main_cat_name) + +def add_categories(categories, main_class, sub_class): + for main_cat_name, sub_cat_names in categories: + main_cat = main_class(name=main_cat_name) for i, sub_cat_name in enumerate(sub_cat_names): # Composite keys can't autoincrement, set sub_cat id manually (1-index) - sub_cat = models.SubCategory(id=i+1, name=sub_cat_name, main_category=main_cat) + sub_cat = sub_class(id=i+1, name=sub_cat_name, main_category=main_cat) db.session.add(main_cat) + +if __name__ == '__main__': + # Test for the user table, assume db is empty if it's not created + database_empty = False + try: + models.User.query.first() + except (sqlalchemy.exc.ProgrammingError, sqlalchemy.exc.OperationalError): + database_empty = True + + + print('Creating all tables...') + db.create_all() + + + nyaa_category_test = models.NyaaMainCategory.query.first() + if not nyaa_category_test: + print('Adding Nyaa categories...') + add_categories(NYAA_CATEGORIES, models.NyaaMainCategory, models.NyaaSubCategory) + + sukebei_category_test = models.SukebeiMainCategory.query.first() + if not sukebei_category_test: + print('Adding Sukebei categories...') + add_categories(SUKEBEI_CATEGORIES, models.SukebeiMainCategory, models.SukebeiSubCategory) + db.session.commit() + + if database_empty: + print('Remember to run the following to mark the database up-to-date for Alembic:') + print('./db_migrate.py stamp head') + # Technically we should be able to do this here, but when you have + # Flask-Migrate and Flask-SQA and everything... I didn't get it working. \ No newline at end of file diff --git a/db_migrate.py b/db_migrate.py index 8d4f8f0..c02ff2f 100644 --- a/db_migrate.py +++ b/db_migrate.py @@ -1,5 +1,6 @@ -#!/usr/bin/python3 +#!/usr/bin/env python3 # -*- coding: utf-8 -*- +import sys from nyaa import app, db from flask_script import Manager from flask_migrate import Migrate, MigrateCommand @@ -10,4 +11,7 @@ manager = Manager(app) manager.add_command("db", MigrateCommand) if __name__ == "__main__": + # Patch sys.argv to default to 'db' + sys.argv.insert(1, 'db') + manager.run() diff --git a/es_mapping.yml b/es_mapping.yml index 1fc72ad..3490fd9 100644 --- a/es_mapping.yml +++ b/es_mapping.yml @@ -91,6 +91,8 @@ mappings: type: long seed_count: type: long + comment_count: + type: long # these ids are really only for filtering, thus keyword uploader_id: type: keyword diff --git a/es_sync_config.example.json b/es_sync_config.example.json new file mode 100644 index 0000000..b2dc524 --- /dev/null +++ b/es_sync_config.example.json @@ -0,0 +1,11 @@ +{ + "save_loc": "/tmp/pos.json", + "mysql_host": "127.0.0.1", + "mysql_port": 3306, + "mysql_user": "nyaa", + "mysql_password": "some_password", + "database": "nyaav2", + "internal_queue_depth": 10000, + "es_chunk_size": 10000, + "flush_interval": 5 +} diff --git a/import_to_es.py b/import_to_es.py index 106cbc1..b3bc3b1 100644 --- a/import_to_es.py +++ b/import_to_es.py @@ -5,22 +5,16 @@ which is assumed to already exist. This is a one-shot deal, so you'd either need to complement it with a cron job or some binlog-reading thing (TODO) """ -from nyaa import app -from nyaa.models import Torrent +import sys +import json +from nyaa import app, db, models + from elasticsearch import Elasticsearch from elasticsearch.client import IndicesClient from elasticsearch import helpers -import progressbar -import sys -bar = progressbar.ProgressBar( - max_value=Torrent.query.count(), - widgets=[ - progressbar.SimpleProgress(), - ' [', progressbar.Timer(), '] ', - progressbar.Bar(), - ' (', progressbar.ETA(), ') ', - ]) +# This should be progressbar33 +import progressbar es = Elasticsearch(timeout=30) ic = IndicesClient(es) @@ -32,11 +26,11 @@ ic = IndicesClient(es) # we don't want to reindex all the user's torrents just because they # changed their name, and we don't really want to FTS search on the user anyway. # Maybe it's more convenient to derefence though. -def mk_es(t): +def mk_es(t, index_name): return { "_id": t.id, "_type": "torrent", - "_index": app.config['ES_INDEX_NAME'], + "_index": index_name, "_source": { # we're also indexing the id as a number so you can # order by it. seems like this is just equivalent to @@ -51,6 +45,7 @@ def mk_es(t): "uploader_id": t.uploader_id, "main_category_id": t.main_category_id, "sub_category_id": t.sub_category_id, + "comment_count": t.comment_count, # XXX all the bitflags are numbers "anonymous": bool(t.anonymous), "trusted": bool(t.trusted), @@ -72,7 +67,7 @@ def mk_es(t): # page through an sqlalchemy query, like the per_fetch but # doesn't break the eager joins its doing against the stats table. # annoying that this isn't built in somehow. -def page_query(query, limit=sys.maxsize, batch_size=10000): +def page_query(query, limit=sys.maxsize, batch_size=10000, progress_bar=None): start = 0 while True: # XXX very inelegant way to do this, i'm confus @@ -88,13 +83,46 @@ def page_query(query, limit=sys.maxsize, batch_size=10000): yield(thing) if not had_things or stop == limit: break - bar.update(start) + if progress_bar: + progress_bar.update(start) start = min(limit, start + batch_size) -# turn off refreshes while bulk loading -ic.put_settings(body={'index': {'refresh_interval': '-1'}}, index=app.config['ES_INDEX_NAME']) +FLAVORS = [ + ('nyaa', models.NyaaTorrent), + ('sukebei', models.SukebeiTorrent) +] -helpers.bulk(es, (mk_es(t) for t in page_query(Torrent.query)), chunk_size=10000) +# Get binlog status from mysql +master_status = db.engine.execute('SHOW MASTER STATUS;').fetchone() -# restore to near-enough real time -ic.put_settings(body={'index': {'refresh_interval': '30s'}}, index=app.config['ES_INDEX_NAME']) +position_json = { + 'log_file': master_status[0], + 'log_pos': master_status[1] +} + +print('Save the following in the file configured in your ES sync config JSON:') +print(json.dumps(position_json)) + +for flavor, torrent_class in FLAVORS: + print('Importing torrents for index', flavor, 'from', torrent_class) + bar = progressbar.ProgressBar( + maxval=torrent_class.query.count(), + widgets=[ progressbar.SimpleProgress(), + ' [', progressbar.Timer(), '] ', + progressbar.Bar(), + ' (', progressbar.ETA(), ') ', + ]) + + # turn off refreshes while bulk loading + ic.put_settings(body={'index': {'refresh_interval': '-1'}}, index=flavor) + + bar.start() + helpers.bulk(es, (mk_es(t, flavor) for t in page_query(torrent_class.query, progress_bar=bar)), chunk_size=10000) + bar.finish() + + # Refresh the index immideately + ic.refresh(index=flavor) + print('Index refresh done.') + + # restore to near-enough real time + ic.put_settings(body={'index': {'refresh_interval': '30s'}}, index=flavor) diff --git a/migrations/versions/2bceb2cb4d7c_add_comment_count_to_torrent.py b/migrations/versions/2bceb2cb4d7c_add_comment_count_to_torrent.py new file mode 100644 index 0000000..3114e27 --- /dev/null +++ b/migrations/versions/2bceb2cb4d7c_add_comment_count_to_torrent.py @@ -0,0 +1,52 @@ +"""Add comment_count to Torrent + +Revision ID: 2bceb2cb4d7c +Revises: d0eeb8049623 +Create Date: 2017-05-26 15:07:21.114331 + +""" +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision = '2bceb2cb4d7c' +down_revision = 'd0eeb8049623' +branch_labels = None +depends_on = None + +COMMENT_UPDATE_SQL = '''UPDATE {0}_torrents + SET comment_count = ( + SELECT COUNT(*) FROM {0}_comments + WHERE {0}_torrents.id = {0}_comments.torrent_id + );''' + + +def upgrade(): + # ### commands auto generated by Alembic - please adjust! ### + op.add_column('nyaa_torrents', sa.Column('comment_count', sa.Integer(), nullable=False)) + op.create_index(op.f('ix_nyaa_torrents_comment_count'), 'nyaa_torrents', ['comment_count'], unique=False) + + op.add_column('sukebei_torrents', sa.Column('comment_count', sa.Integer(), nullable=False)) + op.create_index(op.f('ix_sukebei_torrents_comment_count'), 'sukebei_torrents', ['comment_count'], unique=False) + # ### end Alembic commands ### + + connection = op.get_bind() + + print('Updating comment counts on nyaa_torrents...') + connection.execute(sa.sql.text(COMMENT_UPDATE_SQL.format('nyaa'))) + print('Done.') + + print('Updating comment counts on sukebei_torrents...') + connection.execute(sa.sql.text(COMMENT_UPDATE_SQL.format('sukebei'))) + print('Done.') + + +def downgrade(): + # ### commands auto generated by Alembic - please adjust! ### + op.drop_index(op.f('ix_nyaa_torrents_comment_count'), table_name='nyaa_torrents') + op.drop_column('nyaa_torrents', 'comment_count') + + op.drop_index(op.f('ix_sukebei_torrents_comment_count'), table_name='sukebei_torrents') + op.drop_column('sukebei_torrents', 'comment_count') + # ### end Alembic commands ### diff --git a/migrations/versions/3001f79b7722_add_torrents.uploader_ip.py b/migrations/versions/3001f79b7722_add_torrents.uploader_ip.py index 152c440..03ed87c 100644 --- a/migrations/versions/3001f79b7722_add_torrents.uploader_ip.py +++ b/migrations/versions/3001f79b7722_add_torrents.uploader_ip.py @@ -11,20 +11,19 @@ import sqlalchemy as sa # revision identifiers, used by Alembic. revision = '3001f79b7722' -down_revision = None +down_revision = '97ddefed1834' branch_labels = None depends_on = None +TABLE_PREFIXES = ('nyaa', 'sukebei') + def upgrade(): - # ### commands auto generated by Alembic - please adjust! ### - op.add_column('nyaa_torrents', sa.Column('uploader_ip', sa.Binary(), nullable=True)) - op.add_column('sukebei_torrents', sa.Column('uploader_ip', sa.Binary(), nullable=True)) - # ### end Alembic commands ### + for prefix in TABLE_PREFIXES: + op.add_column(prefix + '_torrents', sa.Column('uploader_ip', sa.Binary(), nullable=True)) + # ### end Alembic commands ### def downgrade(): - # ### commands auto generated by Alembic - please adjust! ### - op.drop_column('nyaa_torrents', 'uploader_ip') - op.drop_column('sukebei_torrents', 'uploader_ip') - # ### end Alembic commands ### + for prefix in TABLE_PREFIXES: + op.drop_column(prefix + '_torrents', 'uploader_ip') diff --git a/migrations/versions/97ddefed1834_initial_database_state.py b/migrations/versions/97ddefed1834_initial_database_state.py new file mode 100644 index 0000000..fcb1dcb --- /dev/null +++ b/migrations/versions/97ddefed1834_initial_database_state.py @@ -0,0 +1,166 @@ +"""Initial database state + +Revision ID: 97ddefed1834 +Revises: +Create Date: 2017-05-26 18:46:14.440040 + +""" +from alembic import op +import sqlalchemy as sa +import sqlalchemy_utils +from sqlalchemy.dialects import mysql + +# revision identifiers, used by Alembic. +revision = '97ddefed1834' +down_revision = None +branch_labels = None +depends_on = None + +TABLE_PREFIXES = ('nyaa', 'sukebei') + +def upgrade(): + # Shared tables + op.create_table('users', + sa.Column('id', sa.Integer(), nullable=False), + sa.Column('username', sa.String(length=32, collation='ascii_general_ci'), nullable=False), + sa.Column('email', sqlalchemy_utils.types.email.EmailType(length=255), nullable=True), + + # These are actually PasswordType, UserStatusType and UserLevelType, + # but database-wise binary and integers are what's being used + sa.Column('password_hash', sa.Binary(length=255), nullable=False), + sa.Column('status', sa.Integer(), nullable=False), + sa.Column('level', sa.Integer(), nullable=False), + + sa.Column('created_time', sa.DateTime(), nullable=True), + sa.Column('last_login_date', sa.DateTime(), nullable=True), + sa.Column('last_login_ip', sa.Binary(), nullable=True), + sa.PrimaryKeyConstraint('id'), + sa.UniqueConstraint('email'), + sa.UniqueConstraint('username') + ) + + op.create_table('trackers', + sa.Column('id', sa.Integer(), nullable=False), + sa.Column('uri', sa.String(length=255, collation='utf8_general_ci'), nullable=False), + sa.Column('disabled', sa.Boolean(), nullable=False), + sa.PrimaryKeyConstraint('id'), + sa.UniqueConstraint('uri') + ) + + # Nyaa and Sukebei + for prefix in TABLE_PREFIXES: + # Main categories + op.create_table(prefix + '_main_categories', + sa.Column('id', sa.Integer(), nullable=False), + sa.Column('name', sa.String(length=64), nullable=False), + sa.PrimaryKeyConstraint('id') + ) + # Sub categories + op.create_table(prefix + '_sub_categories', + sa.Column('id', sa.Integer(), nullable=False), + sa.Column('main_category_id', sa.Integer(), nullable=False), + sa.Column('name', sa.String(length=64), nullable=False), + sa.ForeignKeyConstraint(['main_category_id'], [prefix + '_main_categories.id'], ), + sa.PrimaryKeyConstraint('id', 'main_category_id') + ) + # Main torrent table + op.create_table(prefix + '_torrents', + sa.Column('id', sa.Integer(), nullable=False), + sa.Column('info_hash', sa.BINARY(length=20), nullable=False), + sa.Column('display_name', sa.String(length=255, collation='utf8_general_ci'), nullable=False), + sa.Column('torrent_name', sa.String(length=255), nullable=False), + sa.Column('information', sa.String(length=255), nullable=False), + sa.Column('description', mysql.TEXT(collation='utf8mb4_bin'), nullable=False), + sa.Column('filesize', sa.BIGINT(), nullable=False), + sa.Column('encoding', sa.String(length=32), nullable=False), + sa.Column('flags', sa.Integer(), nullable=False), + sa.Column('uploader_id', sa.Integer(), nullable=True), + sa.Column('has_torrent', sa.Boolean(), nullable=False), + sa.Column('created_time', sa.DateTime(), nullable=False), + sa.Column('updated_time', sa.DateTime(), nullable=False), + sa.Column('main_category_id', sa.Integer(), nullable=False), + sa.Column('sub_category_id', sa.Integer(), nullable=False), + sa.Column('redirect', sa.Integer(), nullable=True), + sa.ForeignKeyConstraint(['main_category_id', 'sub_category_id'], [prefix + '_sub_categories.main_category_id', prefix + '_sub_categories.id'], ), + sa.ForeignKeyConstraint(['main_category_id'], [prefix + '_main_categories.id'], ), + sa.ForeignKeyConstraint(['redirect'], [prefix + '_torrents.id'], ), + sa.ForeignKeyConstraint(['uploader_id'], ['users.id'], ), + sa.PrimaryKeyConstraint('id') + ) + op.create_index(op.f('ix_' + prefix + '_torrents_display_name'), prefix + '_torrents', ['display_name'], unique=False) + op.create_index(op.f('ix_' + prefix + '_torrents_filesize'), prefix + '_torrents', ['filesize'], unique=False) + op.create_index(op.f('ix_' + prefix + '_torrents_flags'), prefix + '_torrents', ['flags'], unique=False) + op.create_index(op.f('ix_' + prefix + '_torrents_info_hash'), prefix + '_torrents', ['info_hash'], unique=True) + op.create_index(prefix + '_uploader_flag_idx', prefix + '_torrents', ['uploader_id', 'flags'], unique=False) + + # Statistics for torrents + op.create_table(prefix + '_statistics', + sa.Column('torrent_id', sa.Integer(), nullable=False), + sa.Column('seed_count', sa.Integer(), nullable=False), + sa.Column('leech_count', sa.Integer(), nullable=False), + sa.Column('download_count', sa.Integer(), nullable=False), + sa.Column('last_updated', sa.DateTime(), nullable=True), + sa.ForeignKeyConstraint(['torrent_id'], [prefix + '_torrents.id'], ondelete='CASCADE'), + sa.PrimaryKeyConstraint('torrent_id') + ) + op.create_index(op.f('ix_' + prefix + '_statistics_download_count'), prefix + '_statistics', ['download_count'], unique=False) + op.create_index(op.f('ix_' + prefix + '_statistics_leech_count'), prefix + '_statistics', ['leech_count'], unique=False) + op.create_index(op.f('ix_' + prefix + '_statistics_seed_count'), prefix + '_statistics', ['seed_count'], unique=False) + + # Trackers relationships for torrents + op.create_table(prefix + '_torrent_trackers', + sa.Column('torrent_id', sa.Integer(), nullable=False), + sa.Column('tracker_id', sa.Integer(), nullable=False), + sa.Column('order', sa.Integer(), nullable=False), + sa.ForeignKeyConstraint(['torrent_id'], [prefix + '_torrents.id'], ondelete='CASCADE'), + sa.ForeignKeyConstraint(['tracker_id'], ['trackers.id'], ondelete='CASCADE'), + sa.PrimaryKeyConstraint('torrent_id', 'tracker_id') + ) + op.create_index(op.f('ix_' + prefix + '_torrent_trackers_order'), prefix + '_torrent_trackers', ['order'], unique=False) + + # Torrent filelists + op.create_table(prefix + '_torrents_filelist', + sa.Column('torrent_id', sa.Integer(), nullable=False), + sa.Column('filelist_blob', mysql.MEDIUMBLOB(), nullable=True), + sa.ForeignKeyConstraint(['torrent_id'], [prefix + '_torrents.id'], ondelete='CASCADE'), + sa.PrimaryKeyConstraint('torrent_id'), + mysql_row_format='COMPRESSED' + ) + + # Torrent info_dicts + op.create_table(prefix + '_torrents_info', + sa.Column('torrent_id', sa.Integer(), nullable=False), + sa.Column('info_dict', mysql.MEDIUMBLOB(), nullable=True), + sa.ForeignKeyConstraint(['torrent_id'], [prefix + '_torrents.id'], ondelete='CASCADE'), + sa.PrimaryKeyConstraint('torrent_id'), + mysql_row_format='COMPRESSED' + ) + # ### end Alembic commands ### + + +def downgrade(): + # Note: this may fail. It's better to just drop all tables instead (or reset the database) + + # Nyaa and Sukebei + for prefix in TABLE_PREFIXES: + op.drop_table(prefix + '_torrents_info') + op.drop_table(prefix + '_torrents_filelist') + op.drop_index(op.f('ix_' + prefix + '_torrent_trackers_order'), table_name=prefix + '_torrent_trackers') + op.drop_table(prefix + '_torrent_trackers') + op.drop_index(op.f('ix_' + prefix + '_statistics_seed_count'), table_name=prefix + '_statistics') + op.drop_index(op.f('ix_' + prefix + '_statistics_leech_count'), table_name=prefix + '_statistics') + op.drop_index(op.f('ix_' + prefix + '_statistics_download_count'), table_name=prefix + '_statistics') + op.drop_table(prefix + '_statistics') + op.drop_table(prefix + '_torrents') + op.drop_index(prefix + '_uploader_flag_idx', table_name=prefix + '_torrents') + op.drop_index(op.f('ix_' + prefix + '_torrents_info_hash'), table_name=prefix + '_torrents') + op.drop_index(op.f('ix_' + prefix + '_torrents_flags'), table_name=prefix + '_torrents') + op.drop_index(op.f('ix_' + prefix + '_torrents_filesize'), table_name=prefix + '_torrents') + op.drop_index(op.f('ix_' + prefix + '_torrents_display_name'), table_name=prefix + '_torrents') + op.drop_table(prefix + '_sub_categories') + op.drop_table(prefix + '_main_categories') + + # Shared tables + op.drop_table('users') + op.drop_table('trackers') + # ### end Alembic commands ### diff --git a/migrations/versions/d0eeb8049623_add_comments.py b/migrations/versions/d0eeb8049623_add_comments.py index f6fab5b..47e2a7a 100644 --- a/migrations/versions/d0eeb8049623_add_comments.py +++ b/migrations/versions/d0eeb8049623_add_comments.py @@ -15,34 +15,23 @@ down_revision = '3001f79b7722' branch_labels = None depends_on = None +TABLE_PREFIXES = ('nyaa', 'sukebei') + def upgrade(): - # ### commands auto generated by Alembic - please adjust! ### - op.create_table('nyaa_comments', - sa.Column('id', sa.Integer(), nullable=False), - sa.Column('torrent_id', sa.Integer(), nullable=False), - sa.Column('user_id', sa.Integer(), nullable=True), - sa.Column('created_time', sa.DateTime(), nullable=True), - sa.Column('text', sa.String(length=255, collation='utf8mb4_bin'), nullable=False), - sa.ForeignKeyConstraint(['torrent_id'], ['nyaa_torrents.id'], ondelete='CASCADE'), - sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'), - sa.PrimaryKeyConstraint('id') - ) - op.create_table('sukebei_comments', - sa.Column('id', sa.Integer(), nullable=False), - sa.Column('torrent_id', sa.Integer(), nullable=False), - sa.Column('user_id', sa.Integer(), nullable=True), - sa.Column('created_time', sa.DateTime(), nullable=True), - sa.Column('text', sa.String(length=255, collation='utf8mb4_bin'), nullable=False), - sa.ForeignKeyConstraint(['torrent_id'], ['sukebei_torrents.id'], ondelete='CASCADE'), - sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'), - sa.PrimaryKeyConstraint('id') - ) - # ### end Alembic commands ### + for prefix in TABLE_PREFIXES: + op.create_table(prefix + '_comments', + sa.Column('id', sa.Integer(), nullable=False), + sa.Column('torrent_id', sa.Integer(), nullable=False), + sa.Column('user_id', sa.Integer(), nullable=True), + sa.Column('created_time', sa.DateTime(), nullable=True), + sa.Column('text', sa.String(length=255, collation='utf8mb4_bin'), nullable=False), + sa.ForeignKeyConstraint(['torrent_id'], [prefix + '_torrents.id'], ondelete='CASCADE'), + sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'), + sa.PrimaryKeyConstraint('id') + ) def downgrade(): - # ### commands auto generated by Alembic - please adjust! ### - op.drop_table('nyaa_comments') - op.drop_table('sukebei_comments') - # ### end Alembic commands ### + for prefix in TABLE_PREFIXES: + op.drop_table(prefix + '_comments') diff --git a/nyaa/__init__.py b/nyaa/__init__.py index d8c04bf..9ef9bfb 100644 --- a/nyaa/__init__.py +++ b/nyaa/__init__.py @@ -22,6 +22,15 @@ if app.config['DEBUG']: app.config['DEBUG_TB_INTERCEPT_REDIRECTS'] = False toolbar = DebugToolbarExtension(app) app.logger.setLevel(logging.DEBUG) + + # Forbid caching + @app.after_request + def forbid_cache(request): + request.headers['Cache-Control'] = 'no-cache, no-store, must-revalidate, max-age=0' + request.headers['Pragma'] = 'no-cache' + request.headers['Expires'] = '0' + return request + else: app.logger.setLevel(logging.WARNING) diff --git a/nyaa/forms.py b/nyaa/forms.py index d29ae82..ba0ad60 100644 --- a/nyaa/forms.py +++ b/nyaa/forms.py @@ -43,7 +43,7 @@ _username_validator = Regexp( class LoginForm(FlaskForm): - username = StringField('Username or email address', [DataRequired(), _username_validator]) + username = StringField('Username or email address', [DataRequired()]) password = PasswordField('Password', [DataRequired()]) diff --git a/nyaa/models.py b/nyaa/models.py index 66d25e2..331dc12 100644 --- a/nyaa/models.py +++ b/nyaa/models.py @@ -3,10 +3,13 @@ from enum import Enum, IntEnum from datetime import datetime, timezone from nyaa import app, db from nyaa.torrents import create_magnet + from sqlalchemy import func, ForeignKeyConstraint, Index +from sqlalchemy.ext import declarative from sqlalchemy_utils import ChoiceType, EmailType, PasswordType -from werkzeug.security import generate_password_hash, check_password_hash from sqlalchemy_fulltext import FullText + +from werkzeug.security import generate_password_hash, check_password_hash from ipaddress import ip_address import re @@ -17,7 +20,6 @@ from hashlib import md5 if app.config['USE_MYSQL']: from sqlalchemy.dialects import mysql - BinaryType = mysql.BINARY DescriptionTextType = mysql.TEXT MediumBlobType = mysql.MEDIUMBLOB @@ -32,10 +34,36 @@ else: COL_UTF8MB4_BIN = None COL_ASCII_GENERAL_CI = 'NOCASE' + # For property timestamps UTC_EPOCH = datetime.utcfromtimestamp(0) +class DeclarativeHelperBase(object): + ''' This class eases our nyaa-sukebei shenanigans by automatically adjusting + __tablename__ and providing class methods for renaming references. ''' + # See http://docs.sqlalchemy.org/en/latest/orm/extensions/declarative/api.html + + __tablename_base__ = None + __flavor__ = None + + @classmethod + def _table_prefix_string(cls): + return cls.__flavor__.lower() + '_' + + @classmethod + def _table_prefix(cls, table_name): + return cls._table_prefix_string() + table_name + + @classmethod + def _flavor_prefix(cls, table_name): + return cls.__flavor__ + table_name + + @declarative.declared_attr + def __tablename__(cls): + return cls._table_prefix(cls.__tablename_base__) + + class TorrentFlags(IntEnum): NONE = 0 ANONYMOUS = 1 @@ -46,16 +74,13 @@ class TorrentFlags(IntEnum): DELETED = 32 -DB_TABLE_PREFIX = app.config['TABLE_PREFIX'] - - -class Torrent(db.Model): - __tablename__ = DB_TABLE_PREFIX + 'torrents' +class TorrentBase(DeclarativeHelperBase): + __tablename_base__ = 'torrents' id = db.Column(db.Integer, primary_key=True) info_hash = db.Column(BinaryType(length=20), unique=True, nullable=False, index=True) - display_name = db.Column( - db.String(length=255, collation=COL_UTF8_GENERAL_CI), nullable=False, index=True) + display_name = db.Column(db.String(length=255, collation=COL_UTF8_GENERAL_CI), + nullable=False, index=True) torrent_name = db.Column(db.String(length=255), nullable=False) information = db.Column(db.String(length=255), nullable=False) description = db.Column(DescriptionTextType(collation=COL_UTF8MB4_BIN), nullable=False) @@ -63,50 +88,95 @@ class Torrent(db.Model): filesize = db.Column(db.BIGINT, default=0, nullable=False, index=True) encoding = db.Column(db.String(length=32), nullable=False) flags = db.Column(db.Integer, default=0, nullable=False, index=True) - uploader_id = db.Column(db.Integer, db.ForeignKey('users.id'), nullable=True) + + @declarative.declared_attr + def uploader_id(cls): + # Even though this is same for both tables, declarative requires this + return db.Column(db.Integer, db.ForeignKey('users.id'), nullable=True) + uploader_ip = db.Column(db.Binary(length=16), default=None, nullable=True) has_torrent = db.Column(db.Boolean, nullable=False, default=False) + comment_count = db.Column(db.Integer, default=0, nullable=False, index=True) + created_time = db.Column(db.DateTime(timezone=False), default=datetime.utcnow, nullable=False) - updated_time = db.Column(db.DateTime(timezone=False), - default=datetime.utcnow, onupdate=datetime.utcnow, nullable=False) + updated_time = db.Column(db.DateTime(timezone=False), default=datetime.utcnow, + onupdate=datetime.utcnow, nullable=False) + + @declarative.declared_attr + def main_category_id(cls): + fk = db.ForeignKey(cls._table_prefix('main_categories.id')) + return db.Column(db.Integer, fk, nullable=False) - main_category_id = db.Column(db.Integer, db.ForeignKey( - DB_TABLE_PREFIX + 'main_categories.id'), nullable=False) sub_category_id = db.Column(db.Integer, nullable=False) - redirect = db.Column(db.Integer, db.ForeignKey( - DB_TABLE_PREFIX + 'torrents.id'), nullable=True) - __table_args__ = ( - Index('uploader_flag_idx', 'uploader_id', 'flags'), - ForeignKeyConstraint( - ['main_category_id', 'sub_category_id'], - [DB_TABLE_PREFIX + 'sub_categories.main_category_id', - DB_TABLE_PREFIX + 'sub_categories.id'] - ), {} - ) + @declarative.declared_attr + def redirect(cls): + fk = db.ForeignKey(cls._table_prefix('torrents.id')) + return db.Column(db.Integer, fk, nullable=True) - user = db.relationship('User', uselist=False, back_populates='torrents') - main_category = db.relationship('MainCategory', uselist=False, - back_populates='torrents', lazy="joined") - sub_category = db.relationship('SubCategory', uselist=False, backref='torrents', lazy="joined", - primaryjoin=( - "and_(SubCategory.id == foreign(Torrent.sub_category_id), " - "SubCategory.main_category_id == Torrent.main_category_id)")) - info = db.relationship('TorrentInfo', uselist=False, - cascade="all, delete-orphan", back_populates='torrent') - filelist = db.relationship('TorrentFilelist', uselist=False, + @declarative.declared_attr + def __table_args__(cls): + return ( + Index(cls._table_prefix('uploader_flag_idx'), 'uploader_id', 'flags'), + ForeignKeyConstraint( + ['main_category_id', 'sub_category_id'], + [cls._table_prefix('sub_categories.main_category_id'), + cls._table_prefix('sub_categories.id')] + ), {} + ) + + @declarative.declared_attr + def user(cls): + return db.relationship('User', uselist=False, back_populates=cls._table_prefix('torrents')) + + @declarative.declared_attr + def main_category(cls): + return db.relationship(cls._flavor_prefix('MainCategory'), uselist=False, + back_populates='torrents', lazy="joined") + + @declarative.declared_attr + def sub_category(cls): + join_sql = ("and_({0}SubCategory.id == foreign({0}Torrent.sub_category_id), " + "{0}SubCategory.main_category_id == {0}Torrent.main_category_id)") + return db.relationship(cls._flavor_prefix('SubCategory'), uselist=False, + backref='torrents', lazy="joined", + primaryjoin=join_sql.format(cls.__flavor__)) + + @declarative.declared_attr + def info(cls): + return db.relationship(cls._flavor_prefix('TorrentInfo'), uselist=False, cascade="all, delete-orphan", back_populates='torrent') - stats = db.relationship('Statistic', uselist=False, - cascade="all, delete-orphan", back_populates='torrent', lazy='joined') - trackers = db.relationship('TorrentTrackers', uselist=True, cascade="all, delete-orphan", - lazy='joined', order_by='TorrentTrackers.order') - comments = db.relationship('Comment', uselist=True, + + @declarative.declared_attr + def filelist(cls): + return db.relationship(cls._flavor_prefix('TorrentFilelist'), uselist=False, + cascade="all, delete-orphan", back_populates='torrent') + + @declarative.declared_attr + def stats(cls): + return db.relationship(cls._flavor_prefix('Statistic'), uselist=False, + cascade="all, delete-orphan", back_populates='torrent', + lazy='joined') + + @declarative.declared_attr + def trackers(cls): + return db.relationship(cls._flavor_prefix('TorrentTrackers'), uselist=True, + cascade="all, delete-orphan", lazy='joined', + order_by=cls._flavor_prefix('TorrentTrackers.order')) + + @declarative.declared_attr + def comments(cls): + return db.relationship(cls._flavor_prefix('Comment'), uselist=True, cascade="all, delete-orphan") def __repr__(self): return '<{0} #{1.id} \'{1.display_name}\' {1.filesize}b>'.format(type(self).__name__, self) + def update_comment_count(self): + self.comment_count = Comment.query.filter_by(torrent_id=self.id).count() + return self.comment_count + @property def created_utc_timestamp(self): ''' Returns a UTC POSIX timestamp, as seconds ''' @@ -149,6 +219,8 @@ class Torrent(db.Model): if self.uploader_ip: return str(ip_address(self.uploader_ip)) + # Flag getters and setters below + @property def anonymous(self): return self.flags & TorrentFlags.ANONYMOUS @@ -197,6 +269,8 @@ class Torrent(db.Model): def complete(self, value): self.flags = (self.flags & ~TorrentFlags.COMPLETE) | (value and TorrentFlags.COMPLETE) + # Class methods + @classmethod def by_id(cls, id): return cls.query.get(id) @@ -211,44 +285,57 @@ class Torrent(db.Model): return cls.by_info_hash(info_hash_bytes) -class TorrentNameSearch(FullText, Torrent): - __fulltext_columns__ = ('display_name',) +class TorrentFilelistBase(DeclarativeHelperBase): + __tablename_base__ = 'torrents_filelist' - -class TorrentFilelist(db.Model): - __tablename__ = DB_TABLE_PREFIX + 'torrents_filelist' __table_args__ = {'mysql_row_format': 'COMPRESSED'} - torrent_id = db.Column(db.Integer, db.ForeignKey( - DB_TABLE_PREFIX + 'torrents.id', ondelete="CASCADE"), primary_key=True) + @declarative.declared_attr + def torrent_id(cls): + fk = db.ForeignKey(cls._table_prefix('torrents.id'), ondelete="CASCADE") + return db.Column(db.Integer, fk, primary_key=True) + filelist_blob = db.Column(MediumBlobType, nullable=True) - torrent = db.relationship('Torrent', uselist=False, back_populates='filelist') + @declarative.declared_attr + def torrent(cls): + return db.relationship(cls._flavor_prefix('Torrent'), uselist=False, + back_populates='filelist') -class TorrentInfo(db.Model): - __tablename__ = DB_TABLE_PREFIX + 'torrents_info' +class TorrentInfoBase(DeclarativeHelperBase): + __tablename_base__ = 'torrents_info' + __table_args__ = {'mysql_row_format': 'COMPRESSED'} - torrent_id = db.Column(db.Integer, db.ForeignKey( - DB_TABLE_PREFIX + 'torrents.id', ondelete="CASCADE"), primary_key=True) + @declarative.declared_attr + def torrent_id(cls): + return db.Column(db.Integer, db.ForeignKey( + cls._table_prefix('torrents.id'), ondelete="CASCADE"), primary_key=True) info_dict = db.Column(MediumBlobType, nullable=True) - torrent = db.relationship('Torrent', uselist=False, back_populates='info') + @declarative.declared_attr + def torrent(cls): + return db.relationship(cls._flavor_prefix('Torrent'), uselist=False, back_populates='info') -class Statistic(db.Model): - __tablename__ = DB_TABLE_PREFIX + 'statistics' +class StatisticBase(DeclarativeHelperBase): + __tablename_base__ = 'statistics' - torrent_id = db.Column(db.Integer, db.ForeignKey( - DB_TABLE_PREFIX + 'torrents.id', ondelete="CASCADE"), primary_key=True) + @declarative.declared_attr + def torrent_id(cls): + fk = db.ForeignKey(cls._table_prefix('torrents.id'), ondelete="CASCADE") + return db.Column(db.Integer, fk, primary_key=True) seed_count = db.Column(db.Integer, default=0, nullable=False, index=True) leech_count = db.Column(db.Integer, default=0, nullable=False, index=True) download_count = db.Column(db.Integer, default=0, nullable=False, index=True) last_updated = db.Column(db.DateTime(timezone=False)) - torrent = db.relationship('Torrent', uselist=False, back_populates='stats') + @declarative.declared_attr + def torrent(cls): + return db.relationship(cls._flavor_prefix('Torrent'), uselist=False, + back_populates='stats') class Trackers(db.Model): @@ -264,30 +351,43 @@ class Trackers(db.Model): return cls.query.filter_by(uri=uri).first() -class TorrentTrackers(db.Model): - __tablename__ = DB_TABLE_PREFIX + 'torrent_trackers' +class TorrentTrackersBase(DeclarativeHelperBase): + __tablename_base__ = 'torrent_trackers' + + @declarative.declared_attr + def torrent_id(cls): + fk = db.ForeignKey(cls._table_prefix('torrents.id'), ondelete="CASCADE") + return db.Column(db.Integer, fk, primary_key=True) + + @declarative.declared_attr + def tracker_id(cls): + fk = db.ForeignKey('trackers.id', ondelete="CASCADE") + return db.Column(db.Integer, fk, primary_key=True) - torrent_id = db.Column(db.Integer, db.ForeignKey( - DB_TABLE_PREFIX + 'torrents.id', ondelete="CASCADE"), primary_key=True) - tracker_id = db.Column(db.Integer, db.ForeignKey( - 'trackers.id', ondelete="CASCADE"), primary_key=True) order = db.Column(db.Integer, nullable=False, index=True) - tracker = db.relationship('Trackers', uselist=False, lazy='joined') + @declarative.declared_attr + def tracker(cls): + return db.relationship('Trackers', uselist=False, lazy='joined') @classmethod def by_torrent_id(cls, torrent_id): return cls.query.filter_by(torrent_id=torrent_id).order_by(cls.order.desc()) -class MainCategory(db.Model): - __tablename__ = DB_TABLE_PREFIX + 'main_categories' +class MainCategoryBase(DeclarativeHelperBase): + __tablename_base__ = 'main_categories' id = db.Column(db.Integer, primary_key=True) name = db.Column(db.String(length=64), nullable=False) - sub_categories = db.relationship('SubCategory', back_populates='main_category') - torrents = db.relationship('Torrent', back_populates='main_category') + @declarative.declared_attr + def sub_categories(cls): + return db.relationship(cls._flavor_prefix('SubCategory'), back_populates='main_category') + + @declarative.declared_attr + def torrents(cls): + return db.relationship(cls._flavor_prefix('Torrent'), back_populates='main_category') def get_category_ids(self): return (self.id, 0) @@ -301,18 +401,22 @@ class MainCategory(db.Model): return cls.query.get(id) -class SubCategory(db.Model): - __tablename__ = DB_TABLE_PREFIX + 'sub_categories' +class SubCategoryBase(DeclarativeHelperBase): + __tablename_base__ = 'sub_categories' id = db.Column(db.Integer, primary_key=True) - main_category_id = db.Column(db.Integer, db.ForeignKey( - DB_TABLE_PREFIX + 'main_categories.id'), primary_key=True) + + @declarative.declared_attr + def main_category_id(cls): + fk = db.ForeignKey(cls._table_prefix('main_categories.id')) + return db.Column(db.Integer, fk, primary_key=True) + name = db.Column(db.String(length=64), nullable=False) - main_category = db.relationship('MainCategory', uselist=False, back_populates='sub_categories') -# torrents = db.relationship('Torrent', back_populates='sub_category'), -# primaryjoin="and_(Torrent.sub_category_id == foreign(SubCategory.id), " -# "Torrent.main_category_id == SubCategory.main_category_id)") + @declarative.declared_attr + def main_category(cls): + return db.relationship(cls._flavor_prefix('MainCategory'), uselist=False, + back_populates='sub_categories') def get_category_ids(self): return (self.main_category_id, self.id) @@ -326,17 +430,27 @@ class SubCategory(db.Model): return cls.query.get((sub_cat_id, main_cat_id)) -class Comment(db.Model): - __tablename__ = DB_TABLE_PREFIX + 'comments' +class CommentBase(DeclarativeHelperBase): + __tablename_base__ = 'comments' id = db.Column(db.Integer, primary_key=True) - torrent_id = db.Column(db.Integer, db.ForeignKey( - DB_TABLE_PREFIX + 'torrents.id', ondelete='CASCADE'), nullable=False) - user_id = db.Column(db.Integer, db.ForeignKey('users.id', ondelete='CASCADE')) + + @declarative.declared_attr + def torrent_id(cls): + return db.Column(db.Integer, db.ForeignKey( + cls._table_prefix('torrents.id'), ondelete='CASCADE'), nullable=False) + + @declarative.declared_attr + def user_id(cls): + return db.Column(db.Integer, db.ForeignKey('users.id', ondelete='CASCADE')) + created_time = db.Column(db.DateTime(timezone=False), default=datetime.utcnow) text = db.Column(db.String(length=255, collation=COL_UTF8MB4_BIN), nullable=False) - user = db.relationship('User', uselist=False, back_populates='comments', lazy="joined") + @declarative.declared_attr + def user(cls): + return db.relationship('User', uselist=False, + back_populates=cls._table_prefix('comments'), lazy="joined") def __repr__(self): return '' % self.id @@ -376,9 +490,11 @@ class User(db.Model): last_login_date = db.Column(db.DateTime(timezone=False), default=None, nullable=True) last_login_ip = db.Column(db.Binary(length=16), default=None, nullable=True) - torrents = db.relationship('Torrent', back_populates='user', lazy='dynamic') - comments = db.relationship('Comment', back_populates='user', lazy='dynamic') - # session = db.relationship('Session', uselist=False, back_populates='user') + nyaa_torrents = db.relationship('NyaaTorrent', back_populates='user', lazy='dynamic') + nyaa_comments = db.relationship('NyaaComment', back_populates='user', lazy='dynamic') + + sukebei_torrents = db.relationship('SukebeiTorrent', back_populates='user', lazy='dynamic') + sukebei_comments = db.relationship('SukebeiComment', back_populates='user', lazy='dynamic') def __init__(self, username, email, password): self.username = username @@ -470,20 +586,30 @@ class ReportStatus(IntEnum): INVALID = 2 -class Report(db.Model): - __tablename__ = DB_TABLE_PREFIX + 'reports' +class ReportBase(DeclarativeHelperBase): + __tablename_base__ = 'reports' id = db.Column(db.Integer, primary_key=True) - torrent_id = db.Column(db.Integer, db.ForeignKey( - DB_TABLE_PREFIX + 'torrents.id', ondelete='CASCADE')) - user_id = db.Column(db.Integer, db.ForeignKey( - 'users.id')) created_time = db.Column(db.DateTime(timezone=False), default=datetime.utcnow) reason = db.Column(db.String(length=255), nullable=False) status = db.Column(ChoiceType(ReportStatus, impl=db.Integer()), nullable=False) - user = db.relationship('User', uselist=False, lazy="joined") - torrent = db.relationship('Torrent', uselist=False, lazy="joined") + @declarative.declared_attr + def torrent_id(cls): + return db.Column(db.Integer, db.ForeignKey( + cls._table_prefix('torrents.id'), ondelete='CASCADE'), nullable=False) + + @declarative.declared_attr + def user_id(cls): + return db.Column(db.Integer, db.ForeignKey('users.id')) + + @declarative.declared_attr + def user(cls): + return db.relationship('User', uselist=False, lazy="joined") + + @declarative.declared_attr + def torrent(cls): + return db.relationship(cls._flavor_prefix('Torrent'), uselist=False, lazy="joined") def __init__(self, torrent_id, user_id, reason): self.torrent_id = torrent_id @@ -512,12 +638,130 @@ class Report(db.Model): def remove_reviewed(cls, id): return cls.query.filter(cls.torrent_id == id, cls.status == 0).delete() -# class Session(db.Model): -# __tablename__ = 'sessions' -# -# session_id = db.Column(db.Integer, primary_key=True) -# user_id = db.Column(db.Integer, db.ForeignKey('users.id'), nullable=False) -# login_ip = db.Column(db.Binary(length=16), nullable=True) -# login_date = db.Column(db.DateTime(timezone=False), nullable=True) -# -# user = db.relationship('User', back_populates='session') + +# Actually declare our site-specific classes + +# Torrent +class NyaaTorrent(TorrentBase, db.Model): + __flavor__ = 'Nyaa' + + +class SukebeiTorrent(TorrentBase, db.Model): + __flavor__ = 'Sukebei' + + +# Fulltext models for MySQL +if app.config['USE_MYSQL']: + class NyaaTorrentNameSearch(FullText, NyaaTorrent): + __fulltext_columns__ = ('display_name',) + __table_args__ = {'extend_existing': True} + + class SukebeiTorrentNameSearch(FullText, SukebeiTorrent): + __fulltext_columns__ = ('display_name',) + __table_args__ = {'extend_existing': True} +else: + # Bogus classes for Sqlite + class NyaaTorrentNameSearch(object): + pass + + class SukebeiTorrentNameSearch(object): + pass + + +# TorrentFilelist +class NyaaTorrentFilelist(TorrentFilelistBase, db.Model): + __flavor__ = 'Nyaa' + + +class SukebeiTorrentFilelist(TorrentFilelistBase, db.Model): + __flavor__ = 'Sukebei' + + +# TorrentInfo +class NyaaTorrentInfo(TorrentInfoBase, db.Model): + __flavor__ = 'Nyaa' + + +class SukebeiTorrentInfo(TorrentInfoBase, db.Model): + __flavor__ = 'Sukebei' + + +# Statistic +class NyaaStatistic(StatisticBase, db.Model): + __flavor__ = 'Nyaa' + + +class SukebeiStatistic(StatisticBase, db.Model): + __flavor__ = 'Sukebei' + + +# TorrentTrackers +class NyaaTorrentTrackers(TorrentTrackersBase, db.Model): + __flavor__ = 'Nyaa' + + +class SukebeiTorrentTrackers(TorrentTrackersBase, db.Model): + __flavor__ = 'Sukebei' + + +# MainCategory +class NyaaMainCategory(MainCategoryBase, db.Model): + __flavor__ = 'Nyaa' + + +class SukebeiMainCategory(MainCategoryBase, db.Model): + __flavor__ = 'Sukebei' + + +# SubCategory +class NyaaSubCategory(SubCategoryBase, db.Model): + __flavor__ = 'Nyaa' + + +class SukebeiSubCategory(SubCategoryBase, db.Model): + __flavor__ = 'Sukebei' + + +# Comment +class NyaaComment(CommentBase, db.Model): + __flavor__ = 'Nyaa' + + +class SukebeiComment(CommentBase, db.Model): + __flavor__ = 'Sukebei' + + +# Report +class NyaaReport(ReportBase, db.Model): + __flavor__ = 'Nyaa' + + +class SukebeiReport(ReportBase, db.Model): + __flavor__ = 'Sukebei' + + +# Choose our defaults for models.Torrent etc +if app.config['SITE_FLAVOR'] == 'nyaa': + Torrent = NyaaTorrent + TorrentFilelist = NyaaTorrentFilelist + TorrentInfo = NyaaTorrentInfo + Statistic = NyaaStatistic + TorrentTrackers = NyaaTorrentTrackers + MainCategory = NyaaMainCategory + SubCategory = NyaaSubCategory + Comment = NyaaComment + Report = NyaaReport + + TorrentNameSearch = NyaaTorrentNameSearch +elif app.config['SITE_FLAVOR'] == 'sukebei': + Torrent = SukebeiTorrent + TorrentFilelist = SukebeiTorrentFilelist + TorrentInfo = SukebeiTorrentInfo + Statistic = SukebeiStatistic + TorrentTrackers = SukebeiTorrentTrackers + MainCategory = SukebeiMainCategory + SubCategory = SukebeiSubCategory + Comment = SukebeiComment + Report = SukebeiReport + + TorrentNameSearch = SukebeiTorrentNameSearch diff --git a/nyaa/routes.py b/nyaa/routes.py index 17daf99..eaf1e59 100644 --- a/nyaa/routes.py +++ b/nyaa/routes.py @@ -51,7 +51,7 @@ def redirect_url(): @app.template_global() -def static_cachebuster(static_filename): +def static_cachebuster(filename): ''' Adds a ?t= cachebuster to the given path, if the file exists. Results are cached in memory and persist until app restart! ''' # Instead of timestamps, we could use commit hashes (we already load it in __init__) @@ -60,19 +60,18 @@ def static_cachebuster(static_filename): if app.debug: # Do not bust cache on debug (helps debugging) - return static_filename + return flask.url_for('static', filename=filename) # Get file mtime if not already cached. - if static_filename not in _static_cache: - file_path = os.path.join(app.config['BASE_DIR'], 'nyaa', static_filename[1:]) + if filename not in _static_cache: + file_path = os.path.join(app.static_folder, filename) + file_mtime = None if os.path.exists(file_path): file_mtime = int(os.path.getmtime(file_path)) - _static_cache[static_filename] = static_filename + '?t=' + str(file_mtime) - else: - # Throw a warning? - _static_cache[static_filename] = static_filename - return _static_cache[static_filename] + _static_cache[filename] = file_mtime + + return flask.url_for('static', filename=filename, t=_static_cache[filename]) @app.template_global() @@ -657,9 +656,10 @@ def view_torrent(torrent_id): text=comment_text) db.session.add(comment) - db.session.commit() + db.session.flush() - torrent_count = models.Comment.query.filter_by(torrent_id=torrent.id).count() + torrent_count = torrent.update_comment_count() + db.session.commit() flask.flash('Comment successfully posted.', 'success') @@ -687,6 +687,9 @@ def view_torrent(torrent_id): def delete_comment(torrent_id, comment_id): if not flask.g.user: flask.abort(403) + torrent = models.Torrent.by_id(torrent_id) + if not torrent: + flask.abort(404) comment = models.Comment.query.filter_by(id=comment_id).first() if not comment: @@ -696,6 +699,8 @@ def delete_comment(torrent_id, comment_id): flask.abort(403) db.session.delete(comment) + db.session.flush() + torrent.update_comment_count() db.session.commit() flask.flash('Comment successfully deleted.', 'success') diff --git a/nyaa/search.py b/nyaa/search.py index 9e22f84..976ad5c 100644 --- a/nyaa/search.py +++ b/nyaa/search.py @@ -25,6 +25,7 @@ def search_elastic(term='', user=None, sort='id', order='desc', 'id': 'id', 'size': 'filesize', # 'name': 'display_name', # This is slow and buggy + 'comments': 'comment_count', 'seeders': 'seed_count', 'leechers': 'leech_count', 'downloads': 'download_count' @@ -190,6 +191,7 @@ def search_db(term='', user=None, sort='id', order='desc', category='0_0', 'size': models.Torrent.filesize, # Disable this because we disabled this in search_elastic, for the sake of consistency: # 'name': models.Torrent.display_name, + 'comments': models.Torrent.comment_count, 'seeders': models.Statistic.seed_count, 'leechers': models.Statistic.leech_count, 'downloads': models.Statistic.download_count diff --git a/nyaa/static/css/main.css b/nyaa/static/css/main.css index 3368282..4d70ea0 100644 --- a/nyaa/static/css/main.css +++ b/nyaa/static/css/main.css @@ -62,6 +62,30 @@ table.torrent-list tbody tr td a:visited { color: #1d4568; } +/* comments count */ +table.torrent-list .hdr-comments { + border-left: hidden; + font-size: medium; +} + +table.torrent-list .hdr-comments i { + margin-right: 6px; +} + +table.torrent-list tbody .comments { + position: relative; + float: right; + border: 1px solid #d7d7d7; + border-radius: 3px; + color: #383838; + padding: 0 5px; + font-size: small; + background-color: #ffffff; +} +table.torrent-list tbody .comments i { + padding-right: 2px; +} + #torrent-description img { max-width: 100%; } diff --git a/nyaa/templates/layout.html b/nyaa/templates/layout.html index f4faac7..ffbb18f 100644 --- a/nyaa/templates/layout.html +++ b/nyaa/templates/layout.html @@ -6,9 +6,9 @@ - - - + + + @@ -25,10 +25,10 @@ make the navbar not look awful on tablets. --> {# These are extracted here for the dark mode toggle #} - {% set bootstrap_light = static_cachebuster('/static/css/bootstrap.min.css') %} - {% set bootstrap_dark = static_cachebuster('/static/css/bootstrap-dark.min.css') %} + {% set bootstrap_light = static_cachebuster('css/bootstrap.min.css') %} + {% set bootstrap_dark = static_cachebuster('css/bootstrap-dark.min.css') %} - + - + - - + +