sb745/NyaaV3
1
0
Fork 0
mirror of https://github.com/sb745/NyaaV3.git synced 2025-03-12 05:46:55 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions

Updated for NyaaV3

Browse source
This commit is contained in:
sb745 2025-03-02 15:08:58 +02:00
parent 4fe0ff5b1a
commit f3031cd480
Signed by: sb745
GPG key ID: 1C93C11AC862817B
39 changed files with 752 additions and 577 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
.docker/README.md
View file

@ -1,4 +1,6 @@
# Nyaa on Docker # Nyaa on Docker
> [!CAUTION]
> Docker deployment is out of date and currently unsupported in NyaaV3.
Docker infrastructure is provided to ease setting up a dev environment Docker infrastructure is provided to ease setting up a dev environment

2
.docker/es_sync_config.json
View file

@ -4,7 +4,7 @@
"mysql_port": 3306, "mysql_port": 3306,
"mysql_user": "nyaadev", "mysql_user": "nyaadev",
"mysql_password": "ZmtB2oihHFvc39JaEDoF", "mysql_password": "ZmtB2oihHFvc39JaEDoF",
"database": "nyaav2", "database": "nyaav3",
"internal_queue_depth": 10000, "internal_queue_depth": 10000,
"es_chunk_size": 10000, "es_chunk_size": 10000,
"flush_interval": 5 "flush_interval": 5

2
.docker/full-stack.yml
View file

@ -48,7 +48,7 @@ services:
- MYSQL_RANDOM_ROOT_PASSWORD=yes - MYSQL_RANDOM_ROOT_PASSWORD=yes
- MYSQL_USER=nyaadev - MYSQL_USER=nyaadev
- MYSQL_PASSWORD=ZmtB2oihHFvc39JaEDoF - MYSQL_PASSWORD=ZmtB2oihHFvc39JaEDoF
- MYSQL_DATABASE=nyaav2 - MYSQL_DATABASE=nyaav3
elasticsearch: elasticsearch:
image: elasticsearch:6.5.4 image: elasticsearch:6.5.4

2
.docker/nyaa-config-partial.py
View file

@ -3,7 +3,7 @@
SITE_NAME = 'Nyaa [DEVEL]' SITE_NAME = 'Nyaa [DEVEL]'
GLOBAL_SITE_NAME = 'nyaa.devel' GLOBAL_SITE_NAME = 'nyaa.devel'
SQLALCHEMY_DATABASE_URI = ('mysql://nyaadev:ZmtB2oihHFvc39JaEDoF@mariadb/nyaav2?charset=utf8mb4') SQLALCHEMY_DATABASE_URI = ('mysql://nyaadev:ZmtB2oihHFvc39JaEDoF@mariadb/nyaav3?charset=utf8mb4')
# MAIN_ANNOUNCE_URL = 'http://chihaya:6881/announce' # MAIN_ANNOUNCE_URL = 'http://chihaya:6881/announce'
# TRACKER_API_URL = 'http://chihaya:6881/api' # TRACKER_API_URL = 'http://chihaya:6881/api'
BACKUP_TORRENT_FOLDER = '/nyaa-torrents' BACKUP_TORRENT_FOLDER = '/nyaa-torrents'

6
.github/issue_template.md vendored
View file

@ -1,5 +1,3 @@
Describe your issue/feature request here (you can remove all this text). Describe well and include images, if relevant! Describe your issue/feature request here (you can remove all this text). Describe well and include images if relevant.
Please make sure to skim through the existing issues, your issue/request/etc may have already been noted! Please make sure to skim through the existing issues, as your issue/request/etc. may have already been noted!
IMPORTANT: only submit issues that are relevant to the code. We do not offer support for any deployments of the project here; make your way to the IRC channel in such cases.

7
.travis.yml
View file

@ -1,9 +1,8 @@
language: python language: python
python: "3.7" python: "3.13"
dist: xenial dist: jammy
sudo: required
matrix: matrix:
fast_finish: true fast_finish: true
@ -14,7 +13,7 @@ services:
mysql mysql
before_install: before_install:
- mysql -u root -e 'CREATE DATABASE nyaav2 DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;' - mysql -u root -e 'CREATE DATABASE nyaav3 DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;'
install: install:
- pip install -r requirements.txt - pip install -r requirements.txt

85
README.md
View file

@ -1,47 +1,54 @@
# NyaaV2 [![Build Status](https://travis-ci.org/nyaadevs/nyaa.svg?branch=master)](https://travis-ci.org/nyaadevs/nyaa) # NyaaV3 [![python](https://img.shields.io/badge/Python-3.13.2-3776AB.svg?style=flat&logo=python&logoColor=white)](https://www.python.org) ![Maintenance](https://img.shields.io/maintenance/yes/2025)
## Setting up for development ## Setting up for development
This project uses Python 3.7. There are features used that do not exist in 3.6, so make sure to use Python 3.7. This project uses Python 3.13. The codebase has been updated from the original Python 3.7 version to use modern Python features and updated dependencies.
This guide also assumes you 1) are using Linux and 2) are somewhat capable with the commandline. This guide assumes you are using Linux and are somewhat capable with the commandline.
It's not impossible to run Nyaa on Windows, but this guide doesn't focus on that. Running Nyaa on Windows may be possible, but it's currently unsupported.
### Major changes from NyaaV2
- Updated from Python 3.7 to Python 3.13
- Updated all dependencies to their latest versions
- Modernized code patterns for Flask 3.0 and SQLAlchemy 2.0
- Replaced deprecated Flask-Script, orderedset and `flask.Markup` with Flask CLI, orderly-set and markupsafe
- Implemented mail error handling
### Code Quality: ### Code Quality:
- Before we get any deeper, remember to follow PEP8 style guidelines and run `./dev.py lint` before committing to see a list of warnings/problems. - Before we get any deeper, remember to follow PEP8 style guidelines and run `python dev.py lint` before committing to see a list of warnings/problems.
- You may also use `./dev.py fix && ./dev.py isort` to automatically fix some of the issues reported by the previous command. - You may also use `python dev.py fix && python dev.py isort` to automatically fix some of the issues reported by the previous command.
- Other than PEP8, try to keep your code clean and easy to understand, as well. It's only polite! - Other than PEP8, try to keep your code clean and easy to understand, as well. It's only polite!
### Running Tests ### Running Tests
The `tests` folder contains tests for the the `nyaa` module and the webserver. To run the tests: The `tests` folder contains tests for the the `nyaa` module and the webserver. To run the tests:
- Make sure that you are in the python virtual environment. - Make sure that you are in the Python virtual environment.
- Run `./dev.py test` while in the repository directory. - Run `python dev.py test` while in the repository directory.
### Setting up Pyenv ### Setting up Pyenv
pyenv eases the use of different Python versions, and as not all Linux distros offer 3.7 packages, it's right up our alley. pyenv eases the use of different Python versions, and as not all Linux distros offer 3.13 packages, it's right up our alley.
- Install dependencies https://github.com/pyenv/pyenv/wiki/Common-build-problems - Install [dependencies](https://github.com/pyenv/pyenv/wiki/Common-build-problems)
- Install `pyenv` https://github.com/pyenv/pyenv/blob/master/README.md#installation - Install [pyenv](https://github.com/pyenv/pyenv/blob/master/README.md#installation)
- Install `pyenv-virtualenv` https://github.com/pyenv/pyenv-virtualenv/blob/master/README.md - Install [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv/blob/master/README.md)
- Install Python 3.7.2 with `pyenv` and create a virtualenv for the project: - Install Python 3.13 with `pyenv` and create a virtualenv for the project:
- `pyenv install 3.7.2` - `pyenv install 3.13.2`
- `pyenv virtualenv 3.7.2 nyaa` - `pyenv virtualenv 3.13.2 nyaa`
- `pyenv activate nyaa` - `pyenv activate nyaa`
- Install dependencies with `pip install -r requirements.txt` - Install dependencies with `pip install -r requirements.txt`
- Copy `config.example.py` into `config.py` - Copy `config.example.py` into `config.py`
- Change `SITE_FLAVOR` in your `config.py` depending on which instance you want to host - Change `SITE_FLAVOR` in your `config.py` depending on which instance you want to host
### Setting up MySQL/MariaDB database ### Setting up MySQL/MariaDB database
You *may* use SQLite but the current support for it in this project is outdated and rather unsupported. > [!WARNING]
> You *may* use SQLite but it is currently untested and unsupported.
- Enable `USE_MYSQL` flag in config.py - Enable `USE_MYSQL` flag in config.py
- Install latest mariadb by following instructions here https://downloads.mariadb.org/mariadb/repositories/ - Install MariaDB by following instructions [here](https://downloads.mariadb.org/mariadb/repositories/)
- Tested versions: `mysql Ver 15.1 Distrib 10.0.30-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2`
- Run the following commands logged in as your root db user (substitute for your own `config.py` values if desired): - Run the following commands logged in as your root db user (substitute for your own `config.py` values if desired):
- `CREATE USER 'test'@'localhost' IDENTIFIED BY 'test123';` - `CREATE USER 'test'@'localhost' IDENTIFIED BY 'test123';`
- `GRANT ALL PRIVILEGES ON *.* TO 'test'@'localhost';` - `GRANT ALL PRIVILEGES ON *.* TO 'test'@'localhost';`
- `FLUSH PRIVILEGES;` - `FLUSH PRIVILEGES;`
- `CREATE DATABASE nyaav2 DEFAULT CHARACTER SET utf8 COLLATE utf8_bin;` - `CREATE DATABASE nyaav3 DEFAULT CHARACTER SET utf8 COLLATE utf8_bin;`
### Finishing up ### Finishing up
- Run `python db_create.py` to create the database and import categories - Run `python db_create.py` to create the database and import categories
- Follow the advice of `db_create.py` and run `./db_migrate.py stamp head` to mark the database version for Alembic - Follow the advice of `db_create.py` and run `python db_migrate.py stamp head` to mark the database version for Alembic
- Start the dev server with `python run.py` - Start the dev server with `python run.py`
- When you are finished developing, deactivate your virtualenv with `pyenv deactivate` or `source deactivate` (or just close your shell session) - When you are finished developing, deactivate your virtualenv with `pyenv deactivate` or `source deactivate` (or just close your shell session)
@ -50,32 +57,35 @@ Continue below to learn about database migrations and enabling the advanced sear
## Database migrations ## Database migrations
- Database migrations are done with [flask-Migrate](https://flask-migrate.readthedocs.io/), a wrapper around [Alembic](http://alembic.zzzcomputing.com/en/latest/). > [!WARNING]
> The database migration feature has been updated but will no longer be supported in NyaaV3.
- Database migrations are done with [Flask-Migrate](https://flask-migrate.readthedocs.io/), a wrapper around [Alembic](http://alembic.zzzcomputing.com/en/latest/).
- The migration system has been updated to use Flask CLI instead of the deprecated Flask-Script.
- If someone has made changes in the database schema and included a new migration script: - If someone has made changes in the database schema and included a new migration script:
- If your database has never been marked by Alembic (you're on a database from before the migrations), run `./db_migrate.py stamp head` before pulling the new migration script(s). - If your database has never been marked by Alembic (you're on a database from before the migrations), run `python db_migrate.py db stamp head` before pulling the new migration script(s).
- If you already have the new scripts, check the output of `./db_migrate.py history` instead and choose a hash that matches your current database state, then run `./db_migrate.py stamp <hash>`. - If you already have the new scripts, check the output of `python db_migrate.py db history` instead and choose a hash that matches your current database state, then run `python db_migrate.py db stamp <hash>`.
- Update your branch (eg. `git fetch && git rebase origin/master`) - Update your branch (eg. `git fetch && git rebase origin/master`)
- Run `./db_migrate.py upgrade head` to run the migration. Done! - Run `python db_migrate.py db upgrade head` to run the migration. Done!
- If *you* have made a change in the database schema: - If *you* have made a change in the database schema:
- Save your changes in `models.py` and ensure the database schema matches the previous version (ie. your new tables/columns are not added to the live database) - Save your changes in `models.py` and ensure the database schema matches the previous version (ie. your new tables/columns are not added to the live database)
- Run `./db_migrate.py migrate -m "Short description of changes"` to automatically generate a migration script for the changes - Run `python db_migrate.py db migrate -m "Short description of changes"` to automatically generate a migration script for the changes
- Check the script (`migrations/versions/...`) and make sure it works! Alembic may not able to notice all changes. - Check the script (`migrations/versions/...`) and make sure it works! Alembic may not able to notice all changes.
- Run `./db_migrate.py upgrade` to run the migration and verify the upgrade works. - Run `python db_migrate.py db upgrade` to run the migration and verify the upgrade works.
- (Run `./db_migrate.py downgrade` to verify the downgrade works as well, then upgrade again) - (Run `python db_migrate.py db downgrade` to verify the downgrade works as well, then upgrade again)
## Setting up and enabling Elasticsearch ## Setting up and enabling Elasticsearch
### Installing Elasticsearch ### Installing Elasticsearch
- Install JDK with `sudo apt-get install openjdk-8-jdk` - Install JDK with `sudo apt-get install openjdk-8-jdk`
- Install [Elasticsearch](https://www.elastic.co/downloads/elasticsearch) - Install Elasticsearch
- [From packages...](https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html) - [From packages](https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html)
- Enable the service: - Enable the service:
- `sudo systemctl enable elasticsearch.service` - `sudo systemctl enable elasticsearch.service`
- `sudo systemctl start elasticsearch.service` - `sudo systemctl start elasticsearch.service`
- or [simply extracting the archives and running the files](https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html), if you don't feel like permantently installing ES - or [simply extracting the archives and running the files](https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html), if you don't feel like permanently installing ES
- Run `curl -XGET 'localhost:9200'` and make sure ES is running - Run `curl -XGET 'localhost:9200'` and make sure ES is running
- Optional: install [Kibana](https://www.elastic.co/products/kibana) as a search debug frontend for ES - Install [Kibana](https://www.elastic.co/products/kibana) as a search debug frontend for ES (*optional*)
### Enabling MySQL Binlogging ### Enabling MySQL Binlogging
- Edit your MariaDB/MySQL server configuration and add the following under `[mariadb]`: - Edit your MariaDB/MySQL server configuration and add the following under `[mariadb]`:
@ -113,3 +123,16 @@ However, take note that binglog is not necessary for simple ES testing and devel
You're done! The script should now be feeding updates from the database to Elasticsearch. You're done! The script should now be feeding updates from the database to Elasticsearch.
Take note, however, that the specified ES index refresh interval is 30 seconds, which may feel like a long time on local development. Feel free to adjust it or [poke Elasticsearch yourself!](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-refresh.html) Take note, however, that the specified ES index refresh interval is 30 seconds, which may feel like a long time on local development. Feel free to adjust it or [poke Elasticsearch yourself!](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-refresh.html)
## License
This project is licensed under the GNU General Public License v3.0 (GPL-3.0). See the [LICENSE](LICENSE) file for more details.
## Disclaimer
> [!CAUTION]
> **This project was created as a learning experience, and while it’s a torrent tracker, I can’t control how people choose to use it.**
By using this software, you're agreeing to a few things:
- I'm not responsible for any legal issues that might come up from using this tracker, especially if it's used to share copyrighted content without permission.
- It's your responsibility to make sure you're following the laws in your area when using this software.
**Please use this project wisely and stay on the right side of the law.** Happy coding!

9
WSGI.py
View file

@ -1,11 +1,16 @@
#!/usr/bin/python3 #!/usr/bin/env python3
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
"""
WSGI entry point for the Nyaa application.
Compatible with Python 3.13.
"""
import gevent.monkey import gevent.monkey
gevent.monkey.patch_all() gevent.monkey.patch_all()
from nyaa import create_app from nyaa import create_app
from flask import Flask
app = create_app('config') app: Flask = create_app('config')
if app.config['DEBUG']: if app.config['DEBUG']:
from werkzeug.debug import DebuggedApplication from werkzeug.debug import DebuggedApplication

8
config.example.py
View file

@ -42,6 +42,12 @@ EXTERNAL_URLS = {'fap':'***', 'main':'***'}
CSRF_SESSION_KEY = '***' CSRF_SESSION_KEY = '***'
SECRET_KEY = '***' SECRET_KEY = '***'
# Session cookie configuration
SESSION_COOKIE_NAME = 'nyaav3_session'
SESSION_COOKIE_SECURE = True
SESSION_COOKIE_HTTPONLY = True
SESSION_COOKIE_SAMESITE = 'Lax'
# Present a recaptcha for anonymous uploaders # Present a recaptcha for anonymous uploaders
USE_RECAPTCHA = False USE_RECAPTCHA = False
# Require email validation # Require email validation
@ -82,7 +88,7 @@ RECAPTCHA_PRIVATE_KEY = '***'
BASE_DIR = os.path.abspath(os.path.dirname(__file__)) BASE_DIR = os.path.abspath(os.path.dirname(__file__))
if USE_MYSQL: if USE_MYSQL:
SQLALCHEMY_DATABASE_URI = ('mysql://test:test123@localhost/nyaav2?charset=utf8mb4') SQLALCHEMY_DATABASE_URI = ('mysql://test:test123@localhost/nyaav3?charset=utf8mb4')
else: else:
SQLALCHEMY_DATABASE_URI = ( SQLALCHEMY_DATABASE_URI = (
'sqlite:///' + os.path.join(BASE_DIR, 'test.db') + '?check_same_thread=False') 'sqlite:///' + os.path.join(BASE_DIR, 'test.db') + '?check_same_thread=False')

34
db_create.py
View file

@ -1,12 +1,19 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""
Database creation script for Nyaa.
Compatible with Python 3.13 and SQLAlchemy 2.0.
"""
from typing import List, Tuple, Type
import sqlalchemy import sqlalchemy
from sqlalchemy import select
from nyaa import create_app, models from nyaa import create_app, models
from nyaa.extensions import db from nyaa.extensions import db
app = create_app('config') app = create_app('config')
NYAA_CATEGORIES = [ NYAA_CATEGORIES: List[Tuple[str, List[str]]] = [
('Anime', ['Anime Music Video', 'English-translated', 'Non-English-translated', 'Raw']), ('Anime', ['Anime Music Video', 'English-translated', 'Non-English-translated', 'Raw']),
('Audio', ['Lossless', 'Lossy']), ('Audio', ['Lossless', 'Lossy']),
('Literature', ['English-translated', 'Non-English-translated', 'Raw']), ('Literature', ['English-translated', 'Non-English-translated', 'Raw']),
@ -16,13 +23,23 @@ NYAA_CATEGORIES = [
] ]
SUKEBEI_CATEGORIES = [ SUKEBEI_CATEGORIES: List[Tuple[str, List[str]]] = [
('Art', ['Anime', 'Doujinshi', 'Games', 'Manga', 'Pictures']), ('Art', ['Anime', 'Doujinshi', 'Games', 'Manga', 'Pictures']),
('Real Life', ['Photobooks / Pictures', 'Videos']), ('Real Life', ['Photobooks / Pictures', 'Videos']),
] ]
def add_categories(categories, main_class, sub_class): def add_categories(categories: List[Tuple[str, List[str]]],
main_class: Type[models.MainCategoryBase],
sub_class: Type[models.SubCategoryBase]) -> None:
"""
Add categories to the database.
Args:
categories: List of tuples containing main category name and list of subcategory names
main_class: Main category model class
sub_class: Subcategory model class
"""
for main_cat_name, sub_cat_names in categories: for main_cat_name, sub_cat_names in categories:
main_cat = main_class(name=main_cat_name) main_cat = main_class(name=main_cat_name)
for i, sub_cat_name in enumerate(sub_cat_names): for i, sub_cat_name in enumerate(sub_cat_names):
@ -36,19 +53,24 @@ if __name__ == '__main__':
# Test for the user table, assume db is empty if it's not created # Test for the user table, assume db is empty if it's not created
database_empty = False database_empty = False
try: try:
models.User.query.first() stmt = select(models.User).limit(1)
db.session.execute(stmt).scalar_one_or_none()
except (sqlalchemy.exc.ProgrammingError, sqlalchemy.exc.OperationalError): except (sqlalchemy.exc.ProgrammingError, sqlalchemy.exc.OperationalError):
database_empty = True database_empty = True
print('Creating all tables...') print('Creating all tables...')
db.create_all() db.create_all()
nyaa_category_test = models.NyaaMainCategory.query.first() # Check if Nyaa categories exist
stmt = select(models.NyaaMainCategory).limit(1)
nyaa_category_test = db.session.execute(stmt).scalar_one_or_none()
if not nyaa_category_test: if not nyaa_category_test:
print('Adding Nyaa categories...') print('Adding Nyaa categories...')
add_categories(NYAA_CATEGORIES, models.NyaaMainCategory, models.NyaaSubCategory) add_categories(NYAA_CATEGORIES, models.NyaaMainCategory, models.NyaaSubCategory)
sukebei_category_test = models.SukebeiMainCategory.query.first() # Check if Sukebei categories exist
stmt = select(models.SukebeiMainCategory).limit(1)
sukebei_category_test = db.session.execute(stmt).scalar_one_or_none()
if not sukebei_category_test: if not sukebei_category_test:
print('Adding Sukebei categories...') print('Adding Sukebei categories...')
add_categories(SUKEBEI_CATEGORIES, models.SukebeiMainCategory, models.SukebeiSubCategory) add_categories(SUKEBEI_CATEGORIES, models.SukebeiMainCategory, models.SukebeiSubCategory)

23
db_migrate.py
View file

@ -1,9 +1,14 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
"""
Database migration script for Nyaa.
Compatible with Python 3.13 and Flask-Migrate 4.0.
"""
import sys import sys
from typing import List
from flask_script import Manager from flask_migrate import Migrate
from flask_migrate import Migrate, MigrateCommand from flask.cli import FlaskGroup
from nyaa import create_app from nyaa import create_app
from nyaa.extensions import db from nyaa.extensions import db
@ -11,11 +16,17 @@ from nyaa.extensions import db
app = create_app('config') app = create_app('config')
migrate = Migrate(app, db) migrate = Migrate(app, db)
manager = Manager(app) def create_cli_app():
manager.add_command("db", MigrateCommand) return app
cli = FlaskGroup(create_app=create_cli_app)
if __name__ == "__main__": if __name__ == "__main__":
# Patch sys.argv to default to 'db' # Patch sys.argv to default to 'db'
sys.argv.insert(1, 'db') if len(sys.argv) > 1 and sys.argv[1] not in ['--help', '-h']:
if sys.argv[1] not in ['db', 'routes', 'shell', 'run']:
args: List[str] = sys.argv.copy()
args.insert(1, 'db')
sys.argv = args
manager.run() cli()

13
dev.py
View file

@ -4,8 +4,11 @@
This tool is designed to assist developers run common tasks, such as This tool is designed to assist developers run common tasks, such as
checking the code for lint issues, auto fixing some lint issues and running tests. checking the code for lint issues, auto fixing some lint issues and running tests.
It imports modules lazily (as-needed basis), so it runs faster! It imports modules lazily (as-needed basis), so it runs faster!
Compatible with Python 3.13.
""" """
import sys import sys
from typing import List, Optional, Generator, Any, Union
LINT_PATHS = [ LINT_PATHS = [
'nyaa/', 'nyaa/',
@ -14,14 +17,14 @@ LINT_PATHS = [
TEST_PATHS = ['tests'] TEST_PATHS = ['tests']
def print_cmd(cmd, args): def print_cmd(cmd: str, args: List[str]) -> None:
""" Prints the command and args as you would run them manually. """ """ Prints the command and args as you would run them manually. """
print('Running: {0}\n'.format( print('Running: {0}\n'.format(
' '.join([('\'' + a + '\'' if ' ' in a else a) for a in [cmd] + args]))) ' '.join([('\'' + a + '\'' if ' ' in a else a) for a in [cmd] + args])))
sys.stdout.flush() # Make sure stdout is flushed before continuing. sys.stdout.flush() # Make sure stdout is flushed before continuing.
def check_config_values(): def check_config_values() -> bool:
""" Verify that all max_line_length values match. """ """ Verify that all max_line_length values match. """
import configparser import configparser
config = configparser.ConfigParser() config = configparser.ConfigParser()
@ -32,7 +35,7 @@ def check_config_values():
autopep8 = config.get('pycodestyle', 'max_line_length', fallback=None) autopep8 = config.get('pycodestyle', 'max_line_length', fallback=None)
isort = config.get('isort', 'line_length', fallback=None) isort = config.get('isort', 'line_length', fallback=None)
values = (v for v in (flake8, autopep8, isort) if v is not None) values: Generator[str, None, None] = (v for v in (flake8, autopep8, isort) if v is not None)
found = next(values, False) found = next(values, False)
if not found: if not found:
print('Warning: No max line length setting set in setup.cfg.') print('Warning: No max line length setting set in setup.cfg.')
@ -44,7 +47,7 @@ def check_config_values():
return True return True
def print_help(): def print_help() -> int:
print('Nyaa Development Helper') print('Nyaa Development Helper')
print('=======================\n') print('=======================\n')
print('Usage: {0} command [different arguments]'.format(sys.argv[0])) print('Usage: {0} command [different arguments]'.format(sys.argv[0]))
@ -62,7 +65,7 @@ def print_help():
if __name__ == '__main__': if __name__ == '__main__':
assert sys.version_info >= (3, 6), "Python 3.6 is required" assert sys.version_info >= (3, 13), "Python 3.13 is required"
check_config_values() check_config_values()

2
es_sync_config.example.json
View file

@ -4,7 +4,7 @@
"mysql_port": 3306, "mysql_port": 3306,
"mysql_user": "nyaa", "mysql_user": "nyaa",
"mysql_password": "some_password", "mysql_password": "some_password",
"database": "nyaav2", "database": "nyaav3",
"internal_queue_depth": 10000, "internal_queue_depth": 10000,
"es_chunk_size": 10000, "es_chunk_size": 10000,
"flush_interval": 5 "flush_interval": 5

2
import_to_es.py
View file

@ -1,6 +1,6 @@
#!/usr/bin/env python #!/usr/bin/env python
""" """
Bulk load torents from mysql into elasticsearch `nyaav2` index, Bulk load torents from mysql into elasticsearch `nyaav3` index,
which is assumed to already exist. which is assumed to already exist.
This is a one-shot deal, so you'd either need to complement it This is a one-shot deal, so you'd either need to complement it
with a cron job or some binlog-reading thing (TODO) with a cron job or some binlog-reading thing (TODO)

3
migrations/README
View file

@ -1 +1,4 @@
> [!WARNING]
> No longer supported in NyaaV3.
Generic single-database configuration. Generic single-database configuration.

43
nyaa/__init__.py
View file

@ -1,8 +1,10 @@
import logging import logging
import os import os
import string import string
from typing import Any, Optional
import flask import flask
from flask import Flask
from flask_assets import Bundle # noqa F401 from flask_assets import Bundle # noqa F401
from nyaa.api_handler import api_blueprint from nyaa.api_handler import api_blueprint
@ -18,11 +20,17 @@ from nyaa.views import register_views
flask.url_for = caching_url_for flask.url_for = caching_url_for
def create_app(config): def create_app(config: Any) -> Flask:
""" Nyaa app factory """ """ Nyaa app factory """
app = flask.Flask(__name__) app = flask.Flask(__name__)
app.config.from_object(config) app.config.from_object(config)
# Session cookie configuration
app.config['SESSION_COOKIE_NAME'] = 'nyaav3_session'
app.config['SESSION_COOKIE_SECURE'] = True
app.config['SESSION_COOKIE_HTTPONLY'] = True
app.config['SESSION_COOKIE_SAMESITE'] = 'Lax'
# Don't refresh cookie each request # Don't refresh cookie each request
app.config['SESSION_REFRESH_EACH_REQUEST'] = False app.config['SESSION_REFRESH_EACH_REQUEST'] = False
@ -34,24 +42,24 @@ def create_app(config):
# Forbid caching # Forbid caching
@app.after_request @app.after_request
def forbid_cache(request): def forbid_cache(response: flask.Response) -> flask.Response:
request.headers['Cache-Control'] = 'no-cache, no-store, must-revalidate, max-age=0' response.headers['Cache-Control'] = 'no-cache, no-store, must-revalidate, max-age=0'
request.headers['Pragma'] = 'no-cache' response.headers['Pragma'] = 'no-cache'
request.headers['Expires'] = '0' response.headers['Expires'] = '0'
return request return response
# Add a timer header to the requests when debugging # Add a timer header to the requests when debugging
# This gives us a simple way to benchmark requests off-app # This gives us a simple way to benchmark requests off-app
import time import time
@app.before_request @app.before_request
def timer_before_request(): def timer_before_request() -> None:
flask.g.request_start_time = time.time() flask.g.request_start_time = time.time()
@app.after_request @app.after_request
def timer_after_request(request): def timer_after_request(response: flask.Response) -> flask.Response:
request.headers['X-Timer'] = time.time() - flask.g.request_start_time response.headers['X-Timer'] = str(time.time() - flask.g.request_start_time)
return request return response
else: else:
app.logger.setLevel(logging.WARNING) app.logger.setLevel(logging.WARNING)
@ -63,17 +71,17 @@ def create_app(config):
app.config['LOG_FILE'], maxBytes=10000, backupCount=1) app.config['LOG_FILE'], maxBytes=10000, backupCount=1)
app.logger.addHandler(app.log_handler) app.logger.addHandler(app.log_handler)
# Log errors and display a message to the user in production mdode # Log errors and display a message to the user in production mode
if not app.config['DEBUG']: if not app.config['DEBUG']:
@app.errorhandler(500) @app.errorhandler(500)
def internal_error(exception): def internal_error(exception: Exception) -> flask.Response:
random_id = random_string(8, string.ascii_uppercase + string.digits) random_id = random_string(8, string.ascii_uppercase + string.digits)
# Pst. Not actually unique, but don't tell anyone! # Pst. Not actually unique, but don't tell anyone!
app.logger.error('Exception occurred! Unique ID: %s', random_id, exc_info=exception) app.logger.error(f'Exception occurred! Unique ID: {random_id}', exc_info=exception)
markup_source = ' '.join([ markup_source = ' '.join([
'<strong>An error occurred!</strong>', '<strong>An error occurred!</strong>',
'Debug information has been logged.', 'Debug information has been logged.',
'Please pass along this ID: <kbd>{}</kbd>'.format(random_id) f'Please pass along this ID: <kbd>{random_id}</kbd>'
]) ])
flask.flash(flask.Markup(markup_source), 'danger') flask.flash(flask.Markup(markup_source), 'danger')
@ -101,10 +109,15 @@ def create_app(config):
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
app.config['MYSQL_DATABASE_CHARSET'] = 'utf8mb4' app.config['MYSQL_DATABASE_CHARSET'] = 'utf8mb4'
db.init_app(app) db.init_app(app)
# Import the fixed Ban.banned method
with app.app_context():
import nyaa.fixed_ban
# Assets # Assets
assets.init_app(app) assets.init_app(app)
assets._named_bundles = {} # Hack to fix state carrying over in tests if hasattr(assets, '_named_bundles'):
assets._named_bundles = {} # Hack to fix state carrying over in tests
main_js = Bundle('js/main.js', filters='rjsmin', output='js/main.min.js') main_js = Bundle('js/main.js', filters='rjsmin', output='js/main.min.js')
bs_js = Bundle('js/bootstrap-select.js', filters='rjsmin', bs_js = Bundle('js/bootstrap-select.js', filters='rjsmin',
output='js/bootstrap-select.min.js') output='js/bootstrap-select.min.js')

10
nyaa/backend.py
View file

@ -5,10 +5,10 @@ from datetime import datetime, timedelta
from ipaddress import ip_address from ipaddress import ip_address
import flask import flask
from werkzeug import secure_filename from werkzeug.utils import secure_filename
import sqlalchemy import sqlalchemy
from orderedset import OrderedSet from orderly_set import OrderlySet
from nyaa import models, utils from nyaa import models, utils
from nyaa.extensions import db from nyaa.extensions import db
@ -304,7 +304,7 @@ def handle_torrent_upload(upload_form, uploading_user=None, fromAPI=False):
db.session.flush() db.session.flush()
# Store the users trackers # Store the users trackers
trackers = OrderedSet() trackers = OrderlySet()
announce = torrent_data.torrent_dict.get('announce', b'').decode('ascii') announce = torrent_data.torrent_dict.get('announce', b'').decode('ascii')
if announce: if announce:
trackers.add(announce) trackers.add(announce)
@ -319,12 +319,12 @@ def handle_torrent_upload(upload_form, uploading_user=None, fromAPI=False):
webseed_list = torrent_data.torrent_dict.get('url-list') or [] webseed_list = torrent_data.torrent_dict.get('url-list') or []
if isinstance(webseed_list, bytes): if isinstance(webseed_list, bytes):
webseed_list = [webseed_list] # qB doesn't contain a sole url in a list webseed_list = [webseed_list] # qB doesn't contain a sole url in a list
webseeds = OrderedSet(webseed.decode('utf-8') for webseed in webseed_list) webseeds = OrderlySet(webseed.decode('utf-8') for webseed in webseed_list)
# Remove our trackers, maybe? TODO ? # Remove our trackers, maybe? TODO ?
# Search for/Add trackers in DB # Search for/Add trackers in DB
db_trackers = OrderedSet() db_trackers = OrderlySet()
for announce in trackers: for announce in trackers:
tracker = models.Trackers.by_uri(announce) tracker = models.Trackers.by_uri(announce)

99
nyaa/custom_pagination.py Normal file
View file

@ -0,0 +1,99 @@
from typing import Any, List, Optional, Sequence, TypeVar, Union
T = TypeVar('T')
class CustomPagination:
"""
A custom pagination class that mimics the interface of Flask-SQLAlchemy's Pagination
but doesn't rely on the _query_items method.
"""
def __init__(self, query: Any, page: int, per_page: int, total: int, items: List[T]):
"""
Initialize a new CustomPagination object.
Args:
query: The query object (not used, but kept for compatibility)
page: The current page number (1-indexed)
per_page: The number of items per page
total: The total number of items
items: The items on the current page
"""
self.query = query
self.page = page
self.per_page = per_page
self.total = total
self.items = items
# For compatibility with LimitedPagination
self.actual_count = total
@property
def has_prev(self) -> bool:
"""Return True if there is a previous page."""
return self.page > 1
@property
def has_next(self) -> bool:
"""Return True if there is a next page."""
return self.page < self.pages
@property
def pages(self) -> int:
"""The total number of pages."""
if self.per_page == 0 or self.total == 0:
return 0
return max(1, (self.total + self.per_page - 1) // self.per_page)
@property
def prev_num(self) -> Optional[int]:
"""The previous page number, or None if this is the first page."""
if self.has_prev:
return self.page - 1
return None
@property
def next_num(self) -> Optional[int]:
"""The next page number, or None if this is the last page."""
if self.has_next:
return self.page + 1
return None
@property
def first(self) -> int:
"""The number of the first item on the page, starting from 1, or 0 if there are no items."""
if not self.items:
return 0
return (self.page - 1) * self.per_page + 1
@property
def last(self) -> int:
"""The number of the last item on the page, starting from 1, inclusive, or 0 if there are no items."""
if not self.items:
return 0
return min(self.total, self.page * self.per_page)
def iter_pages(self, left_edge: int = 2, left_current: int = 2,
right_current: int = 5, right_edge: int = 2) -> Sequence[Optional[int]]:
"""
Yield page numbers for a pagination widget.
Skipped pages between the edges and middle are represented by a None.
"""
last = 0
for num in range(1, self.pages + 1):
if (num <= left_edge or
(num > self.page - left_current - 1 and num < self.page + right_current) or
num > self.pages - right_edge):
if last + 1 != num:
yield None
yield num
last = num
def __iter__(self):
"""Iterate over the items on the current page."""
return iter(self.items)
def __len__(self):
"""Return the number of items on the current page."""
return len(self.items)

67
nyaa/email.py
View file

@ -45,29 +45,58 @@ class EmailHolder(object):
def send_email(email_holder): def send_email(email_holder):
"""Send an email using the configured mail backend."""
mail_backend = app.config.get('MAIL_BACKEND') mail_backend = app.config.get('MAIL_BACKEND')
if mail_backend == 'mailgun':
_send_mailgun(email_holder) if not mail_backend:
elif mail_backend == 'smtp': app.logger.warning('No mail backend configured, skipping email send')
_send_smtp(email_holder) return False
elif mail_backend:
# TODO: Do this in logging.error when we have that set up try:
print('Unknown mail backend:', mail_backend) if mail_backend == 'mailgun':
success = _send_mailgun(email_holder)
elif mail_backend == 'smtp':
success = _send_smtp(email_holder)
else:
app.logger.error(f'Unknown mail backend: {mail_backend}')
return False
if not success:
app.logger.error(f'Failed to send email using {mail_backend} backend')
return False
app.logger.info(f'Email successfully sent using {mail_backend} backend')
return True
except Exception as e:
app.logger.error(f'Error sending email: {str(e)}')
return False
def _send_mailgun(email_holder): def _send_mailgun(email_holder):
mailgun_endpoint = app.config['MAILGUN_API_BASE'] + '/messages' """Send an email using Mailgun API with proper error handling."""
auth = ('api', app.config['MAILGUN_API_KEY']) try:
data = { mailgun_endpoint = app.config['MAILGUN_API_BASE'] + '/messages'
'from': app.config['MAIL_FROM_ADDRESS'], auth = ('api', app.config['MAILGUN_API_KEY'])
'to': email_holder.format_recipient(), data = {
'subject': email_holder.subject, 'from': app.config['MAIL_FROM_ADDRESS'],
'text': email_holder.text, 'to': email_holder.format_recipient(),
'html': email_holder.html 'subject': email_holder.subject,
} 'text': email_holder.text,
r = requests.post(mailgun_endpoint, data=data, auth=auth) 'html': email_holder.html
# TODO real error handling? }
assert r.status_code == 200
r = requests.post(mailgun_endpoint, data=data, auth=auth)
if r.status_code != 200:
app.logger.error(f'Mailgun API error: {r.status_code} - {r.text}')
return False
return True
except Exception as e:
app.logger.error(f'Error sending email via Mailgun: {str(e)}')
return False
def _send_smtp(email_holder): def _send_smtp(email_holder):

48
nyaa/extensions.py
View file

@ -1,4 +1,5 @@
import os.path import os.path
from typing import Any, Optional, Sequence, TypeVar, Union
from flask import abort from flask import abort
from flask.config import Config from flask.config import Config
@ -7,7 +8,9 @@ from flask_caching import Cache
from flask_debugtoolbar import DebugToolbarExtension from flask_debugtoolbar import DebugToolbarExtension
from flask_limiter import Limiter from flask_limiter import Limiter
from flask_limiter.util import get_remote_address from flask_limiter.util import get_remote_address
from flask_sqlalchemy import BaseQuery, Pagination, SQLAlchemy from flask_sqlalchemy import SQLAlchemy
from flask_sqlalchemy.pagination import Pagination
from sqlalchemy.orm import Query
assets = Environment() assets = Environment()
db = SQLAlchemy() db = SQLAlchemy()
@ -15,16 +18,28 @@ toolbar = DebugToolbarExtension()
cache = Cache() cache = Cache()
limiter = Limiter(key_func=get_remote_address) limiter = Limiter(key_func=get_remote_address)
# Type variable for query results
T = TypeVar('T')
class LimitedPagination(Pagination): class LimitedPagination(Pagination):
def __init__(self, actual_count, *args, **kwargs): def __init__(self, actual_count: int, *args: Any, **kwargs: Any) -> None:
self.actual_count = actual_count self.actual_count = actual_count
super().__init__(*args, **kwargs) super().__init__(*args, **kwargs)
def fix_paginate(): def fix_paginate() -> None:
"""Add custom pagination method to SQLAlchemy Query."""
def paginate_faste(self, page=1, per_page=50, max_page=None, step=5, count_query=None):
def paginate_faste(
self: Query[T],
page: int = 1,
per_page: int = 50,
max_page: Optional[int] = None,
step: int = 5,
count_query: Optional[Query[int]] = None
) -> LimitedPagination:
"""Custom pagination that supports max_page and count_query."""
if page < 1: if page < 1:
abort(404) abort(404)
@ -36,6 +51,10 @@ def fix_paginate():
total_query_count = count_query.scalar() total_query_count = count_query.scalar()
else: else:
total_query_count = self.count() total_query_count = self.count()
if total_query_count is None:
total_query_count = 0
actual_query_count = total_query_count actual_query_count = total_query_count
if max_page: if max_page:
total_query_count = min(total_query_count, max_page * per_page) total_query_count = min(total_query_count, max_page * per_page)
@ -49,17 +68,20 @@ def fix_paginate():
return LimitedPagination(actual_query_count, self, page, per_page, total_query_count, return LimitedPagination(actual_query_count, self, page, per_page, total_query_count,
items) items)
BaseQuery.paginate_faste = paginate_faste # Monkey patch the Query class
setattr(Query, 'paginate_faste', paginate_faste)
def _get_config(): def _get_config() -> Config:
# Workaround to get an available config object before the app is initiallized """
# Only needed/used in top-level and class statements Workaround to get an available config object before the app is initialized.
# https://stackoverflow.com/a/18138250/7597273 Only needed/used in top-level and class statements.
https://stackoverflow.com/a/18138250/7597273
"""
root_path = os.path.abspath(os.path.join(os.path.dirname(__file__), '..')) root_path = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
config = Config(root_path) config_obj = Config(root_path)
config.from_object('config') config_obj.from_object('config')
return config return config_obj
config = _get_config() config = _get_config()

26
nyaa/fixed_ban.py Normal file
View file

@ -0,0 +1,26 @@
from typing import Optional, Union
from sqlalchemy import or_, select
from nyaa.extensions import db
from nyaa.models import Ban
# Fix the banned method to return a query object instead of a list
@classmethod
def fixed_banned(cls, user_id: Optional[int], user_ip: Optional[bytes]):
"""Check if a user or IP is banned.
Returns a query object that can be further filtered or used with .first(), .all(), etc.
"""
if not user_id and not user_ip:
# Return an empty query that will return no results
return db.session.query(cls).filter(cls.id < 0)
clauses = []
if user_id:
clauses.append(cls.user_id == user_id)
if user_ip:
clauses.append(cls.user_ip == user_ip)
return db.session.query(cls).filter(or_(*clauses))
# Replace the original method with our fixed version
Ban.banned = fixed_banned

7
nyaa/forms.py
View file

@ -11,7 +11,8 @@ from wtforms import (BooleanField, HiddenField, PasswordField, SelectField, Stri
SubmitField, TextAreaField) SubmitField, TextAreaField)
from wtforms.validators import (DataRequired, Email, EqualTo, Length, Optional, Regexp, from wtforms.validators import (DataRequired, Email, EqualTo, Length, Optional, Regexp,
StopValidation, ValidationError) StopValidation, ValidationError)
from wtforms.widgets import HTMLString # For DisabledSelectField # from wtforms.widgets import HTMLString # For DisabledSelectField
from markupsafe import Markup
from wtforms.widgets import Select as SelectWidget # For DisabledSelectField from wtforms.widgets import Select as SelectWidget # For DisabledSelectField
from wtforms.widgets import html_params from wtforms.widgets import html_params
@ -223,7 +224,7 @@ class DisabledSelectWidget(SelectWidget):
extra = disabled and {'disabled': ''} or {} extra = disabled and {'disabled': ''} or {}
html.append(self.render_option(val, label, selected, **extra)) html.append(self.render_option(val, label, selected, **extra))
html.append('</select>') html.append('</select>')
return HTMLString(''.join(html)) return Markup(''.join(html))
class DisabledSelectField(SelectField): class DisabledSelectField(SelectField):
@ -265,7 +266,7 @@ class InlineButtonWidget(object):
kwargs.setdefault('type', self.input_type) kwargs.setdefault('type', self.input_type)
if not label: if not label:
label = field.label.text label = field.label.text
return HTMLString('<button %s>' % self.html_params(name=field.name, **kwargs) + label) return Markup('<button %s>' % self.html_params(name=field.name, **kwargs) + label)
class StringSubmitField(StringField): class StringSubmitField(StringField):

204
nyaa/models.py
View file

@ -5,13 +5,14 @@ from datetime import datetime
from enum import Enum, IntEnum from enum import Enum, IntEnum
from hashlib import md5 from hashlib import md5
from ipaddress import ip_address from ipaddress import ip_address
from typing import Any, Dict, List, Optional, Tuple, Union
from urllib.parse import unquote as unquote_url from urllib.parse import unquote as unquote_url
from urllib.parse import urlencode from urllib.parse import urlencode
import flask import flask
from markupsafe import escape as escape_markup from markupsafe import escape as escape_markup
from sqlalchemy import ForeignKeyConstraint, Index, func from sqlalchemy import ForeignKeyConstraint, Index, func, select
from sqlalchemy.ext import declarative from sqlalchemy.ext import declarative
from sqlalchemy.ext.hybrid import hybrid_property from sqlalchemy.ext.hybrid import hybrid_property
from sqlalchemy_fulltext import FullText from sqlalchemy_fulltext import FullText
@ -31,7 +32,7 @@ if config['USE_MYSQL']:
COL_UTF8MB4_BIN = 'utf8mb4_bin' COL_UTF8MB4_BIN = 'utf8mb4_bin'
COL_ASCII_GENERAL_CI = 'ascii_general_ci' COL_ASCII_GENERAL_CI = 'ascii_general_ci'
else: else:
BinaryType = db.Binary BinaryType = db.LargeBinary
TextType = db.String TextType = db.String
MediumBlobType = db.BLOB MediumBlobType = db.BLOB
COL_UTF8_GENERAL_CI = 'NOCASE' COL_UTF8_GENERAL_CI = 'NOCASE'
@ -48,23 +49,29 @@ class DeclarativeHelperBase(object):
__tablename__ and providing class methods for renaming references. ''' __tablename__ and providing class methods for renaming references. '''
# See http://docs.sqlalchemy.org/en/latest/orm/extensions/declarative/api.html # See http://docs.sqlalchemy.org/en/latest/orm/extensions/declarative/api.html
__tablename_base__ = None __tablename_base__: Optional[str] = None
__flavor__ = None __flavor__: Optional[str] = None
@classmethod @classmethod
def _table_prefix_string(cls): def _table_prefix_string(cls) -> str:
if cls.__flavor__ is None:
raise ValueError("__flavor__ must be set")
return cls.__flavor__.lower() + '_' return cls.__flavor__.lower() + '_'
@classmethod @classmethod
def _table_prefix(cls, table_name): def _table_prefix(cls, table_name: str) -> str:
return cls._table_prefix_string() + table_name return cls._table_prefix_string() + table_name
@classmethod @classmethod
def _flavor_prefix(cls, table_name): def _flavor_prefix(cls, table_name: str) -> str:
if cls.__flavor__ is None:
raise ValueError("__flavor__ must be set")
return cls.__flavor__ + table_name return cls.__flavor__ + table_name
@declarative.declared_attr @declarative.declared_attr
def __tablename__(cls): def __tablename__(cls) -> str:
if cls.__tablename_base__ is None:
raise ValueError("__tablename_base__ must be set")
return cls._table_prefix(cls.__tablename_base__) return cls._table_prefix(cls.__tablename_base__)
@ -72,22 +79,22 @@ class FlagProperty(object):
''' This class will act as a wrapper between the given flag and the class's ''' This class will act as a wrapper between the given flag and the class's
flag collection. ''' flag collection. '''
def __init__(self, flag, flags_attr='flags'): def __init__(self, flag: int, flags_attr: str = 'flags'):
self._flag = flag self._flag = flag
self._flags_attr_name = flags_attr self._flags_attr_name = flags_attr
def _get_flags(self, instance): def _get_flags(self, instance: Any) -> int:
return getattr(instance, self._flags_attr_name) return getattr(instance, self._flags_attr_name)
def _set_flags(self, instance, value): def _set_flags(self, instance: Any, value: int) -> None:
return setattr(instance, self._flags_attr_name, value) return setattr(instance, self._flags_attr_name, value)
def __get__(self, instance, owner_class): def __get__(self, instance: Any, owner_class: Any) -> bool:
if instance is None: if instance is None:
raise AttributeError() raise AttributeError()
return bool(self._get_flags(instance) & self._flag) return bool(self._get_flags(instance) & self._flag)
def __set__(self, instance, value): def __set__(self, instance: Any, value: bool) -> None:
new_flags = (self._get_flags(instance) & ~self._flag) | (bool(value) and self._flag) new_flags = (self._get_flags(instance) & ~self._flag) | (bool(value) and self._flag)
self._set_flags(instance, new_flags) self._set_flags(instance, new_flags)
@ -124,7 +131,7 @@ class TorrentBase(DeclarativeHelperBase):
# Even though this is same for both tables, declarative requires this # Even though this is same for both tables, declarative requires this
return db.Column(db.Integer, db.ForeignKey('users.id'), nullable=True) return db.Column(db.Integer, db.ForeignKey('users.id'), nullable=True)
uploader_ip = db.Column(db.Binary(length=16), default=None, nullable=True) uploader_ip = db.Column(db.LargeBinary(length=16), default=None, nullable=True)
has_torrent = db.Column(db.Boolean, nullable=False, default=False) has_torrent = db.Column(db.Boolean, nullable=False, default=False)
comment_count = db.Column(db.Integer, default=0, nullable=False, index=True) comment_count = db.Column(db.Integer, default=0, nullable=False, index=True)
@ -198,15 +205,22 @@ class TorrentBase(DeclarativeHelperBase):
def __repr__(self): def __repr__(self):
return '<{0} #{1.id} \'{1.display_name}\' {1.filesize}b>'.format(type(self).__name__, self) return '<{0} #{1.id} \'{1.display_name}\' {1.filesize}b>'.format(type(self).__name__, self)
def update_comment_count(self): def update_comment_count(self) -> int:
self.comment_count = db.session.query(func.count( """Update the comment count for this torrent and return the new count."""
Comment.id)).filter_by(torrent_id=self.id).first()[0] stmt = select(func.count(Comment.id)).filter_by(torrent_id=self.id)
result = db.session.execute(stmt).scalar_one_or_none() or 0
self.comment_count = result
return self.comment_count return self.comment_count
@classmethod @classmethod
def update_comment_count_db(cls, torrent_id): def update_comment_count_db(cls, torrent_id: int) -> None:
cls.query.filter_by(id=torrent_id).update({'comment_count': db.session.query( """Update the comment count in the database for the given torrent ID."""
func.count(Comment.id)).filter_by(torrent_id=torrent_id).as_scalar()}, False) stmt = select(func.count(Comment.id)).filter_by(torrent_id=torrent_id)
count = db.session.execute(stmt).scalar_one_or_none() or 0
# Use the new update() style
stmt = db.update(cls).filter_by(id=torrent_id).values(comment_count=count)
db.session.execute(stmt)
@property @property
def created_utc_timestamp(self): def created_utc_timestamp(self):
@ -272,15 +286,20 @@ class TorrentBase(DeclarativeHelperBase):
# Class methods # Class methods
@classmethod @classmethod
def by_id(cls, id): def by_id(cls, id: int) -> Optional['TorrentBase']:
return cls.query.get(id) """Get a torrent by its ID."""
stmt = select(cls).filter_by(id=id)
return db.session.execute(stmt).scalar_one_or_none()
@classmethod @classmethod
def by_info_hash(cls, info_hash): def by_info_hash(cls, info_hash: bytes) -> Optional['TorrentBase']:
return cls.query.filter_by(info_hash=info_hash).first() """Get a torrent by its info hash."""
stmt = select(cls).filter_by(info_hash=info_hash)
return db.session.execute(stmt).scalar_one_or_none()
@classmethod @classmethod
def by_info_hash_hex(cls, info_hash_hex): def by_info_hash_hex(cls, info_hash_hex: str) -> Optional['TorrentBase']:
"""Get a torrent by its hex-encoded info hash."""
info_hash_bytes = bytearray.fromhex(info_hash_hex) info_hash_bytes = bytearray.fromhex(info_hash_hex)
return cls.by_info_hash(info_hash_bytes) return cls.by_info_hash(info_hash_bytes)
@ -332,8 +351,10 @@ class Trackers(db.Model):
disabled = db.Column(db.Boolean, nullable=False, default=False) disabled = db.Column(db.Boolean, nullable=False, default=False)
@classmethod @classmethod
def by_uri(cls, uri): def by_uri(cls, uri: str) -> Optional['Trackers']:
return cls.query.filter_by(uri=uri).first() """Get a tracker by its URI."""
stmt = select(cls).filter_by(uri=uri)
return db.session.execute(stmt).scalar_one_or_none()
class TorrentTrackersBase(DeclarativeHelperBase): class TorrentTrackersBase(DeclarativeHelperBase):
@ -356,8 +377,10 @@ class TorrentTrackersBase(DeclarativeHelperBase):
return db.relationship('Trackers', uselist=False, lazy='joined') return db.relationship('Trackers', uselist=False, lazy='joined')
@classmethod @classmethod
def by_torrent_id(cls, torrent_id): def by_torrent_id(cls, torrent_id: int) -> List['TorrentTrackersBase']:
return cls.query.filter_by(torrent_id=torrent_id).order_by(cls.order.desc()) """Get all trackers for a torrent, ordered by their order field."""
stmt = select(cls).filter_by(torrent_id=torrent_id).order_by(cls.order.desc())
return db.session.execute(stmt).scalars().all()
class MainCategoryBase(DeclarativeHelperBase): class MainCategoryBase(DeclarativeHelperBase):
@ -382,8 +405,10 @@ class MainCategoryBase(DeclarativeHelperBase):
return '_'.join(str(x) for x in self.get_category_ids()) return '_'.join(str(x) for x in self.get_category_ids())
@classmethod @classmethod
def by_id(cls, id): def by_id(cls, id: int) -> Optional['MainCategoryBase']:
return cls.query.get(id) """Get a main category by its ID."""
stmt = select(cls).filter_by(id=id)
return db.session.execute(stmt).scalar_one_or_none()
class SubCategoryBase(DeclarativeHelperBase): class SubCategoryBase(DeclarativeHelperBase):
@ -411,8 +436,10 @@ class SubCategoryBase(DeclarativeHelperBase):
return '_'.join(str(x) for x in self.get_category_ids()) return '_'.join(str(x) for x in self.get_category_ids())
@classmethod @classmethod
def by_category_ids(cls, main_cat_id, sub_cat_id): def by_category_ids(cls, main_cat_id: int, sub_cat_id: int) -> Optional['SubCategoryBase']:
return cls.query.get((sub_cat_id, main_cat_id)) """Get a subcategory by its main category ID and subcategory ID."""
stmt = select(cls).filter_by(id=sub_cat_id, main_category_id=main_cat_id)
return db.session.execute(stmt).scalar_one_or_none()
class CommentBase(DeclarativeHelperBase): class CommentBase(DeclarativeHelperBase):
@ -493,8 +520,8 @@ class User(db.Model):
created_time = db.Column(db.DateTime(timezone=False), default=datetime.utcnow) created_time = db.Column(db.DateTime(timezone=False), default=datetime.utcnow)
last_login_date = db.Column(db.DateTime(timezone=False), default=None, nullable=True) last_login_date = db.Column(db.DateTime(timezone=False), default=None, nullable=True)
last_login_ip = db.Column(db.Binary(length=16), default=None, nullable=True) last_login_ip = db.Column(db.LargeBinary(length=16), default=None, nullable=True)
registration_ip = db.Column(db.Binary(length=16), default=None, nullable=True) registration_ip = db.Column(db.LargeBinary(length=16), default=None, nullable=True)
nyaa_torrents = db.relationship('NyaaTorrent', back_populates='user', lazy='dynamic') nyaa_torrents = db.relationship('NyaaTorrent', back_populates='user', lazy='dynamic')
nyaa_comments = db.relationship('NyaaComment', back_populates='user', lazy='dynamic') nyaa_comments = db.relationship('NyaaComment', back_populates='user', lazy='dynamic')
@ -597,22 +624,26 @@ class User(db.Model):
return str(ip_address(self.registration_ip)) return str(ip_address(self.registration_ip))
@classmethod @classmethod
def by_id(cls, id): def by_id(cls, id: int) -> Optional['User']:
return cls.query.get(id) """Get a user by their ID."""
stmt = select(cls).filter_by(id=id)
return db.session.execute(stmt).scalar_one_or_none()
@classmethod @classmethod
def by_username(cls, username): def by_username(cls, username: str) -> Optional['User']:
"""Get a user by their username."""
def isascii(s): return len(s) == len(s.encode()) def isascii(s): return len(s) == len(s.encode())
if not isascii(username): if not isascii(username):
return None return None
user = cls.query.filter_by(username=username).first() stmt = select(cls).filter_by(username=username)
return user return db.session.execute(stmt).scalar_one_or_none()
@classmethod @classmethod
def by_email(cls, email): def by_email(cls, email: str) -> Optional['User']:
user = cls.query.filter_by(email=email).first() """Get a user by their email."""
return user stmt = select(cls).filter_by(email=email)
return db.session.execute(stmt).scalar_one_or_none()
@classmethod @classmethod
def by_username_or_email(cls, username_or_email): def by_username_or_email(cls, username_or_email):
@ -649,20 +680,28 @@ class User(db.Model):
return (self.created_time - UTC_EPOCH).total_seconds() return (self.created_time - UTC_EPOCH).total_seconds()
@property @property
def satisfies_trusted_reqs(self): def satisfies_trusted_reqs(self) -> bool:
"""Check if the user meets the requirements to be trusted."""
num_total = 0 num_total = 0
downloads_total = 0 downloads_total = 0
for ts_flavor, t_flavor in ((NyaaStatistic, NyaaTorrent), for ts_flavor, t_flavor in ((NyaaStatistic, NyaaTorrent),
(SukebeiStatistic, SukebeiTorrent)): (SukebeiStatistic, SukebeiTorrent)):
uploads = db.session.query(func.count(t_flavor.id)).\ # Count uploads that aren't remakes
stmt = select(func.count(t_flavor.id)).\
filter(t_flavor.user == self).\ filter(t_flavor.user == self).\
filter(t_flavor.flags.op('&')(int(TorrentFlags.REMAKE)).is_(False)).scalar() filter(t_flavor.flags.op('&')(int(TorrentFlags.REMAKE)).is_(False))
dls = db.session.query(func.sum(ts_flavor.download_count)).\ uploads = db.session.execute(stmt).scalar_one_or_none() or 0
# Sum download counts for user's torrents that aren't remakes
stmt = select(func.sum(ts_flavor.download_count)).\
join(t_flavor).\ join(t_flavor).\
filter(t_flavor.user == self).\ filter(t_flavor.user == self).\
filter(t_flavor.flags.op('&')(int(TorrentFlags.REMAKE)).is_(False)).scalar() filter(t_flavor.flags.op('&')(int(TorrentFlags.REMAKE)).is_(False))
num_total += uploads or 0 dls = db.session.execute(stmt).scalar_one_or_none() or 0
downloads_total += dls or 0
num_total += uploads
downloads_total += dls
return (num_total >= config['TRUSTED_MIN_UPLOADS'] and return (num_total >= config['TRUSTED_MIN_UPLOADS'] and
downloads_total >= config['TRUSTED_MIN_DOWNLOADS']) downloads_total >= config['TRUSTED_MIN_DOWNLOADS'])
@ -711,7 +750,8 @@ class AdminLogBase(DeclarativeHelperBase):
@classmethod @classmethod
def all_logs(cls): def all_logs(cls):
return cls.query """Get a query for all admin logs."""
return db.session.query(cls)
class ReportStatus(IntEnum): class ReportStatus(IntEnum):
@ -760,17 +800,26 @@ class ReportBase(DeclarativeHelperBase):
return (self.created_time - UTC_EPOCH).total_seconds() return (self.created_time - UTC_EPOCH).total_seconds()
@classmethod @classmethod
def by_id(cls, id): def by_id(cls, id: int) -> Optional['ReportBase']:
return cls.query.get(id) """Get a report by its ID."""
stmt = select(cls).filter_by(id=id)
return db.session.execute(stmt).scalar_one_or_none()
@classmethod @classmethod
def not_reviewed(cls, page): def not_reviewed(cls, page: int):
reports = cls.query.filter_by(status=0).paginate(page=page, per_page=20) """Get paginated reports that haven't been reviewed yet."""
return reports # Note: paginate is a Flask-SQLAlchemy extension method, not standard SQLAlchemy
# We'll keep using it for now, but it should be updated to use the new pagination API
# in a future update
stmt = select(cls).filter_by(status=0)
return db.paginate(stmt, page=page, per_page=20)
@classmethod @classmethod
def remove_reviewed(cls, id): def remove_reviewed(cls, id: int) -> int:
return cls.query.filter(cls.torrent_id == id, cls.status == 0).delete() """Remove all reports for a torrent that haven't been reviewed yet."""
stmt = db.delete(cls).filter(cls.torrent_id == id, cls.status == 0)
result = db.session.execute(stmt)
return result.rowcount
class Ban(db.Model): class Ban(db.Model):
@ -780,7 +829,7 @@ class Ban(db.Model):
created_time = db.Column(db.DateTime(timezone=False), default=datetime.utcnow) created_time = db.Column(db.DateTime(timezone=False), default=datetime.utcnow)
admin_id = db.Column(db.Integer, db.ForeignKey('users.id'), nullable=False) admin_id = db.Column(db.Integer, db.ForeignKey('users.id'), nullable=False)
user_id = db.Column(db.Integer, db.ForeignKey('users.id'), nullable=True) user_id = db.Column(db.Integer, db.ForeignKey('users.id'), nullable=True)
user_ip = db.Column(db.Binary(length=16), nullable=True) user_ip = db.Column(db.LargeBinary(length=16), nullable=True)
reason = db.Column(db.String(length=2048), nullable=False) reason = db.Column(db.String(length=2048), nullable=False)
admin = db.relationship('User', uselist=False, lazy='joined', foreign_keys=[admin_id]) admin = db.relationship('User', uselist=False, lazy='joined', foreign_keys=[admin_id])
@ -801,20 +850,27 @@ class Ban(db.Model):
@classmethod @classmethod
def all_bans(cls): def all_bans(cls):
return cls.query """Get a query for all bans."""
return db.session.query(cls)
@classmethod @classmethod
def by_id(cls, id): def by_id(cls, id: int) -> Optional['Ban']:
return cls.query.get(id) """Get a ban by its ID."""
stmt = select(cls).filter_by(id=id)
return db.session.execute(stmt).scalar_one_or_none()
@classmethod @classmethod
def banned(cls, user_id, user_ip): def banned(cls, user_id: Optional[int], user_ip: Optional[bytes]):
"""Check if a user or IP is banned."""
if user_id: if user_id:
if user_ip: if user_ip:
return cls.query.filter((cls.user_id == user_id) | (cls.user_ip == user_ip)) stmt = select(cls).filter((cls.user_id == user_id) | (cls.user_ip == user_ip))
return cls.query.filter(cls.user_id == user_id) return db.session.execute(stmt).scalars().all()
stmt = select(cls).filter(cls.user_id == user_id)
return db.session.execute(stmt).scalars().all()
if user_ip: if user_ip:
return cls.query.filter(cls.user_ip == user_ip) stmt = select(cls).filter(cls.user_ip == user_ip)
return db.session.execute(stmt).scalars().all()
return None return None
@ -858,15 +914,17 @@ class RangeBan(db.Model):
self._cidr_string = s self._cidr_string = s
@classmethod @classmethod
def is_rangebanned(cls, ip): def is_rangebanned(cls, ip: bytes) -> bool:
"""Check if an IP is within a banned range."""
if len(ip) > 4: if len(ip) > 4:
raise NotImplementedError("IPv6 is unsupported.") raise NotImplementedError("IPv6 is unsupported.")
elif len(ip) < 4: elif len(ip) < 4:
raise ValueError("Not an IP address.") raise ValueError("Not an IP address.")
ip_int = int.from_bytes(ip, 'big') ip_int = int.from_bytes(ip, 'big')
q = cls.query.filter(cls.mask.op('&')(ip_int) == cls.masked_cidr, stmt = select(cls).filter(cls.mask.op('&')(ip_int) == cls.masked_cidr,
cls.enabled) cls.enabled)
return q.count() > 0 count = db.session.execute(select(func.count()).select_from(stmt.subquery())).scalar_one()
return count > 0
class TrustedApplicationStatus(IntEnum): class TrustedApplicationStatus(IntEnum):
@ -915,8 +973,10 @@ class TrustedApplication(db.Model):
return (self.created_time - UTC_EPOCH).total_seconds() return (self.created_time - UTC_EPOCH).total_seconds()
@classmethod @classmethod
def by_id(cls, id): def by_id(cls, id: int) -> Optional['TrustedApplication']:
return cls.query.get(id) """Get a trusted application by its ID."""
stmt = select(cls).filter_by(id=id)
return db.session.execute(stmt).scalar_one_or_none()
class TrustedRecommendation(IntEnum): class TrustedRecommendation(IntEnum):

393
nyaa/search.py
View file

@ -3,15 +3,16 @@ import re
import shlex import shlex
import threading import threading
import time import time
from typing import Any, Dict, List, Optional, Tuple, Union
import flask import flask
from flask_sqlalchemy import Pagination from nyaa.custom_pagination import CustomPagination
import sqlalchemy import sqlalchemy
from sqlalchemy import select, func, bindparam
import sqlalchemy_fulltext.modes as FullTextMode import sqlalchemy_fulltext.modes as FullTextMode
from elasticsearch import Elasticsearch from elasticsearch import Elasticsearch
from elasticsearch_dsl import Q, Search from elasticsearch_dsl import Q, Search
from sqlalchemy.ext import baked
from sqlalchemy_fulltext import FullTextSearch from sqlalchemy_fulltext import FullTextSearch
from nyaa import models from nyaa import models
@ -30,7 +31,7 @@ SERACH_PAGINATE_DISPLAY_MSG = ('Displaying results {start}-{end} out of {total}
_index_name_cache = {} _index_name_cache = {}
def _get_index_name(column): def _get_index_name(column) -> Optional[str]:
''' Returns an index name for a given column, or None. ''' Returns an index name for a given column, or None.
Only considers single-column indexes. Only considers single-column indexes.
Results are cached in memory (until app restart). ''' Results are cached in memory (until app restart). '''
@ -43,7 +44,7 @@ def _get_index_name(column):
try: try:
column_table = sqlalchemy.Table(column_table_name, column_table = sqlalchemy.Table(column_table_name,
sqlalchemy.MetaData(), sqlalchemy.MetaData(),
autoload=True, autoload_with=db.engine) autoload_with=db.engine)
except sqlalchemy.exc.NoSuchTableError: except sqlalchemy.exc.NoSuchTableError:
# Trust the developer to notice this? # Trust the developer to notice this?
pass pass
@ -60,7 +61,8 @@ def _get_index_name(column):
return table_indexes.get(column.name) return table_indexes.get(column.name)
def _generate_query_string(term, category, filter, user): def _generate_query_string(term: Optional[str], category: Optional[str],
filter: Optional[str], user: Optional[int]) -> Dict[str, str]:
params = {} params = {}
if term: if term:
params['q'] = str(term) params['q'] = str(term)
@ -370,16 +372,23 @@ class QueryPairCaller(object):
return wrapper return wrapper
def search_db(term='', user=None, sort='id', order='desc', category='0_0', def search_db(term: str = '', user: Optional[int] = None, sort: str = 'id',
quality_filter='0', page=1, rss=False, admin=False, order: str = 'desc', category: str = '0_0',
logged_in_user=None, per_page=75): quality_filter: str = '0', page: int = 1, rss: bool = False,
admin: bool = False, logged_in_user: Optional[models.User] = None,
per_page: int = 75) -> Union[CustomPagination, List[models.Torrent]]:
"""
Search the database for torrents matching the given criteria.
This is the SQLAlchemy 2.0 compatible version of the search function.
"""
if page > 4294967295: if page > 4294967295:
flask.abort(404) flask.abort(404)
MAX_PAGES = app.config.get("MAX_PAGES", 0) MAX_PAGES = app.config.get("MAX_PAGES", 0)
same_user = False same_user = False
if logged_in_user: if logged_in_user and user:
same_user = logged_in_user.id == user same_user = logged_in_user.id == user
# Logged in users should always be able to view their full listing. # Logged in users should always be able to view their full listing.
@ -427,10 +436,10 @@ def search_db(term='', user=None, sort='id', order='desc', category='0_0',
flask.abort(400) flask.abort(400)
if user: if user:
user = models.User.by_id(user) user_obj = models.User.by_id(user)
if not user: if not user_obj:
flask.abort(404) flask.abort(404)
user = user.id user = user_obj.id
main_category = None main_category = None
sub_category = None sub_category = None
@ -460,22 +469,22 @@ def search_db(term='', user=None, sort='id', order='desc', category='0_0',
model_class = models.TorrentNameSearch if term else models.Torrent model_class = models.TorrentNameSearch if term else models.Torrent
query = db.session.query(model_class) # Create the base query
query = select(model_class)
# This is... eh. Optimize the COUNT() query since MySQL is bad at that. count_query = select(func.count(model_class.id))
# See http://docs.sqlalchemy.org/en/rel_1_1/orm/query.html#sqlalchemy.orm.query.Query.count
# Wrap the queries into the helper class to deduplicate code and apply filters to both in one go
count_query = db.session.query(sqlalchemy.func.count(model_class.id))
qpc = QueryPairCaller(query, count_query)
# User view (/user/username) # User view (/user/username)
if user: if user:
qpc.filter(models.Torrent.uploader_id == user) query = query.where(models.Torrent.uploader_id == user)
count_query = count_query.where(models.Torrent.uploader_id == user)
if not admin: if not admin:
# Hide all DELETED torrents if regular user # Hide all DELETED torrents if regular user
qpc.filter(models.Torrent.flags.op('&')( deleted_filter = models.Torrent.flags.op('&')(
int(models.TorrentFlags.DELETED)).is_(False)) int(models.TorrentFlags.DELETED)).is_(False)
query = query.where(deleted_filter)
count_query = count_query.where(deleted_filter)
# If logged in user is not the same as the user being viewed, # If logged in user is not the same as the user being viewed,
# show only torrents that aren't hidden or anonymous # show only torrents that aren't hidden or anonymous
# #
@ -485,277 +494,99 @@ def search_db(term='', user=None, sort='id', order='desc', category='0_0',
# On RSS pages in user view, # On RSS pages in user view,
# show only torrents that aren't hidden or anonymous no matter what # show only torrents that aren't hidden or anonymous no matter what
if not same_user or rss: if not same_user or rss:
qpc.filter(models.Torrent.flags.op('&')( hidden_anon_filter = models.Torrent.flags.op('&')(
int(models.TorrentFlags.HIDDEN | models.TorrentFlags.ANONYMOUS)).is_(False)) int(models.TorrentFlags.HIDDEN | models.TorrentFlags.ANONYMOUS)).is_(False)
query = query.where(hidden_anon_filter)
count_query = count_query.where(hidden_anon_filter)
# General view (homepage, general search view) # General view (homepage, general search view)
else: else:
if not admin: if not admin:
# Hide all DELETED torrents if regular user # Hide all DELETED torrents if regular user
qpc.filter(models.Torrent.flags.op('&')( deleted_filter = models.Torrent.flags.op('&')(
int(models.TorrentFlags.DELETED)).is_(False)) int(models.TorrentFlags.DELETED)).is_(False)
query = query.where(deleted_filter)
count_query = count_query.where(deleted_filter)
# If logged in, show all torrents that aren't hidden unless they belong to you # If logged in, show all torrents that aren't hidden unless they belong to you
# On RSS pages, show all public torrents and nothing more. # On RSS pages, show all public torrents and nothing more.
if logged_in_user and not rss: if logged_in_user and not rss:
qpc.filter( hidden_or_user_filter = (
(models.Torrent.flags.op('&')(int(models.TorrentFlags.HIDDEN)).is_(False)) | (models.Torrent.flags.op('&')(int(models.TorrentFlags.HIDDEN)).is_(False)) |
(models.Torrent.uploader_id == logged_in_user.id)) (models.Torrent.uploader_id == logged_in_user.id)
)
query = query.where(hidden_or_user_filter)
count_query = count_query.where(hidden_or_user_filter)
# Otherwise, show all torrents that aren't hidden # Otherwise, show all torrents that aren't hidden
else: else:
qpc.filter(models.Torrent.flags.op('&')( hidden_filter = models.Torrent.flags.op('&')(
int(models.TorrentFlags.HIDDEN)).is_(False)) int(models.TorrentFlags.HIDDEN)).is_(False)
query = query.where(hidden_filter)
count_query = count_query.where(hidden_filter)
if main_category: if main_category:
qpc.filter(models.Torrent.main_category_id == main_cat_id) main_cat_filter = models.Torrent.main_category_id == main_cat_id
query = query.where(main_cat_filter)
count_query = count_query.where(main_cat_filter)
elif sub_category: elif sub_category:
qpc.filter((models.Torrent.main_category_id == main_cat_id) & sub_cat_filter = (
(models.Torrent.sub_category_id == sub_cat_id)) (models.Torrent.main_category_id == main_cat_id) &
(models.Torrent.sub_category_id == sub_cat_id)
)
query = query.where(sub_cat_filter)
count_query = count_query.where(sub_cat_filter)
if filter_tuple: if filter_tuple:
qpc.filter(models.Torrent.flags.op('&')( filter_condition = models.Torrent.flags.op('&')(
int(filter_tuple[0])).is_(filter_tuple[1])) int(filter_tuple[0])).is_(filter_tuple[1])
query = query.where(filter_condition)
count_query = count_query.where(filter_condition)
if term: if term:
for item in shlex.split(term, posix=False): for item in shlex.split(term, posix=False):
if len(item) >= 2: if len(item) >= 2:
qpc.filter(FullTextSearch( fulltext_filter = FullTextSearch(
item, models.TorrentNameSearch, FullTextMode.NATURAL)) item, models.TorrentNameSearch, FullTextMode.NATURAL)
query, count_query = qpc.items query = query.where(fulltext_filter)
count_query = count_query.where(fulltext_filter)
# Sort and order # Sort and order
if sort_column.class_ != models.Torrent: if sort_column.class_ != models.Torrent:
index_name = _get_index_name(sort_column) index_name = _get_index_name(sort_column)
query = query.join(sort_column.class_) query = query.join(sort_column.class_)
query = query.with_hint(sort_column.class_, 'USE INDEX ({0})'.format(index_name))
# Add index hint for MySQL if available
query = query.order_by(getattr(sort_column, order)()) if index_name and hasattr(db.engine.dialect, 'name') and db.engine.dialect.name == 'mysql':
# In SQLAlchemy 2.0, we use execution_options instead of with_hint
# This is MySQL specific - for other databases, different approaches would be needed
query = query.execution_options(
mysql_hint=f"USE INDEX ({index_name})"
)
if order_ == 'desc':
query = query.order_by(sort_column.desc())
else:
query = query.order_by(sort_column.asc())
if rss: if rss:
query = query.limit(per_page) query = query.limit(per_page)
return db.session.execute(query).scalars().all()
else: else:
query = query.paginate_faste(page, per_page=per_page, step=5, count_query=count_query, # Get the total count
max_page=MAX_PAGES) total_count = db.session.execute(count_query).scalar_one()
return query # Apply pagination
query = query.limit(per_page).offset((page - 1) * per_page)
items = db.session.execute(query).scalars().all()
# Baked queries follow
if not items and page != 1:
class BakedPair(object):
def __init__(self, *items):
self.items = list(items)
def __iadd__(self, other):
for item in self.items:
item += other
return self
bakery = baked.bakery()
BAKED_SORT_KEYS = {
'id': models.Torrent.id,
'size': models.Torrent.filesize,
'comments': models.Torrent.comment_count,
'seeders': models.Statistic.seed_count,
'leechers': models.Statistic.leech_count,
'downloads': models.Statistic.download_count
}
BAKED_SORT_LAMBDAS = {
'id-asc': lambda q: q.order_by(models.Torrent.id.asc()),
'id-desc': lambda q: q.order_by(models.Torrent.id.desc()),
'size-asc': lambda q: q.order_by(models.Torrent.filesize.asc()),
'size-desc': lambda q: q.order_by(models.Torrent.filesize.desc()),
'comments-asc': lambda q: q.order_by(models.Torrent.comment_count.asc()),
'comments-desc': lambda q: q.order_by(models.Torrent.comment_count.desc()),
# This is a bit stupid, but programmatically generating these mixed up the baked keys, so deal.
'seeders-asc': lambda q: q.join(models.Statistic).with_hint(
models.Statistic, 'USE INDEX (idx_nyaa_statistics_seed_count)'
).order_by(models.Statistic.seed_count.asc(), models.Torrent.id.asc()),
'seeders-desc': lambda q: q.join(models.Statistic).with_hint(
models.Statistic, 'USE INDEX (idx_nyaa_statistics_seed_count)'
).order_by(models.Statistic.seed_count.desc(), models.Torrent.id.desc()),
'leechers-asc': lambda q: q.join(models.Statistic).with_hint(
models.Statistic, 'USE INDEX (idx_nyaa_statistics_leech_count)'
).order_by(models.Statistic.leech_count.asc(), models.Torrent.id.asc()),
'leechers-desc': lambda q: q.join(models.Statistic).with_hint(
models.Statistic, 'USE INDEX (idx_nyaa_statistics_leech_count)'
).order_by(models.Statistic.leech_count.desc(), models.Torrent.id.desc()),
'downloads-asc': lambda q: q.join(models.Statistic).with_hint(
models.Statistic, 'USE INDEX (idx_nyaa_statistics_download_count)'
).order_by(models.Statistic.download_count.asc(), models.Torrent.id.asc()),
'downloads-desc': lambda q: q.join(models.Statistic).with_hint(
models.Statistic, 'USE INDEX (idx_nyaa_statistics_download_count)'
).order_by(models.Statistic.download_count.desc(), models.Torrent.id.desc()),
}
BAKED_FILTER_LAMBDAS = {
'0': None,
'1': lambda q: (
q.filter(models.Torrent.flags.op('&')(models.TorrentFlags.REMAKE.value).is_(False))
),
'2': lambda q: (
q.filter(models.Torrent.flags.op('&')(models.TorrentFlags.TRUSTED.value).is_(True))
),
'3': lambda q: (
q.filter(models.Torrent.flags.op('&')(models.TorrentFlags.COMPLETE.value).is_(True))
),
}
def search_db_baked(term='', user=None, sort='id', order='desc', category='0_0',
quality_filter='0', page=1, rss=False, admin=False,
logged_in_user=None, per_page=75):
if page > 4294967295:
flask.abort(404)
MAX_PAGES = app.config.get("MAX_PAGES", 0)
if MAX_PAGES and page > MAX_PAGES:
flask.abort(flask.Response("You've exceeded the maximum number of pages. Please "
"make your search query less broad.", 403))
sort_lambda = BAKED_SORT_LAMBDAS.get('{}-{}'.format(sort, order).lower())
if not sort_lambda:
flask.abort(400)
sentinel = object()
filter_lambda = BAKED_FILTER_LAMBDAS.get(quality_filter.lower(), sentinel)
if filter_lambda is sentinel:
flask.abort(400)
if user:
user = models.User.by_id(user)
if not user:
flask.abort(404) flask.abort(404)
user = user.id
# Create a pagination object
return CustomPagination(query, page, per_page, total_count, items)
main_cat_id = 0
sub_cat_id = 0
if category: # Alias for backward compatibility
cat_match = re.match(r'^(\d+)_(\d+)$', category) search_db_baked = search_db
if not cat_match:
flask.abort(400)
main_cat_id = int(cat_match.group(1))
sub_cat_id = int(cat_match.group(2))
if main_cat_id > 0:
if sub_cat_id > 0:
sub_category = models.SubCategory.by_category_ids(main_cat_id, sub_cat_id)
if not sub_category:
flask.abort(400)
else:
main_category = models.MainCategory.by_id(main_cat_id)
if not main_category:
flask.abort(400)
# Force sort by id desc if rss
if rss:
sort_lambda = BAKED_SORT_LAMBDAS['id-desc']
same_user = False
if logged_in_user:
same_user = logged_in_user.id == user
if term:
query = bakery(lambda session: session.query(models.TorrentNameSearch))
count_query = bakery(lambda session: session.query(
sqlalchemy.func.count(models.TorrentNameSearch.id)))
else:
query = bakery(lambda session: session.query(models.Torrent))
# This is... eh. Optimize the COUNT() query since MySQL is bad at that.
# See http://docs.sqlalchemy.org/en/rel_1_1/orm/query.html#sqlalchemy.orm.query.Query.count
# Wrap the queries into the helper class to deduplicate code and
# apply filters to both in one go
count_query = bakery(lambda session: session.query(
sqlalchemy.func.count(models.Torrent.id)))
qpc = BakedPair(query, count_query)
bp = sqlalchemy.bindparam
baked_params = {}
# User view (/user/username)
if user:
qpc += lambda q: q.filter(models.Torrent.uploader_id == bp('user'))
baked_params['user'] = user
if not admin:
# Hide all DELETED torrents if regular user
qpc += lambda q: q.filter(models.Torrent.flags.op('&')
(int(models.TorrentFlags.DELETED)).is_(False))
# If logged in user is not the same as the user being viewed,
# show only torrents that aren't hidden or anonymous
#
# If logged in user is the same as the user being viewed,
# show all torrents including hidden and anonymous ones
#
# On RSS pages in user view,
# show only torrents that aren't hidden or anonymous no matter what
if not same_user or rss:
qpc += lambda q: (
q.filter(
models.Torrent.flags.op('&')(
int(models.TorrentFlags.HIDDEN | models.TorrentFlags.ANONYMOUS)
).is_(False)
)
)
# General view (homepage, general search view)
else:
if not admin:
# Hide all DELETED torrents if regular user
qpc += lambda q: q.filter(models.Torrent.flags.op('&')
(int(models.TorrentFlags.DELETED)).is_(False))
# If logged in, show all torrents that aren't hidden unless they belong to you
# On RSS pages, show all public torrents and nothing more.
if logged_in_user and not rss:
qpc += lambda q: q.filter(
(models.Torrent.flags.op('&')(int(models.TorrentFlags.HIDDEN)).is_(False)) |
(models.Torrent.uploader_id == bp('logged_in_user'))
)
baked_params['logged_in_user'] = logged_in_user
# Otherwise, show all torrents that aren't hidden
else:
qpc += lambda q: q.filter(models.Torrent.flags.op('&')
(int(models.TorrentFlags.HIDDEN)).is_(False))
if sub_cat_id:
qpc += lambda q: q.filter(
(models.Torrent.main_category_id == bp('main_cat_id')),
(models.Torrent.sub_category_id == bp('sub_cat_id'))
)
baked_params['main_cat_id'] = main_cat_id
baked_params['sub_cat_id'] = sub_cat_id
elif main_cat_id:
qpc += lambda q: q.filter(models.Torrent.main_category_id == bp('main_cat_id'))
baked_params['main_cat_id'] = main_cat_id
if filter_lambda:
qpc += filter_lambda
if term:
raise Exception('Baked search does not support search terms')
# Sort and order
query += sort_lambda
if rss:
query += lambda q: q.limit(bp('per_page'))
baked_params['per_page'] = per_page
return query(db.session()).params(**baked_params).all()
return baked_paginate(query, count_query, baked_params,
page, per_page=per_page, step=5, max_page=MAX_PAGES)
class ShoddyLRU(object): class ShoddyLRU(object):
@ -800,33 +631,33 @@ class ShoddyLRU(object):
LRU_CACHE = ShoddyLRU(256, 60) LRU_CACHE = ShoddyLRU(256, 60)
def baked_paginate(query, count_query, params, page=1, per_page=50, max_page=None, step=5): def paginate_query(query, count_query, page=1, per_page=50, max_page=None):
"""
Paginate a SQLAlchemy 2.0 query.
This is a replacement for the baked_paginate function that uses SQLAlchemy 2.0 style.
"""
if page < 1: if page < 1:
flask.abort(404) flask.abort(404)
if max_page and page > max_page: if max_page and page > max_page:
flask.abort(404) flask.abort(404)
bp = sqlalchemy.bindparam
ses = db.session()
# Count all items, use cache # Count all items, use cache
if app.config['COUNT_CACHE_DURATION']: if app.config.get('COUNT_CACHE_DURATION'):
query_key = (count_query._effective_key(ses), tuple(sorted(params.items()))) # Create a cache key based on the query and parameters
# This is a simplified version compared to the bakery's _effective_key
query_key = str(count_query)
total_query_count = LRU_CACHE.get(query_key) total_query_count = LRU_CACHE.get(query_key)
if total_query_count is None: if total_query_count is None:
total_query_count = count_query(ses).params(**params).scalar() total_query_count = db.session.execute(count_query).scalar_one()
LRU_CACHE.put(query_key, total_query_count, expiry=app.config['COUNT_CACHE_DURATION']) LRU_CACHE.put(query_key, total_query_count, expiry=app.config['COUNT_CACHE_DURATION'])
else: else:
total_query_count = count_query(ses).params(**params).scalar() total_query_count = db.session.execute(count_query).scalar_one()
# Grab items on current page # Apply pagination
query += lambda q: q.limit(bp('limit')).offset(bp('offset')) paginated_query = query.limit(per_page).offset((page - 1) * per_page)
params['limit'] = per_page items = db.session.execute(paginated_query).scalars().all()
params['offset'] = (page - 1) * per_page
res = query(ses).params(**params)
items = res.all()
if max_page: if max_page:
total_query_count = min(total_query_count, max_page * per_page) total_query_count = min(total_query_count, max_page * per_page)
@ -837,4 +668,8 @@ def baked_paginate(query, count_query, params, page=1, per_page=50, max_page=Non
if not items and page != 1: if not items and page != 1:
flask.abort(404) flask.abort(404)
return Pagination(None, page, per_page, total_query_count, items) return CustomPagination(None, page, per_page, total_query_count, items)
# Alias for backward compatibility
baked_paginate = paginate_query

5
nyaa/template_utils.py
View file

@ -5,7 +5,8 @@ from datetime import datetime
from email.utils import formatdate from email.utils import formatdate
import flask import flask
from werkzeug.urls import url_encode #from werkzeug.utils import url_encode
from urllib.parse import urlencode # now using Python's built-in urlencode
from nyaa.backend import get_category_id_map from nyaa.backend import get_category_id_map
from nyaa.torrents import create_magnet from nyaa.torrents import create_magnet
@ -81,7 +82,7 @@ def modify_query(**new_values):
for key, value in new_values.items(): for key, value in new_values.items():
args[key] = value args[key] = value
return '{}?{}'.format(flask.request.path, url_encode(args)) return '{}?{}'.format(flask.request.path, urlencode(args))
@bp.app_template_global() @bp.app_template_global()

6
nyaa/templates/help.html
View file

@ -108,8 +108,8 @@
{{ linkable_header("IRC Help Channel Policies", "irchelp") }} {{ linkable_header("IRC Help Channel Policies", "irchelp") }}
<div> <div>
<p>Our IRC help channel is at Rizon <a href="irc://irc.rizon.net/nyaa-help">#nyaa-help</a>. A webchat link <p>Our IRC help channel is at Rizon <a>#nyaa-help</a>. A webchat link
pre-filled with our channel is available <a href="https://qchat.rizon.net/?channels=nyaa-help">right here</a>.</p> pre-filled with our channel is available <a>right here</a>.</p>
<b>Read this to avoid getting banned:</b> <b>Read this to avoid getting banned:</b>
<ul> <ul>
@ -199,7 +199,7 @@
</li> </li>
</ul> </ul>
<h1>IRC help channel</h1><a href="irc://irc-server:port/channel?key"> <h1>IRC help channel</h1><a href="irc://irc-server:port/channel?key">
<h1>NyaaV2 IRC</h1></a> <h1>NyaaV3 IRC</h1></a>
<p>The IRC channel is only for site support.<br></p> <p>The IRC channel is only for site support.<br></p>
<p><b>Read this to avoid getting banned:</b></p> <p><b>Read this to avoid getting banned:</b></p>
<ul> <ul>

12
nyaa/torrents.py
View file

@ -5,11 +5,11 @@ from urllib.parse import quote, urlencode
import flask import flask
from flask import current_app as app from flask import current_app as app
from orderedset import OrderedSet from orderly_set import OrderlySet
from nyaa import bencode from nyaa import bencode
USED_TRACKERS = OrderedSet() USED_TRACKERS = OrderlySet()
def read_trackers_from_file(file_object): def read_trackers_from_file(file_object):
@ -37,8 +37,8 @@ def default_trackers():
def get_trackers_and_webseeds(torrent): def get_trackers_and_webseeds(torrent):
trackers = OrderedSet() trackers = OrderlySet()
webseeds = OrderedSet() webseeds = OrderlySet()
# Our main one first # Our main one first
main_announce_url = app.config.get('MAIN_ANNOUNCE_URL') main_announce_url = app.config.get('MAIN_ANNOUNCE_URL')
@ -63,7 +63,7 @@ def get_trackers_and_webseeds(torrent):
def get_default_trackers(): def get_default_trackers():
trackers = OrderedSet() trackers = OrderlySet()
# Our main one first # Our main one first
main_announce_url = app.config.get('MAIN_ANNOUNCE_URL') main_announce_url = app.config.get('MAIN_ANNOUNCE_URL')
@ -114,7 +114,7 @@ def create_default_metadata_base(torrent, trackers=None, webseeds=None):
webseeds = db_webseeds if webseeds is None else webseeds webseeds = db_webseeds if webseeds is None else webseeds
metadata_base = { metadata_base = {
'created by': 'NyaaV2', 'created by': 'NyaaV3',
'creation date': int(torrent.created_utc_timestamp), 'creation date': int(torrent.created_utc_timestamp),
'comment': flask.url_for('torrents.view', 'comment': flask.url_for('torrents.view',
torrent_id=torrent.id, torrent_id=torrent.id,

3
nyaa/views/__init__.py
View file

@ -1,4 +1,5 @@
import flask import flask
from markupsafe import Markup
from nyaa.views import ( # isort:skip from nyaa.views import ( # isort:skip
account, account,
@ -26,7 +27,7 @@ def _maintenance_mode_hook():
return resp return resp
else: else:
# Otherwise redirect to the target page and flash a message # Otherwise redirect to the target page and flash a message
flask.flash(flask.Markup(message), 'danger') flask.flash(Markup(message), 'danger')
try: try:
target_url = flask.url_for(endpoint) target_url = flask.url_for(endpoint)
except Exception: except Exception:

40
nyaa/views/account.py
View file

@ -2,6 +2,7 @@ import binascii
import time import time
from datetime import datetime, timedelta from datetime import datetime, timedelta
from ipaddress import ip_address from ipaddress import ip_address
from markupsafe import Markup
import flask import flask
@ -24,7 +25,7 @@ def login():
form = forms.LoginForm(flask.request.form) form = forms.LoginForm(flask.request.form)
if flask.request.method == 'POST' and form.validate(): if flask.request.method == 'POST' and form.validate():
if app.config['MAINTENANCE_MODE'] and not app.config['MAINTENANCE_MODE_LOGINS']: if app.config['MAINTENANCE_MODE'] and not app.config['MAINTENANCE_MODE_LOGINS']:
flask.flash(flask.Markup('<strong>Logins are currently disabled.</strong>'), 'danger') flask.flash(Markup('<strong>Logins are currently disabled.</strong>'), 'danger')
return flask.redirect(flask.url_for('account.login')) return flask.redirect(flask.url_for('account.login'))
username = form.username.data.strip() username = form.username.data.strip()
@ -38,20 +39,21 @@ def login():
user = models.User.by_email(username) user = models.User.by_email(username)
if not user or password != user.password_hash: if not user or password != user.password_hash:
flask.flash(flask.Markup( flask.flash(Markup(
'<strong>Login failed!</strong> Incorrect username or password.'), 'danger') '<strong>Login failed!</strong> Incorrect username or password.'), 'danger')
return flask.redirect(flask.url_for('account.login')) return flask.redirect(flask.url_for('account.login'))
if user.is_banned: if user.is_banned:
ban_reason = models.Ban.banned(user.id, None).first().reason ban = models.Ban.banned(user.id, None).first()
ban_str = ('<strong>Login failed!</strong> You are banned with the ' ban_reason = ban.reason if ban else '[No reason provided]'
'reason "{0}" If you believe that this is a mistake, contact ' ban_str = ('<strong>Login failed!</strong> You are banned. '
'Reason: "{0}" If you believe this is a mistake, contact '
'a moderator on IRC.'.format(ban_reason)) 'a moderator on IRC.'.format(ban_reason))
flask.flash(flask.Markup(ban_str), 'danger') flask.flash(Markup(ban_str), 'danger')
return flask.redirect(flask.url_for('account.login')) return flask.redirect(flask.url_for('account.login'))
if user.status != models.UserStatusType.ACTIVE: if user.status != models.UserStatusType.ACTIVE:
flask.flash(flask.Markup( flask.flash(Markup(
'<strong>Login failed!</strong> Account is not activated.'), 'danger') '<strong>Login failed!</strong> Account is not activated.'), 'danger')
return flask.redirect(flask.url_for('account.login')) return flask.redirect(flask.url_for('account.login'))
@ -78,7 +80,7 @@ def logout():
flask.session.modified = False flask.session.modified = False
response = flask.make_response(flask.redirect(redirect_url())) response = flask.make_response(flask.redirect(redirect_url()))
response.set_cookie(app.session_cookie_name, expires=0) response.set_cookie(app.config['SESSION_COOKIE_NAME'], expires=0)
return response return response
@ -113,7 +115,7 @@ def register():
db.session.commit() db.session.commit()
if app.config['RAID_MODE_LIMIT_REGISTER']: if app.config['RAID_MODE_LIMIT_REGISTER']:
flask.flash(flask.Markup(app.config['RAID_MODE_REGISTER_MESSAGE'] + ' ' flask.flash(Markup(app.config['RAID_MODE_REGISTER_MESSAGE'] + ' '
'Please <a href="{}">ask a moderator</a> to manually ' 'Please <a href="{}">ask a moderator</a> to manually '
'activate your account <a href="{}">\'{}\'</a>.' 'activate your account <a href="{}">\'{}\'</a>.'
.format(flask.url_for('site.help') + '#irchelp', .format(flask.url_for('site.help') + '#irchelp',
@ -122,7 +124,7 @@ def register():
user.username)), 'warning') user.username)), 'warning')
elif models.RangeBan.is_rangebanned(user.registration_ip): elif models.RangeBan.is_rangebanned(user.registration_ip):
flask.flash(flask.Markup('Your IP is blocked from creating new accounts. ' flask.flash(Markup('Your IP is blocked from creating new accounts. '
'Please <a href="{}">ask a moderator</a> to manually ' 'Please <a href="{}">ask a moderator</a> to manually '
'activate your account <a href="{}">\'{}\'</a>.' 'activate your account <a href="{}">\'{}\'</a>.'
.format(flask.url_for('site.help') + '#irchelp', .format(flask.url_for('site.help') + '#irchelp',
@ -162,7 +164,7 @@ def password_reset(payload=None):
if user: if user:
send_password_reset_request_email(user) send_password_reset_request_email(user)
flask.flash(flask.Markup( flask.flash(Markup(
'A password reset request was sent to the provided email, ' 'A password reset request was sent to the provided email, '
'if a matching account was found.'), 'info') 'if a matching account was found.'), 'info')
return flask.redirect(flask.url_for('main.home')) return flask.redirect(flask.url_for('main.home'))
@ -196,7 +198,7 @@ def password_reset(payload=None):
send_password_reset_email(user) send_password_reset_email(user)
flask.flash(flask.Markup('Your password was reset. Log in now.'), 'info') flask.flash(Markup('Your password was reset. Log in now.'), 'info')
return flask.redirect(flask.url_for('account.login')) return flask.redirect(flask.url_for('account.login'))
return flask.render_template('password_reset.html', form=form) return flask.render_template('password_reset.html', form=form)
@ -212,25 +214,25 @@ def profile():
if flask.request.method == 'POST': if flask.request.method == 'POST':
if form.authorized_submit and form.validate(): if form.authorized_submit and form.validate():
user = flask.g.user user = flask.g.user
new_email = form.email.data.strip() new_email = form.email.data.strip() if form.email.data else None
new_password = form.new_password.data new_password = form.new_password.data
if new_email: if new_email and new_email.strip():
if form.current_password.data != user.password_hash: if form.current_password.data != user.password_hash:
flask.flash(flask.Markup( flask.flash(Markup(
'<strong>Email change failed!</strong> Incorrect password.'), 'danger') '<strong>Email change failed!</strong> Incorrect password.'), 'danger')
return flask.redirect('/profile') return flask.redirect('/profile')
user.email = form.email.data user.email = form.email.data
flask.flash(flask.Markup( flask.flash(Markup(
'<strong>Email successfully changed!</strong>'), 'success') '<strong>Email successfully changed!</strong>'), 'success')
if new_password: if new_password:
if form.current_password.data != user.password_hash: if form.current_password.data != user.password_hash:
flask.flash(flask.Markup( flask.flash(Markup(
'<strong>Password change failed!</strong> Incorrect password.'), 'danger') '<strong>Password change failed!</strong> Incorrect password.'), 'danger')
return flask.redirect('/profile') return flask.redirect('/profile')
user.password_hash = form.new_password.data user.password_hash = form.new_password.data
flask.flash(flask.Markup( flask.flash(Markup(
'<strong>Password successfully changed!</strong>'), 'success') '<strong>Password successfully changed!</strong>'), 'success')
db.session.add(user) db.session.add(user)
db.session.commit() db.session.commit()
@ -244,7 +246,7 @@ def profile():
db.session.add(preferences) db.session.add(preferences)
db.session.commit() db.session.commit()
user.preferences.hide_comments = form.hide_comments.data user.preferences.hide_comments = form.hide_comments.data
flask.flash(flask.Markup( flask.flash(Markup(
'<strong>Preferences successfully changed!</strong>'), 'success') '<strong>Preferences successfully changed!</strong>'), 'success')
db.session.add(user) db.session.add(user)
db.session.commit() db.session.commit()

5
nyaa/views/admin.py
View file

@ -1,5 +1,6 @@
from datetime import datetime from datetime import datetime
from ipaddress import ip_address from ipaddress import ip_address
from markupsafe import Markup
import flask import flask
@ -162,10 +163,10 @@ def view_trusted_application(app_id):
if decision_form.accept.data: if decision_form.accept.data:
app.status = models.TrustedApplicationStatus.ACCEPTED app.status = models.TrustedApplicationStatus.ACCEPTED
app.submitter.level = models.UserLevelType.TRUSTED app.submitter.level = models.UserLevelType.TRUSTED
flask.flash(flask.Markup('Application has been <b>accepted</b>.'), 'success') flask.flash(Markup('Application has been <b>accepted</b>.'), 'success')
elif decision_form.reject.data: elif decision_form.reject.data:
app.status = models.TrustedApplicationStatus.REJECTED app.status = models.TrustedApplicationStatus.REJECTED
flask.flash(flask.Markup('Application has been <b>rejected</b>.'), 'success') flask.flash(Markup('Application has been <b>rejected</b>.'), 'success')
_send_trusted_decision_email(app.submitter, bool(decision_form.accept.data)) _send_trusted_decision_email(app.submitter, bool(decision_form.accept.data))
db.session.commit() db.session.commit()
return flask.redirect(flask.url_for('admin.trusted_application', app_id=app_id)) return flask.redirect(flask.url_for('admin.trusted_application', app_id=app_id))

9
nyaa/views/main.py
View file

@ -1,8 +1,9 @@
import base64 import base64
import math import math
import re import re
from datetime import datetime, timedelta from datetime import datetime, timedelta, timezone
from ipaddress import ip_address from ipaddress import ip_address
from markupsafe import Markup
import flask import flask
from flask_paginate import Pagination from flask_paginate import Pagination
@ -37,8 +38,8 @@ def before_request():
flask.g.user = user flask.g.user = user
if 'timeout' not in flask.session or flask.session['timeout'] < datetime.now(): if 'timeout' not in flask.session or flask.session['timeout'] < datetime.now(timezone.utc):
flask.session['timeout'] = datetime.now() + timedelta(days=7) flask.session['timeout'] = datetime.now(timezone.utc) + timedelta(days=7)
flask.session.permanent = True flask.session.permanent = True
flask.session.modified = True flask.session.modified = True
@ -140,7 +141,7 @@ def home(rss):
infohash_torrent = special_results.get('infohash_torrent') infohash_torrent = special_results.get('infohash_torrent')
if infohash_torrent: if infohash_torrent:
# infohash_torrent is only set if this is not RSS or userpage search # infohash_torrent is only set if this is not RSS or userpage search
flask.flash(flask.Markup('You were redirected here because ' flask.flash(Markup('You were redirected here because '
'the given hash matched this torrent.'), 'info') 'the given hash matched this torrent.'), 'info')
# Redirect user from search to the torrent if we found one with the specific info_hash # Redirect user from search to the torrent if we found one with the specific info_hash
return flask.redirect(flask.url_for('torrents.view', torrent_id=infohash_torrent.id)) return flask.redirect(flask.url_for('torrents.view', torrent_id=infohash_torrent.id))

13
nyaa/views/torrents.py
View file

@ -1,6 +1,7 @@
import json import json
from ipaddress import ip_address from ipaddress import ip_address
from urllib.parse import quote from urllib.parse import quote
from markupsafe import Markup
import flask import flask
from werkzeug.datastructures import CombinedMultiDict from werkzeug.datastructures import CombinedMultiDict
@ -21,7 +22,7 @@ def view_torrent(torrent_id):
torrent = models.Torrent.by_id(torrent_id) torrent = models.Torrent.by_id(torrent_id)
else: else:
torrent = models.Torrent.query \ torrent = models.Torrent.query \
.options(joinedload('filelist')) \ .options(joinedload(models.Torrent.filelist)) \
.filter_by(id=torrent_id) \ .filter_by(id=torrent_id) \
.first() .first()
@ -135,7 +136,7 @@ def edit_torrent(torrent_id):
db.session.commit() db.session.commit()
flask.flash(flask.Markup( flask.flash(Markup(
'Torrent has been successfully edited! Changes might take a few minutes to show up.'), 'Torrent has been successfully edited! Changes might take a few minutes to show up.'),
'success') 'success')
@ -227,7 +228,7 @@ def _delete_torrent(torrent, form, banform):
db.session.add(torrent) db.session.add(torrent)
if not action and not ban_torrent: if not action and not ban_torrent:
flask.flash(flask.Markup('What the fuck are you doing?'), 'danger') flask.flash(Markup('What the fuck are you doing?'), 'danger')
return flask.redirect(flask.url_for('torrents.edit', torrent_id=torrent.id)) return flask.redirect(flask.url_for('torrents.edit', torrent_id=torrent.id))
if action and editor.is_moderator: if action and editor.is_moderator:
@ -239,7 +240,7 @@ def _delete_torrent(torrent, form, banform):
if action: if action:
db.session.commit() db.session.commit()
flask.flash(flask.Markup('Torrent has been successfully {0}.'.format(action)), 'success') flask.flash(Markup('Torrent has been successfully {0}.'.format(action)), 'success')
if not banform or not (banform.ban_user.data or banform.ban_userip.data): if not banform or not (banform.ban_user.data or banform.ban_userip.data):
return flask.redirect(url) return flask.redirect(url)
@ -253,7 +254,7 @@ def _delete_torrent(torrent, form, banform):
if (banform.ban_user.data and (not uploader or uploader.is_banned)) or \ if (banform.ban_user.data and (not uploader or uploader.is_banned)) or \
(banform.ban_userip.data and ipbanned): (banform.ban_userip.data and ipbanned):
flask.flash(flask.Markup('What the fuck are you doing?'), 'danger') flask.flash(Markup('What the fuck are you doing?'), 'danger')
return flask.redirect(flask.url_for('torrents.edit', torrent_id=torrent.id)) return flask.redirect(flask.url_for('torrents.edit', torrent_id=torrent.id))
flavor = "Nyaa" if app.config['SITE_FLAVOR'] == 'nyaa' else "Sukebei" flavor = "Nyaa" if app.config['SITE_FLAVOR'] == 'nyaa' else "Sukebei"
@ -298,7 +299,7 @@ def _delete_torrent(torrent, form, banform):
db.session.commit() db.session.commit()
flask.flash(flask.Markup('Uploader has been successfully banned.'), 'success') flask.flash(Markup('Uploader has been successfully banned.'), 'success')
return flask.redirect(url) return flask.redirect(url)

9
nyaa/views/users.py
View file

@ -3,6 +3,7 @@ import math
import time import time
from ipaddress import ip_address from ipaddress import ip_address
from itertools import chain from itertools import chain
from markupsafe import Markup
import flask import flask
from flask_paginate import Pagination from flask_paginate import Pagination
@ -74,7 +75,7 @@ def view_user(user_name):
if (ban_form.ban_user.data and user.is_banned) or \ if (ban_form.ban_user.data and user.is_banned) or \
(ban_form.ban_userip.data and ipbanned) or \ (ban_form.ban_userip.data and ipbanned) or \
(ban_form.unban.data and not user.is_banned and not bans): (ban_form.unban.data and not user.is_banned and not bans):
flask.flash(flask.Markup('What the fuck are you doing?'), 'danger') flask.flash(Markup('What the fuck are you doing?'), 'danger')
return flask.redirect(url) return flask.redirect(url)
user_str = "[{0}]({1})".format(user.username, url) user_str = "[{0}]({1})".format(user.username, url)
@ -107,7 +108,7 @@ def view_user(user_name):
db.session.commit() db.session.commit()
flask.flash(flask.Markup('User has been successfully {0}.'.format(action)), 'success') flask.flash(Markup('User has been successfully {0}.'.format(action)), 'success')
return flask.redirect(url) return flask.redirect(url)
req_args = flask.request.args req_args = flask.request.args
@ -233,7 +234,7 @@ def view_user_comments(user_name):
@bp.route('/user/activate/<payload>') @bp.route('/user/activate/<payload>')
def activate_user(payload): def activate_user(payload):
if app.config['MAINTENANCE_MODE']: if app.config['MAINTENANCE_MODE']:
flask.flash(flask.Markup('<strong>Activations are currently disabled.</strong>'), 'danger') flask.flash(Markup('<strong>Activations are currently disabled.</strong>'), 'danger')
return flask.redirect(flask.url_for('main.home')) return flask.redirect(flask.url_for('main.home'))
s = get_serializer() s = get_serializer()
@ -259,7 +260,7 @@ def activate_user(payload):
flask.session.permanent = True flask.session.permanent = True
flask.session.modified = True flask.session.modified = True
flask.flash(flask.Markup("You've successfully verified your account!"), 'success') flask.flash(Markup("You've successfully verified your account!"), 'success')
return flask.redirect(flask.url_for('main.home')) return flask.redirect(flask.url_for('main.home'))

110
requirements.txt
View file

@ -1,55 +1,59 @@
alembic==1.0.11 alembic==1.14.1
appdirs==1.4.3 appdirs==1.4.4
argon2-cffi==19.1.0 argon2-cffi==23.1.0
autopep8==1.4.4 autopep8==2.3.2
blinker==1.4 blinker==1.9.0
cffi==1.12.3 cffi==1.17.1
click==7.0 click==8.1.8
dnspython==1.16.0 dnspython==2.7.0
elasticsearch==7.0.2 elasticsearch==8.17.1
elasticsearch-dsl==7.0.0 elasticsearch-dsl==8.17.1
flake8==3.7.8 flake8==7.1.2
flake8-isort==2.7.0 flake8-isort==6.1.2
Flask==1.1.1 Flask==3.1.0
Flask-Assets==0.12 Flask-Assets==2.1.0
Flask-DebugToolbar==0.10.1 Flask-DebugToolbar==0.16.0
Flask-Migrate==2.5.2 Flask-Migrate==4.1.0
flask-paginate==0.5.3 flask-paginate==2024.4.12
Flask-Script==2.0.6 # Flask-Script removed as it's deprecated and replaced with Flask CLI
Flask-SQLAlchemy==2.4.0 Flask-SQLAlchemy==3.1.1
Flask-WTF==0.14.2 Flask-WTF==1.2.2
gevent==1.4.0 gevent==24.11.1
greenlet==0.4.15 greenlet==3.1.1
isort==4.3.21 isort==6.0.1
itsdangerous==1.1.0 itsdangerous==2.2.0
Jinja2==2.10.1 Jinja2==3.1.5
Mako==1.1.0 Mako==1.3.9
MarkupSafe==1.1.1 MarkupSafe==3.0.2
mysql-replication==0.19 mysql-replication==1.0.9
mysqlclient==1.4.3 mysqlclient==2.2.7
orderedset==2.0.1 # orderedset removed as it's deprecated and replaced with Flask CLI
packaging==19.1 orderly-set==5.3.0
passlib==1.7.1 packaging==24.2
passlib==1.7.4
progressbar33==2.4 progressbar33==2.4
py==1.8.0 py==1.11.0
pycodestyle==2.5.0 pycodestyle==2.12.1
pycparser==2.19 pycparser==2.22
PyMySQL==0.9.3 PyMySQL==1.1.1
pyparsing==2.4.2 pyparsing==3.2.1
pytest==5.0.1 pytest==8.3.4
python-dateutil==2.8.0 python-dateutil==2.9.0
python-editor==1.0.4 python-editor==1.0.4
python-utils==2.3.0 python-utils==3.9.1
requests==2.22.0 requests==2.32.3
SQLAlchemy==1.3.6 SQLAlchemy==2.0.38
SQLAlchemy-FullText-Search==0.2.5 SQLAlchemy-FullText-Search==0.3.0
SQLAlchemy-Utils==0.34.1 SQLAlchemy-Utils==0.41.2
statsd==3.3.0 statsd==4.0.1
urllib3==1.25.3 urllib3==2.3.0
uWSGI==2.0.18 uWSGI==2.0.28
redis==3.2.1 redis==5.2.1
webassets==0.12.1 webassets==2.0
Werkzeug==0.15.5 Werkzeug==3.1.3
WTForms==2.2.1 WTForms==3.2.1
Flask-Caching==1.7.2 Flask-Caching==2.3.1
Flask-Limiter==1.0.1 Flask-Limiter==3.10.1
mypy==1.15.0
typing-extensions==4.12.2
email-validator==2.2.0

8
run.py
View file

@ -1,5 +1,11 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""
Main entry point for running the Nyaa application.
Compatible with Python 3.13.
"""
from nyaa import create_app from nyaa import create_app
app = create_app('config') app = create_app('config')
app.run(host='0.0.0.0', port=5500, debug=True)
if __name__ == '__main__':
app.run(host='0.0.0.0', port=5500, debug=True)

2
tests/__init__.py
View file

@ -18,7 +18,7 @@ class NyaaTestCase(unittest.TestCase):
# Use a separate database for testing # Use a separate database for testing
# if USE_MYSQL: # if USE_MYSQL:
# cls.db_name = 'nyaav2_tests' # cls.db_name = 'nyaav3_tests'
# db_uri = 'mysql://root:@localhost/{}?charset=utf8mb4'.format(cls.db_name) # db_uri = 'mysql://root:@localhost/{}?charset=utf8mb4'.format(cls.db_name)
# else: # else:
# cls.db_name = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'test.db') # cls.db_name = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'test.db')

4
utils/api_info.py
View file

@ -5,8 +5,8 @@ import re
import requests import requests
NYAA_HOST = 'https://nyaa.si' NYAA_HOST = 'https://your.nyaa.instance'
SUKEBEI_HOST = 'https://sukebei.nyaa.si' SUKEBEI_HOST = 'https://your.sukebei.instance'
API_BASE = '/api' API_BASE = '/api'
API_INFO = API_BASE + '/info' API_INFO = API_BASE + '/info'

4
utils/api_uploader_v2.py
View file

@ -5,8 +5,8 @@ import os
import requests import requests
NYAA_HOST = 'https://nyaa.si' NYAA_HOST = 'https://your.nyaa.instance'
SUKEBEI_HOST = 'https://sukebei.nyaa.si' SUKEBEI_HOST = 'https://your.sukebei.instance'
API_BASE = '/api' API_BASE = '/api'
API_UPLOAD = API_BASE + '/upload' API_UPLOAD = API_BASE + '/upload'

2
utils/infodict_mysql2file.py
View file

@ -18,7 +18,7 @@ if not os.path.exists(outdir):
db = MySQLdb.connect(host='localhost', db = MySQLdb.connect(host='localhost',
user='test', user='test',
passwd='test123', passwd='test123',
db='nyaav2', db='nyaav3',
cursorclass=MySQLdb.cursors.SSCursor) cursorclass=MySQLdb.cursors.SSCursor)
cur = db.cursor() cur = db.cursor()