sb745/NyaaV3
1
0
Fork 0
mirror of https://github.com/sb745/NyaaV3.git synced 2025-12-23 04:35:45 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Merge branch 'master' into markdown-ins-and-mark

Browse source
This commit is contained in:
sb745 2025-12-22 06:08:01 +02:00 • committed by GitHub
commit 6f4634c0f6
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
121 changed files with 3822 additions and 1030 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

17
.docker/Dockerfile Normal file
View file

@ -0,0 +1,17 @@
FROM ubuntu:18.04
ENV LANG=en_US.utf-8 LC_ALL=en_US.utf-8 DEBIAN_FRONTEND=noninteractive
RUN apt-get -y update
COPY ./ /nyaa/
RUN cat /nyaa/config.example.py /nyaa/.docker/nyaa-config-partial.py > /nyaa/config.py
# Requirements for running the Flask app
RUN apt-get -y install build-essential git python3 python3-pip libmysqlclient-dev curl
# Helpful stuff for the docker entrypoint.sh script
RUN apt-get -y install mariadb-client netcat
WORKDIR /nyaa
RUN pip3 install -r requirements.txt
CMD ["/nyaa/.docker/entrypoint.sh"]

48
.docker/README.md Normal file
View file

@ -0,0 +1,48 @@
# Nyaa on Docker
> [!CAUTION]
> Docker deployment is out of date and currently unsupported in NyaaV3.
Docker infrastructure is provided to ease setting up a dev environment
## Quickstart
Get started by running (from the root of the project):
docker-compose -f .docker/full-stack.yml -p nyaa build nyaa-flask
docker-compose -f .docker/full-stack.yml -p nyaa up -d
This builds the Flask app container, then starts up the project. You can then go
to [localhost:8080](http://localhost:8080/) (note that some of the
services are somewhat slow to start so it may not be available for 30s or so).
You can shut it down with:
docker-compose -f .docker/full-stack.yml -p nyaa down
## Details
The environment includes:
- [nginx frontend](http://localhost:8080/) (on port 8080)
- uwsgi running the flask app
- the ES<>MariaDB sync process
- MariaDB
- ElasticSearch
- [Kibana](http://localhost:8080/kibana/) (at /kibana/)
MariaDB, ElasticSearch, the sync process, and uploaded torrents will
persistently store their data in volumes which makes future start ups faster.
To make it more useful to develop with, you can copy `.docker/full-stack.yml` and
edit the copy and uncomment the `- "${NYAA_SRC_DIR}:/nyaa"` line, then
`export NYAA_SRC_DIR=$(pwd)` and start up the environment using the new compose
file:
cp -a .docker/full-stack.yml .docker/local-dev.yml
cat config.example.py .docker/nyaa-config-partial.py > ./config.py
$EDITOR .docker/local-dev.yml
export NYAA_SRC_DIR=$(pwd)
docker-compose -f .docker/local-dev.yml -p nyaa up -d
This will mount the local copy of the project files into the Flask container,
which combined with live-reloading in uWSGI should let you make changes and see
them take effect immediately (technically with a ~2 second delay).

32
.docker/entrypoint-sync.sh Executable file
View file

@ -0,0 +1,32 @@
#!/bin/bash
# set +x
pushd /nyaa
echo 'Waiting for MySQL to start up'
while ! echo HELO | nc mariadb 3306 &>/dev/null; do
sleep 1
done
echo 'DONE'
echo 'Waiting for ES to start up'
while ! echo HELO | nc elasticsearch 9200 &>/dev/null; do
sleep 1
done
echo 'DONE'
echo 'Waiting for ES to be ready'
while ! curl -s -XGET 'elasticsearch:9200/_cluster/health?pretty=true&wait_for_status=green' &>/dev/null; do
sleep 1
done
echo 'DONE'
echo 'Waiting for sync data file to exist'
while ! [ -f /elasticsearch-sync/pos.json ]; do
sleep 1
done
echo 'DONE'
echo 'Starting the sync process'
/usr/bin/python3 /nyaa/sync_es.py /nyaa/.docker/es_sync_config.json

50
.docker/entrypoint.sh Executable file
View file

@ -0,0 +1,50 @@
#!/bin/bash
# set +x
pushd /nyaa
echo 'Waiting for MySQL to start up'
while ! echo HELO | nc mariadb 3306 &>/dev/null; do
sleep 1
done
echo 'DONE'
if ! [ -f /elasticsearch-sync/flag-db_create ]; then
python3 ./db_create.py
touch /elasticsearch-sync/flag-db_create
fi
if ! [ -f /elasticsearch-sync/flag-db_migrate ]; then
python3 ./db_migrate.py stamp head
touch /elasticsearch-sync/flag-db_migrate
fi
echo 'Waiting for ES to start up'
while ! echo HELO | nc elasticsearch 9200 &>/dev/null; do
sleep 1
done
echo 'DONE'
echo 'Waiting for ES to be ready'
while ! curl -s -XGET 'elasticsearch:9200/_cluster/health?pretty=true&wait_for_status=green' &>/dev/null; do
sleep 1
done
echo 'DONE'
if ! [ -f /elasticsearch-sync/flag-create_es ]; then
# @source create_es.sh
# create indices named "nyaa" and "sukebei", these are hardcoded
curl -v -XPUT 'elasticsearch:9200/nyaa?pretty' -H"Content-Type: application/yaml" --data-binary @es_mapping.yml
curl -v -XPUT 'elasticsearch:9200/sukebei?pretty' -H"Content-Type: application/yaml" --data-binary @es_mapping.yml
touch /elasticsearch-sync/flag-create_es
fi
if ! [ -f /elasticsearch-sync/flag-import_to_es ]; then
python3 ./import_to_es.py | tee /elasticsearch-sync/import.out
grep -A1 'Save the following' /elasticsearch-sync/import.out | tail -1 > /elasticsearch-sync/pos.json
touch /elasticsearch-sync/flag-import_to_es
fi
echo 'Starting the Flask app'
/usr/local/bin/uwsgi /nyaa/.docker/uwsgi.config.ini

11
.docker/es_sync_config.json Normal file
View file

@ -0,0 +1,11 @@
{
"save_loc": "/elasticsearch-sync/pos.json",
"mysql_host": "mariadb",
"mysql_port": 3306,
"mysql_user": "nyaadev",
"mysql_password": "ZmtB2oihHFvc39JaEDoF",
"database": "nyaav3",
"internal_queue_depth": 10000,
"es_chunk_size": 10000,
"flush_interval": 5
}

71
.docker/full-stack.yml Normal file
View file

@ -0,0 +1,71 @@
---
version: "3"
services:
nginx:
image: nginx:1.15-alpine
ports:
- '8080:80'
volumes:
- './nginx.conf:/etc/nginx/nginx.conf:ro'
- '../nyaa/static:/nyaa-static:ro'
depends_on:
- nyaa-flask
- kibana
nyaa-flask:
image: local/nyaa:devel
volumes:
- 'nyaa-torrents:/nyaa-torrents'
- 'nyaa-sync-data:/elasticsearch-sync'
## Uncomment this line to have to mount the local dir to the running
## instance for live changes (after setting NYAA_SRC_DIR env var)
# - "${NYAA_SRC_DIR}:/nyaa"
depends_on:
- mariadb
- elasticsearch
build:
context: ../
dockerfile: ./.docker/Dockerfile
nyaa-sync:
image: local/nyaa:devel
volumes:
- 'nyaa-sync-data:/elasticsearch-sync'
command: /nyaa/.docker/entrypoint-sync.sh
depends_on:
- mariadb
- elasticsearch
restart: on-failure
mariadb:
image: mariadb:10.0
volumes:
- './mariadb-init-sql:/docker-entrypoint-initdb.d:ro'
- '../configs/my.cnf:/etc/mysql/conf.d/50-binlog.cnf:ro'
- 'mariadb-data:/var/lib/mysql'
environment:
- MYSQL_RANDOM_ROOT_PASSWORD=yes
- MYSQL_USER=nyaadev
- MYSQL_PASSWORD=ZmtB2oihHFvc39JaEDoF
- MYSQL_DATABASE=nyaav3
elasticsearch:
image: elasticsearch:6.5.4
volumes:
- elasticsearch-data:/usr/share/elasticsearch/data
depends_on:
- mariadb
kibana:
image: kibana:6.5.4
volumes:
- './kibana.config.yml:/usr/share/kibana/config/kibana.yml:ro'
depends_on:
- elasticsearch
volumes:
nyaa-torrents:
nyaa-sync-data:
mariadb-data:
elasticsearch-data:

9
.docker/kibana.config.yml Normal file
View file

@ -0,0 +1,9 @@
---
server.name: kibana
server.host: 'kibana'
server.basePath: /kibana
# server.rewriteBasePath: true
# server.defaultRoute: /kibana/app/kibana
elasticsearch.url: http://elasticsearch:9200
xpack.monitoring.ui.container.elasticsearch.enabled: true

1
.docker/mariadb-init-sql/.gitignore vendored Normal file
View file

@ -0,0 +1 @@
!*.sql

3
.docker/mariadb-init-sql/50-grant-binlog-access.sql Normal file
View file

@ -0,0 +1,3 @@
GRANT REPLICATION SLAVE ON *.* TO 'nyaadev'@'%';
GRANT REPLICATION CLIENT ON *.* TO 'nyaadev'@'%';
FLUSH PRIVILEGES;

59
.docker/nginx.conf Normal file
View file

@ -0,0 +1,59 @@
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
charset utf-8;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
gzip on;
server {
listen 80;
server_name localhost default;
location /static {
alias /nyaa-static;
}
# fix kibana redirecting to localhost/kibana (without the port)
rewrite ^/kibana$ http://$http_host/kibana/ permanent;
location /kibana/ {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_cache_bypass $http_upgrade;
proxy_set_header Host 'kibana';
proxy_set_header X-Real-IP $remote_addr;
proxy_pass http://kibana:5601/;
}
location / {
include /etc/nginx/uwsgi_params;
uwsgi_pass nyaa-flask:5000;
}
}
}

10
.docker/nyaa-config-partial.py Normal file
View file

@ -0,0 +1,10 @@
# This is only a partial config file that will be appended to the end of
# config.example.py to build the full config for the docker environment
SITE_NAME = 'Nyaa [DEVEL]'
GLOBAL_SITE_NAME = 'nyaa.devel'
SQLALCHEMY_DATABASE_URI = ('mysql://nyaadev:ZmtB2oihHFvc39JaEDoF@mariadb/nyaav3?charset=utf8mb4')
# MAIN_ANNOUNCE_URL = 'http://chihaya:6881/announce'
# TRACKER_API_URL = 'http://chihaya:6881/api'
BACKUP_TORRENT_FOLDER = '/nyaa-torrents'
ES_HOSTS = ['elasticsearch:9200']

34
.docker/uwsgi.config.ini Normal file
View file

@ -0,0 +1,34 @@
[uwsgi]
# socket = [addr:port]
socket = 0.0.0.0:5000
#chmod-socket = 664
die-on-term = true
# logging
#disable-logging = True
#logger = file:uwsgi.log
# Base application directory
chdir = /nyaa
# WSGI module and callable
# module = [wsgi_module_name]:[application_callable_name]
module = WSGI:app
# master = [master process (true of false)]
master = true
# debugging
catch-exceptions = true
# performance
processes = 4
buffer-size = 8192
loop = gevent
socket-timeout = 10
gevent = 1000
gevent-monkey-patch = true
py-autoreload = 2

6
.github/issue_template.md vendored
View file

@ -1,5 +1,3 @@
Describe your issue/feature request here (you can remove all this text). Describe well and include images, if relevant!
Describe your issue/feature request here (you can remove all this text). Describe well and include images if relevant.
Please make sure to skim through the existing issues, your issue/request/etc may have already been noted!
IMPORTANT: only submit issues that are relevant to the code. We do not offer support for any deployments of the project here; make your way to the IRC channel in such cases.
Please make sure to skim through the existing issues, as your issue/request/etc. may have already been noted!

100
.github/workflows/codeql.yml vendored Normal file
View file

@ -0,0 +1,100 @@
# For most projects, this workflow file will not need changing; you simply need
# to commit it to your repository.
#
# You may wish to alter this file to override the set of languages analyzed,
# or to provide custom queries or build logic.
#
# ******** NOTE ********
# We have attempted to detect the languages in your repository. Please check
# the `language` matrix defined below to confirm you have the correct set of
# supported CodeQL languages.
#
name: "CodeQL Advanced"
on:
push:
branches: [ "master" ]
pull_request:
branches: [ "master" ]
schedule:
- cron: '37 6 * * 4'
jobs:
analyze:
name: Analyze (${{ matrix.language }})
# Runner size impacts CodeQL analysis time. To learn more, please see:
# - https://gh.io/recommended-hardware-resources-for-running-codeql
# - https://gh.io/supported-runners-and-hardware-resources
# - https://gh.io/using-larger-runners (GitHub.com only)
# Consider using larger runners or machines with greater resources for possible analysis time improvements.
runs-on: ${{ (matrix.language == 'swift' && 'macos-latest') || 'ubuntu-latest' }}
permissions:
# required for all workflows
security-events: write
# required to fetch internal or private CodeQL packs
packages: read
# only required for workflows in private repositories
actions: read
contents: read
strategy:
fail-fast: false
matrix:
include:
- language: javascript-typescript
build-mode: none
- language: python
build-mode: none
# CodeQL supports the following values keywords for 'language': 'c-cpp', 'csharp', 'go', 'java-kotlin', 'javascript-typescript', 'python', 'ruby', 'swift'
# Use `c-cpp` to analyze code written in C, C++ or both
# Use 'java-kotlin' to analyze code written in Java, Kotlin or both
# Use 'javascript-typescript' to analyze code written in JavaScript, TypeScript or both
# To learn more about changing the languages that are analyzed or customizing the build mode for your analysis,
# see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning.
# If you are analyzing a compiled language, you can modify the 'build-mode' for that language to customize how
# your codebase is analyzed, see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages
steps:
- name: Checkout repository
uses: actions/checkout@v4
# Add any setup steps before running the `github/codeql-action/init` action.
# This includes steps like installing compilers or runtimes (`actions/setup-node`
# or others). This is typically only required for manual builds.
# - name: Setup runtime (example)
# uses: actions/setup-example@v1
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v3
with:
languages: ${{ matrix.language }}
build-mode: ${{ matrix.build-mode }}
# If you wish to specify custom queries, you can do so here or in a config file.
# By default, queries listed here will override any specified in a config file.
# Prefix the list here with "+" to use these queries and those in the config file.
# For more details on CodeQL's query packs, refer to: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
# queries: security-extended,security-and-quality
# If the analyze step fails for one of the languages you are analyzing with
# "We were unable to automatically build your code", modify the matrix above
# to set the build mode to "manual" for that language. Then modify this step
# to build your code.
# ℹ️ Command-line programs to run using the OS shell.
# 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
- if: matrix.build-mode == 'manual'
shell: bash
run: |
echo 'If you are using a "manual" build mode for one or more of the' \
'languages you are analyzing, replace this with the commands to build' \
'your code, for example:'
echo ' make bootstrap'
echo ' make release'
exit 1
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v3
with:
category: "/language:${{matrix.language}}"

14
.gitignore vendored
View file

@ -14,16 +14,20 @@ __pycache__
# Databases
*.sql
test.db
/test.db
# Webserver
uwsgi.sock
/uwsgi.sock
# Application
install/*
config.py
/install/*
/config.py
/es_sync_config.json
/test_torrent_batch
torrents
# Build Output
nyaa/static/js/bootstrap-select.min.js
nyaa/static/js/main.min.js
# Other
*.swp

7
.travis.yml
View file

@ -1,9 +1,8 @@
language: python
python: "3.6"
python: "3.13"
dist: trusty
sudo: required
dist: jammy
matrix:
fast_finish: true
@ -14,7 +13,7 @@ services:
mysql
before_install:
- mysql -u root -e 'CREATE DATABASE nyaav2 DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;'
- mysql -u root -e 'CREATE DATABASE nyaav3 DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;'
install:
- pip install -r requirements.txt

117
README.md
View file

@ -1,47 +1,54 @@
# NyaaV2 [![Build Status](https://travis-ci.org/nyaadevs/nyaa.svg?branch=master)](https://travis-ci.org/nyaadevs/nyaa)
# NyaaV3 [![python](https://img.shields.io/badge/Python-3.13-3776AB.svg?style=flat&logo=python&logoColor=white)](https://www.python.org) ![Maintenance](https://img.shields.io/maintenance/yes/2025)
## Setting up for development
This project uses Python 3.6. There are features used that do not exist in 3.5, so make sure to use Python 3.6.
This guide also assumes you 1) are using Linux and 2) are somewhat capable with the commandline.
It's not impossible to run Nyaa on Windows, but this guide doesn't focus on that.
This project uses Python 3.13. The codebase has been updated from the original Python 3.7 version to use modern Python features and updated dependencies.
This guide assumes you are using Linux and are somewhat capable with the commandline.
Running Nyaa on Windows may be possible, but it's currently unsupported.
### Code Quality:
- Before we get any deeper, remember to follow PEP8 style guidelines and run `./dev.py lint` before committing to see a list of warnings/problems.
- You may also use `./dev.py fix && ./dev.py isort` to automatically fix some of the issues reported by the previous command.
### Major changes from NyaaV2
- Updated from Python 3.7 to Python 3.13
- Updated all dependencies to their latest versions
- Modernized code patterns for Flask 3.0 and SQLAlchemy 2.0
- Replaced deprecated Flask-Script, orderedset and `flask.Markup` with Flask CLI, orderly-set and markupsafe
- Implemented mail error handling
### Code Quality
- Before we get any deeper, remember to follow PEP8 style guidelines and run `python dev.py lint` before committing to see a list of warnings/problems.
- You may also use `python dev.py fix && python dev.py isort` to automatically fix some of the issues reported by the previous command.
- Other than PEP8, try to keep your code clean and easy to understand, as well. It's only polite!
### Running Tests
### Running tests
The `tests` folder contains tests for the the `nyaa` module and the webserver. To run the tests:
- Make sure that you are in the python virtual environment.
- Run `./dev.py test` while in the repository directory.
- Make sure that you are in the Python virtual environment.
- Run `python dev.py test` while in the repository directory.
### Setting up Pyenv
pyenv eases the use of different Python versions, and as not all Linux distros offer 3.6 packages, it's right up our alley.
- Install dependencies https://github.com/pyenv/pyenv/wiki/Common-build-problems
- Install `pyenv` https://github.com/pyenv/pyenv/blob/master/README.md#installation
- Install `pyenv-virtualenv` https://github.com/pyenv/pyenv-virtualenv/blob/master/README.md
- Install Python 3.6.1 with `pyenv` and create a virtualenv for the project:
- `pyenv install 3.6.1`
- `pyenv virtualenv 3.6.1 nyaa`
pyenv eases the use of different Python versions, and as not all Linux distros offer 3.13 packages, it's right up our alley.
- Install [dependencies](https://github.com/pyenv/pyenv/wiki/Common-build-problems)
- Install [pyenv](https://github.com/pyenv/pyenv/blob/master/README.md#installation)
- Install [pyenv-virtualenv](https://github.com/pyenv/pyenv-virtualenv/blob/master/README.md)
- Install Python 3.13 with `pyenv` and create a virtualenv for the project:
- `pyenv install 3.13.2`
- `pyenv virtualenv 3.13.2 nyaa`
- `pyenv activate nyaa`
- Install dependencies with `pip install -r requirements.txt`
- Copy `config.example.py` into `config.py`
- Change `SITE_FLAVOR` in your `config.py` depending on which instance you want to host
### Setting up MySQL/MariaDB database
You *may* use SQLite but the current support for it in this project is outdated and rather unsupported.
> [!WARNING]
> You *may* use SQLite but it is currently untested and unsupported.
- Enable `USE_MYSQL` flag in config.py
- Install latest mariadb by following instructions here https://downloads.mariadb.org/mariadb/repositories/
- Tested versions: `mysql Ver 15.1 Distrib 10.0.30-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2`
- Install MariaDB by following instructions [here](https://downloads.mariadb.org/mariadb/repositories/)
- Run the following commands logged in as your root db user (substitute for your own `config.py` values if desired):
- `CREATE USER 'test'@'localhost' IDENTIFIED BY 'test123';`
- `GRANT ALL PRIVILEGES ON *.* TO 'test'@'localhost';`
- `CREATE USER 'nyaauser'@'localhost' IDENTIFIED BY 'nyaapass';`
- `CREATE DATABASE nyaav3 DEFAULT CHARACTER SET utf8 COLLATE utf8_bin;`
- `GRANT ALL PRIVILEGES ON nyaav3.* TO 'nyaauser'@'localhost';`
- `FLUSH PRIVILEGES;`
- `CREATE DATABASE nyaav2 DEFAULT CHARACTER SET utf8 COLLATE utf8_bin;`
### Finishing up
- Run `python db_create.py` to create the database and import categories
- Follow the advice of `db_create.py` and run `./db_migrate.py stamp head` to mark the database version for Alembic
- ~~Follow the advice of `db_create.py` and run `python db_migrate.py stamp head` to mark the database version for Alembic~~
- Start the dev server with `python run.py`
- When you are finished developing, deactivate your virtualenv with `pyenv deactivate` or `source deactivate` (or just close your shell session)
@ -50,44 +57,35 @@ Continue below to learn about database migrations and enabling the advanced sear
## Database migrations
- Database migrations are done with [flask-Migrate](https://flask-migrate.readthedocs.io/), a wrapper around [Alembic](http://alembic.zzzcomputing.com/en/latest/).
> [!WARNING]
> The database migration feature has been updated but will no longer be supported in NyaaV3.
- Database migrations are done with [Flask-Migrate](https://flask-migrate.readthedocs.io/), a wrapper around [Alembic](http://alembic.zzzcomputing.com/en/latest/).
- The migration system has been updated to use Flask CLI instead of the deprecated Flask-Script.
- If someone has made changes in the database schema and included a new migration script:
- If your database has never been marked by Alembic (you're on a database from before the migrations), run `./db_migrate.py stamp head` before pulling the new migration script(s).
- If you already have the new scripts, check the output of `./db_migrate.py history` instead and choose a hash that matches your current database state, then run `./db_migrate.py stamp <hash>`.
- If your database has never been marked by Alembic (you're on a database from before the migrations), run `python db_migrate.py db stamp head` before pulling the new migration script(s).
- If you already have the new scripts, check the output of `python db_migrate.py db history` instead and choose a hash that matches your current database state, then run `python db_migrate.py db stamp <hash>`.
- Update your branch (eg. `git fetch && git rebase origin/master`)
- Run `./db_migrate.py upgrade head` to run the migration. Done!
- Run `python db_migrate.py db upgrade head` to run the migration. Done!
- If *you* have made a change in the database schema:
- Save your changes in `models.py` and ensure the database schema matches the previous version (ie. your new tables/columns are not added to the live database)
- Run `./db_migrate.py migrate -m "Short description of changes"` to automatically generate a migration script for the changes
- Run `python db_migrate.py db migrate -m "Short description of changes"` to automatically generate a migration script for the changes
- Check the script (`migrations/versions/...`) and make sure it works! Alembic may not able to notice all changes.
- Run `./db_migrate.py upgrade` to run the migration and verify the upgrade works.
- (Run `./db_migrate.py downgrade` to verify the downgrade works as well, then upgrade again)
- Run `python db_migrate.py db upgrade` to run the migration and verify the upgrade works.
- (Run `python db_migrate.py db downgrade` to verify the downgrade works as well, then upgrade again)
## Setting up and enabling Elasticsearch
### Installing Elasticsearch
- Install JDK with `sudo apt-get install openjdk-8-jdk`
- Install [Elasticsearch](https://www.elastic.co/downloads/elasticsearch)
- [From packages...](https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html)
- Install Elasticsearch
- [From packages](https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html)
- Enable the service:
- `sudo systemctl enable elasticsearch.service`
- `sudo systemctl start elasticsearch.service`
- or [simply extracting the archives and running the files](https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html), if you don't feel like permantently installing ES
- or [simply extracting the archives and running the files](https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html), if you don't feel like permanently installing ES
- Run `curl -XGET 'localhost:9200'` and make sure ES is running
- Optional: install [Kibana](https://www.elastic.co/products/kibana) as a search debug frontend for ES
### Setting up ES
- Run `./create_es.sh` to create the indices for the torrents: `nyaa` and `sukebei`
- The output should show `acknowledged: true` twice
- Stop the Nyaa app if you haven't already
- Run `python import_to_es.py` to import all the torrents (on nyaa and sukebei) into the ES indices.
- This may take some time to run if you have plenty of torrents in your database.
Enable the `USE_ELASTIC_SEARCH` flag in `config.py` and (re)start the application.
Elasticsearch should now be functional! The ES indices won't be updated "live" with the current setup, continue below for instructions on how to hook Elasticsearch up to MySQL binlog.
However, take note that binglog is not necessary for simple ES testing and development; you can simply run `import_to_es.py` from time to time to reindex all the torrents.
- Install [Kibana](https://www.elastic.co/products/kibana) as a search debug frontend for ES (*optional*)
### Enabling MySQL Binlogging
- Edit your MariaDB/MySQL server configuration and add the following under `[mariadb]`:
@ -103,6 +101,18 @@ However, take note that binglog is not necessary for simple ES testing and devel
- Verify that the result of `SHOW VARIABLES LIKE 'binlog_format';` is `ROW`
- Execute `GRANT REPLICATION SLAVE ON *.* TO 'username'@'localhost';` to allow your configured user access to the binlog
### Setting up ES
- Run `./create_es.sh` to create the indices for the torrents: `nyaa` and `sukebei`
- The output should show `acknowledged: true` twice
- Stop the Nyaa app if you haven't already
- Run `python import_to_es.py` to import all the torrents (on nyaa and sukebei) into the ES indices.
- This may take some time to run if you have plenty of torrents in your database.
Enable the `USE_ELASTIC_SEARCH` flag in `config.py` and (re)start the application.
Elasticsearch should now be functional! The ES indices won't be updated "live" with the current setup, continue below for instructions on how to hook Elasticsearch up to MySQL binlog.
However, take note that binglog is not necessary for simple ES testing and development; you can simply run `import_to_es.py` from time to time to reindex all the torrents.
### Setting up sync_es.py
`sync_es.py` keeps the Elasticsearch indices updated by reading the binlog and pushing the changes to the ES indices.
@ -113,3 +123,16 @@ However, take note that binglog is not necessary for simple ES testing and devel
You're done! The script should now be feeding updates from the database to Elasticsearch.
Take note, however, that the specified ES index refresh interval is 30 seconds, which may feel like a long time on local development. Feel free to adjust it or [poke Elasticsearch yourself!](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-refresh.html)
## License
This project is licensed under the GNU General Public License v3.0 (GPL-3.0). See the [LICENSE](LICENSE) file for more details.
## Disclaimer
> [!CAUTION]
> **This project was created as a learning experience, and while it's a torrent tracker, I can't control how people choose to use it.**
By using this software, you're agreeing to a few things:
- I'm not responsible for any legal issues that might come up from using this tracker, especially if it's used to share copyrighted content without permission.
- It's your responsibility to make sure you're following the laws in your area when using this software.
**Please use this project wisely and stay on the right side of the law.** Happy coding!

9
WSGI.py
View file

@ -1,11 +1,16 @@
#!/usr/bin/python3
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
WSGI entry point for the Nyaa application.
Compatible with Python 3.13.
"""
import gevent.monkey
gevent.monkey.patch_all()
from nyaa import create_app
from flask import Flask
app = create_app('config')
app: Flask = create_app('config')
if app.config['DEBUG']:
from werkzeug.debug import DebuggedApplication

101
config.example.py
View file

@ -1,4 +1,5 @@
import os
import re
DEBUG = True
@ -13,6 +14,16 @@ MAINTENANCE_MODE_MESSAGE = 'Site is currently in read-only maintenance mode.'
# Allow logging in during maintenance (without updating last login date)
MAINTENANCE_MODE_LOGINS = True
# Block *anonymous* uploads completely
RAID_MODE_LIMIT_UPLOADS = False
# Message prepended to the full error message (account.py)
RAID_MODE_UPLOADS_MESSAGE = 'Anonymous uploads are currently disabled.'
# Require manual activation for newly registered accounts
RAID_MODE_LIMIT_REGISTER = False
# Message prepended to the full error message (account.py)
RAID_MODE_REGISTER_MESSAGE = 'Registration is currently being limited.'
#############
## General ##
#############
@ -31,6 +42,12 @@ EXTERNAL_URLS = {'fap':'***', 'main':'***'}
CSRF_SESSION_KEY = '***'
SECRET_KEY = '***'
# Session cookie configuration
SESSION_COOKIE_NAME = 'nyaav3_session'
SESSION_COOKIE_SECURE = True
SESSION_COOKIE_HTTPONLY = True
SESSION_COOKIE_SAMESITE = 'Lax'
# Present a recaptcha for anonymous uploaders
USE_RECAPTCHA = False
# Require email validation
@ -44,13 +61,34 @@ ENABLE_SHOW_STATS = True
# Depends on email support!
ALLOW_PASSWORD_RESET = True
# A list of strings or compiled regexes to deny registering emails by.
# Regexes will be .search()'d against emails,
# while strings will be a simple 'string in email.lower()' check.
# Leave empty to disable the blacklist.
EMAIL_BLACKLIST = (
# Hotmail completely rejects "untrusted" emails,
# so it's less of a headache to blacklist them as users can't receive the mails anyway.
# (Hopefully) complete list of Microsoft email domains follows:
re.compile(r'(?i)@hotmail\.(co|co\.uk|com|de|dk|eu|fr|it|net|org|se)'),
re.compile(r'(?i)@live\.(co|co.uk|com|de|dk|eu|fr|it|net|org|se|no)'),
re.compile(r'(?i)@outlook\.(at|be|cl|co|co\.(id|il|nz|th)|com|com\.(ar|au|au|br|gr|pe|tr|vn)|cz|de|de|dk|dk|es|eu|fr|fr|hu|ie|in|it|it|jp|kr|lv|my|org|ph|pt|sa|se|sg|sk)'),
re.compile(r'(?i)@(msn\.com|passport\.(com|net))'),
# '@dodgydomain.tk'
)
EMAIL_SERVER_BLACKLIST = (
# Bad mailserver IPs here (MX server.com -> A mail.server.com > 11.22.33.44)
# '1.2.3.4', '11.22.33.44'
)
# Recaptcha keys (https://www.google.com/recaptcha)
RECAPTCHA_PUBLIC_KEY = '***'
RECAPTCHA_PRIVATE_KEY = '***'
BASE_DIR = os.path.abspath(os.path.dirname(__file__))
if USE_MYSQL:
SQLALCHEMY_DATABASE_URI = ('mysql://test:test123@localhost/nyaav2?charset=utf8mb4')
SQLALCHEMY_DATABASE_URI = ('mysql://test:test123@localhost/nyaav3?charset=utf8mb4')
else:
SQLALCHEMY_DATABASE_URI = (
'sqlite:///' + os.path.join(BASE_DIR, 'test.db') + '?check_same_thread=False')
@ -107,6 +145,10 @@ MINIMUM_ANONYMOUS_TORRENT_SIZE = 1 * 1024 * 1024
# Relies on USE_RECAPTCHA. Set to 0 to disable.
ACCOUNT_RECAPTCHA_AGE = 7 * 24 * 3600 # A week
# Seconds after which an IP is allowed to register another account
# (0 disables the limitation)
PER_IP_ACCOUNT_COOLDOWN = 24 * 3600
# Backup original .torrent uploads
BACKUP_TORRENT_FOLDER = 'torrents'
@ -117,6 +159,16 @@ BACKUP_TORRENT_FOLDER = 'torrents'
# How many results should a page contain. Applies to RSS as well.
RESULTS_PER_PAGE = 75
# How many pages we'll return at most
MAX_PAGES = 100
# How long and how many entries to cache for count queries
COUNT_CACHE_SIZE = 256
COUNT_CACHE_DURATION = 30
# Use baked queries for database search
USE_BAKED_SEARCH = False
# Use better searching with ElasticSearch
# See README.MD on setup!
USE_ELASTIC_SEARCH = False
@ -127,6 +179,8 @@ ENABLE_ELASTIC_SEARCH_HIGHLIGHT = False
ES_MAX_SEARCH_RESULT = 1000
# ES index name generally (nyaa or sukebei)
ES_INDEX_NAME = SITE_FLAVOR
# ES hosts
ES_HOSTS = ['localhost:9200']
################
## Commenting ##
@ -135,3 +189,48 @@ ES_INDEX_NAME = SITE_FLAVOR
# Time limit for editing a comment after it has been posted (seconds)
# Set to 0 to disable
EDITING_TIME_LIMIT = 0
# Whether to use Gravatar or just always use the default avatar
# (Useful if run as development instance behind NAT/firewall)
ENABLE_GRAVATAR = True
##########################
## Trusted Requirements ##
##########################
# Minimum number of uploads the user needs to have in order to apply for trusted
TRUSTED_MIN_UPLOADS = 10
# Minimum number of cumulative downloads the user needs to have across their
# torrents in order to apply for trusted
TRUSTED_MIN_DOWNLOADS = 10000
# Number of days an applicant needs to wait before re-applying
TRUSTED_REAPPLY_COOLDOWN = 90
###########
## Cache ##
###########
# Interesting types include "simple", "redis" and "uwsgi"
# See https://pythonhosted.org/Flask-Caching/#configuring-flask-caching
CACHE_TYPE = "simple"
# Maximum number of items the cache will store
# Only applies to "simple" and "filesystem" cache types
CACHE_THRESHOLD = 8192
# If you want to use redis, try this
# CACHE_TYPE = "redis"
# CACHE_REDIS_HOST = "127.0.0.1"
# CACHE_KEY_PREFIX = "catcache_"
###############
## Ratelimit ##
###############
# To actually make this work across multiple worker processes, use redis
# RATELIMIT_STORAGE_URL="redis://host:port"
RATELIMIT_KEY_PREFIX="nyaaratelimit_"
# Use this to show the commit hash in the footer (see layout.html)
# COMMIT_HASH="[enter your commit hash here]";

3
create_es.sh
View file

@ -1,5 +1,6 @@
#!/usr/bin/env bash
set -e
# create indicies named "nyaa" and "sukebei", these are hardcoded
# create indices named "nyaa" and "sukebei", these are hardcoded
curl -v -XPUT 'localhost:9200/nyaa?pretty' -H"Content-Type: application/yaml" --data-binary @es_mapping.yml
curl -v -XPUT 'localhost:9200/sukebei?pretty' -H"Content-Type: application/yaml" --data-binary @es_mapping.yml

34
db_create.py
View file

@ -1,12 +1,19 @@
#!/usr/bin/env python3
"""
Database creation script for Nyaa.
Compatible with Python 3.13 and SQLAlchemy 2.0.
"""
from typing import List, Tuple, Type
import sqlalchemy
from sqlalchemy import select
from nyaa import create_app, models
from nyaa.extensions import db
app = create_app('config')
NYAA_CATEGORIES = [
NYAA_CATEGORIES: List[Tuple[str, List[str]]] = [
('Anime', ['Anime Music Video', 'English-translated', 'Non-English-translated', 'Raw']),
('Audio', ['Lossless', 'Lossy']),
('Literature', ['English-translated', 'Non-English-translated', 'Raw']),
@ -16,13 +23,23 @@ NYAA_CATEGORIES = [
]
SUKEBEI_CATEGORIES = [
SUKEBEI_CATEGORIES: List[Tuple[str, List[str]]] = [
('Art', ['Anime', 'Doujinshi', 'Games', 'Manga', 'Pictures']),
('Real Life', ['Photobooks / Pictures', 'Videos']),
]
def add_categories(categories, main_class, sub_class):
def add_categories(categories: List[Tuple[str, List[str]]],
main_class: Type[models.MainCategoryBase],
sub_class: Type[models.SubCategoryBase]) -> None:
"""
Add categories to the database.
Args:
categories: List of tuples containing main category name and list of subcategory names
main_class: Main category model class
sub_class: Subcategory model class
"""
for main_cat_name, sub_cat_names in categories:
main_cat = main_class(name=main_cat_name)
for i, sub_cat_name in enumerate(sub_cat_names):
@ -36,19 +53,24 @@ if __name__ == '__main__':
# Test for the user table, assume db is empty if it's not created
database_empty = False
try:
models.User.query.first()
stmt = select(models.User).limit(1)
db.session.execute(stmt).scalar_one_or_none()
except (sqlalchemy.exc.ProgrammingError, sqlalchemy.exc.OperationalError):
database_empty = True
print('Creating all tables...')
db.create_all()
nyaa_category_test = models.NyaaMainCategory.query.first()
# Check if Nyaa categories exist
stmt = select(models.NyaaMainCategory).limit(1)
nyaa_category_test = db.session.execute(stmt).scalar_one_or_none()
if not nyaa_category_test:
print('Adding Nyaa categories...')
add_categories(NYAA_CATEGORIES, models.NyaaMainCategory, models.NyaaSubCategory)
sukebei_category_test = models.SukebeiMainCategory.query.first()
# Check if Sukebei categories exist
stmt = select(models.SukebeiMainCategory).limit(1)
sukebei_category_test = db.session.execute(stmt).scalar_one_or_none()
if not sukebei_category_test:
print('Adding Sukebei categories...')
add_categories(SUKEBEI_CATEGORIES, models.SukebeiMainCategory, models.SukebeiSubCategory)

23
db_migrate.py
View file

@ -1,9 +1,14 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Database migration script for Nyaa.
Compatible with Python 3.13 and Flask-Migrate 4.0.
"""
import sys
from typing import List
from flask_script import Manager
from flask_migrate import Migrate, MigrateCommand
from flask_migrate import Migrate
from flask.cli import FlaskGroup
from nyaa import create_app
from nyaa.extensions import db
@ -11,11 +16,17 @@ from nyaa.extensions import db
app = create_app('config')
migrate = Migrate(app, db)
manager = Manager(app)
manager.add_command("db", MigrateCommand)
def create_cli_app():
return app
cli = FlaskGroup(create_app=create_cli_app)
if __name__ == "__main__":
# Patch sys.argv to default to 'db'
sys.argv.insert(1, 'db')
if len(sys.argv) > 1 and sys.argv[1] not in ['--help', '-h']:
if sys.argv[1] not in ['db', 'routes', 'shell', 'run']:
args: List[str] = sys.argv.copy()
args.insert(1, 'db')
sys.argv = args
manager.run()
cli()

13
dev.py
View file

@ -4,8 +4,11 @@
This tool is designed to assist developers run common tasks, such as
checking the code for lint issues, auto fixing some lint issues and running tests.
It imports modules lazily (as-needed basis), so it runs faster!
Compatible with Python 3.13.
"""
import sys
from typing import List, Optional, Generator, Any, Union
LINT_PATHS = [
'nyaa/',
@ -14,14 +17,14 @@ LINT_PATHS = [
TEST_PATHS = ['tests']
def print_cmd(cmd, args):
def print_cmd(cmd: str, args: List[str]) -> None:
""" Prints the command and args as you would run them manually. """
print('Running: {0}\n'.format(
' '.join([('\'' + a + '\'' if ' ' in a else a) for a in [cmd] + args])))
sys.stdout.flush() # Make sure stdout is flushed before continuing.
def check_config_values():
def check_config_values() -> bool:
""" Verify that all max_line_length values match. """
import configparser
config = configparser.ConfigParser()
@ -32,7 +35,7 @@ def check_config_values():
autopep8 = config.get('pycodestyle', 'max_line_length', fallback=None)
isort = config.get('isort', 'line_length', fallback=None)
values = (v for v in (flake8, autopep8, isort) if v is not None)
values: Generator[str, None, None] = (v for v in (flake8, autopep8, isort) if v is not None)
found = next(values, False)
if not found:
print('Warning: No max line length setting set in setup.cfg.')
@ -44,7 +47,7 @@ def check_config_values():
return True
def print_help():
def print_help() -> int:
print('Nyaa Development Helper')
print('=======================\n')
print('Usage: {0} command [different arguments]'.format(sys.argv[0]))
@ -62,7 +65,7 @@ def print_help():
if __name__ == '__main__':
assert sys.version_info >= (3, 6), "Python 3.6 is required"
assert sys.version_info >= (3, 13), "Python 3.13 is required"
check_config_values()

152
es_mapping.yml
View file

@ -10,7 +10,6 @@ settings:
char_filter:
- my_char_filter
filter:
- standard
- lowercase
my_index_analyzer:
type: custom
@ -20,9 +19,15 @@ settings:
filter:
- resolution
- lowercase
- my_ngram
- word_delimit
- my_ngram
- trim_zero
- unique
# For exact matching - separate each character for substring matching + lowercase
exact_analyzer:
tokenizer: exact_tokenizer
filter:
- lowercase
# For matching full words longer than the ngram limit (15 chars)
my_fullword_index_analyzer:
type: custom
@ -32,13 +37,27 @@ settings:
filter:
- lowercase
- word_delimit
# These should be enough, as my_index_analyzer will match the rest
# Skip tokens shorter than N characters,
# since they're already indexed in the main field
- fullword_min
- unique
tokenizer:
# Splits input into characters, for exact substring matching
exact_tokenizer:
type: pattern
pattern: "(.)"
group: 1
filter:
my_ngram:
type: edgeNGram
type: edge_ngram
min_gram: 1
max_gram: 15
fullword_min:
type: length
# Remember to change this if you change the max_gram below!
min: 16
resolution:
type: pattern_capture
patterns: ["(\\d+)[xX](\\d+)"]
@ -46,9 +65,13 @@ settings:
type: pattern_capture
patterns: ["0*([0-9]*)"]
word_delimit:
type: word_delimiter
type: word_delimiter_graph
preserve_original: true
split_on_numerics: false
# https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-word-delimiter-graph-tokenfilter.html#word-delimiter-graph-tokenfilter-configure-parms
# since we're using "trim" filters downstream, otherwise
# you get weird lucene errors about startOffset
adjust_offsets: false
char_filter:
my_char_filter:
type: mapping
@ -58,66 +81,65 @@ settings:
# plus replicas don't really help either.
number_of_shards: 1
number_of_replicas : 0
mapper:
# disable elasticsearch's "helpful" autoschema
dynamic: false
# since we disabled the _all field, default query the
# name of the torrent.
query:
default_field: display_name
mappings:
torrent:
# don't want everything concatenated
_all:
enabled: false
properties:
id:
type: long
display_name:
# TODO could do a fancier tokenizer here to parse out the
# the scene convention of stuff in brackets, plus stuff like k-on
type: text
analyzer: my_index_analyzer
fielddata: true # Is this required?
fields:
# Multi-field for full-word matching (when going over ngram limits)
# Note: will have to be queried for, not automatic
fullword:
type: text
analyzer: my_fullword_index_analyzer
created_time:
type: date
# Only in the ES index for generating magnet links
info_hash:
enabled: false
filesize:
type: long
anonymous:
type: boolean
trusted:
type: boolean
remake:
type: boolean
complete:
type: boolean
hidden:
type: boolean
deleted:
type: boolean
has_torrent:
type: boolean
download_count:
type: long
leech_count:
type: long
seed_count:
type: long
comment_count:
type: long
# these ids are really only for filtering, thus keyword
uploader_id:
type: keyword
main_category_id:
type: keyword
sub_category_id:
type: keyword
# disable elasticsearch's "helpful" autoschema
dynamic: false
properties:
id:
type: long
display_name:
# TODO could do a fancier tokenizer here to parse out the
# the scene convention of stuff in brackets, plus stuff like k-on
type: text
analyzer: my_index_analyzer
fielddata: true # Is this required?
fields:
# Multi-field for full-word matching (when going over ngram limits)
# Note: will have to be queried for, not automatic
fullword:
type: text
analyzer: my_fullword_index_analyzer
# Stored for exact phrase matching
exact:
type: text
analyzer: exact_analyzer
created_time:
type: date
#
# Only in the ES index for generating magnet links
info_hash:
type: keyword
index: false
filesize:
type: long
anonymous:
type: boolean
trusted:
type: boolean
remake:
type: boolean
complete:
type: boolean
hidden:
type: boolean
deleted:
type: boolean
has_torrent:
type: boolean
download_count:
type: long
leech_count:
type: long
seed_count:
type: long
comment_count:
type: long
# these ids are really only for filtering, thus keyword
uploader_id:
type: keyword
main_category_id:
type: keyword
sub_category_id:
type: keyword

2
es_sync_config.example.json
View file

@ -4,7 +4,7 @@
"mysql_port": 3306,
"mysql_user": "nyaa",
"mysql_password": "some_password",
"database": "nyaav2",
"database": "nyaav3",
"internal_queue_depth": 10000,
"es_chunk_size": 10000,
"flush_interval": 5

10
import_to_es.py
View file

@ -1,6 +1,6 @@
#!/usr/bin/env python
"""
Bulk load torents from mysql into elasticsearch `nyaav2` index,
Bulk load torents from mysql into elasticsearch `nyaav3` index,
which is assumed to already exist.
This is a one-shot deal, so you'd either need to complement it
with a cron job or some binlog-reading thing (TODO)
@ -18,9 +18,12 @@ from nyaa import create_app, models
from nyaa.extensions import db
app = create_app('config')
es = Elasticsearch(timeout=30)
es = Elasticsearch(hosts=app.config['ES_HOSTS'], timeout=30)
ic = IndicesClient(es)
def pad_bytes(in_bytes, size):
return in_bytes + (b'\x00' * max(0, size - len(in_bytes)))
# turn into thing that elasticsearch indexes. We flatten in
# the stats (seeders/leechers) so we can order by them in es naturally.
# we _don't_ dereference uploader_id to the user's display name however,
@ -31,7 +34,6 @@ ic = IndicesClient(es)
def mk_es(t, index_name):
return {
"_id": t.id,
"_type": "torrent",
"_index": index_name,
"_source": {
# we're also indexing the id as a number so you can
@ -42,7 +44,7 @@ def mk_es(t, index_name):
"created_time": t.created_time,
# not analyzed but included so we can render magnet links
# without querying sql again.
"info_hash": t.info_hash.hex(),
"info_hash": pad_bytes(t.info_hash, 20).hex(),
"filesize": t.filesize,
"uploader_id": t.uploader_id,
"main_category_id": t.main_category_id,

2
info_dicts/.gitignore vendored Normal file
View file

@ -0,0 +1,2 @@
*
!.gitignore

1
migrations/README
View file

@ -1 +0,0 @@
Generic single-database configuration.

4
migrations/README.md Normal file
View file

@ -0,0 +1,4 @@
> [!WARNING]
> No longer supported in NyaaV3.
Generic single-database configuration.

47
migrations/versions/5cbcee17bece_add_trusted_applications.py Normal file
View file

@ -0,0 +1,47 @@
"""Add trusted applications
Revision ID: 5cbcee17bece
Revises: 8a6a7662eb37
Create Date: 2018-11-05 15:16:07.497898
"""
from alembic import op
import sqlalchemy as sa
import sqlalchemy_utils
# revision identifiers, used by Alembic.
revision = '5cbcee17bece'
down_revision = '8a6a7662eb37'
branch_labels = None
depends_on = None
def upgrade():
op.create_table('trusted_applications',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('submitter_id', sa.Integer(), nullable=False, index=True),
sa.Column('created_time', sa.DateTime(), nullable=True),
sa.Column('closed_time', sa.DateTime(), nullable=True),
sa.Column('why_want', sa.String(length=4000), nullable=False),
sa.Column('why_give', sa.String(length=4000), nullable=False),
sa.Column('status', sa.Integer(), nullable=False),
sa.ForeignKeyConstraint(['submitter_id'], ['users.id'], ),
sa.PrimaryKeyConstraint('id')
)
op.create_table('trusted_reviews',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('reviewer_id', sa.Integer(), nullable=False),
sa.Column('app_id', sa.Integer(), nullable=False),
sa.Column('created_time', sa.DateTime(), nullable=True),
sa.Column('comment', sa.String(length=4000), nullable=False),
sa.Column('recommendation', sa.Integer(), nullable=False),
sa.ForeignKeyConstraint(['app_id'], ['trusted_applications.id'], ),
sa.ForeignKeyConstraint(['reviewer_id'], ['users.id'], ),
sa.PrimaryKeyConstraint('id')
)
def downgrade():
op.drop_table('trusted_reviews')
op.drop_table('trusted_applications')

40
migrations/versions/6cc823948c5a_add_trackerapi.py Normal file
View file

@ -0,0 +1,40 @@
"""Add trackerapi table
Revision ID: 6cc823948c5a
Revises: b61e4f6a88cc
Create Date: 2018-02-11 20:57:15.244171
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = '6cc823948c5a'
down_revision = 'b61e4f6a88cc'
branch_labels = None
depends_on = None
def upgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('nyaa_trackerapi',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('info_hash', sa.BINARY(length=20), nullable=False),
sa.Column('method', sa.String(length=255), nullable=False),
sa.PrimaryKeyConstraint('id')
)
op.create_table('sukebei_trackerapi',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('info_hash', sa.BINARY(length=20), nullable=False),
sa.Column('method', sa.String(length=255), nullable=False),
sa.PrimaryKeyConstraint('id')
)
# ### end Alembic commands ###
def downgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.drop_table('sukebei_trackerapi')
op.drop_table('nyaa_trackerapi')
# ### end Alembic commands ###

39
migrations/versions/8a6a7662eb37_add_user_preferences_table.py Normal file
View file

@ -0,0 +1,39 @@
"""Add user preferences table
Revision ID: 8a6a7662eb37
Revises: f703f911d4ae
Create Date: 2018-11-20 17:02:26.408532
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = '8a6a7662eb37'
down_revision = 'f703f911d4ae'
branch_labels = None
depends_on = None
def upgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('user_preferences',
sa.Column('user_id', sa.Integer(), nullable=False),
sa.Column('hide_comments', sa.Boolean(), server_default=sa.sql.expression.false(), nullable=False),
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('user_id')
)
connection = op.get_bind()
print('Populating user_preferences...')
connection.execute(sa.sql.text('INSERT INTO user_preferences (user_id) SELECT id FROM users'))
print('Done.')
# ### end Alembic commands ###
def downgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.drop_table('user_preferences')
# ### end Alembic commands ###

57
migrations/versions/b61e4f6a88cc_del_torrents_info.py Normal file
View file

@ -0,0 +1,57 @@
"""Remove bencoded info dicts from mysql
Revision ID: b61e4f6a88cc
Revises: cf7bf6d0e6bd
Create Date: 2017-08-29 01:45:08.357936
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects import mysql
import sys
# revision identifiers, used by Alembic.
revision = 'b61e4f6a88cc'
down_revision = 'cf7bf6d0e6bd'
branch_labels = None
depends_on = None
def upgrade():
print("--- WARNING ---")
print("This migration drops the torrent_info tables.")
print("You will lose all of your .torrent files if you have not converted them beforehand.")
print("Use the migration script at utils/infodict_mysql2file.py")
print("Type OKAY and hit Enter to continue, CTRL-C to abort.")
print("--- WARNING ---")
try:
if input() != "OKAY":
sys.exit(1)
except KeyboardInterrupt:
sys.exit(1)
op.drop_table('sukebei_torrents_info')
op.drop_table('nyaa_torrents_info')
def downgrade():
op.create_table('nyaa_torrents_info',
sa.Column('info_dict', mysql.MEDIUMBLOB(), nullable=True),
sa.Column('torrent_id', mysql.INTEGER(display_width=11), autoincrement=False, nullable=False),
sa.ForeignKeyConstraint(['torrent_id'], ['nyaa_torrents.id'], name='nyaa_torrents_info_ibfk_1', ondelete='CASCADE'),
sa.PrimaryKeyConstraint('torrent_id'),
mysql_collate='utf8_bin',
mysql_default_charset='utf8',
mysql_engine='InnoDB',
mysql_row_format='COMPRESSED'
)
op.create_table('sukebei_torrents_info',
sa.Column('info_dict', mysql.MEDIUMBLOB(), nullable=True),
sa.Column('torrent_id', mysql.INTEGER(display_width=11), autoincrement=False, nullable=False),
sa.ForeignKeyConstraint(['torrent_id'], ['sukebei_torrents.id'], name='sukebei_torrents_info_ibfk_1', ondelete='CASCADE'),
sa.PrimaryKeyConstraint('torrent_id'),
mysql_collate='utf8_bin',
mysql_default_charset='utf8',
mysql_engine='InnoDB',
mysql_row_format='COMPRESSED'
)

40
migrations/versions/f69d7fec88d6_add_rangebans.py Normal file
View file

@ -0,0 +1,40 @@
"""add rangebans
Revision ID: f69d7fec88d6
Revises: 6cc823948c5a
Create Date: 2018-06-01 14:01:49.596007
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = 'f69d7fec88d6'
down_revision = '6cc823948c5a'
branch_labels = None
depends_on = None
def upgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('rangebans',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('cidr_string', sa.String(length=18), nullable=False),
sa.Column('masked_cidr', sa.BigInteger(), nullable=False),
sa.Column('mask', sa.BigInteger(), nullable=False),
sa.Column('enabled', sa.Boolean(), nullable=False),
sa.Column('temp', sa.DateTime(), nullable=True),
sa.PrimaryKeyConstraint('id')
)
op.create_index(op.f('ix_rangebans_mask'), 'rangebans', ['mask'], unique=False)
op.create_index(op.f('ix_rangebans_masked_cidr'), 'rangebans', ['masked_cidr'], unique=False)
# ### end Alembic commands ###
def downgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.drop_index(op.f('ix_rangebans_masked_cidr'), table_name='rangebans')
op.drop_index(op.f('ix_rangebans_mask'), table_name='rangebans')
op.drop_table('rangebans')
# ### end Alembic commands ###

28
migrations/versions/f703f911d4ae_add_registration_ip.py Normal file
View file

@ -0,0 +1,28 @@
"""add registration IP
Revision ID: f703f911d4ae
Revises: f69d7fec88d6
Create Date: 2018-07-09 13:04:50.652781
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = 'f703f911d4ae'
down_revision = 'f69d7fec88d6'
branch_labels = None
depends_on = None
def upgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.add_column('users', sa.Column('registration_ip', sa.Binary(), nullable=True))
# ### end Alembic commands ###
def downgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.drop_column('users', 'registration_ip')
# ### end Alembic commands ###

76
nyaa/__init__.py
View file

@ -1,22 +1,36 @@
import logging
import os
import string
from typing import Any, Optional
import flask
from flask import Flask
from flask_assets import Bundle # noqa F401
from nyaa.api_handler import api_blueprint
from nyaa.extensions import assets, db, fix_paginate, toolbar
from nyaa.extensions import assets, cache, db, fix_paginate, limiter, toolbar
from nyaa.template_utils import bp as template_utils_bp
from nyaa.template_utils import caching_url_for
from nyaa.utils import random_string
from nyaa.views import register_views
# Replace the Flask url_for with our cached version, since there's no real harm in doing so
# (caching_url_for has stored a reference to the OG url_for, so we won't recurse)
# Touching globals like this is a bit dirty, but nicer than replacing every url_for usage
flask.url_for = caching_url_for
def create_app(config):
def create_app(config: Any) -> Flask:
""" Nyaa app factory """
app = flask.Flask(__name__)
app.config.from_object(config)
# Session cookie configuration
app.config['SESSION_COOKIE_NAME'] = 'nyaav3_session'
app.config['SESSION_COOKIE_SECURE'] = True
app.config['SESSION_COOKIE_HTTPONLY'] = True
app.config['SESSION_COOKIE_SAMESITE'] = 'Lax'
# Don't refresh cookie each request
app.config['SESSION_REFRESH_EACH_REQUEST'] = False
@ -28,11 +42,24 @@ def create_app(config):
# Forbid caching
@app.after_request
def forbid_cache(request):
request.headers['Cache-Control'] = 'no-cache, no-store, must-revalidate, max-age=0'
request.headers['Pragma'] = 'no-cache'
request.headers['Expires'] = '0'
return request
def forbid_cache(response: flask.Response) -> flask.Response:
response.headers['Cache-Control'] = 'no-cache, no-store, must-revalidate, max-age=0'
response.headers['Pragma'] = 'no-cache'
response.headers['Expires'] = '0'
return response
# Add a timer header to the requests when debugging
# This gives us a simple way to benchmark requests off-app
import time
@app.before_request
def timer_before_request() -> None:
flask.g.request_start_time = time.time()
@app.after_request
def timer_after_request(response: flask.Response) -> flask.Response:
response.headers['X-Timer'] = str(time.time() - flask.g.request_start_time)
return response
else:
app.logger.setLevel(logging.WARNING)
@ -44,17 +71,17 @@ def create_app(config):
app.config['LOG_FILE'], maxBytes=10000, backupCount=1)
app.logger.addHandler(app.log_handler)
# Log errors and display a message to the user in production mdode
# Log errors and display a message to the user in production mode
if not app.config['DEBUG']:
@app.errorhandler(500)
def internal_error(exception):
def internal_error(exception: Exception) -> flask.Response:
random_id = random_string(8, string.ascii_uppercase + string.digits)
# Pst. Not actually unique, but don't tell anyone!
app.logger.error('Exception occurred! Unique ID: %s', random_id, exc_info=exception)
app.logger.error(f'Exception occurred! Unique ID: {random_id}', exc_info=exception)
markup_source = ' '.join([
'<strong>An error occurred!</strong>',
'Debug information has been logged.',
'Please pass along this ID: <kbd>{}</kbd>'.format(random_id)
f'Please pass along this ID: <kbd>{random_id}</kbd>'
])
flask.flash(flask.Markup(markup_source), 'danger')
@ -73,14 +100,29 @@ def create_app(config):
app.jinja_env.lstrip_blocks = True
app.jinja_env.trim_blocks = True
# The default jinja_env has the OG Flask url_for (from before we replaced it),
# so update the globals with our version
app.jinja_env.globals['url_for'] = flask.url_for
# Database
fix_paginate() # This has to be before the database is initialized
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
app.config['MYSQL_DATABASE_CHARSET'] = 'utf8mb4'
db.init_app(app)
# Import the fixed Ban.banned method
with app.app_context():
import nyaa.fixed_ban
# Assets
assets.init_app(app)
if hasattr(assets, '_named_bundles'):
assets._named_bundles = {} # Hack to fix state carrying over in tests
main_js = Bundle('js/main.js', filters='rjsmin', output='js/main.min.js')
bs_js = Bundle('js/bootstrap-select.js', filters='rjsmin',
output='js/bootstrap-select.min.js')
assets.register('main_js', main_js)
assets.register('bs_js', bs_js)
# css = Bundle('style.scss', filters='libsass',
# output='style.css', depends='**/*.scss')
# assets.register('style_all', css)
@ -90,4 +132,16 @@ def create_app(config):
app.register_blueprint(api_blueprint)
register_views(app)
# Pregenerate some URLs to avoid repeat url_for calls
if 'SERVER_NAME' in app.config and app.config['SERVER_NAME']:
with app.app_context():
url = flask.url_for('static', filename='img/avatar/default.png', _external=True)
app.config['DEFAULT_GRAVATAR_URL'] = url
# Cache
cache.init_app(app, config=app.config)
# Rate Limiting, reads app.config itself
limiter.init_app(app)
return app

140
nyaa/api_handler.py
View file

@ -1,13 +1,11 @@
import binascii
import functools
import json
import os.path
import re
import flask
from nyaa import backend, bencode, forms, models, utils
from nyaa.extensions import db
from nyaa import backend, forms, models
from nyaa.views.torrents import _create_upload_category_choices
api_blueprint = flask.Blueprint('api', __name__, url_prefix='/api')
@ -120,142 +118,6 @@ def v2_api_upload():
return flask.jsonify({'errors': mapped_errors}), 400
# #################################### TEMPORARY ####################################
from orderedset import OrderedSet # noqa: E402 isort:skip
@api_blueprint.route('/ghetto_import', methods=['POST'])
def ghetto_import():
if flask.request.remote_addr != '127.0.0.1':
return flask.error(403)
torrent_file = flask.request.files.get('torrent')
try:
torrent_dict = bencode.decode(torrent_file)
# field.data.close()
except (bencode.MalformedBencodeException, UnicodeError):
return 'Malformed torrent file', 500
try:
forms._validate_torrent_metadata(torrent_dict)
except AssertionError as e:
return 'Malformed torrent metadata ({})'.format(e.args[0]), 500
try:
tracker_found = forms._validate_trackers(torrent_dict) # noqa F841
except AssertionError as e:
return 'Malformed torrent trackers ({})'.format(e.args[0]), 500
bencoded_info_dict = bencode.encode(torrent_dict['info'])
info_hash = utils.sha1_hash(bencoded_info_dict)
# Check if the info_hash exists already in the database
torrent = models.Torrent.by_info_hash(info_hash)
if not torrent:
return 'This torrent does not exists', 500
if torrent.has_torrent:
return 'This torrent already has_torrent', 500
# Torrent is legit, pass original filename and dict along
torrent_data = forms.TorrentFileData(filename=os.path.basename(torrent_file.filename),
torrent_dict=torrent_dict,
info_hash=info_hash,
bencoded_info_dict=bencoded_info_dict)
# The torrent has been validated and is safe to access with ['foo'] etc - all relevant
# keys and values have been checked for (see UploadForm in forms.py for details)
info_dict = torrent_data.torrent_dict['info']
changed_to_utf8 = backend._replace_utf8_values(torrent_data.torrent_dict)
torrent_filesize = info_dict.get('length') or sum(
f['length'] for f in info_dict.get('files'))
# In case no encoding, assume UTF-8.
torrent_encoding = torrent_data.torrent_dict.get('encoding', b'utf-8').decode('utf-8')
# Store bencoded info_dict
torrent.info = models.TorrentInfo(info_dict=torrent_data.bencoded_info_dict)
torrent.has_torrent = True
# To simplify parsing the filelist, turn single-file torrent into a list
torrent_filelist = info_dict.get('files')
used_path_encoding = changed_to_utf8 and 'utf-8' or torrent_encoding
parsed_file_tree = dict()
if not torrent_filelist:
# If single-file, the root will be the file-tree (no directory)
file_tree_root = parsed_file_tree
torrent_filelist = [{'length': torrent_filesize, 'path': [info_dict['name']]}]
else:
# If multi-file, use the directory name as root for files
file_tree_root = parsed_file_tree.setdefault(
info_dict['name'].decode(used_path_encoding), {})
# Parse file dicts into a tree
for file_dict in torrent_filelist:
# Decode path parts from utf8-bytes
path_parts = [path_part.decode(used_path_encoding) for path_part in file_dict['path']]
filename = path_parts.pop()
current_directory = file_tree_root
for directory in path_parts:
current_directory = current_directory.setdefault(directory, {})
# Don't add empty filenames (BitComet directory)
if filename:
current_directory[filename] = file_dict['length']
parsed_file_tree = utils.sorted_pathdict(parsed_file_tree)
json_bytes = json.dumps(parsed_file_tree, separators=(',', ':')).encode('utf8')
torrent.filelist = models.TorrentFilelist(filelist_blob=json_bytes)
db.session.add(torrent)
db.session.flush()
# Store the users trackers
trackers = OrderedSet()
announce = torrent_data.torrent_dict.get('announce', b'').decode('ascii')
if announce:
trackers.add(announce)
# List of lists with single item
announce_list = torrent_data.torrent_dict.get('announce-list', [])
for announce in announce_list:
trackers.add(announce[0].decode('ascii'))
# Remove our trackers, maybe? TODO ?
# Search for/Add trackers in DB
db_trackers = OrderedSet()
for announce in trackers:
tracker = models.Trackers.by_uri(announce)
# Insert new tracker if not found
if not tracker:
tracker = models.Trackers(uri=announce)
db.session.add(tracker)
db.session.flush()
db_trackers.add(tracker)
# Store tracker refs in DB
for order, tracker in enumerate(db_trackers):
torrent_tracker = models.TorrentTrackers(torrent_id=torrent.id,
tracker_id=tracker.id, order=order)
db.session.add(torrent_tracker)
db.session.commit()
return 'success'
# ####################################### INFO #######################################
ID_PATTERN = '^[0-9]+$'
INFO_HASH_PATTERN = '^[0-9a-fA-F]{40}$' # INFO_HASH as string

133
nyaa/backend.py
View file

@ -1,21 +1,43 @@
import json
import os
import re
from datetime import datetime, timedelta
from ipaddress import ip_address
from urllib.parse import urlencode
from urllib.request import urlopen
import flask
from werkzeug import secure_filename
from werkzeug.utils import secure_filename
import sqlalchemy
from orderedset import OrderedSet
from orderly_set import OrderlySet
from nyaa import models, utils
from nyaa.extensions import db
app = flask.current_app
# Blacklists for _validate_torrent_filenames
# TODO: consider moving to config.py?
CHARACTER_BLACKLIST = [
'\u202E', # RIGHT-TO-LEFT OVERRIDE
]
FILENAME_BLACKLIST = [
# Windows reserved filenames
'con',
'nul',
'prn',
'aux',
'com0', 'com1', 'com2', 'com3', 'com4', 'com5', 'com6', 'com7', 'com8', 'com9',
'lpt0', 'lpt1', 'lpt2', 'lpt3', 'lpt4', 'lpt5', 'lpt6', 'lpt7', 'lpt8', 'lpt9',
]
# Invalid RSS characters regex, used to sanitize some strings
ILLEGAL_XML_CHARS_RE = re.compile(u'[\x00-\x08\x0b\x0c\x0e-\x1F\uD800-\uDFFF\uFFFE\uFFFF]')
def sanitize_string(string, replacement='\uFFFD'):
''' Simply replaces characters based on a regex '''
return ILLEGAL_XML_CHARS_RE.sub(replacement, string)
class TorrentExtraValidationException(Exception):
def __init__(self, errors={}):
@ -64,16 +86,14 @@ def _recursive_dict_iterator(source):
def _validate_torrent_filenames(torrent):
''' Checks path parts of a torrent's filetree against blacklisted characters,
returning False on rejection '''
# TODO Move to config.py
character_blacklist = [
'\u202E', # RIGHT-TO-LEFT OVERRIDE
]
''' Checks path parts of a torrent's filetree against blacklisted characters
and filenames, returning False on rejection '''
file_tree = json.loads(torrent.filelist.filelist_blob.decode('utf-8'))
for path_part, value in _recursive_dict_iterator(file_tree):
if any(True for c in character_blacklist if c in path_part):
if path_part.rsplit('.', 1)[0].lower() in FILENAME_BLACKLIST:
return False
if any(True for c in CHARACTER_BLACKLIST if c in path_part):
return False
return True
@ -119,7 +139,9 @@ def check_uploader_ratelimit(user):
def filter_uploader(query):
if user:
return query.filter(Torrent.user == user)
return query.filter(sqlalchemy.or_(
Torrent.user == user,
Torrent.uploader_ip == ip_address(flask.request.remote_addr).packed))
else:
return query.filter(Torrent.uploader_ip == ip_address(flask.request.remote_addr).packed)
@ -160,11 +182,23 @@ def handle_torrent_upload(upload_form, uploading_user=None, fromAPI=False):
upload_form.ratelimit.errors = ["You've gone over the upload ratelimit."]
raise TorrentExtraValidationException()
# Delete exisiting torrent which is marked as deleted
if not uploading_user:
if app.config['RAID_MODE_LIMIT_UPLOADS']:
# XXX TODO: rename rangebanned to something more generic
upload_form.rangebanned.errors = [app.config['RAID_MODE_UPLOADS_MESSAGE']]
raise TorrentExtraValidationException()
elif models.RangeBan.is_rangebanned(ip_address(flask.request.remote_addr).packed):
upload_form.rangebanned.errors = ["Your IP is banned from "
"uploading anonymously."]
raise TorrentExtraValidationException()
# Delete existing torrent which is marked as deleted
if torrent_data.db_id is not None:
models.Torrent.query.filter_by(id=torrent_data.db_id).delete()
old_torrent = models.Torrent.by_id(torrent_data.db_id)
db.session.delete(old_torrent)
db.session.commit()
_delete_cached_torrent_file(torrent_data.db_id)
# Delete physical file after transaction has been committed
_delete_info_dict(old_torrent)
# The torrent has been validated and is safe to access with ['foo'] etc - all relevant
# keys and values have been checked for (see UploadForm in forms.py for details)
@ -177,6 +211,11 @@ def handle_torrent_upload(upload_form, uploading_user=None, fromAPI=False):
information = (upload_form.information.data or '').strip()
description = (upload_form.description.data or '').strip()
# Sanitize fields
display_name = sanitize_string(display_name)
information = sanitize_string(information)
description = sanitize_string(description)
torrent_filesize = info_dict.get('length') or sum(
f['length'] for f in info_dict.get('files'))
@ -195,7 +234,14 @@ def handle_torrent_upload(upload_form, uploading_user=None, fromAPI=False):
uploader_ip=ip_address(flask.request.remote_addr).packed)
# Store bencoded info_dict
torrent.info = models.TorrentInfo(info_dict=torrent_data.bencoded_info_dict)
info_dict_path = torrent.info_dict_path
info_dict_dir = os.path.dirname(info_dict_path)
os.makedirs(info_dict_dir, exist_ok=True)
with open(info_dict_path, 'wb') as out_file:
out_file.write(torrent_data.bencoded_info_dict)
torrent.stats = models.Statistic()
torrent.has_torrent = True
@ -211,6 +257,10 @@ def handle_torrent_upload(upload_form, uploading_user=None, fromAPI=False):
# To do, automatically mark trusted if user is trusted unless user specifies otherwise
torrent.trusted = upload_form.is_trusted.data if can_mark_trusted else False
# Only allow mods to upload locked torrents
can_mark_locked = uploading_user and uploading_user.is_moderator
torrent.comment_locked = upload_form.is_comment_locked.data if can_mark_locked else False
# Set category ids
torrent.main_category_id, torrent.sub_category_id = \
upload_form.category.parsed_data.get_category_ids()
@ -254,7 +304,7 @@ def handle_torrent_upload(upload_form, uploading_user=None, fromAPI=False):
db.session.flush()
# Store the users trackers
trackers = OrderedSet()
trackers = OrderlySet()
announce = torrent_data.torrent_dict.get('announce', b'').decode('ascii')
if announce:
trackers.add(announce)
@ -269,12 +319,12 @@ def handle_torrent_upload(upload_form, uploading_user=None, fromAPI=False):
webseed_list = torrent_data.torrent_dict.get('url-list') or []
if isinstance(webseed_list, bytes):
webseed_list = [webseed_list] # qB doesn't contain a sole url in a list
webseeds = OrderedSet(webseed.decode('utf-8') for webseed in webseed_list)
webseeds = OrderlySet(webseed.decode('utf-8') for webseed in webseed_list)
# Remove our trackers, maybe? TODO ?
# Search for/Add trackers in DB
db_trackers = OrderedSet()
db_trackers = OrderlySet()
for announce in trackers:
tracker = models.Trackers.by_uri(announce)
@ -313,6 +363,9 @@ def handle_torrent_upload(upload_form, uploading_user=None, fromAPI=False):
# Before final commit, validate the torrent again
validate_torrent_post_upload(torrent, upload_form)
# Add to tracker whitelist
db.session.add(models.TrackerApi(torrent.info_hash, 'insert'))
db.session.commit()
# Store the actual torrent file as well
@ -321,8 +374,7 @@ def handle_torrent_upload(upload_form, uploading_user=None, fromAPI=False):
torrent_file.seek(0, 0)
torrent_dir = app.config['BACKUP_TORRENT_FOLDER']
if not os.path.exists(torrent_dir):
os.makedirs(torrent_dir)
os.makedirs(torrent_dir, exist_ok=True)
torrent_path = os.path.join(torrent_dir, '{}.{}'.format(
torrent.id, secure_filename(torrent_file.filename)))
@ -332,38 +384,7 @@ def handle_torrent_upload(upload_form, uploading_user=None, fromAPI=False):
return torrent
def tracker_api(info_hashes, method):
api_url = app.config.get('TRACKER_API_URL')
if not api_url:
return False
# Split list into at most 100 elements
chunk_size = 100
chunk_range = range(0, len(info_hashes), chunk_size)
chunked_info_hashes = (info_hashes[i:i + chunk_size] for i in chunk_range)
for info_hashes_chunk in chunked_info_hashes:
qs = [
('auth', app.config.get('TRACKER_API_AUTH')),
('method', method)
]
qs.extend(('info_hash', info_hash) for info_hash in info_hashes_chunk)
api_url += '?' + urlencode(qs)
try:
req = urlopen(api_url)
except:
return False
if req.status != 200:
return False
return True
def _delete_cached_torrent_file(torrent_id):
# Note: obviously temporary
cached_torrent = os.path.join(app.config['BASE_DIR'],
'torrent_cache', str(torrent_id) + '.torrent')
if os.path.exists(cached_torrent):
os.remove(cached_torrent)
def _delete_info_dict(torrent):
info_dict_path = torrent.info_dict_path
if os.path.exists(info_dict_path):
os.remove(info_dict_path)

4
nyaa/bencode.py
View file

@ -67,7 +67,7 @@ def _bencode_decode(file_object, decode_keys_as_utf8=True):
elif c == _B_END:
try:
return int(int_bytes.decode('utf8'))
except Exception as e:
except Exception:
raise create_ex('Unable to parse int')
# not a digit OR '-' in the middle of the int
@ -109,7 +109,7 @@ def _bencode_decode(file_object, decode_keys_as_utf8=True):
raise create_ex('Unexpected input while reading string length: ' + repr(c))
try:
str_len = int(str_len_bytes.decode())
except Exception as e:
except Exception:
raise create_ex('Unable to parse bytestring length')
bytestring = file_object.read(str_len)

99
nyaa/custom_pagination.py Normal file
View file

@ -0,0 +1,99 @@
from typing import Any, List, Optional, Sequence, TypeVar, Union
T = TypeVar('T')
class CustomPagination:
"""
A custom pagination class that mimics the interface of Flask-SQLAlchemy's Pagination
but doesn't rely on the _query_items method.
"""
def __init__(self, query: Any, page: int, per_page: int, total: int, items: List[T]):
"""
Initialize a new CustomPagination object.
Args:
query: The query object (not used, but kept for compatibility)
page: The current page number (1-indexed)
per_page: The number of items per page
total: The total number of items
items: The items on the current page
"""
self.query = query
self.page = page
self.per_page = per_page
self.total = total
self.items = items
# For compatibility with LimitedPagination
self.actual_count = total
@property
def has_prev(self) -> bool:
"""Return True if there is a previous page."""
return self.page > 1
@property
def has_next(self) -> bool:
"""Return True if there is a next page."""
return self.page < self.pages
@property
def pages(self) -> int:
"""The total number of pages."""
if self.per_page == 0 or self.total == 0:
return 0
return max(1, (self.total + self.per_page - 1) // self.per_page)
@property
def prev_num(self) -> Optional[int]:
"""The previous page number, or None if this is the first page."""
if self.has_prev:
return self.page - 1
return None
@property
def next_num(self) -> Optional[int]:
"""The next page number, or None if this is the last page."""
if self.has_next:
return self.page + 1
return None
@property
def first(self) -> int:
"""The number of the first item on the page, starting from 1, or 0 if there are no items."""
if not self.items:
return 0
return (self.page - 1) * self.per_page + 1
@property
def last(self) -> int:
"""The number of the last item on the page, starting from 1, inclusive, or 0 if there are no items."""
if not self.items:
return 0
return min(self.total, self.page * self.per_page)
def iter_pages(self, left_edge: int = 2, left_current: int = 2,
right_current: int = 5, right_edge: int = 2) -> Sequence[Optional[int]]:
"""
Yield page numbers for a pagination widget.
Skipped pages between the edges and middle are represented by a None.
"""
last = 0
for num in range(1, self.pages + 1):
if (num <= left_edge or
(num > self.page - left_current - 1 and num < self.page + right_current) or
num > self.pages - right_edge):
if last + 1 != num:
yield None
yield num
last = num
def __iter__(self):
"""Iterate over the items on the current page."""
return iter(self.items)
def __len__(self):
"""Return the number of items on the current page."""
return len(self.items)

67
nyaa/email.py
View file

@ -45,29 +45,58 @@ class EmailHolder(object):
def send_email(email_holder):
"""Send an email using the configured mail backend."""
mail_backend = app.config.get('MAIL_BACKEND')
if mail_backend == 'mailgun':
_send_mailgun(email_holder)
elif mail_backend == 'smtp':
_send_smtp(email_holder)
elif mail_backend:
# TODO: Do this in logging.error when we have that set up
print('Unknown mail backend:', mail_backend)
if not mail_backend:
app.logger.warning('No mail backend configured, skipping email send')
return False
try:
if mail_backend == 'mailgun':
success = _send_mailgun(email_holder)
elif mail_backend == 'smtp':
success = _send_smtp(email_holder)
else:
app.logger.error(f'Unknown mail backend: {mail_backend}')
return False
if not success:
app.logger.error(f'Failed to send email using {mail_backend} backend')
return False
app.logger.info(f'Email successfully sent using {mail_backend} backend')
return True
except Exception as e:
app.logger.error(f'Error sending email: {str(e)}')
return False
def _send_mailgun(email_holder):
mailgun_endpoint = app.config['MAILGUN_API_BASE'] + '/messages'
auth = ('api', app.config['MAILGUN_API_KEY'])
data = {
'from': app.config['MAIL_FROM_ADDRESS'],
'to': email_holder.format_recipient(),
'subject': email_holder.subject,
'text': email_holder.text,
'html': email_holder.html
}
r = requests.post(mailgun_endpoint, data=data, auth=auth)
# TODO real error handling?
assert r.status_code == 200
"""Send an email using Mailgun API with proper error handling."""
try:
mailgun_endpoint = app.config['MAILGUN_API_BASE'] + '/messages'
auth = ('api', app.config['MAILGUN_API_KEY'])
data = {
'from': app.config['MAIL_FROM_ADDRESS'],
'to': email_holder.format_recipient(),
'subject': email_holder.subject,
'text': email_holder.text,
'html': email_holder.html
}
r = requests.post(mailgun_endpoint, data=data, auth=auth)
if r.status_code != 200:
app.logger.error(f'Mailgun API error: {r.status_code} - {r.text}')
return False
return True
except Exception as e:
app.logger.error(f'Error sending email via Mailgun: {str(e)}')
return False
def _send_smtp(email_holder):

61
nyaa/extensions.py
View file

@ -1,19 +1,45 @@
import os.path
from typing import Any, Optional, Sequence, TypeVar, Union
from flask import abort
from flask.config import Config
from flask_assets import Environment
from flask_caching import Cache
from flask_debugtoolbar import DebugToolbarExtension
from flask_sqlalchemy import BaseQuery, Pagination, SQLAlchemy
from flask_limiter import Limiter
from flask_limiter.util import get_remote_address
from flask_sqlalchemy import SQLAlchemy
from flask_sqlalchemy.pagination import Pagination
from sqlalchemy.orm import Query
assets = Environment()
db = SQLAlchemy()
toolbar = DebugToolbarExtension()
cache = Cache()
limiter = Limiter(key_func=get_remote_address)
# Type variable for query results
T = TypeVar('T')
def fix_paginate():
class LimitedPagination(Pagination):
def __init__(self, actual_count: int, *args: Any, **kwargs: Any) -> None:
self.actual_count = actual_count
super().__init__(*args, **kwargs)
def paginate_faste(self, page=1, per_page=50, max_page=None, step=5, count_query=None):
def fix_paginate() -> None:
"""Add custom pagination method to SQLAlchemy Query."""
def paginate_faste(
self: Query[T],
page: int = 1,
per_page: int = 50,
max_page: Optional[int] = None,
step: int = 5,
count_query: Optional[Query[int]] = None
) -> LimitedPagination:
"""Custom pagination that supports max_page and count_query."""
if page < 1:
abort(404)
@ -26,25 +52,36 @@ def fix_paginate():
else:
total_query_count = self.count()
if total_query_count is None:
total_query_count = 0
actual_query_count = total_query_count
if max_page:
total_query_count = min(total_query_count, max_page * per_page)
# Grab items on current page
items = self.limit(per_page).offset((page - 1) * per_page).all()
if not items and page != 1:
abort(404)
return Pagination(self, page, per_page, total_query_count, items)
return LimitedPagination(actual_query_count, self, page, per_page, total_query_count,
items)
BaseQuery.paginate_faste = paginate_faste
# Monkey patch the Query class
setattr(Query, 'paginate_faste', paginate_faste)
def _get_config():
# Workaround to get an available config object before the app is initiallized
# Only needed/used in top-level and class statements
# https://stackoverflow.com/a/18138250/7597273
def _get_config() -> Config:
"""
Workaround to get an available config object before the app is initialized.
Only needed/used in top-level and class statements.
https://stackoverflow.com/a/18138250/7597273
"""
root_path = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
config = Config(root_path)
config.from_object('config')
return config
config_obj = Config(root_path)
config_obj.from_object('config')
return config_obj
config = _get_config()

26
nyaa/fixed_ban.py Normal file
View file

@ -0,0 +1,26 @@
from typing import Optional, Union
from sqlalchemy import or_, select
from nyaa.extensions import db
from nyaa.models import Ban
# Fix the banned method to return a query object instead of a list
@classmethod
def fixed_banned(cls, user_id: Optional[int], user_ip: Optional[bytes]):
"""Check if a user or IP is banned.
Returns a query object that can be further filtered or used with .first(), .all(), etc.
"""
if not user_id and not user_ip:
# Return an empty query that will return no results
return db.session.query(cls).filter(cls.id < 0)
clauses = []
if user_id:
clauses.append(cls.user_id == user_id)
if user_ip:
clauses.append(cls.user_ip == user_ip)
return db.session.query(cls).filter(or_(*clauses))
# Replace the original method with our fixed version
Ban.banned = fixed_banned

114
nyaa/forms.py
View file

@ -11,8 +11,13 @@ from wtforms import (BooleanField, HiddenField, PasswordField, SelectField, Stri
SubmitField, TextAreaField)
from wtforms.validators import (DataRequired, Email, EqualTo, Length, Optional, Regexp,
StopValidation, ValidationError)
# from wtforms.widgets import HTMLString # For DisabledSelectField
from markupsafe import Markup
from wtforms.widgets import Select as SelectWidget # For DisabledSelectField
from wtforms.widgets import HTMLString, html_params # For DisabledSelectField
from wtforms.widgets import html_params
import dns.exception
import dns.resolver
from nyaa import bencode, models, utils
from nyaa.extensions import config
@ -69,6 +74,59 @@ def upload_recaptcha_validator_shim(form, field):
return True
def register_email_blacklist_validator(form, field):
email_blacklist = app.config.get('EMAIL_BLACKLIST', [])
email = field.data.strip()
validation_exception = StopValidation('Blacklisted email provider')
for item in email_blacklist:
if isinstance(item, re.Pattern):
if item.search(email):
raise validation_exception
elif isinstance(item, str):
if item in email.lower():
raise validation_exception
else:
raise Exception('Unexpected email validator type {!r} ({!r})'.format(type(item), item))
return True
def register_email_server_validator(form, field):
server_blacklist = app.config.get('EMAIL_SERVER_BLACKLIST', [])
if not server_blacklist:
return True
validation_exception = StopValidation('Blacklisted email provider')
email = field.data.strip()
email_domain = email.split('@', 1)[-1]
try:
# Query domain MX records
mx_records = list(dns.resolver.query(email_domain, 'MX'))
except dns.exception.DNSException:
app.logger.error('Unable to query MX records for email: %s - ignoring',
email, exc_info=False)
return True
for mx_record in mx_records:
try:
# Query mailserver A records
a_records = list(dns.resolver.query(mx_record.exchange))
for a_record in a_records:
# Check for address in blacklist
if a_record.address in server_blacklist:
app.logger.warning('Rejected email %s due to blacklisted mailserver (%s, %s)',
email, a_record.address, mx_record.exchange)
raise validation_exception
except dns.exception.DNSException:
app.logger.warning('Failed to query A records for mailserver: %s (%s) - ignoring',
mx_record.exchange, email, exc_info=False)
return True
_username_validator = Regexp(
r'^[a-zA-Z0-9_\-]+$',
message='Your username must only consist of alphanumerics and _- (a-zA-Z0-9_-)')
@ -105,14 +163,16 @@ class RegisterForm(FlaskForm):
DataRequired(),
Length(min=3, max=32),
stop_on_validation_error(_username_validator),
Unique(User, User.username, 'Username not availiable')
Unique(User, User.username, 'Username not available')
])
email = StringField('Email address', [
Email(),
DataRequired(),
Length(min=5, max=128),
Unique(User, User.email, 'Email already in use by another account')
register_email_blacklist_validator,
Unique(User, User.email, 'Email already in use by another account'),
register_email_server_validator
])
password = PasswordField('Password', [
@ -146,6 +206,10 @@ class ProfileForm(FlaskForm):
])
password_confirm = PasswordField('Repeat New Password')
hide_comments = BooleanField('Hide comments by default')
authorized_submit = SubmitField('Update')
submit_settings = SubmitField('Update')
# Classes for a SelectField that can be set to disable options (id, name, disabled)
@ -160,7 +224,7 @@ class DisabledSelectWidget(SelectWidget):
extra = disabled and {'disabled': ''} or {}
html.append(self.render_option(val, label, selected, **extra))
html.append('</select>')
return HTMLString(''.join(html))
return Markup(''.join(html))
class DisabledSelectField(SelectField):
@ -187,6 +251,8 @@ class CommentForm(FlaskForm):
DataRequired(message='Comment must not be empty.')
])
recaptcha = RecaptchaField(validators=[upload_recaptcha_validator_shim])
class InlineButtonWidget(object):
"""
@ -200,7 +266,7 @@ class InlineButtonWidget(object):
kwargs.setdefault('type', self.input_type)
if not label:
label = field.label.text
return HTMLString('<button %s>' % self.html_params(name=field.name, **kwargs) + label)
return Markup('<button %s>' % self.html_params(name=field.name, **kwargs) + label)
class StringSubmitField(StringField):
@ -239,11 +305,11 @@ class EditForm(FlaskForm):
field.parsed_data = cat
is_hidden = BooleanField('Hidden')
is_deleted = BooleanField('Deleted')
is_remake = BooleanField('Remake')
is_anonymous = BooleanField('Anonymous')
is_complete = BooleanField('Complete')
is_trusted = BooleanField('Trusted')
is_comment_locked = BooleanField('Lock Comments')
information = StringField('Information', [
Length(max=255, message='Information must be at most %(max)d characters long.')
@ -265,7 +331,6 @@ class DeleteForm(FlaskForm):
class BanForm(FlaskForm):
ban_user = SubmitField("Delete & Ban and Ban User")
ban_userip = SubmitField("Delete & Ban and Ban User+IP")
nuke = SubmitField("Delete & Ban all torrents")
unban = SubmitField("Unban")
_validator = DataRequired()
@ -280,6 +345,11 @@ class BanForm(FlaskForm):
])
class NukeForm(FlaskForm):
nuke_torrents = SubmitField("\U0001F4A3 Nuke Torrents")
nuke_comments = SubmitField("\U0001F4A3 Nuke Comments")
class UploadForm(FlaskForm):
torrent_file = FileField('Torrent file', [
FileRequired()
@ -316,6 +386,7 @@ class UploadForm(FlaskForm):
is_anonymous = BooleanField('Anonymous')
is_complete = BooleanField('Complete')
is_trusted = BooleanField('Trusted')
is_comment_locked = BooleanField('Lock Comments')
information = StringField('Information', [
Length(max=255, message='Information must be at most %(max)d characters long.')
@ -325,6 +396,7 @@ class UploadForm(FlaskForm):
])
ratelimit = HiddenField()
rangebanned = HiddenField()
def validate_torrent_file(form, field):
# Decode and ensure data is bencoded data
@ -384,6 +456,7 @@ class UploadForm(FlaskForm):
class UserForm(FlaskForm):
user_class = SelectField('Change User Class')
activate_user = SubmitField('Activate User')
def validate_user_class(form, field):
if not field.data:
@ -415,6 +488,33 @@ class ReportActionForm(FlaskForm):
report = HiddenField()
class TrustedForm(FlaskForm):
why_give_trusted = TextAreaField('Why do you think you should be given trusted status?', [
Length(min=32, max=4000,
message='Please explain why you think you should be given trusted status in at '
'least %(min)d but less than %(max)d characters.'),
DataRequired('Please fill out all of the fields in the form.')
])
why_want_trusted = TextAreaField('Why do you want to become a trusted user?', [
Length(min=32, max=4000,
message='Please explain why you want to become a trusted user in at least %(min)d '
'but less than %(max)d characters.'),
DataRequired('Please fill out all of the fields in the form.')
])
class TrustedReviewForm(FlaskForm):
comment = TextAreaField('Comment',
[Length(min=8, max=4000, message='Please provide a comment')])
recommendation = SelectField(choices=[('abstain', 'Abstain'), ('reject', 'Reject'),
('accept', 'Accept')])
class TrustedDecisionForm(FlaskForm):
accept = SubmitField('Accept')
reject = SubmitField('Reject')
def _validate_trackers(torrent_dict, tracker_to_check_for=None):
announce = torrent_dict.get('announce')
assert announce is not None, 'no tracker in torrent'

441
nyaa/models.py
View file

@ -1,17 +1,20 @@
import base64
import os.path
import re
from datetime import datetime
from enum import Enum, IntEnum
from hashlib import md5
from ipaddress import ip_address
from typing import Any, Dict, List, Optional, Tuple, Union
from urllib.parse import unquote as unquote_url
from urllib.parse import urlencode
import flask
from markupsafe import escape as escape_markup
from sqlalchemy import ForeignKeyConstraint, Index
from sqlalchemy import ForeignKeyConstraint, Index, func, select
from sqlalchemy.ext import declarative
from sqlalchemy.ext.hybrid import hybrid_property
from sqlalchemy_fulltext import FullText
from sqlalchemy_utils import ChoiceType, EmailType, PasswordType
@ -29,7 +32,7 @@ if config['USE_MYSQL']:
COL_UTF8MB4_BIN = 'utf8mb4_bin'
COL_ASCII_GENERAL_CI = 'ascii_general_ci'
else:
BinaryType = db.Binary
BinaryType = db.LargeBinary
TextType = db.String
MediumBlobType = db.BLOB
COL_UTF8_GENERAL_CI = 'NOCASE'
@ -46,23 +49,29 @@ class DeclarativeHelperBase(object):
__tablename__ and providing class methods for renaming references. '''
# See http://docs.sqlalchemy.org/en/latest/orm/extensions/declarative/api.html
__tablename_base__ = None
__flavor__ = None
__tablename_base__: Optional[str] = None
__flavor__: Optional[str] = None
@classmethod
def _table_prefix_string(cls):
def _table_prefix_string(cls) -> str:
if cls.__flavor__ is None:
raise ValueError("__flavor__ must be set")
return cls.__flavor__.lower() + '_'
@classmethod
def _table_prefix(cls, table_name):
def _table_prefix(cls, table_name: str) -> str:
return cls._table_prefix_string() + table_name
@classmethod
def _flavor_prefix(cls, table_name):
def _flavor_prefix(cls, table_name: str) -> str:
if cls.__flavor__ is None:
raise ValueError("__flavor__ must be set")
return cls.__flavor__ + table_name
@declarative.declared_attr
def __tablename__(cls):
def __tablename__(cls) -> str:
if cls.__tablename_base__ is None:
raise ValueError("__tablename_base__ must be set")
return cls._table_prefix(cls.__tablename_base__)
@ -70,22 +79,22 @@ class FlagProperty(object):
''' This class will act as a wrapper between the given flag and the class's
flag collection. '''
def __init__(self, flag, flags_attr='flags'):
def __init__(self, flag: int, flags_attr: str = 'flags'):
self._flag = flag
self._flags_attr_name = flags_attr
def _get_flags(self, instance):
def _get_flags(self, instance: Any) -> int:
return getattr(instance, self._flags_attr_name)
def _set_flags(self, instance, value):
def _set_flags(self, instance: Any, value: int) -> None:
return setattr(instance, self._flags_attr_name, value)
def __get__(self, instance, owner_class):
def __get__(self, instance: Any, owner_class: Any) -> bool:
if instance is None:
raise AttributeError()
return bool(self._get_flags(instance) & self._flag)
def __set__(self, instance, value):
def __set__(self, instance: Any, value: bool) -> None:
new_flags = (self._get_flags(instance) & ~self._flag) | (bool(value) and self._flag)
self._set_flags(instance, new_flags)
@ -99,6 +108,7 @@ class TorrentFlags(IntEnum):
COMPLETE = 16
DELETED = 32
BANNED = 64
COMMENT_LOCKED = 128
class TorrentBase(DeclarativeHelperBase):
@ -121,7 +131,7 @@ class TorrentBase(DeclarativeHelperBase):
# Even though this is same for both tables, declarative requires this
return db.Column(db.Integer, db.ForeignKey('users.id'), nullable=True)
uploader_ip = db.Column(db.Binary(length=16), default=None, nullable=True)
uploader_ip = db.Column(db.LargeBinary(length=16), default=None, nullable=True)
has_torrent = db.Column(db.Boolean, nullable=False, default=False)
comment_count = db.Column(db.Integer, default=0, nullable=False, index=True)
@ -170,11 +180,6 @@ class TorrentBase(DeclarativeHelperBase):
backref='torrents', lazy="joined",
primaryjoin=join_sql.format(cls.__flavor__))
@declarative.declared_attr
def info(cls):
return db.relationship(cls._flavor_prefix('TorrentInfo'), uselist=False,
cascade="all, delete-orphan", back_populates='torrent')
@declarative.declared_attr
def filelist(cls):
return db.relationship(cls._flavor_prefix('TorrentFilelist'), uselist=False,
@ -189,7 +194,7 @@ class TorrentBase(DeclarativeHelperBase):
@declarative.declared_attr
def trackers(cls):
return db.relationship(cls._flavor_prefix('TorrentTrackers'), uselist=True,
cascade="all, delete-orphan", lazy='select',
cascade="all, delete-orphan",
order_by=cls._flavor_prefix('TorrentTrackers.order'))
@declarative.declared_attr
@ -200,10 +205,23 @@ class TorrentBase(DeclarativeHelperBase):
def __repr__(self):
return '<{0} #{1.id} \'{1.display_name}\' {1.filesize}b>'.format(type(self).__name__, self)
def update_comment_count(self):
self.comment_count = Comment.query.filter_by(torrent_id=self.id).count()
def update_comment_count(self) -> int:
"""Update the comment count for this torrent and return the new count."""
stmt = select(func.count(Comment.id)).filter_by(torrent_id=self.id)
result = db.session.execute(stmt).scalar_one_or_none() or 0
self.comment_count = result
return self.comment_count
@classmethod
def update_comment_count_db(cls, torrent_id: int) -> None:
"""Update the comment count in the database for the given torrent ID."""
stmt = select(func.count(Comment.id)).filter_by(torrent_id=torrent_id)
count = db.session.execute(stmt).scalar_one_or_none() or 0
# Use the new update() style
stmt = db.update(cls).filter_by(id=torrent_id).values(comment_count=count)
db.session.execute(stmt)
@property
def created_utc_timestamp(self):
''' Returns a UTC POSIX timestamp, as seconds '''
@ -225,10 +243,18 @@ class TorrentBase(DeclarativeHelperBase):
invalid_url_characters = '<>"'
# Check if url contains invalid characters
if not any(c in url for c in invalid_url_characters):
return '<a href="{0}">{1}</a>'.format(url, escape_markup(unquote_url(url)))
return('<a rel="noopener noreferrer nofollow" '
'href="{0}">{1}</a>'.format(url, escape_markup(unquote_url(url))))
# Escaped
return escape_markup(self.information)
@property
def info_dict_path(self):
''' Returns a path to the info_dict file in form of 'info_dicts/aa/bb/aabbccddee...' '''
info_hash = self.info_hash_as_hex
return os.path.join(app.config['BASE_DIR'], 'info_dicts',
info_hash[0:2], info_hash[2:4], info_hash)
@property
def info_hash_as_b32(self):
return base64.b32encode(self.info_hash).decode('utf-8')
@ -255,19 +281,25 @@ class TorrentBase(DeclarativeHelperBase):
trusted = FlagProperty(TorrentFlags.TRUSTED)
remake = FlagProperty(TorrentFlags.REMAKE)
complete = FlagProperty(TorrentFlags.COMPLETE)
comment_locked = FlagProperty(TorrentFlags.COMMENT_LOCKED)
# Class methods
@classmethod
def by_id(cls, id):
return cls.query.get(id)
def by_id(cls, id: int) -> Optional['TorrentBase']:
"""Get a torrent by its ID."""
stmt = select(cls).filter_by(id=id)
return db.session.execute(stmt).scalar_one_or_none()
@classmethod
def by_info_hash(cls, info_hash):
return cls.query.filter_by(info_hash=info_hash).first()
def by_info_hash(cls, info_hash: bytes) -> Optional['TorrentBase']:
"""Get a torrent by its info hash."""
stmt = select(cls).filter_by(info_hash=info_hash)
return db.session.execute(stmt).scalar_one_or_none()
@classmethod
def by_info_hash_hex(cls, info_hash_hex):
def by_info_hash_hex(cls, info_hash_hex: str) -> Optional['TorrentBase']:
"""Get a torrent by its hex-encoded info hash."""
info_hash_bytes = bytearray.fromhex(info_hash_hex)
return cls.by_info_hash(info_hash_bytes)
@ -290,22 +322,6 @@ class TorrentFilelistBase(DeclarativeHelperBase):
back_populates='filelist')
class TorrentInfoBase(DeclarativeHelperBase):
__tablename_base__ = 'torrents_info'
__table_args__ = {'mysql_row_format': 'COMPRESSED'}
@declarative.declared_attr
def torrent_id(cls):
return db.Column(db.Integer, db.ForeignKey(
cls._table_prefix('torrents.id'), ondelete="CASCADE"), primary_key=True)
info_dict = db.Column(MediumBlobType, nullable=True)
@declarative.declared_attr
def torrent(cls):
return db.relationship(cls._flavor_prefix('Torrent'), uselist=False, back_populates='info')
class StatisticBase(DeclarativeHelperBase):
__tablename_base__ = 'statistics'
@ -335,8 +351,10 @@ class Trackers(db.Model):
disabled = db.Column(db.Boolean, nullable=False, default=False)
@classmethod
def by_uri(cls, uri):
return cls.query.filter_by(uri=uri).first()
def by_uri(cls, uri: str) -> Optional['Trackers']:
"""Get a tracker by its URI."""
stmt = select(cls).filter_by(uri=uri)
return db.session.execute(stmt).scalar_one_or_none()
class TorrentTrackersBase(DeclarativeHelperBase):
@ -359,8 +377,10 @@ class TorrentTrackersBase(DeclarativeHelperBase):
return db.relationship('Trackers', uselist=False, lazy='joined')
@classmethod
def by_torrent_id(cls, torrent_id):
return cls.query.filter_by(torrent_id=torrent_id).order_by(cls.order.desc())
def by_torrent_id(cls, torrent_id: int) -> List['TorrentTrackersBase']:
"""Get all trackers for a torrent, ordered by their order field."""
stmt = select(cls).filter_by(torrent_id=torrent_id).order_by(cls.order.desc())
return db.session.execute(stmt).scalars().all()
class MainCategoryBase(DeclarativeHelperBase):
@ -385,8 +405,10 @@ class MainCategoryBase(DeclarativeHelperBase):
return '_'.join(str(x) for x in self.get_category_ids())
@classmethod
def by_id(cls, id):
return cls.query.get(id)
def by_id(cls, id: int) -> Optional['MainCategoryBase']:
"""Get a main category by its ID."""
stmt = select(cls).filter_by(id=id)
return db.session.execute(stmt).scalar_one_or_none()
class SubCategoryBase(DeclarativeHelperBase):
@ -414,8 +436,10 @@ class SubCategoryBase(DeclarativeHelperBase):
return '_'.join(str(x) for x in self.get_category_ids())
@classmethod
def by_category_ids(cls, main_cat_id, sub_cat_id):
return cls.query.get((sub_cat_id, main_cat_id))
def by_category_ids(cls, main_cat_id: int, sub_cat_id: int) -> Optional['SubCategoryBase']:
"""Get a subcategory by its main category ID and subcategory ID."""
stmt = select(cls).filter_by(id=sub_cat_id, main_category_id=main_cat_id)
return db.session.execute(stmt).scalar_one_or_none()
class CommentBase(DeclarativeHelperBase):
@ -441,6 +465,11 @@ class CommentBase(DeclarativeHelperBase):
return db.relationship('User', uselist=False,
back_populates=cls._table_prefix('comments'), lazy="joined")
@declarative.declared_attr
def torrent(cls):
return db.relationship(cls._flavor_prefix('Torrent'), uselist=False,
back_populates='comments')
def __repr__(self):
return '<Comment %r>' % self.id
@ -491,7 +520,8 @@ class User(db.Model):
created_time = db.Column(db.DateTime(timezone=False), default=datetime.utcnow)
last_login_date = db.Column(db.DateTime(timezone=False), default=None, nullable=True)
last_login_ip = db.Column(db.Binary(length=16), default=None, nullable=True)
last_login_ip = db.Column(db.LargeBinary(length=16), default=None, nullable=True)
registration_ip = db.Column(db.LargeBinary(length=16), default=None, nullable=True)
nyaa_torrents = db.relationship('NyaaTorrent', back_populates='user', lazy='dynamic')
nyaa_comments = db.relationship('NyaaComment', back_populates='user', lazy='dynamic')
@ -501,6 +531,8 @@ class User(db.Model):
bans = db.relationship('Ban', uselist=True, foreign_keys='Ban.user_id')
preferences = db.relationship('UserPreferences', back_populates='user', uselist=False)
def __init__(self, username, email, password):
self.username = username
self.email = email
@ -522,19 +554,27 @@ class User(db.Model):
return all(checks)
def gravatar_url(self):
# from http://en.gravatar.com/site/implement/images/python/
params = {
# Image size (https://en.gravatar.com/site/implement/images/#size)
's': 120,
# Default image (https://en.gravatar.com/site/implement/images/#default-image)
'd': flask.url_for('static', filename='img/avatar/default.png', _external=True),
# Image rating (https://en.gravatar.com/site/implement/images/#rating)
# Nyaa: PG-rated, Sukebei: X-rated
'r': 'pg' if app.config['SITE_FLAVOR'] == 'nyaa' else 'x',
}
# construct the url
return 'https://www.gravatar.com/avatar/{}?{}'.format(
md5(self.email.encode('utf-8').lower()).hexdigest(), urlencode(params))
if 'DEFAULT_GRAVATAR_URL' in app.config:
default_url = app.config['DEFAULT_GRAVATAR_URL']
else:
default_url = flask.url_for('static', filename='img/avatar/default.png',
_external=True)
if app.config['ENABLE_GRAVATAR']:
# from http://en.gravatar.com/site/implement/images/python/
params = {
# Image size (https://en.gravatar.com/site/implement/images/#size)
's': 120,
# Default image (https://en.gravatar.com/site/implement/images/#default-image)
'd': default_url,
# Image rating (https://en.gravatar.com/site/implement/images/#rating)
# Nyaa: PG-rated, Sukebei: X-rated
'r': 'pg' if app.config['SITE_FLAVOR'] == 'nyaa' else 'x',
}
# construct the url
return 'https://www.gravatar.com/avatar/{}?{}'.format(
md5(self.email.encode('utf-8').lower()).hexdigest(), urlencode(params))
else:
return default_url
@property
def userlevel_str(self):
@ -578,23 +618,32 @@ class User(db.Model):
if self.last_login_ip:
return str(ip_address(self.last_login_ip))
@classmethod
def by_id(cls, id):
return cls.query.get(id)
@property
def reg_ip_string(self):
if self.registration_ip:
return str(ip_address(self.registration_ip))
@classmethod
def by_username(cls, username):
def by_id(cls, id: int) -> Optional['User']:
"""Get a user by their ID."""
stmt = select(cls).filter_by(id=id)
return db.session.execute(stmt).scalar_one_or_none()
@classmethod
def by_username(cls, username: str) -> Optional['User']:
"""Get a user by their username."""
def isascii(s): return len(s) == len(s.encode())
if not isascii(username):
return None
user = cls.query.filter_by(username=username).first()
return user
stmt = select(cls).filter_by(username=username)
return db.session.execute(stmt).scalar_one_or_none()
@classmethod
def by_email(cls, email):
user = cls.query.filter_by(email=email).first()
return user
def by_email(cls, email: str) -> Optional['User']:
"""Get a user by their email."""
stmt = select(cls).filter_by(email=email)
return db.session.execute(stmt).scalar_one_or_none()
@classmethod
def by_username_or_email(cls, username_or_email):
@ -616,6 +665,10 @@ class User(db.Model):
def is_banned(self):
return self.status == UserStatusType.BANNED
@property
def is_active(self):
return self.status != UserStatusType.INACTIVE
@property
def age(self):
'''Account age in seconds'''
@ -626,6 +679,47 @@ class User(db.Model):
''' Returns a UTC POSIX timestamp, as seconds '''
return (self.created_time - UTC_EPOCH).total_seconds()
@property
def satisfies_trusted_reqs(self) -> bool:
"""Check if the user meets the requirements to be trusted."""
num_total = 0
downloads_total = 0
for ts_flavor, t_flavor in ((NyaaStatistic, NyaaTorrent),
(SukebeiStatistic, SukebeiTorrent)):
# Count uploads that aren't remakes
stmt = select(func.count(t_flavor.id)).\
filter(t_flavor.user == self).\
filter(t_flavor.flags.op('&')(int(TorrentFlags.REMAKE)).is_(False))
uploads = db.session.execute(stmt).scalar_one_or_none() or 0
# Sum download counts for user's torrents that aren't remakes
stmt = select(func.sum(ts_flavor.download_count)).\
join(t_flavor).\
filter(t_flavor.user == self).\
filter(t_flavor.flags.op('&')(int(TorrentFlags.REMAKE)).is_(False))
dls = db.session.execute(stmt).scalar_one_or_none() or 0
num_total += uploads
downloads_total += dls
return (num_total >= config['TRUSTED_MIN_UPLOADS'] and
downloads_total >= config['TRUSTED_MIN_DOWNLOADS'])
class UserPreferences(db.Model):
__tablename__ = 'user_preferences'
user_id = db.Column(db.Integer, db.ForeignKey('users.id', ondelete='CASCADE'), primary_key=True)
def __init__(self, user_id):
self.user_id = user_id
def __repr__(self):
return '<UserPreferences %r>' % self.user_id
user = db.relationship('User', back_populates='preferences')
hide_comments = db.Column(db.Boolean, nullable=False, default=False)
class AdminLogBase(DeclarativeHelperBase):
__tablename_base__ = 'adminlog'
@ -656,7 +750,8 @@ class AdminLogBase(DeclarativeHelperBase):
@classmethod
def all_logs(cls):
return cls.query
"""Get a query for all admin logs."""
return db.session.query(cls)
class ReportStatus(IntEnum):
@ -705,17 +800,26 @@ class ReportBase(DeclarativeHelperBase):
return (self.created_time - UTC_EPOCH).total_seconds()
@classmethod
def by_id(cls, id):
return cls.query.get(id)
def by_id(cls, id: int) -> Optional['ReportBase']:
"""Get a report by its ID."""
stmt = select(cls).filter_by(id=id)
return db.session.execute(stmt).scalar_one_or_none()
@classmethod
def not_reviewed(cls, page):
reports = cls.query.filter_by(status=0).paginate(page=page, per_page=20)
return reports
def not_reviewed(cls, page: int):
"""Get paginated reports that haven't been reviewed yet."""
# Note: paginate is a Flask-SQLAlchemy extension method, not standard SQLAlchemy
# We'll keep using it for now, but it should be updated to use the new pagination API
# in a future update
stmt = select(cls).filter_by(status=0)
return db.paginate(stmt, page=page, per_page=20)
@classmethod
def remove_reviewed(cls, id):
return cls.query.filter(cls.torrent_id == id, cls.status == 0).delete()
def remove_reviewed(cls, id: int) -> int:
"""Remove all reports for a torrent that haven't been reviewed yet."""
stmt = db.delete(cls).filter(cls.torrent_id == id, cls.status == 0)
result = db.session.execute(stmt)
return result.rowcount
class Ban(db.Model):
@ -725,7 +829,7 @@ class Ban(db.Model):
created_time = db.Column(db.DateTime(timezone=False), default=datetime.utcnow)
admin_id = db.Column(db.Integer, db.ForeignKey('users.id'), nullable=False)
user_id = db.Column(db.Integer, db.ForeignKey('users.id'), nullable=True)
user_ip = db.Column(db.Binary(length=16), nullable=True)
user_ip = db.Column(db.LargeBinary(length=16), nullable=True)
reason = db.Column(db.String(length=2048), nullable=False)
admin = db.relationship('User', uselist=False, lazy='joined', foreign_keys=[admin_id])
@ -746,23 +850,156 @@ class Ban(db.Model):
@classmethod
def all_bans(cls):
return cls.query
"""Get a query for all bans."""
return db.session.query(cls)
@classmethod
def by_id(cls, id):
return cls.query.get(id)
def by_id(cls, id: int) -> Optional['Ban']:
"""Get a ban by its ID."""
stmt = select(cls).filter_by(id=id)
return db.session.execute(stmt).scalar_one_or_none()
@classmethod
def banned(cls, user_id, user_ip):
def banned(cls, user_id: Optional[int], user_ip: Optional[bytes]):
"""Check if a user or IP is banned."""
if user_id:
if user_ip:
return cls.query.filter((cls.user_id == user_id) | (cls.user_ip == user_ip))
return cls.query.filter(cls.user_id == user_id)
stmt = select(cls).filter((cls.user_id == user_id) | (cls.user_ip == user_ip))
return db.session.execute(stmt).scalars().all()
stmt = select(cls).filter(cls.user_id == user_id)
return db.session.execute(stmt).scalars().all()
if user_ip:
return cls.query.filter(cls.user_ip == user_ip)
stmt = select(cls).filter(cls.user_ip == user_ip)
return db.session.execute(stmt).scalars().all()
return None
class TrackerApiBase(DeclarativeHelperBase):
__tablename_base__ = 'trackerapi'
id = db.Column(db.Integer, primary_key=True)
info_hash = db.Column(BinaryType(length=20), nullable=False)
method = db.Column(db.String(length=255), nullable=False)
# Methods = insert, remove
def __init__(self, info_hash, method):
self.info_hash = info_hash
self.method = method
class RangeBan(db.Model):
__tablename__ = 'rangebans'
id = db.Column(db.Integer, primary_key=True)
_cidr_string = db.Column('cidr_string', db.String(length=18), nullable=False)
masked_cidr = db.Column(db.BigInteger, nullable=False,
index=True)
mask = db.Column(db.BigInteger, nullable=False, index=True)
enabled = db.Column(db.Boolean, nullable=False, default=True)
# If this rangeban may be automatically cleared once it becomes
# out of date, set this column to the creation time of the ban.
# None (or NULL in the db) is understood as the ban being permanent.
temp = db.Column(db.DateTime(timezone=False), nullable=True, default=None)
@property
def cidr_string(self):
return self._cidr_string
@cidr_string.setter
def cidr_string(self, s):
subnet, masked_bits = s.split('/')
subnet_b = ip_address(subnet).packed
self.mask = (1 << 32) - (1 << (32 - int(masked_bits)))
self.masked_cidr = int.from_bytes(subnet_b, 'big') & self.mask
self._cidr_string = s
@classmethod
def is_rangebanned(cls, ip: bytes) -> bool:
"""Check if an IP is within a banned range."""
if len(ip) > 4:
raise NotImplementedError("IPv6 is unsupported.")
elif len(ip) < 4:
raise ValueError("Not an IP address.")
ip_int = int.from_bytes(ip, 'big')
stmt = select(cls).filter(cls.mask.op('&')(ip_int) == cls.masked_cidr,
cls.enabled)
count = db.session.execute(select(func.count()).select_from(stmt.subquery())).scalar_one()
return count > 0
class TrustedApplicationStatus(IntEnum):
# If you change these, don't forget to change is_closed in TrustedApplication
NEW = 0
REVIEWED = 1
ACCEPTED = 2
REJECTED = 3
class TrustedApplication(db.Model):
__tablename__ = 'trusted_applications'
id = db.Column(db.Integer, primary_key=True)
submitter_id = db.Column(db.Integer, db.ForeignKey('users.id'), nullable=False, index=True)
created_time = db.Column(db.DateTime(timezone=False), default=datetime.utcnow)
closed_time = db.Column(db.DateTime(timezone=False))
why_want = db.Column(db.String(length=4000), nullable=False)
why_give = db.Column(db.String(length=4000), nullable=False)
status = db.Column(ChoiceType(TrustedApplicationStatus, impl=db.Integer()), nullable=False,
default=TrustedApplicationStatus.NEW)
reviews = db.relationship('TrustedReview', backref='trusted_applications')
submitter = db.relationship('User', uselist=False, lazy='joined', foreign_keys=[submitter_id])
@hybrid_property
def is_closed(self):
# We can't use the attribute names from TrustedApplicationStatus in an or here because of
# SQLAlchemy jank. It'll generate the wrong query.
return self.status > 1
@hybrid_property
def is_new(self):
return self.status == TrustedApplicationStatus.NEW
@hybrid_property
def is_reviewed(self):
return self.status == TrustedApplicationStatus.REVIEWED
@hybrid_property
def is_rejected(self):
return self.status == TrustedApplicationStatus.REJECTED
@property
def created_utc_timestamp(self):
''' Returns a UTC POSIX timestamp, as seconds '''
return (self.created_time - UTC_EPOCH).total_seconds()
@classmethod
def by_id(cls, id: int) -> Optional['TrustedApplication']:
"""Get a trusted application by its ID."""
stmt = select(cls).filter_by(id=id)
return db.session.execute(stmt).scalar_one_or_none()
class TrustedRecommendation(IntEnum):
ACCEPT = 0
REJECT = 1
ABSTAIN = 2
class TrustedReview(db.Model):
__tablename__ = 'trusted_reviews'
id = db.Column(db.Integer, primary_key=True)
reviewer_id = db.Column(db.Integer, db.ForeignKey('users.id'), nullable=False)
app_id = db.Column(db.Integer, db.ForeignKey('trusted_applications.id'), nullable=False)
created_time = db.Column(db.DateTime(timezone=False), default=datetime.utcnow)
comment = db.Column(db.String(length=4000), nullable=False)
recommendation = db.Column(ChoiceType(TrustedRecommendation, impl=db.Integer()),
nullable=False)
reviewer = db.relationship('User', uselist=False, lazy='joined', foreign_keys=[reviewer_id])
application = db.relationship('TrustedApplication', uselist=False, lazy='joined',
foreign_keys=[app_id])
# Actually declare our site-specific classes
# Torrent
@ -801,15 +1038,6 @@ class SukebeiTorrentFilelist(TorrentFilelistBase, db.Model):
__flavor__ = 'Sukebei'
# TorrentInfo
class NyaaTorrentInfo(TorrentInfoBase, db.Model):
__flavor__ = 'Nyaa'
class SukebeiTorrentInfo(TorrentInfoBase, db.Model):
__flavor__ = 'Sukebei'
# Statistic
class NyaaStatistic(StatisticBase, db.Model):
__flavor__ = 'Nyaa'
@ -873,11 +1101,19 @@ class SukebeiReport(ReportBase, db.Model):
__flavor__ = 'Sukebei'
# TrackerApi
class NyaaTrackerApi(TrackerApiBase, db.Model):
__flavor__ = 'Nyaa'
class SukebeiTrackerApi(TrackerApiBase, db.Model):
__flavor__ = 'Sukebei'
# Choose our defaults for models.Torrent etc
if config['SITE_FLAVOR'] == 'nyaa':
Torrent = NyaaTorrent
TorrentFilelist = NyaaTorrentFilelist
TorrentInfo = NyaaTorrentInfo
Statistic = NyaaStatistic
TorrentTrackers = NyaaTorrentTrackers
MainCategory = NyaaMainCategory
@ -886,11 +1122,11 @@ if config['SITE_FLAVOR'] == 'nyaa':
AdminLog = NyaaAdminLog
Report = NyaaReport
TorrentNameSearch = NyaaTorrentNameSearch
TrackerApi = NyaaTrackerApi
elif config['SITE_FLAVOR'] == 'sukebei':
Torrent = SukebeiTorrent
TorrentFilelist = SukebeiTorrentFilelist
TorrentInfo = SukebeiTorrentInfo
Statistic = SukebeiStatistic
TorrentTrackers = SukebeiTorrentTrackers
MainCategory = SukebeiMainCategory
@ -899,3 +1135,4 @@ elif config['SITE_FLAVOR'] == 'sukebei':
AdminLog = SukebeiAdminLog
Report = SukebeiReport
TorrentNameSearch = SukebeiTorrentNameSearch
TrackerApi = SukebeiTrackerApi

356
nyaa/search.py
View file

@ -1,10 +1,15 @@
import math
import re
import shlex
import threading
import time
from typing import Any, Dict, List, Optional, Tuple, Union
import flask
from nyaa.custom_pagination import CustomPagination
import sqlalchemy
from sqlalchemy import select, func, bindparam
import sqlalchemy_fulltext.modes as FullTextMode
from elasticsearch import Elasticsearch
from elasticsearch_dsl import Q, Search
@ -26,7 +31,7 @@ SERACH_PAGINATE_DISPLAY_MSG = ('Displaying results {start}-{end} out of {total}
_index_name_cache = {}
def _get_index_name(column):
def _get_index_name(column) -> Optional[str]:
''' Returns an index name for a given column, or None.
Only considers single-column indexes.
Results are cached in memory (until app restart). '''
@ -39,7 +44,7 @@ def _get_index_name(column):
try:
column_table = sqlalchemy.Table(column_table_name,
sqlalchemy.MetaData(),
autoload=True, autoload_with=db.engine)
autoload_with=db.engine)
except sqlalchemy.exc.NoSuchTableError:
# Trust the developer to notice this?
pass
@ -56,7 +61,8 @@ def _get_index_name(column):
return table_indexes.get(column.name)
def _generate_query_string(term, category, filter, user):
def _generate_query_string(term: Optional[str], category: Optional[str],
filter: Optional[str], user: Optional[int]) -> Dict[str, str]:
params = {}
if term:
params['q'] = str(term)
@ -69,6 +75,114 @@ def _generate_query_string(term, category, filter, user):
return params
# For preprocessing ES search terms in _parse_es_search_terms
QUOTED_LITERAL_REGEX = re.compile(r'(?i)(-)?"(.+?)"')
QUOTED_LITERAL_GROUP_REGEX = re.compile(r'''
(?i)
(-)? # Negate entire group at once
(
".+?" # First literal
(?:
\| # OR
".+?" # Second literal
)+ # repeating
)
''', re.X)
def _es_name_exact_phrase(literal):
''' Returns a Query for a phrase match on the display_name for a given literal '''
return Q({
'match_phrase': {
'display_name.exact': {
'query': literal,
'analyzer': 'exact_analyzer'
}
}
})
def _parse_es_search_terms(search, search_terms):
''' Parse search terms into a query with properly handled literal phrases
(the simple_query_string is not so great with exact results).
For example:
foo bar "hello world" -"exclude this"
will become a must simple_query_string for "foo bar", a must phrase_match for
"hello world" and a must_not for "exclude this".
Returns the search with the generated bool-query added to it. '''
# Literal must and must-not sets
must_set = set()
must_not_set = set()
must_or_groups = []
must_not_or_groups = []
def must_group_matcher(match):
''' Grabs [-]"foo"|"bar"[|"baz"...] groups from the search terms '''
negated = bool(match.group(1))
literal_group = match.group(2)
literals = QUOTED_LITERAL_REGEX.findall(literal_group)
group_query = Q(
'bool',
should=[_es_name_exact_phrase(lit_m[1]) for lit_m in literals]
)
if negated:
must_not_or_groups.append(group_query)
else:
must_or_groups.append(group_query)
# Remove the parsed group from search terms
return ''
def must_matcher(match):
''' Grabs [-]"foo" literals from the search terms '''
negated = bool(match.group(1))
literal = match.group(2)
if negated:
must_not_set.add(literal)
else:
must_set.add(literal)
# Remove the parsed literal from search terms
return ''
# Remove quoted parts (optionally prepended with -) and store them in the sets
parsed_search_terms = QUOTED_LITERAL_GROUP_REGEX.sub(must_group_matcher, search_terms).strip()
parsed_search_terms = QUOTED_LITERAL_REGEX.sub(must_matcher, parsed_search_terms).strip()
# Create phrase matches (if any)
must_queries = [_es_name_exact_phrase(lit) for lit in must_set] + must_or_groups
must_not_queries = [_es_name_exact_phrase(lit) for lit in must_not_set] + must_not_or_groups
if parsed_search_terms:
# Normal text search without the quoted parts
must_queries.append(
Q(
'simple_query_string',
# Query both fields, latter for words with >15 chars
fields=['display_name', 'display_name.fullword'],
analyzer='my_search_analyzer',
default_operator="AND",
query=parsed_search_terms
)
)
if must_queries or must_not_queries:
# Create a combined Query with the positive and negative matches
combined_search_query = Q(
'bool',
must=must_queries,
must_not=must_not_queries
)
search = search.query(combined_search_query)
return search
def search_elastic(term='', user=None, sort='id', order='desc',
category='0_0', quality_filter='0', page=1,
rss=False, admin=False, logged_in_user=None,
@ -77,7 +191,7 @@ def search_elastic(term='', user=None, sort='id', order='desc',
if page > 4294967295:
flask.abort(404)
es_client = Elasticsearch()
es_client = Elasticsearch(hosts=app.config['ES_HOSTS'])
es_sort_keys = {
'id': 'id',
@ -165,12 +279,8 @@ def search_elastic(term='', user=None, sort='id', order='desc',
# Apply search term
if term:
s = s.query('simple_query_string',
# Query both fields, latter for words with >15 chars
fields=['display_name', 'display_name.fullword'],
analyzer='my_search_analyzer',
default_operator="AND",
query=term)
# Do some preprocessing on the search terms for literal "" matching
s = _parse_es_search_terms(s, term)
# User view (/user/username)
if user:
@ -262,12 +372,33 @@ class QueryPairCaller(object):
return wrapper
def search_db(term='', user=None, sort='id', order='desc', category='0_0',
quality_filter='0', page=1, rss=False, admin=False,
logged_in_user=None, per_page=75):
def search_db(term: str = '', user: Optional[int] = None, sort: str = 'id',
order: str = 'desc', category: str = '0_0',
quality_filter: str = '0', page: int = 1, rss: bool = False,
admin: bool = False, logged_in_user: Optional[models.User] = None,
per_page: int = 75) -> Union[CustomPagination, List[models.Torrent]]:
"""
Search the database for torrents matching the given criteria.
This is the SQLAlchemy 2.0 compatible version of the search function.
"""
if page > 4294967295:
flask.abort(404)
MAX_PAGES = app.config.get("MAX_PAGES", 0)
same_user = False
if logged_in_user and user:
same_user = logged_in_user.id == user
# Logged in users should always be able to view their full listing.
if same_user or admin:
MAX_PAGES = 0
if MAX_PAGES and page > MAX_PAGES:
flask.abort(flask.Response("You've exceeded the maximum number of pages. Please "
"make your search query less broad.", 403))
sort_keys = {
'id': models.Torrent.id,
'size': models.Torrent.filesize,
@ -305,10 +436,10 @@ def search_db(term='', user=None, sort='id', order='desc', category='0_0',
flask.abort(400)
if user:
user = models.User.by_id(user)
if not user:
user_obj = models.User.by_id(user)
if not user_obj:
flask.abort(404)
user = user.id
user = user_obj.id
main_category = None
sub_category = None
@ -336,28 +467,24 @@ def search_db(term='', user=None, sort='id', order='desc', category='0_0',
sort_column = sort_keys['id']
order = 'desc'
same_user = False
if logged_in_user:
same_user = logged_in_user.id == user
model_class = models.TorrentNameSearch if term else models.Torrent
query = db.session.query(model_class)
# This is... eh. Optimize the COUNT() query since MySQL is bad at that.
# See http://docs.sqlalchemy.org/en/rel_1_1/orm/query.html#sqlalchemy.orm.query.Query.count
# Wrap the queries into the helper class to deduplicate code and apply filters to both in one go
count_query = db.session.query(sqlalchemy.func.count(model_class.id))
qpc = QueryPairCaller(query, count_query)
# Create the base query
query = select(model_class)
count_query = select(func.count(model_class.id))
# User view (/user/username)
if user:
qpc.filter(models.Torrent.uploader_id == user)
query = query.where(models.Torrent.uploader_id == user)
count_query = count_query.where(models.Torrent.uploader_id == user)
if not admin:
# Hide all DELETED torrents if regular user
qpc.filter(models.Torrent.flags.op('&')(
int(models.TorrentFlags.DELETED)).is_(False))
deleted_filter = models.Torrent.flags.op('&')(
int(models.TorrentFlags.DELETED)).is_(False)
query = query.where(deleted_filter)
count_query = count_query.where(deleted_filter)
# If logged in user is not the same as the user being viewed,
# show only torrents that aren't hidden or anonymous
#
@ -367,53 +494,182 @@ def search_db(term='', user=None, sort='id', order='desc', category='0_0',
# On RSS pages in user view,
# show only torrents that aren't hidden or anonymous no matter what
if not same_user or rss:
qpc.filter(models.Torrent.flags.op('&')(
int(models.TorrentFlags.HIDDEN | models.TorrentFlags.ANONYMOUS)).is_(False))
hidden_anon_filter = models.Torrent.flags.op('&')(
int(models.TorrentFlags.HIDDEN | models.TorrentFlags.ANONYMOUS)).is_(False)
query = query.where(hidden_anon_filter)
count_query = count_query.where(hidden_anon_filter)
# General view (homepage, general search view)
else:
if not admin:
# Hide all DELETED torrents if regular user
qpc.filter(models.Torrent.flags.op('&')(
int(models.TorrentFlags.DELETED)).is_(False))
deleted_filter = models.Torrent.flags.op('&')(
int(models.TorrentFlags.DELETED)).is_(False)
query = query.where(deleted_filter)
count_query = count_query.where(deleted_filter)
# If logged in, show all torrents that aren't hidden unless they belong to you
# On RSS pages, show all public torrents and nothing more.
if logged_in_user and not rss:
qpc.filter(
hidden_or_user_filter = (
(models.Torrent.flags.op('&')(int(models.TorrentFlags.HIDDEN)).is_(False)) |
(models.Torrent.uploader_id == logged_in_user.id))
(models.Torrent.uploader_id == logged_in_user.id)
)
query = query.where(hidden_or_user_filter)
count_query = count_query.where(hidden_or_user_filter)
# Otherwise, show all torrents that aren't hidden
else:
qpc.filter(models.Torrent.flags.op('&')(
int(models.TorrentFlags.HIDDEN)).is_(False))
hidden_filter = models.Torrent.flags.op('&')(
int(models.TorrentFlags.HIDDEN)).is_(False)
query = query.where(hidden_filter)
count_query = count_query.where(hidden_filter)
if main_category:
qpc.filter(models.Torrent.main_category_id == main_cat_id)
main_cat_filter = models.Torrent.main_category_id == main_cat_id
query = query.where(main_cat_filter)
count_query = count_query.where(main_cat_filter)
elif sub_category:
qpc.filter((models.Torrent.main_category_id == main_cat_id) &
(models.Torrent.sub_category_id == sub_cat_id))
sub_cat_filter = (
(models.Torrent.main_category_id == main_cat_id) &
(models.Torrent.sub_category_id == sub_cat_id)
)
query = query.where(sub_cat_filter)
count_query = count_query.where(sub_cat_filter)
if filter_tuple:
qpc.filter(models.Torrent.flags.op('&')(
int(filter_tuple[0])).is_(filter_tuple[1]))
filter_condition = models.Torrent.flags.op('&')(
int(filter_tuple[0])).is_(filter_tuple[1])
query = query.where(filter_condition)
count_query = count_query.where(filter_condition)
if term:
for item in shlex.split(term, posix=False):
if len(item) >= 2:
qpc.filter(FullTextSearch(
item, models.TorrentNameSearch, FullTextMode.NATURAL))
fulltext_filter = FullTextSearch(
item, models.TorrentNameSearch, FullTextMode.NATURAL)
query = query.where(fulltext_filter)
count_query = count_query.where(fulltext_filter)
query, count_query = qpc.items
# Sort and order
if sort_column.class_ != models.Torrent:
index_name = _get_index_name(sort_column)
query = query.join(sort_column.class_)
query = query.with_hint(sort_column.class_, 'USE INDEX ({0})'.format(index_name))
query = query.order_by(getattr(sort_column, order)())
# Add index hint for MySQL if available
if index_name and hasattr(db.engine.dialect, 'name') and db.engine.dialect.name == 'mysql':
# In SQLAlchemy 2.0, we use execution_options instead of with_hint
# This is MySQL specific - for other databases, different approaches would be needed
query = query.execution_options(
mysql_hint=f"USE INDEX ({index_name})"
)
if order_ == 'desc':
query = query.order_by(sort_column.desc())
else:
query = query.order_by(sort_column.asc())
if rss:
query = query.limit(per_page)
return db.session.execute(query).scalars().all()
else:
query = query.paginate_faste(page, per_page=per_page, step=5, count_query=count_query)
# Get the total count
total_count = db.session.execute(count_query).scalar_one()
return query
# Apply pagination
query = query.limit(per_page).offset((page - 1) * per_page)
items = db.session.execute(query).scalars().all()
if not items and page != 1:
flask.abort(404)
# Create a pagination object
return CustomPagination(query, page, per_page, total_count, items)
# Alias for backward compatibility
search_db_baked = search_db
class ShoddyLRU(object):
def __init__(self, max_entries=128, expiry=60):
self.max_entries = max_entries
self.expiry = expiry
# Contains [value, last_used, expires_at]
self.entries = {}
self._lock = threading.Lock()
self._sentinel = object()
def get(self, key, default=None):
entry = self.entries.get(key)
if entry is None:
return default
now = time.time()
if now > entry[2]:
with self._lock:
del self.entries[key]
return default
entry[1] = now
return entry[0]
def put(self, key, value, expiry=None):
with self._lock:
overflow = len(self.entries) - self.max_entries
if overflow > 0:
# Pick the least recently used keys
removed_keys = [key for key, value in sorted(
self.entries.items(), key=lambda t:t[1][1])][:overflow]
for key in removed_keys:
del self.entries[key]
now = time.time()
self.entries[key] = [value, now, now + (expiry or self.expiry)]
LRU_CACHE = ShoddyLRU(256, 60)
def paginate_query(query, count_query, page=1, per_page=50, max_page=None):
"""
Paginate a SQLAlchemy 2.0 query.
This is a replacement for the baked_paginate function that uses SQLAlchemy 2.0 style.
"""
if page < 1:
flask.abort(404)
if max_page and page > max_page:
flask.abort(404)
# Count all items, use cache
if app.config.get('COUNT_CACHE_DURATION'):
# Create a cache key based on the query and parameters
# This is a simplified version compared to the bakery's _effective_key
query_key = str(count_query)
total_query_count = LRU_CACHE.get(query_key)
if total_query_count is None:
total_query_count = db.session.execute(count_query).scalar_one()
LRU_CACHE.put(query_key, total_query_count, expiry=app.config['COUNT_CACHE_DURATION'])
else:
total_query_count = db.session.execute(count_query).scalar_one()
# Apply pagination
paginated_query = query.limit(per_page).offset((page - 1) * per_page)
items = db.session.execute(paginated_query).scalars().all()
if max_page:
total_query_count = min(total_query_count, max_page * per_page)
# Handle case where we've had no results but then have some while in cache
total_query_count = max(total_query_count, len(items))
if not items and page != 1:
flask.abort(404)
return CustomPagination(None, page, per_page, total_query_count, items)
# Alias for backward compatibility
baked_paginate = paginate_query

70
nyaa/static/css/main.css
View file

@ -36,6 +36,11 @@ table.torrent-list thead th a {
filter: alpha(opacity=1);
}
.category-icon {
width: 80px;
height: 28px;
}
table.torrent-list thead th.sorting:after,
table.torrent-list thead th.sorting_asc:after,
table.torrent-list thead th.sorting_desc:after {
@ -88,6 +93,23 @@ table.torrent-list tbody .comments i {
padding-right: 2px;
}
table.torrent-list td:first-child {
padding: 0 4px;
}
table.torrent-list td:nth-child(4) {
white-space: nowrap;
}
table.torrent-list td:nth-child(6),
body.dark table.torrent-list > tbody > tr.success > td:nth-child(6),
body.dark table.torrent-list > tbody > tr.danger > td:nth-child(6) {
color: green;
}
table.torrent-list td:nth-child(7),
body.dark table.torrent-list > tbody > tr.success > td:nth-child(7),
body.dark table.torrent-list > tbody > tr.danger > td:nth-child(7) {
color: red;
}
#torrent-description img {
max-width: 100%;
}
@ -275,6 +297,11 @@ a.text-purple:hover, a.text-purple:active, a.text-purple:focus {
margin-bottom: 10px;
}
/* workaround for Mozilla whitespace copypaste dumbfuckery */
.comment-body {
-moz-user-select: text;
}
.comment-content {
overflow-wrap: break-word;
}
@ -421,6 +448,9 @@ h6:hover .header-anchor {
visibility: visible;
display: inline-block;
}
.trusted-form textarea {
height:12em;
}
/* Dark theme */
@ -432,7 +462,8 @@ body.dark .navbar a {
color: #e2e2e2;
}
body.dark kbd {
body.dark kbd,
body.dark .btn.edit-comment {
background-color: #4a4a4a;
}
@ -450,8 +481,9 @@ body.dark .torrent-list tbody tr td a:visited {
color: #205c90;
}
body.dark .torrent-list > thead > tr, body.dark tbody > tr,
body.dark .panel > .panel-heading {
body.dark thead > tr, body.dark tbody > tr,
body.dark .panel > .panel-heading,
body.dark .report-action-column select {
color: #cbcbcb;
}
@ -474,6 +506,14 @@ body.dark table.torrent-list tbody .comments {
background-color: #2f2c2c;
}
body.dark .comment-panel:target {
border-color: white;
}
body.dark .table > table {
background-color: #323232;
}
/* trusted */
body.dark .torrent-list > tbody > tr.success > td {
color: inherit;
@ -534,6 +574,26 @@ body.dark .panel-deleted > .panel-heading {
.search-container > .search-bar {
margin-top: 15px;
}
.torrent-list .hdr-date,
.torrent-list .hdr-downloads,
.torrent-list td: nth-of-type(5),
.torrent-list td:nth-of-type(8) {
display: none;
}
.table-responsive > .table > tbody > tr > td:nth-of-type(2) {
white-space: unset;
word-break: break-all;
}
.container {
width: unset;
}
.container > .row {
margin: unset;
}
}
@media (min-width: 992px) {
@ -564,6 +624,10 @@ td.report-action-column {
.search-container {
width: 400px;
}
.table-responsive > .table > tbody > tr > td:nth-of-type(2) {
white-space: unset;
}
}

BIN
nyaa/static/img/avatar/default.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 4.8 KiB

After

Width:  |  Height:  |  Size: 3.5 KiB

BIN
nyaa/static/img/icons/nyaa/1_1.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 3.2 KiB

After

Width:  |  Height:  |  Size: 2.7 KiB

BIN
nyaa/static/img/icons/nyaa/1_2.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 3.1 KiB

After

Width:  |  Height:  |  Size: 2.7 KiB

BIN
nyaa/static/img/icons/nyaa/1_3.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 3.1 KiB

After

Width:  |  Height:  |  Size: 2.7 KiB

BIN
nyaa/static/img/icons/nyaa/1_4.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 3.8 KiB

After

Width:  |  Height:  |  Size: 3.3 KiB

BIN
nyaa/static/img/icons/nyaa/2_1.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 2.8 KiB

After

Width:  |  Height:  |  Size: 2.4 KiB

BIN
nyaa/static/img/icons/nyaa/2_2.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 3.5 KiB

After

Width:  |  Height:  |  Size: 3 KiB

BIN
nyaa/static/img/icons/nyaa/3_1.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 2.6 KiB

After

Width:  |  Height:  |  Size: 2.3 KiB

BIN
nyaa/static/img/icons/nyaa/3_2.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 2.4 KiB

After

Width:  |  Height:  |  Size: 2.1 KiB

BIN
nyaa/static/img/icons/nyaa/3_3.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 4.7 KiB

After

Width:  |  Height:  |  Size: 4.1 KiB

BIN
nyaa/static/img/icons/nyaa/4_1.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 3.4 KiB

After

Width:  |  Height:  |  Size: 2.9 KiB

BIN
nyaa/static/img/icons/nyaa/4_2.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 3.3 KiB

After

Width:  |  Height:  |  Size: 2.8 KiB

BIN
nyaa/static/img/icons/nyaa/4_3.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 3.1 KiB

After

Width:  |  Height:  |  Size: 2.7 KiB

BIN
nyaa/static/img/icons/nyaa/4_4.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 3.6 KiB

After

Width:  |  Height:  |  Size: 3.1 KiB

BIN
nyaa/static/img/icons/nyaa/5_1.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 4.3 KiB

After

Width:  |  Height:  |  Size: 3.9 KiB

BIN
nyaa/static/img/icons/nyaa/5_2.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 4.3 KiB

After

Width:  |  Height:  |  Size: 3.9 KiB

BIN
nyaa/static/img/icons/nyaa/6_1.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 3.6 KiB

After

Width:  |  Height:  |  Size: 3.2 KiB

BIN
nyaa/static/img/icons/nyaa/6_2.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 3.5 KiB

After

Width:  |  Height:  |  Size: 3.2 KiB

BIN
nyaa/static/img/icons/sukebei/1_1.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 4 KiB

After

Width:  |  Height:  |  Size: 3.4 KiB

BIN
nyaa/static/img/icons/sukebei/1_2.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 4.3 KiB

After

Width:  |  Height:  |  Size: 3.8 KiB

BIN
nyaa/static/img/icons/sukebei/1_3.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 3.7 KiB

After

Width:  |  Height:  |  Size: 3.2 KiB

BIN
nyaa/static/img/icons/sukebei/1_4.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 3.1 KiB

After

Width:  |  Height:  |  Size: 2.7 KiB

BIN
nyaa/static/img/icons/sukebei/1_5.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 4 KiB

After

Width:  |  Height:  |  Size: 3.5 KiB

BIN
nyaa/static/img/icons/sukebei/2_1.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 4.3 KiB

After

Width:  |  Height:  |  Size: 3.7 KiB

BIN
nyaa/static/img/icons/sukebei/2_2.png
View file

Binary file not shown.
Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 4 KiB

After

Width:  |  Height:  |  Size: 3.4 KiB

25
nyaa/static/js/main.js
View file

@ -118,6 +118,9 @@ $(document).ready(function() {
$errorStatus.text(error);
}).always(function() {
$submitButton.removeAttr('disabled');
if (grecaptcha) {
grecaptcha.reset();
}
$waitIndicator.hide();
});
})
@ -212,6 +215,13 @@ markdown.renderer.rules.table_open = function (tokens, idx) {
// Format tables nicer (bootstrap). Force auto-width (default is 100%)
return '<table class="table table-striped table-bordered" style="width: auto;">';
}
var defaultRender = markdown.renderer.rules.link_open || function(tokens, idx, options, env, self) {
return self.renderToken(tokens, idx, options);
};
markdown.renderer.rules.link_open = function (tokens, idx, options, env, self) {
tokens[idx].attrPush(['rel', 'noopener nofollow noreferrer']);
return defaultRender(tokens, idx, options, env, self);
}
// Initialise markdown editors on page
document.addEventListener("DOMContentLoaded", function() {
@ -241,6 +251,11 @@ document.addEventListener("DOMContentLoaded", function() {
var target = markdownTargets[i];
var rendered;
var markdownSource = htmlDecode(target.innerHTML);
if (target.attributes["markdown-no-images"]) {
markdown.disable('image');
} else {
markdown.enable('image');
}
if (target.attributes["markdown-text-inline"]) {
rendered = markdown.renderInline(markdownSource);
} else {
@ -250,6 +265,16 @@ document.addEventListener("DOMContentLoaded", function() {
}
});
// Info bubble stuff
document.addEventListener("DOMContentLoaded", function() {
var bubble = document.getElementById('infobubble');
if (Number(localStorage.getItem('infobubble_dismiss_ts')) < Number(bubble.dataset.ts)) {
bubble.removeAttribute('hidden');
}
$('#infobubble').on('close.bs.alert', function () {
localStorage.setItem('infobubble_dismiss_ts', bubble.dataset.ts);
})
});
// Decode HTML entities (&gt; etc), used for decoding comment markdown from escaped text
function htmlDecode(input){

49
nyaa/template_utils.py
View file

@ -1,15 +1,15 @@
import functools
import os.path
import re
from base64 import b32encode
from datetime import datetime
from email.utils import formatdate
from urllib.parse import urlencode
import flask
from werkzeug.urls import url_encode
#from werkzeug.utils import url_encode
from urllib.parse import urlencode # now using Python's built-in urlencode
from nyaa.backend import get_category_id_map
from nyaa.torrents import get_default_trackers
from nyaa.torrents import create_magnet
app = flask.current_app
bp = flask.Blueprint('template-utils', __name__)
@ -20,24 +20,35 @@ _static_cache = {} # For static_cachebuster
# For processing ES links
@bp.app_context_processor
def create_magnet_from_es_info():
def _create_magnet_from_es_info(display_name, info_hash, max_trackers=5, trackers=None):
if trackers is None:
trackers = get_default_trackers()
magnet_parts = [
('dn', display_name)
]
for tracker in trackers[:max_trackers]:
magnet_parts.append(('tr', tracker))
b32_info_hash = b32encode(bytes.fromhex(info_hash)).decode('utf-8')
return 'magnet:?xt=urn:btih:' + b32_info_hash + '&' + urlencode(magnet_parts)
return dict(create_magnet_from_es_info=_create_magnet_from_es_info)
def create_magnet_from_es_torrent():
# Since ES entries look like ducks, we can use the create_magnet as-is
return dict(create_magnet_from_es_torrent=create_magnet)
# ######################### TEMPLATE GLOBALS #########################
flask_url_for = flask.url_for
@functools.lru_cache(maxsize=1024 * 4)
def _caching_url_for(endpoint, **values):
return flask_url_for(endpoint, **values)
@bp.app_template_global()
def caching_url_for(*args, **kwargs):
try:
# lru_cache requires the arguments to be hashable.
# Majority of the time, they are! But there are some small edge-cases,
# like our copypasted pagination, parameters can be lists.
# Attempt caching first:
return _caching_url_for(*args, **kwargs)
except TypeError:
# Then fall back to the original url_for.
# We could convert the lists to tuples, but the savings are marginal.
return flask_url_for(*args, **kwargs)
@bp.app_template_global()
def static_cachebuster(filename):
""" Adds a ?t=<mtime> cachebuster to the given path, if the file exists.
@ -71,7 +82,7 @@ def modify_query(**new_values):
for key, value in new_values.items():
args[key] = value
return '{}?{}'.format(flask.request.path, url_encode(args))
return '{}?{}'.format(flask.request.path, urlencode(args))
@bp.app_template_global()

10
nyaa/templates/_formhelpers.html
View file

@ -113,15 +113,19 @@
</div>
{% endmacro %}
{% macro render_menu_with_button(field) %}
{% macro render_menu_with_button(field, button_label='Apply') %}
{% if field.errors %}
<div class="form-group has-error">
{% else %}
<div class="form-group">
{% endif %}
{{ field.label(class='control-label') }}
{{ field(title=field.description,**kwargs) | safe }}
<button type="submit" class="btn btn-primary">Apply</button>
<div class="input-group input-group-sm">
{{ field(title=field.description, class_="form-control",**kwargs) | safe }}
<div class="input-group-btn">
<button type="submit" class="btn btn-primary">{{ button_label }}</button>
</div>
</div>
{% if field.errors %}
<div class="help-block">
{% if field.errors|length < 2 %}

54
nyaa/templates/admin_trusted.html Normal file
View file

@ -0,0 +1,54 @@
{% extends "layout.html" %}
{% macro render_filter_tab(name) %}
<li class="nav-item{% if list_filter == name %} active{% endif %}">
<a class="nav-link{% if list_filter == name %} active{% endif %}" href="{{ url_for('admin.trusted', list_filter=name) }}">
{% if name %}
{{ name.capitalize() }}
{% else %}
Open
{% endif %}
</a>
</li>
{% endmacro %}
{% block title %}Trusted Applications :: {{ config.SITE_NAME }}{% endblock %}
{% block body %}
<ul class="nav nav-tabs" role="tablist">
{{ render_filter_tab(None) }}
{{ render_filter_tab('new') }}
{{ render_filter_tab('reviewed') }}
{{ render_filter_tab('closed') }}
</ul>
<div class="table">
<table class="table table-bordered table-hover table-striped table-condensed">
<caption>List of {{ list_filter or 'open' }} applications</caption>
<thead>
<tr>
<th scope="col">#</th>
<th scope="col">Submitter</th>
<th scope="col">Submitted on</th>
<th scope="col">Status</th>
<th scope="col"></th>
</tr>
</thead>
<tbody>
{% for app in apps.items %}
<tr class="reports-row">
<td>{{ app.id }}</td>
<td>
<a href="{{ url_for('users.view_user', user_name=app.submitter.username) }}">
{{ app.submitter.username }}
</a>
</td>
<td data-timestamp="{{ app.created_utc_timestamp | int }}">{{ app.created_time.strftime('%Y-%m-%d %H:%M') }}</td>
<td>{{ app.status.name.capitalize() }}</td>
<td><a class="btn btn-primary btn-sm" style="width:100%" href="{{ url_for('admin.trusted_application', app_id=app.id) }}" role="button">View</a></td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
<div class=pagination>
{% from "bootstrap/pagination.html" import render_pagination %}
{{ render_pagination(apps) }}
</div>
{% endblock %}

114
nyaa/templates/admin_trusted_view.html Normal file
View file

@ -0,0 +1,114 @@
{% extends "layout.html" %}
{% from "_formhelpers.html" import render_field, render_menu_with_button %}
{%- macro review_class(rec) -%}
{%- if rec.name == 'ACCEPT' -%}
{{ 'panel-success' -}}
{%- elif rec.name == 'REJECT' -%}
{{ 'panel-danger' -}}
{%- elif rec.name == 'ABSTAIN' -%}
{{ 'panel-default' -}}
{%- endif -%}
{%- endmacro -%}
{% block title %}{{ app.submitter.username }}'s Application :: {{ config.SITE_NAME }}{% endblock %}
{% block body %}
<div class="panel panel-primary">
<div class="panel-heading">
<h3 class="panel-title">{{ app.submitter.username }}'s Application</h3>
</div>
<div class="panel-body">
<div class="row">
<dl>
<div class="col-xs-4 col-sm-2 col-md-2">
<dt>Submitter</dt>
<dd>
<a href="{{ url_for('users.view_user', user_name=app.submitter.username) }}">
{{ app.submitter.username }}
</a>
</dd>
</div>
<div class="col-xs-4 col-sm-2 col-md-2">
<dt>Submitted on</dt>
<dd data-timestamp="{{ app.created_utc_timestamp | int }}">
{{ app.created_time.strftime('%Y-%m-%d %H:%M') }}
</dd>
</div>
<div class="col-xs-4 col-sm-2 col-md-2">
<dt>Status</dt>
<dd>{{ app.status.name.capitalize() }}</dd>
</div>
</dl>
</div>
<hr>
<div class="row">
<div class="col-md-12">
<h4>Why do you think you should be given trusted status?</h4>
<div class="panel panel-default">
<div class="panel-body" markdown-text markdown-no-images>
{{- app.why_give | escape | replace('\r\n', '\n') | replace('\n', '&#10;'|safe) -}}
</div>
</div>
</div>
</div>
<div class="row">
<div class="col-md-12">
<h4>Why do you want to become a trusted user?</h4>
<div class="panel panel-default">
<div class="panel-body" markdown-text markdown-no-images>
{{- app.why_want | escape | replace('\r\n', '\n') | replace('\n', '&#10;'|safe) -}}
</div>
</div>
</div>
</div>
</div>
{%- if decision_form -%}
<div class="panel-footer">
<form method="POST">
{{ decision_form.csrf_token }}
<div class="btn-group" role="group" aria-label="Decision">
{{ decision_form.reject(class="btn btn-danger") }}
{{ decision_form.accept(class="btn btn-success") }}
</div>
</form>
</div>
{%- endif -%}
</div>
<div class="panel panel-default">
<div class="panel-heading">
<h3 class="panel-title">Reviews - {{ app.reviews | length }}</h3>
</div>
<div class="panel-body">
{% for rev in app.reviews %}
<div class="panel {{ review_class(rev.recommendation) -}}">
<div class="panel-heading">
<h3 class="panel-title">{{ rev.reviewer.username }}'s Review</h3>
</div>
<div class="panel-body">
<div markdown-text>
{{- rev.comment | escape | replace('\r\n', '\n') | replace('\n', '&#10;'|safe) -}}
</div>
</div>
<div class="panel-footer">
{%- if rev.recommendation.name == 'ABSTAIN' -%}
{{ rev.reviewer.username }} does not give an explicit recommendation.
{%- else -%}
{{ rev.reviewer.username }} recommends to <strong>{{ rev.recommendation.name.lower() }}</strong> this application.
{%- endif -%}
</div>
</div>
{% endfor %}
<form method="POST">
{{ review_form.csrf_token }}
<div class="row">
<div class="col-xs-12 col-md-8 col-sm-10">
{{ render_field(review_form.comment, class_="form-control") }}
</div>
</div>
<div class="row">
<div class="col-xs-3">
{{ render_menu_with_button(review_form.recommendation, 'Submit') }}
</div>
</div>
</form>
</div>
</div>
{% endblock %}

12
nyaa/templates/bootstrap/pagination.html
View file

@ -27,7 +27,11 @@
<ul class="pagination{% if size %} pagination-{{size}}{% endif %}"{{kwargs|xmlattr}}>
{# prev and next are only show if a symbol has been passed. #}
{% if prev != None -%}
<li{% if not pagination.has_prev %} class="disabled"{% endif %}><a href="{{_arg_url_for(endpoint, url_args, p=pagination.prev_num) if pagination.has_prev else '#'}}">{{prev}}</a></li>
{% if pagination.has_prev %}
<li><a rel="prev" href="{{_arg_url_for(endpoint, url_args, p=pagination.prev_num)}}">{{prev}}</a></li>
{% else %}
<li class="disabled"><a href="#">{{prev}}</a></li>
{% endif %}
{%- endif -%}
{%- for page in pagination.iter_pages(left_edge=2, left_current=6, right_current=6, right_edge=0) %}
@ -43,7 +47,11 @@
{%- endfor %}
{% if next != None -%}
<li{% if not pagination.has_next %} class="disabled"{% endif %}><a href="{{_arg_url_for(endpoint, url_args, p=pagination.next_num) if pagination.has_next else '#'}}">{{next}}</a></li>
{% if pagination.has_next %}
<li><a rel="next" href="{{_arg_url_for(endpoint, url_args, p=pagination.next_num)}}">{{next}}</a></li>
{% else %}
<li class="disabled"><a href="#">{{next}}</a></li>
{% endif %}
{%- endif -%}
</ul>
</nav>

8
nyaa/templates/edit.html
View file

@ -68,6 +68,14 @@
Trusted
</label>
{% endif %}
{% if g.user.is_moderator %}
<label class="btn btn-default {% if torrent.comment_locked %}active{% endif %}" title="Lock comments">
{{ form.is_comment_locked }}
<span class="glyphicon glyphicon-check"></span>
<span class="glyphicon glyphicon-unchecked"></span>
Lock Comments
</label>
{% endif %}
</div>
</div>
</div>

14
nyaa/templates/email/trusted.html Normal file
View file

@ -0,0 +1,14 @@
<html>
<head>
<title>Your {{ config.GLOBAL_SITE_NAME }} Trusted Application was {{ 'accepted' if is_accepted else 'rejected' }}</title>
</head>
<body>
{% if is_accepted %}
<p>Congratulations! Your Trusted status application on {{ config.GLOBAL_SITE_NAME }} was accepted. You can now edit your torrents and set the Trusted flag on them.</p>
{% else %}
<p>We're sorry to inform you that we've rejected your Trusted status application on {{ config.GLOBAL_SITE_NAME }}. You can re-apply for Trusted status in {{ config.TRUSTED_REAPPLY_COOLDOWN }} days if you wish to do so.</p>
{% endif %}
<p>Regards<br/>
The {{ config.GLOBAL_SITE_NAME }} Moderation Team</p>
</body>
</html>

8
nyaa/templates/email/trusted.txt Normal file
View file

@ -0,0 +1,8 @@
{% if is_accepted %}
Congratulations! Your Trusted status application on {{ config.GLOBAL_SITE_NAME }} was accepted. You can now edit your torrents and set the Trusted flag on them.
{% else %}
We're sorry to inform you that we've rejected your Trusted status application on {{ config.GLOBAL_SITE_NAME }}. You can re-apply for Trusted status in {{ config.TRUSTED_REAPPLY_COOLDOWN }} days if you wish to do so.
{% endif %}
Regards
The {{ config.GLOBAL_SITE_NAME }} Moderation Team

44
nyaa/templates/help.html
View file

@ -14,7 +14,7 @@
<ul>
<li>Torrents uploaded by trusted users.</li>
</ul>
<span style="color:red; font-weight: bold;">Red</span> entries (remake) are torrents that matching any of the following:
<span style="color:red; font-weight: bold;">Red</span> entries (remake) are torrents that match any of the following:
<ul>
<li>Reencode of original release.</li>
<li>Remux of another uploader's original release for hardsubbing and/or fixing purposes.</li>
@ -38,21 +38,25 @@
</div>
<div>
You can combine search terms with the <kbd>|</kbd> operator, such as
<kbd>horrible|cartel</kbd>.
<kbd>foo|bar</kbd>, to match any of the words instead all of them.
</div>
<div>
To exclude results matching a certain word, prefix them with <kbd>-</kbd>,
e.g. <kbd>FFF -memesubs</kbd>, which will return torrents with <em>FFF</em> in the
name, but not those which have <em>memesubs</em> in the name as well.
e.g. <kbd>foo -bar</kbd>, which will return torrents with <em>foo</em> in the
name, but not those which have <em>bar</em> in the name as well.
</div>
<div>
If you want to search for a several-word expression in its entirety, you can
If you want to search for a several-word expression (substring) in its entirety, you can
surround searches with <kbd>"</kbd> (double quotes), such as
<kbd>"foo bar"</kbd>, which would match torrents named <em>foo bar</em> but not
those named <em>bar foo</em>.
those named <em>bar foo</em>. You may also use the aforementioned <kbd>|</kbd> to group
phrases together: <kbd>"foo bar"|"foo baz"</kbd>. You can negate the entire
group with <kbd>-</kbd> (e.g. <kbd>-"foo bar"|"foo baz"</kbd>), but not single items.
</div>
<div>
You can also use <kbd>(</kbd> and <kbd>)</kbd> to signify precedence.
You can also use <kbd>(</kbd> and <kbd>)</kbd> to signify precedence, but quoted strings do
not honor this. Using <kbd>(hello world) "foo bar"</kbd> is fine, but quoted strings inside
the parentheses will lead to unexpected results.
</div>
{{ linkable_header("Reporting Torrents", "reporting") }}
@ -118,6 +122,30 @@
At the moment we have no established process for granting trusted status to users
who did not previously have it. If and when we establish such a process it will be announced.
</div>
{{ linkable_header("IRC Help Channel Policies", "irchelp") }}
<div>
<p>Our IRC help channel is at Rizon <a>#nyaa-help</a>. A webchat link
pre-filled with our channel is available <a>right here</a>.</p>
<b>Read this to avoid getting banned:</b>
<ul>
<li>The IRC channel is for site support <b>only</b>.</li>
<li>XDCC, similar services, and their triggers are not allowed.</li>
<li>Do not idle if you do not need site support unless you have voice/+ access, you may be removed otherwise</li>
<li>We do not know when A or B will be released, if it's authentic, or anything about a particular release. Do not ask.</li>
<li><b>Requests are not allowed.</b> We only manage the site; we do not necessarily have the material you want on hand.</li>
<li>Use English only. Even though we aren't all from English-speaking countries, we need level ground to communicate on.</li>
<li><b>Do NOT under any circumstances send private messages to the staff. Ask your question in the channel on joining and wait; a staff member will respond in due time.</b></li>
</ul>
<b>Keep these things in mind when asking for help:</b>
<ul>
<li>We are not interested in your user name. Paste a link to your account if you want us to do something with it.</li>
<li>Provide as many details as possible. If you are having trouble submitting any kind of entry, we want to know everything about you and what (except any passwords) you supply to the form in question.</li>
</ul>
</div>
{# <div class="content">
<h1>Help</h1>
<p><b>The search engine</b> is located at the top right, and it allows users to search through the torrent titles available on the site. Results matching either word A or B can be included by typing a vertical bar between them (|). Results matching a certain word can be excluded by prefixing that word with a hyphen-minus (-). Phrases can be matched by surrounding them with double-quotes (). Search results can be filtered by category, remake, trusted, and/or A+ status, and then narrowed down further by age and size ranges as well as excluding specific users. Sorting can be done in ascending or descending order by date, amount of seeders/leechers/downloads, size, or name. The search engine adapts to the current view and makes it possible to search for specific torrents in a specific subcategory from a specific user.</p>
@ -188,7 +216,7 @@
</li>
</ul>
<h1>IRC help channel</h1><a href="irc://irc-server:port/channel?key">
<h1>NyaaV2 IRC</h1></a>
<h1>NyaaV3 IRC</h1></a>
<p>The IRC channel is only for site support.<br></p>
<p><b>Read this to avoid getting banned:</b></p>
<ul>

5
nyaa/templates/home.html
View file

@ -12,10 +12,7 @@
{% block body %}
{% if not search.term %}
<div class="alert alert-info">
<p>We welcome you to provide feedback on IRC at <a href="irc://irc.rizon.net/nyaa-dev">#nyaa-dev@irc.rizon.net</a></p>
<p>Our GitHub: <a href="https://github.com/nyaadevs" target="_blank">https://github.com/nyaadevs</a> - creating <a href="https://github.com/nyaadevs/nyaa/issues">issues</a> for features and faults is recommended!</p>
</div>
{% include "infobubble.html" %}
{% endif %}
{% include "search_results.html" %}

16
nyaa/templates/infobubble.html Normal file
View file

@ -0,0 +1,16 @@
{# Update this to a larger timestamp if you change your announcement #}
{# A value of 0 disables the announcements altogether #}
{% set info_ts = 0 %}
{% if info_ts > 0 %}
<div class="alert alert-info alert-dismissible" id="infobubble" data-ts='{{ info_ts }}' hidden>
{% include 'infobubble_content.html' %}
<button type="button" class="close" data-dismiss="alert" aria-label="Close">
<span aria-hidden="true">&times;</span>
</button>
</div>
<noscript>
<div class="alert alert-info" id="infobubble-noscript">
{% include 'infobubble_content.html' %}
</div>
</noscript>
{% endif %}

1
nyaa/templates/infobubble_content.html Normal file
View file

@ -0,0 +1 @@
<strong>Put your announcements into <tt>infobubble_content.html</tt>!</strong>

16
nyaa/templates/layout.html
View file

@ -4,7 +4,7 @@
<meta charset="utf-8">
<title>{% block title %}{{ config.SITE_NAME }}{% endblock %}</title>
<meta name="viewport" content="width=device-width">
<meta name="viewport" content="width=480px">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<link rel="shortcut icon" type="image/png" href="{{ url_for('static', filename='favicon.png') }}">
<link rel="icon" type="image/png" href="{{ url_for('static', filename='favicon.png') }}">
@ -48,8 +48,12 @@
<script src="{{ static_cachebuster('js/lib/markdown-it-ins.min.js') }}"></script>
<script src="{{ static_cachebuster('js/lib/markdown-it-mark.min.js') }}"></script>
<!-- Modified to not apply border-radius to selectpickers and stuff so our navbar looks cool -->
<script src="{{ static_cachebuster('js/bootstrap-select.js') }}"></script>
<script src="{{ static_cachebuster('js/main.js') }}"></script>
{% assets "bs_js" %}
<script src="{{ static_cachebuster('js/bootstrap-select.min.js') }}"></script>
{% endassets %}
{% assets "main_js" %}
<script src="{{ static_cachebuster('js/main.min.js') }}"></script>
{% endassets %}
<!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
<!--[if lt IE 9]>
@ -83,12 +87,13 @@
<li {% if request.path == url_for('torrents.upload') %}class="active"{% endif %}><a href="{{ url_for('torrents.upload') }}">Upload</a></li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">
About
Info
<span class="caret"></span>
</a>
<ul class="dropdown-menu">
<li {% if request.path == url_for('site.rules') %}class="active"{% endif %}><a href="{{ url_for('site.rules') }}">Rules</a></li>
<li {% if request.path == url_for('site.help') %}class="active"{% endif %}><a href="{{ url_for('site.help') }}">Help</a></li>
<li {% if request.path == url_for('site.trusted') %}class="active"{% endif %}><a href="{{ url_for('site.trusted') }}">Trusted</a></li>
</ul>
</li>
<li><a href="{% if rss_filter %}{{ url_for('main.home', page='rss', **rss_filter) }}{% else %}{{ url_for('main.home', page='rss') }}{% endif %}">RSS</a></li>
@ -107,6 +112,7 @@
<li {% if request.path == url_for('admin.reports') %}class="active"{% endif %}><a href="{{ url_for('admin.reports') }}">Reports</a></li>
<li {% if request.path == url_for('admin.log') %}class="active"{% endif %}><a href="{{ url_for('admin.log') }}">Log</a></li>
<li {% if request.path == url_for('admin.bans') %}class="active"{% endif %}><a href="{{ url_for('admin.bans') }}">Bans</a></li>
<li {% if request.path == url_for('admin.trusted') %}class="active"{% endif %}><a href="{{ url_for('admin.trusted') }}">Trusted</a></li>
</ul>
</li>
{% endif %}
@ -328,7 +334,7 @@
<footer style="text-align: center;">
<p>Dark Mode: <a href="#" id="themeToggle">Toggle</a></p>
{% if config.COMMIT_HASH %}
<p>Commit: <a href="https://github.com/nyaadevs/nyaa/tree/{{ config.COMMIT_HASH }}">{{ config.COMMIT_HASH[:7] }}</a></p>
<p>Commit: <a href="https://github.com/sb745/NyaaV3/tree/{{ config.COMMIT_HASH }}">{{ config.COMMIT_HASH[:7] }}</a></p>
{% endif %}
</footer>
</body>

6
nyaa/templates/login.html
View file

@ -12,7 +12,7 @@
<div class="row">
<div class="form-group col-md-4">
{{ render_field(form.username, class_='form-control', placeholder='Username', autofocus='') }}
{{ render_field(form.username, class_='form-control', placeholder='Username', autofocus='', tabindex='1') }}
</div>
</div>
@ -32,7 +32,7 @@
</small>
{% endif%}
{{ form.password(title=form.password.description, class_='form-control') | safe }}
{{ form.password(title=form.password.description, class_='form-control', tabindex='2') | safe }}
{% if form.password.errors %}
<div class="help-block">
{% if form.password.errors|length < 2 %}
@ -54,7 +54,7 @@
<div class="row">
<div class="col-md-4">
<input type="submit" value="Login" class="btn btn-primary">
<input type="submit" value="Login" class="btn btn-primary" tabindex="3">
</div>
</div>
</form>

137
nyaa/templates/profile.html
View file

@ -19,66 +19,89 @@
</div>
<ul class="nav nav-tabs" id="profileTabs" role="tablist">
<li role="presentation" class="active">
<a href="#password-change" id="password-change-tab" role="tab" data-toggle="tab" aria-controls="profile" aria-expanded="true">Password</a>
</li>
<li role="presentation">
<a href="#email-change" id="email-change-tab" role="tab" data-toggle="tab" aria-controls="profile" aria-expanded="false">Email</a>
</li>
<li role="presentation" class="active">
<a href="#password-change" id="password-change-tab" role="tab" data-toggle="tab" aria-controls="profile" aria-expanded="true">Password</a>
</li>
<li role="presentation">
<a href="#email-change" id="email-change-tab" role="tab" data-toggle="tab" aria-controls="profile" aria-expanded="false">Email</a>
</li>
<li role="presentation">
<a href="#preferences-change" id="preferences-change-tab" role="tab" data-toggle="tab" aria-controls="profile" aria-expanded="false">Preferences</a>
</li>
</ul>
<div class="tab-content">
<div class="tab-pane fade active in" role="tabpanel" id="password-change" aria-labelledby="password-change-tab">
<form method="POST">
{{ form.csrf_token }}
<div class="row">
<div class="form-group col-md-4">
{{ render_field(form.current_password, class_='form-control', placeholder='Current password') }}
</div>
</div>
<div class="row">
<div class="form-group col-md-4">
{{ render_field(form.new_password, class_='form-control', placeholder='New password') }}
</div>
</div>
<div class="row">
<div class="form-group col-md-4">
{{ render_field(form.password_confirm, class_='form-control', placeholder='New password (confirm)') }}
</div>
</div>
<div class="row">
<div class="col-md-4">
<input type="submit" value="Update" class="btn btn-primary">
</div>
</div>
</form>
</div>
<div class="tab-pane fade" role="tabpanel" id="email-change" aria-labelledby="email-change-tab">
<form method="POST">
{{ form.csrf_token }}
<div class="row">
<div class="form-group col-md-4">
<label class="control-label" for="current_email">Current Email</label>
<div>{{ g.user.email }}</div>
</div>
</div>
<div class="row">
<div class="form-group col-md-4">
{{ render_field(form.email, class_='form-control', placeholder='New email address') }}
</div>
</div>
<div class="row">
<div class="form-group col-md-4">
{{ render_field(form.current_password, class_='form-control', placeholder='Current password') }}
</div>
</div>
<div class="row">
<div class="col-md-4">
<input type="submit" value="Update" class="btn btn-primary">
</div>
</div>
</form>
</div>
<div class="tab-pane fade active in" role="tabpanel" id="password-change" aria-labelledby="password-change-tab">
<form method="POST">
{{ form.csrf_token }}
<div class="row">
<div class="form-group col-md-4">
{{ render_field(form.current_password, class_='form-control', placeholder='Current password') }}
</div>
</div>
<div class="row">
<div class="form-group col-md-4">
{{ render_field(form.new_password, class_='form-control', placeholder='New password') }}
</div>
</div>
<div class="row">
<div class="form-group col-md-4">
{{ render_field(form.password_confirm, class_='form-control', placeholder='New password (confirm)') }}
</div>
</div>
<div class="row">
<div class="col-md-4">
{{ form.authorized_submit(class_='btn btn-primary') }}
</div>
</div>
</form>
</div>
<div class="tab-pane fade" role="tabpanel" id="email-change" aria-labelledby="email-change-tab">
<form method="POST">
{{ form.csrf_token }}
<div class="row">
<div class="form-group col-md-4">
<label class="control-label" for="current_email">Current Email</label>
<div>{{ g.user.email }}</div>
</div>
</div>
<div class="row">
<div class="form-group col-md-4">
{{ render_field(form.email, class_='form-control', placeholder='New email address') }}
</div>
</div>
<div class="row">
<div class="form-group col-md-4">
{{ render_field(form.current_password, class_='form-control', placeholder='Current password') }}
</div>
</div>
<div class="row">
<div class="col-md-4">
{{ form.authorized_submit(class_='btn btn-primary') }}
</div>
</div>
</form>
</div>
<div class="tab-pane fade" role="tabpanel" id="preferences-change" aria-labelledby="preferences-change-tab">
<form method="POST">
{{ form.csrf_token }}
<div class="row">
<div class="form-group col-md-4">
{% if g.user.preferences.hide_comments %}
{{ form.hide_comments(checked='') }}
{% else %}
{{ form.hide_comments }}
{% endif %}
{{ form.hide_comments.label }}
</div>
</div>
<div class="row">
<div class="col-md-4">
{{ form.submit_settings(class_='btn btn-primary') }}
</div>
</div>
</form>
</div>
</div>
<hr>

1
nyaa/templates/register.html
View file

@ -7,6 +7,7 @@
{% from "_formhelpers.html" import render_field %}
<h1>Register</h1>
<p><strong>Important:</strong> Do not use Outlook (Hotmail/Live/MSN) email addresses, they discard our verification email without sending it to spam. No support is offered if you ignore this warning.</p>
<form method="POST">
{{ form.csrf_token }}

5
nyaa/templates/rss.xml
View file

@ -12,7 +12,7 @@
{% if torrent.has_torrent and not magnet_links %}
<link>{{ url_for('torrents.download', torrent_id=torrent.meta.id, _external=True) }}</link>
{% else %}
<link>{{ create_magnet_from_es_info(torrent.display_name, torrent.info_hash) }}</link>
<link>{{ create_magnet_from_es_torrent(torrent) }}</link>
{% endif %}
<guid isPermaLink="true">{{ url_for('torrents.view', torrent_id=torrent.meta.id, _external=True) }}</guid>
<pubDate>{{ torrent.created_time|rfc822_es }}</pubDate>
@ -40,6 +40,9 @@
<nyaa:categoryId>{{- cat_id }}</nyaa:categoryId>
<nyaa:category> {{- category_name(cat_id) }}</nyaa:category>
<nyaa:size> {{- torrent.filesize | filesizeformat(True) }}</nyaa:size>
<nyaa:comments> {{- torrent.comment_count }}</nyaa:comments>
<nyaa:trusted> {{- torrent.trusted and 'Yes' or 'No' }}</nyaa:trusted>
<nyaa:remake> {{- torrent.remake and 'Yes' or 'No' }}</nyaa:remake>
{% set torrent_id = use_elastic and torrent.meta.id or torrent.id %}
<description><![CDATA[<a href="{{ url_for('torrents.view', torrent_id=torrent_id, _external=True) }}">#{{ torrent_id }} | {{ torrent.display_name }}</a> | {{ torrent.filesize | filesizeformat(True) }} | {{ category_name(cat_id) }} | {{ use_elastic and torrent.info_hash or torrent.info_hash_as_hex | upper }}]]></description>
</item>

92
nyaa/templates/search_results.html
View file

@ -1,11 +1,11 @@
{% macro render_column_header(header_class, header_style, center_text=False, sort_key=None, header_title=None) %}
{% set class_suffix = (search.sort == sort_key) and ("_" + search.order) or "" %}
{% set th_classes = filter_truthy([header_class, sort_key and "sorting" + class_suffix, center_text and "text-center"]) %}
<th {% if th_classes %} class="{{ ' '.join(th_classes) }}"{% endif %} {% if header_title %}title="{{ header_title }}"{% endif %} style="{{ header_style }}">
{% if sort_key %}
<th {% if th_classes %}class="{{ ' '.join(th_classes) }}"{% endif %} {% if header_title %}title="{{ header_title }}" {% endif %}style="{{ header_style }}">
{%- if sort_key -%}
<a href="{% if class_suffix == '_desc' %}{{ modify_query(s=sort_key, o="asc") }}{% else %}{{ modify_query(s=sort_key, o="desc") }}{% endif %}"></a>
{% endif %}
{{ caller() }}
{%- endif -%}
{{- caller() -}}
</th>
{% endmacro %}
@ -17,57 +17,57 @@
{% endif %}
{% endif %}
{% if (use_elastic and torrent_query.hits.total > 0) or (torrent_query.items) %}
{% if (use_elastic and torrent_query.hits.total.value > 0) or (torrent_query.items) %}
<div class="table-responsive">
<table class="table table-bordered table-hover table-striped torrent-list">
<thead>
<tr>
{% call render_column_header("hdr-category", "width:80px;", center_text=True) %}
<div>Category</div>
{% endcall %}
{% call render_column_header("hdr-name", "width:auto;") %}
<div>Name</div>
{% endcall %}
{% call render_column_header("hdr-comments", "width:50px;", center_text=True, sort_key="comments", header_title="Comments") %}
{%+ call render_column_header("hdr-category", "width:80px;", center_text=True) -%}
Category
{%- endcall %}
{%+ call render_column_header("hdr-name", "width:auto;") -%}
Name
{%- endcall %}
{%+ call render_column_header("hdr-comments", "width:50px;", center_text=True, sort_key="comments", header_title="Comments") -%}
<i class="fa fa-comments-o"></i>
{% endcall %}
{% call render_column_header("hdr-link", "width:70px;", center_text=True) %}
<div>Link</div>
{% endcall %}
{% call render_column_header("hdr-size", "width:100px;", center_text=True, sort_key="size") %}
<div>Size</div>
{% endcall %}
{% call render_column_header("hdr-date", "width:140px;", center_text=True, sort_key="id", header_title="In UTC") %}
<div>Date</div>
{% endcall %}
{%- endcall %}
{%+ call render_column_header("hdr-link", "width:70px;", center_text=True) -%}
Link
{%- endcall %}
{%+ call render_column_header("hdr-size", "width:100px;", center_text=True, sort_key="size") -%}
Size
{%- endcall %}
{%+ call render_column_header("hdr-date", "width:140px;", center_text=True, sort_key="id", header_title="In UTC") -%}
Date
{%- endcall %}
{% if config.ENABLE_SHOW_STATS %}
{% call render_column_header("hdr-seeders", "width:50px;", center_text=True, sort_key="seeders", header_title="Seeders") %}
{%+ call render_column_header("hdr-seeders", "width:50px;", center_text=True, sort_key="seeders", header_title="Seeders") -%}
<i class="fa fa-arrow-up" aria-hidden="true"></i>
{% endcall %}
{% call render_column_header("hdr-leechers", "width:50px;", center_text=True, sort_key="leechers", header_title="Leechers") %}
{%- endcall %}
{%+ call render_column_header("hdr-leechers", "width:50px;", center_text=True, sort_key="leechers", header_title="Leechers") -%}
<i class="fa fa-arrow-down" aria-hidden="true"></i>
{% endcall %}
{% call render_column_header("hdr-downloads", "width:50px;", center_text=True, sort_key="downloads", header_title="Completed downloads") %}
{%- endcall %}
{%+ call render_column_header("hdr-downloads", "width:50px;", center_text=True, sort_key="downloads", header_title="Completed downloads") -%}
<i class="fa fa-check" aria-hidden="true"></i>
{% endcall %}
{%- endcall %}
{% endif %}
</tr>
</thead>
<tbody>
{% set icon_dir = config.SITE_FLAVOR %}
{% set torrents = torrent_query if use_elastic else torrent_query.items %}
{% for torrent in torrents %}
<tr class="{% if torrent.deleted %}deleted{% elif torrent.hidden %}warning{% elif torrent.remake %}danger{% elif torrent.trusted %}success{% else %}default{% endif %}">
{% set cat_id = use_elastic and ((torrent.main_category_id|string) + '_' + (torrent.sub_category_id|string)) or torrent.sub_category.id_as_string %}
{% set icon_dir = config.SITE_FLAVOR %}
<td style="padding:0 4px;">
{% if use_elastic %}
<a href="{{ url_for('main.home', c=cat_id) }}" title="{{ category_name(cat_id) }}">
{% else %}
<a href="{{ url_for('main.home', c=cat_id) }}" title="{{ torrent.main_category.name }} - {{ torrent.sub_category.name }}">
{% endif %}
<img src="{{ url_for('static', filename='img/icons/%s/%s.png'|format(icon_dir, cat_id)) }}" alt="{{ category_name(cat_id) }}">
</a>
<td>
{% if use_elastic %}
<a href="{{ url_for('main.home', c=cat_id) }}" title="{{ category_name(cat_id) }}">
{% else %}
<a href="{{ url_for('main.home', c=cat_id) }}" title="{{ torrent.main_category.name }} - {{ torrent.sub_category.name }}">
{% endif %}
<img src="{{ url_for('static', filename='img/icons/%s/%s.png'|format(icon_dir, cat_id)) }}" alt="{{ category_name(cat_id) }}" class="category-icon">
</a>
</td>
<td colspan="2">
{% set torrent_id = torrent.meta.id if use_elastic else torrent.id %}
@ -78,15 +78,17 @@
</a>
{% endif %}
{% if use_elastic %}
<a href="{{ url_for('torrents.view', torrent_id=torrent_id) }}" title="{{ torrent.display_name | escape }}">{%if "highlight" in torrent.meta %}{{ torrent.meta.highlight.display_name[0] | safe }}{% else %}{{torrent.display_name}}{%endif%}</a>
<a href="{{ url_for('torrents.view', torrent_id=torrent_id) }}" title="{{ torrent.display_name | escape }}">{%if "highlight" in torrent.meta %}{{ torrent.meta.highlight.display_name[0] | safe }}{% else %}{{torrent.display_name}}{% endif %}</a>
{% else %}
<a href="{{ url_for('torrents.view', torrent_id=torrent_id) }}" title="{{ torrent.display_name | escape }}">{{ torrent.display_name | escape }}</a>
{% endif %}
</td>
<td class="text-center" style="white-space: nowrap;">
{% if torrent.has_torrent %}<a href="{{ url_for('torrents.download', torrent_id=torrent_id) }}"><i class="fa fa-fw fa-download"></i></a>{% endif %}
<td class="text-center">
{% if torrent.has_torrent %}
<a href="{{ url_for('torrents.download', torrent_id=torrent_id) }}"><i class="fa fa-fw fa-download"></i></a>
{% endif %}
{% if use_elastic %}
<a href="{{ create_magnet_from_es_info(torrent.display_name, torrent.info_hash) }}"><i class="fa fa-fw fa-magnet"></i></a>
<a href="{{ create_magnet_from_es_torrent(torrent) }}"><i class="fa fa-fw fa-magnet"></i></a>
{% else %}
<a href="{{ torrent.magnet_uri }}"><i class="fa fa-fw fa-magnet"></i></a>
{% endif %}
@ -100,12 +102,12 @@
{% if config.ENABLE_SHOW_STATS %}
{% if use_elastic %}
<td class="text-center" style="color: green;">{{ torrent.seed_count }}</td>
<td class="text-center" style="color: red;">{{ torrent.leech_count }}</td>
<td class="text-center">{{ torrent.seed_count }}</td>
<td class="text-center">{{ torrent.leech_count }}</td>
<td class="text-center">{{ torrent.download_count }}</td>
{% else %}
<td class="text-center" style="color: green;">{{ torrent.stats.seed_count }}</td>
<td class="text-center" style="color: red;">{{ torrent.stats.leech_count }}</td>
<td class="text-center">{{ torrent.stats.seed_count }}</td>
<td class="text-center">{{ torrent.stats.leech_count }}</td>
<td class="text-center">{{ torrent.stats.download_count }}</td>
{% endif %}
{% endif %}

17
nyaa/templates/trusted.html Normal file
View file

@ -0,0 +1,17 @@
{% extends "layout.html" %}
{% block title %}Trusted :: {{ config.SITE_NAME }}{% endblock %}
{% block body %}
<div class="content">
<div class="row">
<div class="col-md-12">
{% include "trusted_rules.html" %}
</div>
</div>
<div class="row">
<div class="col-md-12">
<a href="{{ url_for('account.request_trusted') }}" class="btn btn-success btn-lg">Request Trusted Status</a>
</div>
</div>
</div>
{% endblock %}

51
nyaa/templates/trusted_form.html Normal file
View file

@ -0,0 +1,51 @@
{% extends "layout.html" %}
{% from "_formhelpers.html" import render_field %}
{% block title %}Apply for Trusted :: {{ config.SITE_NAME }}{% endblock %}
{% block body %}
<div class="content">
{% if trusted_form %}
<div class="row">
<div class="col-md-12">
<h1>You are eligible to apply for trusted status</h1>
</div>
</div>
<form class="trusted-form" method="POST">
{{ trusted_form.csrf_token }}
<div class="row">
<div class="col-md-6">
{{ render_field(trusted_form.why_give_trusted, class_='form-control') }}
</div>
</div>
<div class="row">
<div class="col-md-6">
{{ render_field(trusted_form.why_want_trusted, class_='form-control') }}
</div>
</div>
<div class="row">
<div class="col-md-2">
<input type="submit" value="Submit" class="btn btn-success btn-sm">
</div>
</div>
</form>
{% else %}
<div class="row">
<div class="col-md-12">
<h1>You are currently not eligible to apply for trusted status</h1>
</div>
</div>
<div class="row">
<div class="col-md-12">
<p>
You currently are not eligible to apply for trusted status for the following
reason{% if deny_reasons|length > 1 %}s{% endif %}:
</p>
<ul>
{% for reason in deny_reasons %}
<li>{{ reason }}</li>
{% endfor %}
</ul>
</div>
</div>
{% endif %}
</div>
{% endblock %}

1
nyaa/templates/trusted_rules.html Normal file
View file

@ -0,0 +1 @@
<h1>Trusted rules go here</h1>

12
nyaa/templates/upload.html
View file

@ -37,6 +37,18 @@
</div>
{% endif %}
{% if upload_form.rangebanned.errors %}
<div class="row">
<div class="col-md-12">
<div class="alert alert-danger" role="alert">
{% for error in upload_form.rangebanned.errors %}
<p>{{ error }}</p>
{% endfor %}
</div>
</div>
</div>
{% endif %}
<div class="row">
<div class="col-md-12">
{{ render_upload(upload_form.torrent_file, accept=".torrent") }}

89
nyaa/templates/user.html
View file

@ -18,31 +18,46 @@
<div class="row" style="margin-bottom: 20px;">
<div class="col-md-2" style="max-width: 150px;">
<img class="avatar" src="{{ user.gravatar_url() }}">
<a href="{{ url_for('users.view_user_comments', user_name=user.username) }}">View all comments</a>
</div>
<div class="col-md-4">
<dl class="dl-horizontal">
<dt>User ID:</dt>
<dd>{{ user.id }}</dd>
<dt>Account created on:</dt>
<dd data-timestamp="{{ user.created_utc_timestamp|int }}">{{ user.created_time.strftime('%Y-%m-%d %H:%M UTC') }}</dd>
<dt>Email address:</dt>
<dd>{{ user.email }}</dd>
<dt>User class:</dt>
<dd>{{ user.userlevel_str }}</dd>
<dt>User status:</dt>
<dd>{{ user.userstatus_str }}</dt>
{%- if g.user.is_superadmin -%}
<dt>Last login IP:</dt>
<dd>{{ user.ip_string }}</dd><br>
{%- endif -%}
</dl>
<div class="row">
<div class="col-md-12">
<dl class="dl-horizontal">
<dt>User ID:</dt>
<dd>{{ user.id }}</dd>
<dt>Account created on:</dt>
<dd data-timestamp="{{ user.created_utc_timestamp|int }}">{{ user.created_time.strftime('%Y-%m-%d %H:%M UTC') }}</dd>
<dt>Email address:</dt>
<dd>{{ user.email }}</dd>
<dt>User class:</dt>
<dd>{{ user.userlevel_str }}</dd>
<dt>User status:</dt>
<dd>{{ user.userstatus_str }}</dt>
{%- if g.user.is_superadmin -%}
<dt>Last login IP:</dt>
<dd>{{ user.ip_string }}</dd>
<dt>Registration IP:</dt>
<dd>{{ user.reg_ip_string }}</dd>
{%- endif -%}
</dl>
</div>
</div>
{% if admin_form %}
<form method="POST">
{{ admin_form.csrf_token }}
<div class="form-group">
{{ render_menu_with_button(admin_form.user_class) }}
<div class="row">
<div class="col-md-6">
{{ render_menu_with_button(admin_form.user_class) }}
</div>
</div>
{% if not user.is_active %}
<div class="row">
<div class="col-md-6">
{{ admin_form.activate_user(class="btn btn-primary") }}
</div>
</div>
{% endif %}
</form>
<br>
{% endif %}
@ -72,13 +87,13 @@
{% if user.is_banned or bans %}
<div class="row">
<div class="col-md-12">
<p>
<ul>
{% for ban in bans %}
#{{ ban.id }}
<li>#{{ ban.id }}
by <a href="{{ url_for('users.view_user', user_name=ban.admin.username) }}">{{ ban.admin.username }}</a>
for <span markdown-text-inline>{{ ban.reason }}</span>
for <span markdown-text-inline>{{ ban.reason }}</span></li>
{% endfor %}
</p>
</ul>
</div>
</div>
<div class="row">
@ -98,30 +113,37 @@
</div>
</div>
<div class="row">
<div class="col-md-4 text-left">
<div class="col-md-6 text-left">
{% if not user.is_banned %}
{{ ban_form.ban_user(value="Ban User", class="btn btn-danger") }}
{% else %}
<button type="button" class="btn btn-danger disabled">Already banned</button>
{% endif %}
</div>
<div class="col-md-4 text-center">
<div class="col-md-6 text-right">
{% if not ipbanned %}
{{ ban_form.ban_userip(value="Ban User+IP", class="btn btn-danger") }}
{% else %}
<button type="button" class="btn btn-danger disabled">Already IP banned</button>
{% endif %}
</div>
<div class="col-md-4 text-right">
{% if g.user.is_superadmin %}
{{ ban_form.nuke(value="\U0001F4A3 Nuke Torrents", class="btn btn-danger") }}
{% else %}
<button type="button" class="btn btn-danger disabled">&#x1f4a3; Nuke Torrents</button>
{% endif %}
</div>
</div>
{% endif %}
</form>
{% if g.user.is_superadmin %}
<hr>
<form method="POST" onsubmit="return prompt('Please type {{ user.username }} to confirm') == '{{ user.username }}'">
{{ nuke_form.csrf_token }}
<div class="row">
<div class="col-md-6 text-left">
{{ nuke_form.nuke_torrents(class="btn btn-danger", formaction=url_for('users.nuke_user_torrents', user_name=user.username)) }}
</div>
<div class="col-md-6 text-right">
{{ nuke_form.nuke_comments(class="btn btn-danger", formaction=url_for('users.nuke_user_comments', user_name=user.username)) }}
</div>
</div>
</form>
{% endif %}
</div>
</div>
</div>
@ -132,6 +154,9 @@
<div class="row">
<h3>
Browsing <span class="text-{{ user.userlevel_color }}" data-toggle="tooltip" title="{{ user.userlevel_str }}">{{ user.username }}</span>'{{ '' if user.username[-1] == 's' else 's' }} torrents
{% if torrent_query.actual_count is number and not search.term: %}
({{ torrent_query.actual_count }})
{% endif %}
</h3>
{% include "search_results.html" %}

72
nyaa/templates/user_comments.html Normal file
View file

@ -0,0 +1,72 @@
{% extends "layout.html" %}
{% block title %}Comments made by {{ user.username }} :: {{ config.SITE_NAME }}{% endblock %}
{% block meta_image %}{{ user.gravatar_url() }}{% endblock %}
{% block metatags %}
<meta property="og:description" content="Comments made by {{ user.username }}">
{% endblock %}
{% block body %}
{% from "_formhelpers.html" import render_menu_with_button %}
{% from "_formhelpers.html" import render_field %}
<h3>
<span class="text-{{ user.userlevel_color }}" data-toggle="tooltip" title="{{ user.userlevel_str }}">{{ user.username }}</span>'{{ '' if user.username[-1] == 's' else 's' }} comments
</h3>
{% if comments_query.items %}
<div id="comments" class="panel panel-default">
<div class="panel-heading">
<h3 class="panel-title">
Total of {{ comments_query.total }} comments
</h3>
</div>
{% for comment in comments_query.items %}
<div class="panel panel-default comment-panel">
<div class="panel-body">
<div class="col-md-2">
<p>
<a class="text-{{ comment.user.userlevel_color }}" href="{{ url_for('users.view_user', user_name=comment.user.username) }}" data-toggle="tooltip" title="{{ comment.user.userlevel_str }}">{{ comment.user.username }}</a>
</p>
<img class="avatar" src="{{ comment.user.gravatar_url() }}" alt="{{ comment.user.userlevel_str }}">
</div>
<div class="col-md-10 comment">
<div class="row comment-details">
<a href="#com-{{ loop.index }}"><small data-timestamp-swap data-timestamp="{{ comment.created_utc_timestamp|int }}">{{ comment.created_time.strftime('%Y-%m-%d %H:%M UTC') }}</small></a>
{% if comment.edited_time %}
<small data-timestamp-swap data-timestamp-title data-timestamp="{{ comment.edited_utc_timestamp }}" title="{{ comment.edited_time }}">(edited)</small>
{% endif %}
<small>on torrent <a href="{{ url_for('torrents.view', torrent_id=comment.torrent_id) }}">#{{comment.torrent_id}} <i>{{ comment.torrent.display_name }}</i></a></small>
{# <div class="comment-actions">
{% if g.user.id == comment.user_id and not comment.editing_limit_exceeded %}
<button class="btn btn-xs edit-comment" title="Edit"{% if config.EDITING_TIME_LIMIT %} data-until="{{ comment.editable_until|int }}"{% endif %}>Edit</button>
{% endif %}
{% if g.user.is_moderator or g.user.id == comment.user_id %}
<form class="delete-comment-form" action="{{ url_for('torrents.delete_comment', torrent_id=comment.torrent_id, comment_id=comment.id) }}" method="POST">
<button name="submit" type="submit" class="btn btn-danger btn-xs" title="Delete">Delete</button>
</form>
{% endif %}
</div> #}
</div>
<div class="row">
{# Escape newlines into html entities because CF strips blank newlines #}
<div markdown-text class="comment-content" id="torrent-comment{{ comment.id }}">{{- comment.text | escape | replace('\r\n', '\n') | replace('\n', '&#10;'|safe) -}}</div>
</div>
</div>
</div>
</div>
{% endfor %}
</div>
{% else %}
<h3>No comments</h3>
{% endif %}
<div class="center">
{% from "bootstrap/pagination.html" import render_pagination %}
{{ render_pagination(comments_query) }}
</div>
{% endblock %}
</div>

74
nyaa/templates/view.html
View file

@ -74,7 +74,7 @@
<div class="panel-footer clearfix">
{% if torrent.has_torrent %}<a href="{{ url_for('torrents.download', torrent_id=torrent.id )}}"><i class="fa fa-download fa-fw"></i>Download Torrent</a> or {% endif %}<a href="{{ torrent.magnet_uri }}" class="card-footer-item"><i class="fa fa-magnet fa-fw"></i>Magnet</a>
{% if g.user %}
{% if g.user and g.user.age > config['RATELIMIT_ACCOUNT_AGE'] %}
<button type="button" class="btn btn-xs btn-danger pull-right" data-toggle="modal" data-target="#reportModal">
Report
</button>
@ -93,6 +93,7 @@
</div>
</div>
{% cache 86400, "filelist", torrent.info_hash_as_hex %}
{% if files and files.__len__() <= config.MAX_FILES_VIEW %}
<div class="panel panel-default">
<div class="panel-heading">
@ -133,13 +134,17 @@
</div>
</div>
{% endif %}
{% endcache %}
<div id="comments" class="panel panel-default">
<div class="panel-heading">
<a data-toggle="collapse" href="#collapse-comments" role="button" aria-expanded="{% if g.user and g.user.preferences.hide_comments %}false{% else %}true{% endif %}" aria-controls="collapse-comments">
<h3 class="panel-title">
Comments - {{ comments | length }}
Comments - {{ torrent.comment_count }}
</h3>
</a>
</div>
<div class="collapse {% if g.user and g.user.preferences.hide_comments %}{% else %}in{% endif %}" id="collapse-comments">
{% for comment in comments %}
<div class="panel panel-default comment-panel" id="com-{{ loop.index }}">
<div class="panel-body">
@ -159,25 +164,42 @@
<small data-timestamp-swap data-timestamp-title data-timestamp="{{ comment.edited_utc_timestamp }}" title="{{ comment.edited_time }}">(edited)</small>
{% endif %}
<div class="comment-actions">
{% if g.user.id == comment.user_id and not comment.editing_limit_exceeded %}
{% if g.user.id == comment.user_id and not comment.editing_limit_exceeded and (not torrent.comment_locked or comment_form) %}
<button class="btn btn-xs edit-comment" title="Edit"{% if config.EDITING_TIME_LIMIT %} data-until="{{ comment.editable_until|int }}"{% endif %}>Edit</button>
{% endif %}
{% if g.user.is_moderator or g.user.id == comment.user_id %}
{% if g.user.is_superadmin or (g.user.id == comment.user_id and not torrent.comment_locked and not comment.editing_limit_exceeded) %}
<form class="delete-comment-form" action="{{ url_for('torrents.delete_comment', torrent_id=torrent.id, comment_id=comment.id) }}" method="POST">
<button name="submit" type="submit" class="btn btn-danger btn-xs" title="Delete">Delete</button>
</form>
{% endif %}
</div>
</div>
<div class="row">
<div class="row comment-body">
{# Escape newlines into html entities because CF strips blank newlines #}
<div markdown-text class="comment-content" id="torrent-comment{{ comment.id }}">{{- comment.text | escape | replace('\r\n', '\n') | replace('\n', '&#10;'|safe) -}}</div>
{% if g.user.id == comment.user_id %}
{% if g.user.id == comment.user_id and comment_form %}
<form class="edit-comment-box" action="{{ url_for('torrents.edit_comment', torrent_id=torrent.id, comment_id=comment.id) }}" method="POST">
{{ comment_form.csrf_token }}
<div class="form-group">
<textarea class="form-control" name="comment" autofocus>{{- comment.text | escape | replace('\r\n', '\n') | replace('\n', '&#10;'|safe) -}}</textarea>
</div>
{% if config.USE_RECAPTCHA and g.user.age < config['ACCOUNT_RECAPTCHA_AGE'] %}
<div class="row">
<div class="col-md-4">
{% if comment_form.recaptcha.errors %}
<div class="alert alert-danger">
<p><strong>CAPTCHA error:</strong></p>
<ul>
{% for error in comment_form.recaptcha.errors %}
<li>{{ error }}</li>
{% endfor %}
</ul>
</div>
{% endif %}
{{ comment_form.recaptcha }}
</div>
</div>
{% endif %}
<input type="submit" value="Submit" class="btn btn-success btn-sm">
<button class="btn btn-sm edit-comment" title="Cancel">Cancel</button>
<span class="edit-error text-danger"></span>
@ -190,16 +212,50 @@
</div>
{% endfor %}
{% if torrent.comment_locked %}
<div class="alert alert-warning">
<p>
<i class="fa fa-lock" aria-hidden="true"></i>
Comments have been locked.
</p>
</div>
{% endif %}
{% if comment_form %}
<form class="comment-box" method="POST">
{{ comment_form.csrf_token }}
{{ render_field(comment_form.comment, class_='form-control') }}
<input type="submit" value="Submit" class="btn btn-success btn-sm">
<div class="row">
<div class="col-md-12">
{{ render_field(comment_form.comment, class_='form-control') }}
</div>
</div>
{% if config.USE_RECAPTCHA and g.user.age < config['ACCOUNT_RECAPTCHA_AGE'] %}
<div class="row">
<div class="col-md-4">
{% if comment_form.recaptcha.errors %}
<div class="alert alert-danger">
<p><strong>CAPTCHA error:</strong></p>
<ul>
{% for error in comment_form.recaptcha.errors %}
<li>{{ error }}</li>
{% endfor %}
</ul>
</div>
{% endif %}
{{ comment_form.recaptcha }}
</div>
</div>
{% endif %}
<div class="row">
<div class="col-md-12">
<input type="submit" value="Submit" class="btn btn-success btn-sm">
</div>
</div>
</form>
{% endif %}
</div>
</div>
{% if g.user %}
{% if g.user and g.user.age > config['RATELIMIT_ACCOUNT_AGE'] %}
<div class="modal fade" id="reportModal" tabindex="-1" role="dialog" aria-labelledby="reportModalLabel">
<div class="modal-dialog" role="document">
<div class="modal-content">

9
nyaa/templates/xmlns.html
View file

@ -27,6 +27,15 @@
<li>
<p><code>&lt;nyaa:size&gt;</code> indicates the torrent's download size to one decimal place, using a magnitude prefix according to ISO/IEC 80000-13.</p>
</li>
<li>
<p><code>&lt;nyaa:trusted&gt;</code> indicates whether the torrent came from a trusted uploader (YES or NO).</p>
</li>
<li>
<p><code>&lt;nyaa:remake&gt;</code> indicates whether the torrent was a remake (YES or NO).</p>
</li>
<li>
<p><code>&lt;nyaa:comments&gt;</code> holds the current amount of comments made on the respective torrent.</p>
</li>
</ul>
</div>
{% endblock %}

61
nyaa/torrents.py
View file

@ -1,18 +1,15 @@
import base64
import functools
import os
import time
from urllib.parse import urlencode
from urllib.parse import quote, urlencode
import flask
from flask import current_app as app
from orderedset import OrderedSet
from orderly_set import OrderlySet
from nyaa import bencode
USED_TRACKERS = OrderedSet()
# Limit the amount of trackers added into .torrent files
MAX_TRACKERS = 5
USED_TRACKERS = OrderlySet()
def read_trackers_from_file(file_object):
@ -20,7 +17,7 @@ def read_trackers_from_file(file_object):
for line in file_object:
line = line.strip()
if line:
if line and not line.startswith('#'):
USED_TRACKERS.add(line)
return USED_TRACKERS
@ -40,8 +37,8 @@ def default_trackers():
def get_trackers_and_webseeds(torrent):
trackers = OrderedSet()
webseeds = OrderedSet()
trackers = OrderlySet()
webseeds = OrderlySet()
# Our main one first
main_announce_url = app.config.get('MAIN_ANNOUNCE_URL')
@ -66,7 +63,7 @@ def get_trackers_and_webseeds(torrent):
def get_default_trackers():
trackers = OrderedSet()
trackers = OrderlySet()
# Our main one first
main_announce_url = app.config.get('MAIN_ANNOUNCE_URL')
@ -79,19 +76,34 @@ def get_default_trackers():
return list(trackers)
def create_magnet(torrent, max_trackers=5, trackers=None):
@functools.lru_cache(maxsize=1024*4)
def _create_magnet(display_name, info_hash, max_trackers=5, trackers=None):
# Unless specified, we just use default trackers
if trackers is None:
trackers = get_default_trackers()
magnet_parts = [
('dn', torrent.display_name)
('dn', display_name)
]
for tracker in trackers[:max_trackers]:
magnet_parts.append(('tr', tracker))
magnet_parts.extend(
('tr', tracker_url)
for tracker_url in trackers[:max_trackers]
)
b32_info_hash = base64.b32encode(torrent.info_hash).decode('utf-8')
return 'magnet:?xt=urn:btih:' + b32_info_hash + '&' + urlencode(magnet_parts)
return ''.join([
'magnet:?xt=urn:btih:', info_hash,
'&', urlencode(magnet_parts, quote_via=quote)
])
def create_magnet(torrent):
# Since we accept both models.Torrents and ES objects,
# we need to make sure the info_hash is a hex string
info_hash = torrent.info_hash
if isinstance(info_hash, (bytes, bytearray)):
info_hash = info_hash.hex()
return _create_magnet(torrent.display_name, info_hash)
def create_default_metadata_base(torrent, trackers=None, webseeds=None):
@ -102,9 +114,11 @@ def create_default_metadata_base(torrent, trackers=None, webseeds=None):
webseeds = db_webseeds if webseeds is None else webseeds
metadata_base = {
'created by': 'NyaaV2',
'creation date': int(time.time()),
'comment': 'NyaaV2 Torrent #' + str(torrent.id), # Throw the url here or something neat
'created by': 'NyaaV3',
'creation date': int(torrent.created_utc_timestamp),
'comment': flask.url_for('torrents.view',
torrent_id=torrent.id,
_external=True)
# 'encoding' : 'UTF-8' # It's almost always UTF-8 and expected, but if it isn't...
}
@ -112,7 +126,7 @@ def create_default_metadata_base(torrent, trackers=None, webseeds=None):
metadata_base['announce'] = trackers[0]
if len(trackers) > 1:
# Yes, it's a list of lists with a single element inside.
metadata_base['announce-list'] = [[tracker] for tracker in trackers[:MAX_TRACKERS]]
metadata_base['announce-list'] = [[tracker] for tracker in trackers]
# Add webseeds
if webseeds:
@ -121,7 +135,7 @@ def create_default_metadata_base(torrent, trackers=None, webseeds=None):
return metadata_base
def create_bencoded_torrent(torrent, metadata_base=None):
def create_bencoded_torrent(torrent, bencoded_info, metadata_base=None):
''' Creates a bencoded torrent metadata for a given torrent,
optionally using a given metadata_base dict (note: 'info' key will be
popped off the dict) '''
@ -138,7 +152,6 @@ def create_bencoded_torrent(torrent, metadata_base=None):
prefix = bencode.encode(prefixed_dict)
suffix = bencode.encode(suffixed_dict)
bencoded_info = torrent.info.info_dict
bencoded_torrent = prefix[:-1] + b'4:info' + bencoded_info + suffix[1:]
return bencoded_torrent

Some files were not shown because too many files have changed in this diff Show more