Documentation for DraciDoupe.cz development

This documentation is intended for developers interested in joining development of DraciDoupe.cz.

It assumes you have local working copy of the site. If not, follow installation guide from repository’s README.

Warning

This project has significant legacy hacks that makes it behave unexpectedly for a seasoned Django developer!

Please note:

Note

Proč to tady neni česky?

Stránka je česky, dokumentace pro uživatele taky. Prográmátoři to ale vždycky měli těžší, protože základní stavební kameny programovacích jazyků jsou anglické, stejně jako mnoho komponent, které jsou zde použity. Původní verze doupěte je česko-anglický mišmaš s českými názvy proměnných apod. Výsledek bohužel není moc dobrý.

Nová verze má proto jasné rozhraní: veřejně viditelné texty (stránky, popisky stránek, URL adresy atd.) jsou české, ale v kódu je konzistentně použita angličtina. Protože mít programátorskou dokumentaci v češtině by vedlo ke zmatení (jazyků), je pro konzistenci použita angličtina i zde.

Pochopitelně to má i své nevýhody. Jednou z nich je reprezentace některých čistě českých názvů v angličtině. K tomuto účelu je udržován konzistentní slovník, který se všude používá.

Pro popis toho co se to tu děje a proč vlastně navštivte sekci QNEA (Questions Not Even Asked).

Product Documentation

Those docs describe how site parts should behave from the user perspective. Those docs are still aimed at developers; user docs are on the site directly.

Tavern

Tavern is the discussion core of the server. Users can create arbitrary Tavern Tables and have a discussion there (entries are called Posts).

There are ways to list Tavern Tables:

  • All Tables
  • Only Tables with new post (shows only Tables that user visited previously and contain a new post since the last visit)
  • Bookmarked Tables
  • Only bookmarked Tables with new post

Tables are listed regardless of whether user can access them or not. The ability to visit a table is denoted by whether the name of the table is a clickable link. In addition:

  • Tables are always shown to the owner and assistant
  • In the original version, tables can be ignored and are then never shown in the listing. This is not supported (yet?), see #291.

Access Model

Tables can be either public (all users can visit and view by default) or private (no user can visit or view by default). The default setting can be modified using nick allow list (allow access even when table is private) or deny list (prevent user from visiting even when table is public).

In both cases, table can be set as read-only: only users in the allow list are able to write posts and noone else.

Owner of the table can designate assistants. Both the owner and the assistants can enter and post, regardless of other settings. In addition, they can both manage Notice Board and Pools.

Tables can have restricted set of users being able to write. This is done by filing any nicknames into the “Write allowed” form. All tables without any users specified are considered publicly writeable.

Note

Implementation Note: There are two models for managing access to ensure compatibility with the old version. See relevant ticket for further information and read the Tavern models carefully before changing anything.

Migration script is in the repository as well.

Developer Documentation

Design, architecture and other documentation aimed at site developers.

Migration and Co-Existence

This section covers design decisions that are driven by how the existing PHP application is structured. All of those should be considered technical debt and eliminated when possible or when the original application is abandoned.

Database

Database is the main integration point. The original structure is just inspection of the original data, so do not pay attention to its design. List of consideration follows.

Database encoding

The original data are misencoded: while stored in field that pretends to be ISO-8859-2/latin2, the data is in fact stored in win1250/cp1250 encoding.

This is transparently handled by ddcz.models.magic; for original tables, MisencodedTextField or MisencodedCharField must be used.

New models respect the connection setting and store data as latin2. Once the old application is shut down, everything should be recoded in a way 21st century people store data (UTF-8).

Warning

While encoding is handled transparently on the model level, it isn’t so during database lookup.

All string lookups on a MisencodedCharField (or MisencodedTextField) has to use a Model.objects.get(field=value.encode("cp1250").decode("latin2")) syntax.

Django’s (database model) migration strategy

Django provides a reasonable framework for handling migration that is used in our application. Initial structure has been done using :cmd:`inspectdb`, which automatically creates unmanaged models and has been placed into ddcz.models.legacy.

When model/table is incorporated into application with all bells and whistles required for it to actually run and be read- and write-able, it’s moved into ddcz.models.used.

There is one problem: unmanaged models are not created during the normal setup, hence tests are failing and application is unusable for anyone without access to database structured backup. To work around it, there is a hack:

  • In the initial migration, the default managed is set depending on SETTINGS.IS_DATABASE_SEEDED. This has to be set depending on whether database is restored from original data
  • This means that migration from unmanaged to managed model will work correctly with seeded database and will be “noop” migration for seeded database
User Model

In order to leverage Django’s authentication framework (meaning reasonable forward-compatible safety), tricks are needed.

Original data is stored in uzivatele table. For usability, this is exposed as UserProfile model and appropriate relation is used.

Warning

Always use ddcz.users.create_user for creating users, instead of django.auth.models.User.create_user

Warning

To avoid the need for complete database migration, django.auth.models.User is not prepopulated and the migration is to be transparently handled on user login until the old application exist.

Hence, the application must use UserProfile model only when displaying user data (i.e. in user stats).

During the initial setup, the arbitrary value of 20000 has been selected for django.auth.models.User’s auto_increment value to distinguish between users from pre-migration to post-migration and to allow old users to retain their IDs.

Author Model

In order to bridge the confusion between article source (zdroj, zdrojmail) and author writing (autor, autmail), we are creating a new model, Author. This contains foreign keys if discovered and allows future normalization of data.

Random discoveries in the legacy data model
  • autor and autmail attribute for authors are denormalized. Author’s email in autmail is never updated if user changes their email in settings

Design

Technically, DDCZ is kinda CMS with simple workflow meeting very simple social network. This section covers most of the legacy design and lists its pages for reference.

Considerations

  • MySQL is the database of choice and integration layer between old and new. For all its awfulness, this will stay. While database pretends to be in latin2 (and even the site’s connectin to it is explicitly set to latin2), the actual content is stored in cp1250. This is a major fuckup that will render most default tools unusable with the database. “Proper” Python way to present a record is to connect using latin2 encoding and then convert individual records using record.encode(“latin2”).decode(“cp1250”).encode(“utf-8”)
  • Page load speed is a concern. Everybody loves fast pages
  • General speed and scalability is not a concern. Whole dataset can fit into modern memory. Please do not optimise on bare metal and single node
  • Maintenance cost is a concern. The less it costs to run, the more likely it will stay with us
  • Stability of an ecosystem is a concern. This site is going to run for another decade. Which technologies do you think are going to stay for so long, preferably with no need for upgrade? That site, this has been always my playground for failed experiments. Judge me

Pages & Data Design

Oh boy, this thing has a lot of dubiously designed data underneath. However, from a high-level perspective, they would be normalised as follows.

There is a Czech/English tension in the naming. Decision TBD.

Public pages

  • Common attributes for all “creations”: Author, External source (for very legacy articles from bootstrapped version of the page), rating (AVG of all votes, expressed in starts, 0-5) and read and print counters
  • Creation pages: “Articles”, containing annotation and text
  • Creation pages: System extensions (roughly dozen of them), containing arbitrary amount of additional attributes, rendered in a table-like structure
  • Creation pages: Gallery and Photogallery (photo uploads included)
  • Creation page: Adventures / Dungeons (random uploads included)
  • Creation page: Downloads (various programs available for download)

All of the “articles” are known as Creations (no equivalent in the old world; let’s call them “Tvorba”). They differ from the rest that they originally:

  • Have to be approved by editor and they do have a workflow attached to them
  • They are rated by both editor and visitors

—

  • Ad section and dating section (both surprisingly active)
  • Links to other sites (now defuct)
  • Few system pages (search in users, news, newsfeed, artices being in the approval process, top stats etc.)
  • Forum (as in public discussion place)
  • Head of a Golden Dragon competition (best articles for a given time period)
  • Polls
  • Chat
  • Registration (subject to approval)
  • Wise Owl / Phorum (as in Phorum installation, separate from the page, with various hand-made wires to make it kinda works)

Private pages

  • “Table discussions” (~600 tables): categories, “flat” discussion list with post rating, book/ignore, notice board, access stats and polls
  • Discussion voting
  • User levels (based on activity)
  • Private Messages (with groups)
  • Ability to send new creations into approval queue (with cathegory-specific directions)
  • Settings
    • Personal settings (user attributes change…ICQ status icon present!)
    • Mentoring system for new users
    • Mailing list for new creations
    • Default filters configuration
    • Login configuration / API key / skin settings
    • News groups (used in private messages)

Administration pages

Now simply know as “Dragon” in order to distinguish from Admin, which has now more php-style interface.

  • Approval and comments for approval queue
  • User registration approvals
  • Article edit
  • Head of Gold Dragon award management
  • Level system adjustments

Model Design

All Creations have Creation abstract model as a simplification. Unfortunately, it doesn’t provide unified interface since because of legacy database design, there are no relation captured.

Hence, the creation instances of all subclases are not treated as entirely standalone as we can’t do Creation unified queries anyway. For all practical purposes, creations has to be treated as (CreationPage, Creation) pair.

Infrastructure

Production

This section contains production-specific information and runbook.

Architecture and Setup

We are using:

Old version is running on an [EC2 instance](https://aws.amazon.com/ec2/).

User Content Hosting

User-uploaded content (user icons, gallery pictures etc.) is hosted on S3 im the uploady.dracidoupe.cz budket/domain.

There is no sharing with the old version: content from current production needs to be uploaded/synchronized manually. This needs to happen on a bastion host as the EC2 instance can’t talk to Amazon APIs because of obsoleted openssl. Use [aws s3 sync . s3://uploady.dracidoupe.cz/whatever](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html) command.

Static File Hosting

Static files (like CSS) are now hosted from within Heroku [using whitenoise](http://whitenoise.evans.io/en/stable/django.html). They [should be migrated to CDN](https://github.com/dracidoupe/graveyard/issues/2).

Error Reporting

Exceptions are sent to [Sentry](https://sentry.io/welcome/). Sentry is configured to push information about new exceptions into [#production-feed Slack channel](https://dracidoupe.slack.com/archives/C029JA38AAV).

Logs are scanned using [Papertrail](https://www.papertrail.com/) and matching events are send to [#production-feed](https://dracidoupe.slack.com/archives/C029JA38AAV). Alerts currently configured:

  • Warning: OR [WARNING] -(“[WARNING] Worker with pid” AND (“was terminated due to signal 19” OR “was terminated due to signal 15”)) -“[WARNING] Not Found: /favicon.ico” (sent every 10 minutes)
Installation

Graveyard is not yet intended to be universally installable, but it plans to be. Current assumptions about production follows.

Configuration

The application assumes following environment variables to be set in production (see settings directory for more info):

  • DJANGO_SECRET_KEY
  • SENTRY_DSN
  • DB_NAME
  • DB_USERNAME
  • DB_PASSWORD
  • DB_HOST
  • DB_PORT

THE FOLLOWING IS LIKELY DEPRECATED: * Production code is currently copied into a directory (let’s call it appdir) * In appdir’s parent directory, production.py is assumed. That should contain all configuration directives from production’s example file, but with correct secrets

  • Graveyard currently assumes Sentry account. This should change (we should run our own Sentry instance)

Testing

For automated testing, we’re using standard Django tests suite. In case of end-to-end browser tests, Selenium is used.

All Selenium tests are in the ddcz/tests/test_ui directory.

Test can be run locally, but there is a testing infrastructure run on every push. We are using Github Actions for that.

To have the Selenium running properly, we are running the Docker container version of everything and with dedicated docker-compose.circle.yml.

Main web container exposes the application server to the Docker network and the Selenium server uses the generated hostname.

Dictionary

Dictionary of words used during move and translation.

Universal domain names

  • Sekce = Categories
  • Nazev stranky / rubriky = Page Heading
  • Nickname = nick (always shortened)
  • Statistiky = User Stats

Pages

In this context, page basically corresponds to URL.

  • Aktuality = News
  • Novinky = Newsfeed
  • Fórum = Forum
  • Inzerce = Market
  • Seznamka = Dating
  • Otázky a odpovědi = FAQ
  • Dračí manuál = Dragon Manual

Users

  • Okres = Shire (yes, it’s a bad translation, but it’s fun)

Creative Pages

In this context, Crative Pages correspond to sections of the site where content is provided by our authors, after review. Creative page has Categories (= Sekce) that serve to further classify the Creations.

  • schváleno = published
  • hlasování = voting
  • hlasující = voter
  • hodnocení (díla, ve smyslu počtu zvězdiček) = rating
  • Rubriky (= Creative Pages)
  • Sekce (= Section)
  • Koncepce (rubriky) = Charter
  • Dílo = Creation (attributes follow naming conventions from Article on schema.org)
  • Anotace = Synopsis
  • Diskuze = Discussion (critique, ratings and suggestions)
  • Příspěvek = Comment
  • Dlouhé příspěvky / Běžné příspěvky = Common Articles (not used publicly, but powers a lot of internal engine)
    (shouldn’t be creations instead of articles?)
  • Bestiář (= Monsters)
  • Dovednosti (= Skills)
  • Hřbitov (= Graveyard)
  • Články a Eseje (= Articles and Essays)
  • Galerie (= Gallery)
  • Fotogalerie (= Photogallery)
  • Alchymistické Předměty (= Alchemist Tools)
  • Alchymista (= Alchemist)
  • Hraničář (= Ranger)
  • Hraničářská Kouzla (= Ranger Spells)
  • Válečník (= Warrior)
  • Zloděj (= Rogue)
  • Kouzelník (= Wizard)
  • Kouzelnická kouzla (= Wizard Spells)
  • Nové rasy (= New Races)
  • Nová povolání (= New Classes)
  • Rozvoj DrD (= Game Expansions)
  • Předměty (= Items)
  • Downloady (= Downloads and DownloadItem)
  • Dobrodružství (= Quests)

Tavern

Tavern (Putyka) is the discussion section of the site, old Putyka counting around 500 different “tables” which are basically separate permanent discussion threads with added functionality (table owner, noticeboards, polls and so on)

  • Staré dělení - Putyka -> Sekce -> Stoly (= Tavern -> Sections -> Tables)
  • Nové dělení (= Tavern -> Tables)
  • Hostinský/á (= Innkeeper)
  • Nástěnka (= notice-board)
  • Anketa (= poll)
  • Příspěvek (= post)
  • Oblíbené stoly (= favorite tables)
  • Označit stůl za oblíbený (= mark as favorite)
  • Ignorovat stůl (= ignore table)
  • Moderátor (kdysi pomocník admina) (= moderator)

Users & Social

  • Reputace (= Reputation)
  • Udělit reputaci (= Give rep)

Decision Log

We capture major decisions about the development here. Similar to architecture decision log, it helps us remember important context and gives us more confidence when revisiting decisions in the future.

2021

September

  • There is small enough active users that we’ll be using Mailgun even for batch sending
  • Batch sending is thus decoupled and send using database as a queue as a starter
  • E-mail list is small enough for blacklisted e-mails to be loaded in memory for every sending. This should be revisited should the userbase grow significantly
  • Backup is still done by cron on the old EC2 instance with upload to S3 there. This can only be reasonably fixed after InnoDB migration
  • Beside registration e-mail, no e-mails should be send to unregistered users. This is because we’d have to track unsubscribe tokens for all combinations and it’s “complicated” to avoid either spamming or unsubscription attack; not worth it for now

June

  • For URLs, we prefer longer and more expressive URLs that explain the particular resource or resource list. Hence, displaying posts in Tavern is /tavern/table/id/posts/ as opposed to just /tavern/table/id/, especially since we have /tavern/table/id/notice-board/ etc. In the same fashion, root url / redirects to /news/ instead of being served directly since we may (and actually want) to redo the landing page.
  • Models missing integer foreign key should have it added (as opposed to working on the character foreign keys). Pattern:
  • For comments and tavern posts, we are using the same endpoint and POST action. Action is designated by a POST attribute. “Correctly”, this would be done better by using different HTTP method, unfortunately Mike’s proposal is not implemented
  • All attributes for all models are now in English and in accordance with the Dictionary. Czech names should be hidden under db_column attributes and under ``Enum``s

QNEA (Questions Not Even Asked)

What is it?

DraciDoupe.cz is a collaboration place for pen-and-paper RPG creativists, designed to host a single (dated) RPG system, Draci Doupe, from a single country, Czech Republic. Back in early 00’s, it also served as a social network for (not only teen) fantasy fans.

In peak of its popularity, it enjoyed thousands of visitors per day (huge number back then). The popularity declined as both technology, the RPG system and the concept of community sites become dated (being consumed by Facebook and likes). It still treasures hundreds of articles and a lot of capture creativity.

Hence, it still attracts few hundred of visitors every day.

Why work on it?

Because I want to keep it available, something to be discovered for both nostalgics and new player generation, if only for nostalgia reasons. That is unsustainable in current state, mainly for security reasons.

Also, I am just grateful. This site shaped my life like nothing else.

Why not fix the product in the process?

That was an idea of RPGPlanet, planned successor to DraciDoupe.cz. It, however, entered development hell, coupled with changing landscape of how sites are consumed. It was also never truly accepted by community, hence dying before bing born.

Thus, I decided to instead fix the current state.

Why reimplementation instead of fix?

Because it has been born in a different time and is very hard to fix incrementally.

Smartphone was not a thing, PDA barely was. Broadband was only available at universities, dial-up connection was a way to connect to internet. PHP just changed its name and came out as PHP 3. CSS was in version 1 and impossible to get working accross browsers consistently; tables were used to do page layouts. Netscape was something to be tested. Mozilla Suite was gaining traction, Firebird (later called Firefox) hasn’t been born yet. Internet Explorer 5 was ruling it.

DDCZ has been kicked out by its hosting provider for consuming more than 1 GB of outbound traffic per month, moving on to be hosted on bare metal. The database was huge for its time, counting data in hundreds of megabytes. That’s why UTF has been considered wasteful at the time and not adopted everywhere.

JavaScript (called JScript in IE) was in its infancy, both DOM and language inconsistent accross browsers (without any façade library; jQuary hasn’t been born yet), painfully slow and with security holes, as well as cookies. The whole site has been designed to work with both JavaScript AND cookies to be turned off.

PHP, the implementation language of choice (not many other options, although ASP 3 implementation has been attempted) hasn’t supported register_globals feature flag and it has been on. This created a huge problem for the future since it can’t be turned on in “modern” PHPs. This renders the whole codebase unworkable and tricky to refactor.

At last, but not at least, the whole site has been architected, designed and implemented by three 16 year olds who learned programming on this.

All of those contribute to architectural decisions that are very hard to reverse, and clean-slate rewrite is better option.

You don’t have time to do this

I don’t and I don’t want this to be another RPGPlanet. Which is why this is going to be implemented in stages, fist one being very easy yet still useful.

What will stay?

Most of the things from user perspective. The goal is to preserve, not to rework. Some changes will be done in the name of usability, though.

  • Public site still usable without cookies
  • Privacy still considered important
  • Original URLs. Redirects will be in place
  • Site being Czech to the bones (and routes)
  • Skins. They are terrible, make the site much harder to develop, makes maintenance costly and doesn’t bring almost any value. I love them
  • Filters, some of them. They are beautifully terrible in the same way skins are
  • User registration approval. It’s elitists, awful, unfriendly and it makes the site the way it is
  • Focus on user privacy. All data is pseudonymous. No personal information collected

What will change?

  • Either cookies or local storage (TBD) will be required for skins and for log in. Those days, it’s more safe than URL parameter, trust me. It also means it is safer to have longer session length than terrible 15 minutes
  • Public site still working without either, though
  • SEO, those articles deserve it. Current site is very unfriendly to search engines
  • Security, on multiple levels
  • Moudra Sova, the over-the-decade, manually-patched version of Phorum is going to either go away, or be sandboxed. It doesn’t see almost any activity anyway
  • Source now open (as opposed to old version)
  • Filters, some of them. In a lot of places they don’t really make sense; I want to keep them, but if they’d complicate implementation too much, they are (at least temporarily) gone