PubliraGitHub

本页尚未翻译成中文,因此以英文显示。

Search

发布于 更新于

The search on every tenant's site finds series, authors, and labels through one engine, shared by every tenant. An install starts on the database itself and needs nothing more; OpenSearch or Elasticsearch is an option for an install that wants results ranked by relevance. This page compares the engines, moves the search from one to another, and covers the index an engine keeps.

Every publiractl command here is shown bare; run it the way The Platform Console and publiractl describes, with PUBLIRA_PLATFORM_DB_URL set to the publira_platform connection and the same PUBLIRA_SECRET_ENCRYPTION_KEYS and PUBLIRA_SECRET_ENCRYPTION_PRIMARY_KEY_ID the processes run with. Every flag is in the publiractl reference.

Choosing an engine

Engine--engineHow it searchesWhat it needs
PostgreSQLsqlFinds the titles and names that contain what was typed, in title or name orderNothing: it is the database the install already has
OpenSearchopensearchRanks the results by relevance, forgives a mistyped letter in a word written in Latin letters, and finds a title written in kanji, kana, or Hangul by a part of it. A language given a text analysis of its own is searched the way that analysis does, such as finding a Japanese title written in kanji from its reading typed in kanaAn OpenSearch cluster
ElasticsearchelasticsearchThe same search as OpenSearchAn Elasticsearch cluster

Whichever engine finds the results, what the storefront shows of each one is read from the database, so a series never shows a stale title, price, or cover because the engine has not caught up. A search can be narrowed and sorted the same ways on every engine.

Every engine finds a series by the title and the synopsis shown in the reader's language: a reader browsing in English finds a series by its English title, and not by a word only its Japanese title has. A series with no English title is found by the one shown in its place, the title in the site's default language. Authors and labels have one name in every language, and are found by it whatever the reader's.

PostgreSQL is enough for a catalog readers browse more than they search, or whose titles they type exactly. Move to an engine when readers look for a title by its reading or misspell it, and come away with nothing.

Preparing OpenSearch or Elasticsearch

Publira does not run the engine for you. Use a cluster you operate or a managed one, and prepare these:

  • The plugins, if any. The default text analysis is built only on what OpenSearch and Elasticsearch include, so an engine from the official image works as it ships. A plugin is needed only for a language you give a text analysis of its own built on one.
  • The URL the engine answers on, http:// or https://, which publira server, publira worker, and publiractl all have to reach.
  • A user, when the engine requires one. Publira signs in with HTTP basic authentication, which it sends only to an https:// URL; it does not support API keys, nor a certificate authority of your own, so an engine with security turned on needs a certificate the servers already trust. The user needs to create, write, and delete indices, and move aliases, whose names start with the index alias below.
  • An index alias, the name Publira's indices are named after: publira-catalog unless you give another. Publira keeps one index for each language a reader can choose, each behind the alias with the language appended — publira-catalog-en, publira-catalog-ja, publira-catalog-zh-hans, and so on — and creates the indices and their aliases itself. Give each install that shares a cluster an alias of its own, since two installs on one alias would overwrite each other's catalog.

Testing the engine

The test asks the engine what it is and which plugins every node has, and reports the product, its version, and the plugins. It fails when the engine does not answer, rejects the credentials, or turns out to be the other product than the one chosen. No plugin decides it: the default text analysis needs none, and a definition of your own is checked against the engine when you save it. Listing the plugins takes a cluster privilege the user needs for nothing else, so a user without it passes the test with the plugins left unlisted.

  • In the Platform Console, Test connection under Search tests the values in the form, before they are saved. It runs from publira server.
  • publiractl search test tests the engine already saved, from wherever publiractl runs, and exits 1 when the test fails. On PostgreSQL there is nothing to test, and it exits 1.

Moving to another engine

Saving an engine does not move the search onto it at once. The index on the new engine is empty, so publira worker builds it first, from the database, and the search moves over once the index holds every tenant's catalog. Until then the storefront keeps answering from where it was, so readers never search an empty index.

From the Platform Console

  1. Choose Search under Services in the sidebar.
  2. Under Engine, choose OpenSearch or Elasticsearch, and enter its URL and, if you chose one, its Index alias (optional).
  3. Under Authentication, choose No authentication, or Username and password and enter both.
  4. Choose Test connection, and save with Save search settings once it passes.

Search engine with OpenSearch chosen: Engine, URL, Index alias (optional), Authentication, the connection test, and Save search settings.

Search status at the top of the screen shows the engine the storefront answers from, and while the index is being built, the engine it will move to. The screen checks again every few seconds, so it shows the move when it happens. An Operator or a Super admin can save; an Auditor only sees the screen.

From publiractl

publiractl search set \
  --engine opensearch \
  --url https://search.example.com:9200 \
  --username publira --password-file /run/secrets/search-password
publiractl search test
publiractl search show

search set replaces every saved setting with the flags it is given, so name all of them each time; leaving --username out removes a saved password. With the same --username and no password, it keeps the saved password. search show prints the engine the search answers from, and a build that is under way or has failed.

How long it takes

The worker looks for an index to build every 30 seconds, and filling it takes as long as the engine needs to take every published series, author, and label of every tenant. Changes made in the meantime are written to the new index as well, so nothing is lost by the wait.

When the build fails

A build that fails — an engine that went away, a plugin a saved definition needs that a node has lost, a user that may not create the index — leaves the search where it was, and the error is shown by Search status and by publiractl search show. The worker tries again every 30 seconds, so once you fix the engine the build goes through without saving the settings again. To give up on the move instead, save the engine the search is still answering from.

Search status after a failed build: answering from PostgreSQL, the message that the OpenSearch index couldn't be built, when it last failed, and the engine's error. The time it last failed is greyed out here, since it changes with every attempt.

Going back to PostgreSQL

Saving PostgreSQL, or publiractl search set --engine sql, moves the search back at once: the database needs no index. The index on the engine is left where it is; delete it yourself if you will not use it again.

When the change reaches every process

publira server and publira worker read the settings again every 30 seconds, so a move reaches every instance within a minute, with no restart. Each save and each test is recorded in Audit logs.

Text analysis

How the engine splits titles, names, and synopses into words is the text analysis, one definition for every tenant, since they share the indices, and the index of every language is built from it unless that language has a definition of its own. The default needs no plugin and suits every language a little: it finds a title in Latin letters by its words, forgiving a mistyped letter, and a title in kanji, kana, or Hangul by any two or more characters in a row of it, ignoring case and the width of letters and kana. It knows nothing of a language beyond that, so it does not, for example, find a title written in kanji from its reading. A language that needs more can be given a definition of its own built on its plugin.

A definition is the settings.analysis object of the index, as JSON of at most 64 KiB, and has to define three names the index refers to: the analyzers written_form and alternate_form and the normalizer exact_match. What each one is for is in the text analysis reference.

  • In the Platform Console, the Shared tab of Text analysis under Search takes the definition in Definition (settings.analysis, JSON) and saves it with Save text analysis. Reset to default goes back to the default definition. On PostgreSQL there is no index, and so no text analysis to save.
  • With publiractl, search set --analysis-file korean-analysis.json saves a definition along with the engine, and --default-analysis goes back to the default. Without either, search set keeps the definition already saved.

Give a language a definition of its own to search it with that language's plugin: its index is built from that definition, and every other language's from the shared one.

  • In the Platform Console, Text analysis has a tab for the shared definition, Shared, which says which languages use it, and a tab for each language. A language without a definition of its own shows the shared one in its tab, to start from. Save text analysis in that tab saves the editor as the language's own definition, and Use the shared definition discards it, so the language is built from the shared one again.
  • With publiractl, search set takes a language's own definition:
publiractl search set --engine opensearch --url https://search.example.com --locale-analysis-file ja=japanese-analysis.json

--locale-analysis-file takes a language and a file as LOCALE=FILE, and --shared-analysis-for ja builds that language from the shared definition again. Repeat either for each language; a language neither names keeps what it has, and so does every language when search set is run without them. The languages are ja, en, ko, zh-Hans, and zh-Hant. publiractl search show lists the languages that have a definition of their own under Own analysis.

Text analysis: the three names a definition has to define, a tab for the shared definition and one for each language, which definition is saved and which languages use it, Definition (settings.analysis, JSON), and Save text analysis.

Before a definition is saved, the engine is asked to create an empty index with it, and a definition it refuses is not saved, with the engine's reason and, for a language's own definition, the language it was given for. A saved definition is built into new indices for the languages it changes, the same way a new engine is, and the search moves onto them when the build is done; the indices of the other languages are left as they are.

Giving a language its own plugin

Each language is searched with the shared definition until you give it one of its own, and doing so is optional. OpenSearch and Elasticsearch both publish an analysis plugin for many languages — analysis-kuromoji and analysis-icu for Japanese, which can match a title written in kanji to its reading typed in kana, analysis-nori for Korean, and analysis-smartcn for Chinese, among others — and a definition built on one searches that language the way the plugin analyzes it:

  1. Install the plugin on every node while it is stopped, with bin/opensearch-plugin install or bin/elasticsearch-plugin install and the plugin's name. The repository's OpenSearch image for development shows the step as a Dockerfile. Test connection then lists it among the plugins every node has.
  2. Write a definition that uses the plugin's tokenizer and filters, as the engine's documentation of the plugin describes, and defines the three names above.
  3. Save it in that language's tab of Text analysis, or with publiractl search set --locale-analysis-file.

The language's index is rebuilt with it, and the search in that language moves onto the new index once it is ready. A series without a title in that language is in its index under the title shown in its place, so the language's analysis applies to that title too.

Rebuilding the index

The worker keeps the indices up to date as tenants publish, edit, and change their default language, and builds new ones whenever the engine or the text analysis changes, or a release changes the languages a reader can choose or the default text analysis, so an index rarely needs rebuilding by hand. While it builds them, Search status and publiractl search show show a build under way, and the storefront keeps answering from the indices it has. publiractl search reindex is for the two cases that are left:

  • A release changed the index. When the release notes say the catalog index changed, run it once the upgrade is done.
  • The index was lost, because the cluster was replaced or the index deleted.
publiractl search reindex

It builds a new index for each language beside the current one, fills them with every tenant's catalog, moves every alias onto its new index at once, and deletes the old ones, so the storefront keeps searching while it runs. It reads the catalog as the publira_content_stats role, so give it PUBLIRA_CONTENT_STATS_DB_URL for that role, along with the encryption keys when the engine has a saved password. It cannot run while the worker is building an index, and exits 1 if one is under way; run it again once the build is done. On PostgreSQL there is no index to rebuild, and it exits 1.

publiractl search reindex --tenant comics.example.com writes one tenant's catalog again, in the index the search answers from, without building a new one. It is for one tenant whose results look wrong, and cannot apply a change to the index itself.