OJSdeSearchConcept - Revision history

Jerico.dev: Mention external file fields.

2012-10-16T14:34:46Z

Mention external file fields.

Jerico.dev: Section-level custom ranking.

2012-10-12T22:15:15Z

Section-level custom ranking.

Jerico.dev: Section-level custom ranking.

2012-10-12T22:12:12Z

Section-level custom ranking.

Jerico.dev: add section_id

2012-10-12T22:06:12Z

add section_id

Jerico.dev: Fix a few details in the faceting section.

2012-10-03T18:45:36Z

Fix a few details in the faceting section.

Jerico.dev: Update data model description to the latest version.

2012-10-03T18:43:08Z

Update data model description to the latest version.

Jerico.dev: Refine facet spec according to lessons learned during implementation.

2012-10-03T18:36:16Z

Refine facet spec according to lessons learned during implementation.

Jerico.dev: Update with lessons learned from implementation

2012-09-26T16:57:25Z

Update with lessons learned from implementation

Jerico.dev: typo

2012-09-04T19:52:23Z

typo

Jerico.dev: Indexing + Subscription-Based Publications

2012-09-04T19:34:51Z

Indexing + Subscription-Based Publications

← Older revision		Revision as of 14:34, 16 October 2012
Line 846:		Line 846:

	If we implement article-level ranking factor adjustments (boosts), then these will probably have to be transferred to solr at indexing time. If we implement section-level ranking factor adjustments, then we can pass those adjustments in as per-section boost queries. This enables the user to change the section-level boost without us having to re-index all articles in that section. Sections that have the "never show" custom ranking setting can be excluded with a filter query.		If we implement article-level ranking factor adjustments (boosts), then these will probably have to be transferred to solr at indexing time. If we implement section-level ranking factor adjustments, then we can pass those adjustments in as per-section boost queries. This enables the user to change the section-level boost without us having to re-index all articles in that section. Sections that have the "never show" custom ranking setting can be excluded with a filter query.
		+
		+	Providers may use an external file field to specify custom ranking factors based on whatever field. In the embedded scenario, we could use external files for article/section ranking adjustments. We prefer to use boost queries, though, as these do not require us to maintain remote ranking files over the network in the central server deployment scenario.

	=== Potential Additional Ranking Metrics ===		=== Potential Additional Ranking Metrics ===

@@ Line 844: / Line 844: @@
 By default we do not change any preconfigured ranking factors. By using the “edismax” query parser (see above) we do keep the option, though, to fine-tune client-side (query-level) ranking parameters should it become necessary.
 It is recommended to avoid client-side ranking adjustments. By configuring different solr search request handlers, different ranking approaches can be provided to different OJS installations by changing their search endpoint configuration (see deployment option configuration above).
 === Potential Additional Ranking Metrics ===
 While TF-IDF is the default scoring/ranking model in Lucene it can be customized by providing so-called boost factors for different entities in the model. These are simple multipliers that can occur on term and document level that increase (multiplier > 1) or decrease (multiplier < 1) the scoring contribution of that entity.

@@ Line 328: / Line 328: @@
 # If the inclusion of additional ranking data (e.g. citation index, usage statistics, etc.) should be possible, then we'll need an interface where such ranking information can be uploaded or integrated from external sources. One possibility is the “Custom Document Ranking Factor Configuration” described below.
 === Custom Document Ranking Factor Configuration ===
-If a custom document ranking factor (e.g. citation index data, usage metrics, etc.) should be supported this can easily be done as a generic input field on the article editing page. When custom document ranking is enabled in the plug-in then such a field will appear there. If editors insert a low value then the article will rank lower than default. A high value will increase it's mean ranking position. Numbers will be linearly normalized so that their mean is one. See the “Ranking” chapter below for internal implementation details and more examples of potential alternative ranking methods.
+If a custom document ranking factor (e.g. citation index data, usage metrics, etc.) should be supported this can easily be done as a generic input field on the article editing page. When custom document ranking is enabled in the plug-in then either such a field will appear there. Alternatively an import plug-in could be realized that allows import of document ranking data from different file formats (CSV, XML, etc.) or even pulling ranking data from an external source via HTTP.
 Alternatively an import plug-in could be realized that allows import of document ranking data from different file formats (CSV, XML, etc.) or even pulling ranking data from an external source via HTTP.
 === Index Administration ===

@@ Line 460: / Line 460: @@
 * the globally unique document ID field ("article_id") concatenating a globally unique installation ID, the journal ID and the article ID,
 * “inst_id” and “journal_id” fields required for administrative purposes,
 * the authors field (“authors_txt”) which is the only multi-valued field,
 * localized article meta-data fields ("title_xx_XX", "abstract_xx_XX", "discipline_xx_XX", "subject_xx_XX", "type_xx_XX", "coverage_xx_XX") where "xx_XX" stands for the locale of the field,

@@ Line 880: / Line 880: @@
 * subject
 * coverage
 Some special filtering of these fields can be applied for faceting:
 * It may make sense to only include the first author for faceting. It is up to the project owner to define this.
 * Faceting on the publication date must be by date range rather than discrete date values.
-* All other fields must be analyzed with the keyword analyzer and otherwise left intact so that facets display with the original spelling.
+* All other fields must be tokenized by separators (e.g. "," and ";") and otherwise left intact so that facets display with the original spelling. Lowercasing facets may make sense, though.
 We propose that we only return faceting results for the currently chosen interface language. This means that the fields for faceting cannot be preconfigured in the OJS search request handler but must be provided in the query string, e.g. “...&facet.field=subjects_de_DE”.
-Facets will be selected as links or check-boxes in the faceting block plug-in as described in the interface section above. Selecting one or more factes will result in the original query being re-issued with an additional filter query, e.g. in the case of a date range “...&fq=publication_date_dt:([2006-01- 01T00:00:00Z TO 2007-01-01T00:00:00Z] NOT "2007-01-01T00:00:00Z") ...”.
+Facets will be selected as links in the faceting block plug-in as described in the interface section above. Selecting one or more facets will result in the original query being re-issued with an additional filter query, e.g. in the case of a date range “...&fq=publication_date_dt:([2006-01- 01T00:00:00Z TO 2007-01-01T00:00:00Z] NOT "2007-01-01T00:00:00Z") ...”.
 == Alternative Spelling Suggestions ==
 Alternative spelling suggestions for a given search query can be provided based on solr's spellcheck search component. If this should be implemented then we recommend creating a multilingual dictionary based on a concatenated field that contains all meta-data and full text. The spellcheck dictionary will be stored in a separate Lucene index in …/files/solr/data/spellchecker.

@@ Line 465: / Line 465: @@
 * a single localized field for supplementary file data ("suppFiles_xx_XX") where "xx_XX" stands for the locale,
 * localized galley full-text fields ("galleyFullText_mmm_xx_XX") where "mmm" stands for data format, eg. "pdf", "html" or "doc" and "xx_XX" stands for the locale of the document.
-* fields to support result set ordering ("title_xx_XX_txtsort, authors_txtsort, issuePublicationDate_dtsort, etc.)
+* fields to support result set ordering ("title_xx_XX_txtsort", "authors_txtsort", "issuePublicationDate_dtsort", etc.), see below
 These fields will be analyzed for search query use cases, potentially including stemming (see “Analysis” below). The exact data schema obviously depends on the number of languages and data formats used by the indexed journals.
@@ Line 475: / Line 477: @@
 If we want to support highlighting then the galley fields need to be stored, too.
-Further specialized fields will be required for certain use cases. If we want to support auto-suggestions, faceting or alternative spelling suggestions then we'll have to provide textual article meta-data fields in a minimally analyzed (lowercase only, non-localized) version. These fields will be called “xxxxx_spell” where “xxxxx” stands for the field name without locale extension.
+Further specialized fields will be required for certain use cases. If we want to support auto-suggestions or alternative spelling suggestions then we'll have to provide textual article meta-data fields in a minimally analyzed (lowercase only, non-localized) version. These fields will be called “xxxxx_spell” where “xxxxx” stands for the field name without locale extension.
-Fields that we want to use as optional sort criteria need to be single valued, indexed, and not-tokenized . This means that sortable values will potentially have to be analyzed separately into “xxxxx_sort_xx_XX” fields where “xxxxx” stands for the field name and “xx_XX” for the locale (if any) of the sort field.
+Fields that we want to use as optional sort criteria need to be single valued, indexed, and not-tokenized . This means that sortable values will potentially have to be analyzed separately into “xxxxx_xx_XX_txtsort” or "xxxxx_dtsort" fields where “xxxxx” stands for the field name and “xx_XX” for the locale (if any) of the sort field.
-If we want to support the “more-like-this” feature then we may have to store term vectors for galley fields if we run into performance problems.
+Faceting fields ("xxxxx_xx_XX_facet") need to be localized. They are minimally analyzed (lower case only) and tokenized by separator (e.g. "," or ";") rather than by whitespace.
 Further technical details of the data model can be found in plugins/generic/solr/embedded/solr/conf/schema.xml.

@@ Line 205: / Line 205: @@
 We may also propose a dynamic list of filter criteria (e.g. authors, publication date ranges, disciplines, type, subject and coverage keywords) to further refine the result list. This is called "faceting". Facets could be provided as a list of links organized by facet category (aka search field) in an optional block plug-in. This allows OJS administrators to easily enable/disable facets and flexibly place them according to their journal design.
-Clicking on one of the facet links will re-execute the original search with an additional filter as defined by the clicked facet. Once a search has been re-executed with a given facet, the facet will be displayed above the result list with a “delete symbol” next to it. Clicking on the delete symbol will re-execute the search without the deleted facet.
+Clicking on one of the facet links will re-execute the original search with an additional filter as defined by the clicked facet. Once a search has been re-executed with a given facet, the facet will be displayed above the result list as a regular filter. As all filters it shows a “delete symbol” next to it. Clicking on the delete symbol will re-execute the search without the facet filter.
 While displaying a facet filtered search, facet categories corresponding to active filters will disappear from the list of available facets. Multiple facet filters can be applied by clicking another facet link while displaying an already filtered search.
 If we want to support selection of multiple facets from one category then we could place check-boxes besides facets rather than implementing them as links. In this case we need a “Search Again” button below the facet list, so that all selected facet filters can be applied. For the sake of interface simplicity we do not recommend enabling selection of multiple facets from one category. Users can enable and delete facets from the same category to achieve similar results. Advanced users can use properly filtered queries directly from the search field.
 === Specification of a common search interface ===
 To achieve our goal to maintain both, the existing and the new search interfaces, as much in sync as possible, we have to create a layer of abstraction that allows us to manipulate certain elements of the search UI while maintaining a common structure for both implementations.

@@ Line 885: / Line 885: @@
 We suggest to issue build commands to the usual OJS search interface with the following parameters: “q=nothing&rows=0&spellcheck=true&spellcheck.build=true&spellcheck.dictionary=default”. The q parameter is not usually required to build the dictionary. In our set-up with a preconfigured search request handler not providing it will result in an error.
 == "More like this" ==
-The “more-like-this” (MLT) feature can be implemented by configuring the solr MLT search component. There are several options how to access the MLT component:
+The “more-like-this” (MLT) feature can be implemented by configuring the solr MLT search component. This component extracts "interesting terms" from a document and then executes a query against the index to identify documents that contain these interesting terms.
 * as a dedicated request handler that usually is supplied a single document id,
 * as a search component of another request handler or
 * as a request handler that ingests full text and proposes similar documents from the index.
-While the first option is the most frequently used option, we do not recommend it as it will require configuration of a second search request handler within OJS which is contrary to our “simplicity” policy. We rather propose to provide MLT functionality through our standard search handler as an optional search component. The search component will be disabled by default and only be activated when doing specific MLT requests.
-We recommend to do MLT requests by sending a GET request with the following parameters to the usual search request handler (“.../search”):
+The first option is the most frequently used option and seems adequate to retrieve "interesting terms" in our case.
 * mlt=on,
-* mlt.fl=xxx where xxx is set to the document field corresponding to the galley formats with the current interface language, e.g. mlt.fl=galley_full_text_pdf_en_US,
+* mlt.fl=xxx where xxx is set to the document field corresponding to the title and abstract with all languages supported for these fields in the index, e.g. mlt.fl=title_en_US title_de_DE abstract_en_US abstract_de_DE,
 * q=id:xxx where xxx is the solr document ID (see “Data Model” above) of the article we wish to base our search on,
-* start=... and rows=... as for usual queries according to the current result set page selected by the user,
+* start=0 and rows=0 as we do not want any direct results,
-As we expect the MLT feature not to be used too frequently in most cases we propose not to store term vectors by default. This means that when an MLT request is being issued the corresponding document fields will have to be re-analyzed to derive term vector information. This is a slower than storing term vectors but saves considerable storage space. Term vectors can always be activated by advanced users if the default configuration is found to be too slow or resource consuming.
+* mlt.boost=off as we do not support boosting in our regular interface.
 = OJS/solr Protocol Specification =

@@ Line 520: / Line 520: @@
 Advantages of pull configuration:
-* Push processing means that editorial activity during daytime will cause update load peaks on the solr server exactly while it also experiences high search volume. This load can be quite erratic and fluctuative in larger system environments and therefore difficult to balance. In pull mode indexing schedules can be configured and co-ordinated in one single place (for scenarios S3 or S4) to balance document import load on the central search server and keep it to off hours.
+* Push processing means that editorial activity during daytime will cause update load peaks on the solr server exactly while it also experiences high search volume. This load can be quite erratic and fluctuating in larger system environments and therefore difficult to balance. In pull mode indexing schedules can be configured and co-ordinated in one single place (for scenarios S3 or S4) to balance document import load on the central search server and keep it to off hours.
 * Pull also means that the process is more resilient against solr server downtime. In case of service outage, updates can be postponed until the server is back online. A full index rebuild is not required in that case.

@@ Line 1,000: / Line 1,000: @@
 When working with push updates then all deleted documents can immediately be pushed for re-indexing if required. When working with pull updates then deleted articles will be marked “not indexed” so that they'll be re-indexed automatically the next time a pull request arrives from the solr server.
 == Search Protocol ==
 The OJS/solr search protocol is the well documented “edismax” query and result format. We do not reproduce the general “edismax” syntax here. Please refer to the official solr documentation for details.