Announcing Vendure v1.3

Video sections: Search index batching (14s), Stock status indexing (5m 24s), Improved list filtering (6m 48s), Entity hydration (7m 14s)

v1.3.0 Changelog

Upgrading from v1.x.x

This minor release contains no breaking schema or GraphQL API changes, so updating should be a matter of changing all @vendure/... dependencies in your package.json file to 1.3.0.

{
  "dependencies": {
-    "@vendure/admin-ui-plugin": "1.1.5",
-    "@vendure/asset-server-plugin": "1.1.5",
-    "@vendure/core": "1.1.5",
-    ... etc
+    "@vendure/admin-ui-plugin": "1.3.0",
+    "@vendure/asset-server-plugin": "1.3.0",
+    "@vendure/core": "1.3.0",
+    ... etc
  },
  "devDependencies": {
-    "@vendure/testing": "1.1.5", 
+    "@vendure/testing": "1.3.0",
  }
}

Also see the Updating Vendure guide for more information.

If you are using the Elasticsearch plugin: you'll need to do a full reindex due to underlying changes to the design of the indexing in this version. You can trigger a reindex by clicking the gear icon next to the search bar in the Admin UI product list screen.

New in v1.3

Batching search index updates

Vendure uses a search index to power its product search. This index is an optimized store of product data that allows searches to execute quickly without needing to perform costly database joins to populate the search results. Whenever a change is made to the invetory (changing a Product description, updating a price, modifying a Collection), these changes must end up in the search index, otherwise the index will contain stale data.

This is achieved by automatically triggering background jobs on the worker process which update those parts of the index affected by any changes. For smaller inventories, this works well and ensures that the search index always stays up-to-date. However, on larger inventories with thousands of ProductVariants and hundreds of Collections, these background jobs can become very resource intensive and inefficient.

As an example, imagine a store with 50,000 ProductVariants and 500 Collections. Each time any change is made to a Product or ProductVariant, all 500 Collections must be processed to see whether their contents have changed. Now imagine that the project runs a daily automated sync operation from a remote data source - the number of generated background jobs will quickly baloon out of control.

With v1.3, we are introducing a solution to this which allows you to configure your Vendure instance to batch updates to the search index. This means that when your make changes to your inventory, not background jobs are triggered. Instead, all of the changes are stored up until you are finished with your changes, at which point you tell Vendure to run all updates at once. Not only that, Vendure will optimize the jobs to perform the minumum amount of work. It does this by aggregating and de-duplicating the jobs.

In practice, this means that no matter how many changes you made to your inventory, the Collections will only get processed once, and multiple discrete changes to the same Product will result in only a single job to update that Product.

This is achieved by means of a brand-new JobBuffer mechanism, which is a general-purpose API which allows very fine-grained control and optimizations over the job queue.

To enable search index batching, update your search plugin in your Vendure config as shown below. Additionally, if you are using the DefaultJobQueuePlugin, you should configure it to buffer jobs in the database, otherwise they will get buffered in-memory which is unsafe for production (turning on this setting will require a DB migration):

const config: VendureConfig = {
  // ...  
  plugins: [
    DefaultJobQueuePlugin.init({ useDatabaseForBuffer: true }),
    // If using the BullMQJobQueuePlugin, no changes are needed
      
    DefaultSearchPlugin.init({ bufferUpdates: true }),  
    // or
    ElasticsearchPlugin.init({
        host: 'http://localhost',
        port: 9200,
        bufferUpdates: true,
    }),  
  ]  
}

Stock status in search

Both The DefaultSearchPlugin and the ElasticsearchPlugin now support indexing the stock status of products, and filtering by inStock. This is enabled by default in the ElasticsearchPlugin, but to preserve backwards-compatibility it is behind a flag in the DefaultSearchPlugin. Enable it as shown below, which will then require a DB migration:

const config: VendureConfig = {
  // ...  
  plugins: [
    DefaultSearchPlugin.init({ indexStockStatus: true }),
  ]  
}

The stock status can then be queried like this:

query SearchProducts {
  search(input: { term: "laptop", inStock: true }) {
    items {
      id
      productName
      productVariantName
      inStock
    }
}

Better data APIs & docs

When building custom logic in your project's plugins, you'll likely have used one of the built-in services which Vendure uses internally to perform common actions on entities. Up until now, these services have not been documented, making it hard to know exactly what some of the methods are for. That changes with this release: here's the documentation for all of our services.

Another pain point when dealing with these services is that you can never be sure what relations of an entity have been joined. Take this example:

const product = await this.productService.findOne(ctx, id);

const facetValueIds = product.facetValues.map(fv => fv.id);
// ERROR: Cannot read property 'map' of undefined

This code will compile just fine, but we get a run-time error because the productService.findOne() method does not perform a join on the facetValues relation!

In v1.3 there's a new helper class, EntityHydrator, which allows us to ensure that the necessary relations are present before we attempt to operate on them:

constructor(private entityHydrator: EntityHydrator) {}

// ...
const product = await this.productService.findOne(ctx, id);
await this.entityHydrator.hydrate(ctx, product, { relations: ['facetValues'] });

const facetValueIds = product.facetValues.map(fv => fv.id);
// No run-time error!

The EntityHydrator will inspect the given entity and will join any missing relations. Moreover, it will automatically translate any translatable entities and can even populate ProductVariant prices - both tasks which involved a lot of tedious and error-prone logic before now.

The last new API to highlight is the new withTransaction() method on the TransactionalConnection class. This method finally makes it possible to run multiple database operations within a transaction in situations outside of the typical request-response lifecycle. Examples include logic which is executed by the worker, or stand-alone scripts run from the command-line.

Improved serverless support

To minimize calls to the database, Vendure caches certain data in-memory which is required every request and rarely changes, such as Channel and Zone information. Unfortunately this makes multi-instance or serverless deployments problematic when this data gets updated but one or more instances still hold the stale data.

In v1.3 we have solved this problem by providing a configurable time-to-live values for this cached data.

There is also a new section in the deployment guide which details the configuration considerations for such a setup.

Elasticsearch Plugin improvements

In addition to indexing stock status as covered above, the Elasticsearch plugin has gained a bunch of other new features in this release:

CustomMappings now support ID types and list types like [String!] or `[Float!]!.
The SearchInput object can now be easily extended to allow custom data to be passed to your Elasticsearch functions.
Additional database entities can be joined when indexing Products and ProductVariants, allowing you to e.g. index all Assets of a Product.

Worker health check disabled

In Vendure v1.2.0, we re-introduced a health check for the worker process, allowing you to see the health of the worker in the Admin UI. Unfortunately, the implementation proved problematic - you can read more about why in this issue - so based on user feedback we decided it was best to remove this feature.

If for some reason you still want the current health check behaviour, you can enable it in your config.

Other notable features

The Admin UI is now available in Portugal Portuguese.
Configurable retries in the DefaultJobQueuePlugin and the BullMQJobQueuePlugin.
All queries that return a PaginatedList now support a new filterOperator input field, allowing you to specify whether multiple filter clauses should be combined with a logical AND or OR operation. You can read more context in issue #1149.
The jobs list view in the Admin UI now displays retry information and more accurate durations.
Event subscribers are now transaction-safe. This means that if you subscribe to an event using the EventBus, you can be sure that all database transactions are complete by the time your subscriber code runs, and any data created or modified inside that transaction will be available to you. You can read more about this in issue #1107.
There have been numerous performance improvements at both the resolver level and at the database-access layer to cut down on uneccessary database calls.

Thank you to our contributors

I'd like to thank the wonderful Vendure community who contribute their ideas, bug reports, and code to the project daily. An extra-special thank you is due to Artem and Kevin for their excellent work on the various Elasticsearch improvements! This release includes contributions by:

Alexandre Couturon:
- fix(elasticsearch-plugin): Elasticsearch Cloud auth is not set during re-indexing (#1108)
Artem Danilov:
- fix(elastic-search-plugin): Refactor indexing to allow searching by fields on all variants (#1086)
- fix(elasticsearch-plugin): fix bug with facetValueMaxSize (#1098)
- feat(elasticsearch-plugin): Add inStock attribute and filter (#1130)
- test(elasticsearch-plugin): Add unit tests for #870
- test(elasticsearch-plugin): Add e2e tests for #870
- feat(elasticsearch-plugin): Custom mappings can return lists & allow additional Product/variant relation hydration
Dushko Jordanovski:
- feat(admin-ui): Improve facet filtering for product search input
- docs(core): Extended payment integration docs (#1139)
- fix(core): Fix wrong event type when a variant is created (#1102)
Filipe Moraes:
- feat(admin-ui): Add admin-ui Portuguese (Portugal) translation (#1069)
Florian Rand:
- fix(admin-ui): Add missing Spanish translation strings and fix a few typos (#1079)
Jorge F Pastor:
- fix(admin-ui): Improved Spanish translation of "facets" (#1122)
Jyotman Singh:
- [Docs] Fix parameter description (#1072)
Kevin Mattutat:
- feat(core): Make extractSessionToken function available in core package (#1138)
- feat(elasticsearch-plugin): Extend config with customScriptFields
- feat(elasticsearch-plugin): Index custom product mappings for products without variants
- chore: Added custom mapping prefix to access the property
- docs: Updated scriptFields documentation