Open Source Ecommerce Platforms Comparison Part 2 — Performance and Extensibility [Summer 2020]

Our previous blog post gave you an overview of the most popular ecommerce platforms based on Ruby, Node.js, and Python. We’ve evaluated them in terms of how quickly you can set up an online shop and what possibilities they give you. Turns out that Vendure.io, Spree, and Saleor are your best bets (choose according to your goals) — they are easy to implement, flexible, and provide room for customization.

Now, we’ll take a closer look to check what exactly you can do in terms of extending their out-of-the-box functionality, and how they perform under pressure.

Let’s dive in!

Vendure.io

Vendure.io is a quite new headless ecommerce platform that leverages Node.js, TypeScript, and GraphQL. The database is relational. Since so far this project has been mainly run by one guy — Michael Bromley — the platform packs only a few essential features.

Vendure.io is quick to set up and thanks to GraphQL it integrates with a frontend and additional features of your choosing really well.

To deploy Vendure on Heroku, you need to:

1. Create a Heroku project.
2. Add ROOT_URL.
3. Add MySQL or Postgres as add-ons.
4. Set an environment at Heroku.
5. Add the heroku-nodejs package-builder.
6. Deploy Vendure.io to Heroku.

Extensibility

As we’ve mentioned earlier, the platform itself is simple and basic, without any additional features. So if you need anything fancy, you’ll have to create it yourself. With that being said, Vendure.io is created with extensibility in mind. As the creator puts it, one of the main reasons to build the platform is:

I need to be able to faithfully model the business requirements and processes without bending to the dictates of the tools we use.

The plugins available right now are:

• AdminUiPlugin: Bundles a default build of the Vendure Admin UI and serves it when the server is started.
• AssetServerPlugin: Stores and serves assets from the local file system, and can perform on-the-fly image transformations.
• ElasticsearchPlugin: Allows your product search to be powered by Elasticsearch — a powerful Open Source search engine.
• EmailPlugin: Creates and sends transactional emails based on Vendure events.

To create custom plugins, you can refer to this document. The process is neatly described — it involves creating a new database entry and a GraphQL mutation.

Performance

Right now Vendure.io best for simple and straightforward shops (those usually don’t face epic amounts of traffic), and for that purpose has a satisfying performance score. (We did all our tests on a free Heroku account so paid hosting will get you much better results.) For a single product, it can pull off 84.89 requests per second, with 33.07KB per second transfer. Here are the exact numbers:

====================
benchmark: query-comparison
 --------------------
 candidate: getproduct on getproduct at https://vendure-monte.herokuapp.com/
     ++++++++++++++++++++
     400Req/s Duration:60s open connections:20
     Running 1m test @ https://vendure-monte.herokuapp.com/
       8 threads and 20 connections
       Thread calibration: mean lat.: 4042.597ms, rate sampling interval: 13705ms
       Thread calibration: mean lat.: 4094.949ms, rate sampling interval: 13828ms
       Thread calibration: mean lat.: 4091.316ms, rate sampling interval: 13787ms
       Thread calibration: mean lat.: 4118.664ms, rate sampling interval: 13975ms
       Thread calibration: mean lat.: 4097.844ms, rate sampling interval: 13844ms
       Thread calibration: mean lat.: 4126.541ms, rate sampling interval: 14090ms
       Thread calibration: mean lat.: 4144.684ms, rate sampling interval: 14057ms
       Thread calibration: mean lat.: 4125.240ms, rate sampling interval: 14008ms
       Thread Stats   Avg      Stdev     Max   +/- Stdev
         Latency    26.24s    11.43s   47.42s    58.47%
         Req/Sec    10.12      0.88    11.00    100.00%
       5095 requests in 1.00m, 1.94MB read
       Non-2xx or 3xx responses: 5095
     Requests/sec:     84.89
     Transfer/sec:     33.07KB

When it comes to requesting a full list of products, Vendure.io can handle 97.11 requests per second with 37.83KB per second transfer. Here are more details:

====================
benchmark: query-comparison
 --------------------
 candidate: getProducts on getProducts at https://vendure-monte.herokuapp.com/
     ++++++++++++++++++++
     400Req/s Duration:60s open connections:20
     Running 1m test @ https://vendure-monte.herokuapp.com/
       8 threads and 20 connections
       Thread calibration: mean lat.: 4112.211ms, rate sampling interval: 13819ms
       Thread calibration: mean lat.: 4120.182ms, rate sampling interval: 13860ms
       Thread calibration: mean lat.: 4150.881ms, rate sampling interval: 13852ms
       Thread calibration: mean lat.: 4142.507ms, rate sampling interval: 14016ms
       Thread calibration: mean lat.: 4166.577ms, rate sampling interval: 13910ms
       Thread calibration: mean lat.: 4175.921ms, rate sampling interval: 14065ms
       Thread calibration: mean lat.: 4193.582ms, rate sampling interval: 14065ms
       Thread calibration: mean lat.: 4193.587ms, rate sampling interval: 14057ms
       Thread Stats   Avg      Stdev     Max   +/- Stdev
         Latency    26.46s    10.76s   45.42s    58.22%
         Req/Sec    12.00      0.00    12.00    100.00%
       5828 requests in 1.00m, 2.22MB read
       Non-2xx or 3xx responses: 5828
     Requests/sec:     97.11
     Transfer/sec:     37.83KB

Spree

Spree Commerce is an open-source ecommerce platform based on Ruby on Rails 6, REST API, and GraphQL. It also leverages relational database. It has a large community and an established reputation.

As we’ve pointed out in our previous blog post, Spree would be best used for new shops (no product importer in Spree) that can make do with the UI shipped together with the platform. In theory, it’s possible to integrate Spree with another frontend, but in our experience this process is not seamless and requires a time investment.

The set up, however, goes very smoothly. To install the platform, you need to:

rails new my_store
cd my_store

Next, in Rails 6.0 or in Rails 5.2:

gem 'spree', '~> 4.1'
gem 'spree_auth_devise', '~> 4.1'
gem 'spree_gateway', '~> 3.7'

Then:

bundle install
# in case of error
bundle update
rails g spree:install --user_class=Spree::User
rails g spree:auth:install
rails g spree_gateway:install
rails g spree:install --migrate=false --sample=false --seed=false --copy_storefront=false
rails s

Extensibility

Spree comes with a lot of advanced features out-of-the-box, with an impressive list of third-party integrations. This means that the majority of payment integrations are taken care of, and many marketing, mobile-first, and customer-facing features are already in place.

If you need something more than the core system provides, you can always check the list of existing extensions. Currently, there are:

• Support for Analytics Trackers
• Avalara AvaTax for tax decision automation
• Devise gem authentication services for Spree
• Paypal Express extension
• Contact Us form
• Extension enabling downloadable products
• Payment Gateways

And more.

In case your need is not covered by the available functionalities and extensions, you can always create your own plugin. You can start committing new functionalities to the local Github repository or create your own sandbox application.

Performance

In terms of performance, Spree packs the most punch out of the three platforms. For a single product request, it handles 456.67 requests per second with 210.05KB per second transfer.

wrk -t12 -c400 -d30s -H "X-Spree-Token: 6e332be3b4fffb003a88ee88c1c9036c8b041147c5e7a0e4" https://spree-e-commerce.herokuapp.com/api/v2/products/1
Running 30s test @ https://spree-e-commerce.herokuapp.com/api/v2/products/1
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   512.14ms   52.55ms 711.39ms   76.98%
    Req/Sec    44.49     29.71   150.00     65.88%
  13747 requests in 30.10s, 6.17MB read
  Socket errors: connect 158, read 0, write 0, timeout 0
Requests/sec:    456.67
Transfer/sec:    210.05KB

When we’re fetching a list of all the products, Spree can handle 546.45 requests per second with 249.21KB per second transfer.

wrk -t12 -c400 -d30s -H "X-Spree-Token: 6e332be3b4fffb003a88ee88c1c9036c8b041147c5e7a0e4" https://spree-e-commerce.herokuapp.com/api/v2/products/
Running 30s test @ https://spree-e-commerce.herokuapp.com/api/v2/products/
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   428.17ms   31.38ms 514.23ms   88.02%
    Req/Sec    51.48     30.98   171.00     61.30%
  16448 requests in 30.10s, 7.33MB read
  Socket errors: connect 158, read 0, write 0, timeout 0
Requests/sec:    546.45
Transfer/sec:    249.21KB

Saleor

Saleor is based on Python and Django, the latest major update introduces a modular front end powered by a GraphQL API and written with React and TypeScript. It’s developed with flexibility in mind so adding new features is easy. It has a big and supportive community, while being mature and well-developed at the same time. Saleor is basically a ready-made open-source product.

As we mentioned in our previous post, the localhost API and Heroku set up is lightning fast (we managed to do it in two hours). To set up the backend on Heroku is quite simple (just keep in mind that you’ll need to set additional environment variables if you’ll pin up the BE with dashboard). There are instructions for deploying Saleor on different platforms. A Heroku environment variables set up might look like this:

ALLOWED_CLIENT_HOSTS=.ecommerce-saleor-dashboard.herokuapp.com
ALLOWED_HOSTS=.ecommerce-saleor-dashboard.herokuapp.com, .ecommerce-saleor.herokuapp.com
DEBUG=False (DEBUG=True by default)

To set up the dashboard, you’ll need to provide API_URI, e.g.:

API_URI=https://ecommerce-saleor.herokuapp.com/graphql/

You can also deploy Saleor locally, using this guide.

Extensibility

As we’ve mentioned, Saleor is basically a ready-made product. In most cases there will be no need to develop additional features. However, to those who wish to extend the platform, Saleor also gives a helping hand.

Most importantly, the backend code is nicely structured and easy to navigate. All developers know this comes a long way when extending an app created by someone else. Additionally, the creators propose two ways of adding new features: through apps, and through plugins.

The documentation states that:

Apps are standalone web applications that are given access to Saleor’s API. Apps can be given permissions and are able to perform most actions that a staff member is able to perform. Apps can also subscribe to certain events as they happen in Saleor and get notified about them using webhooks.

Plugins offer a way to run your code within the same application process as Saleor Core. Plugins are notified about events using callbacks and have direct access to the database.

Tutorials for creating both are neatly written and clear.

Performance

Performance-wise, Saleor stands its ground. For a single product, it handles 92.07 requests per second with a 43.42KB per second transfer.

====================
benchmark: One product
  --------------------
  candidate: OneProduct on python-graphene at https://ecommerce-saleor.herokuapp.com/graphql/
++++++++++++++++++++
     400Req/s Duration:60s open connections:20
      Running 1m test @ https://ecommerce-saleor.herokuapp.com/graphql/
        8 threads and 20 connections
        Thread calibration: mean lat.: 5312.195ms, rate sampling interval: 15056ms
        Thread calibration: mean lat.: 5316.608ms, rate sampling interval: 14991ms
        Thread calibration: mean lat.: 5187.398ms, rate sampling interval: 14991ms
        Thread calibration: mean lat.: 5200.366ms, rate sampling interval: 14942ms
        Thread calibration: mean lat.: 5235.184ms, rate sampling interval: 14983ms
        Thread calibration: mean lat.: 5153.267ms, rate sampling interval: 14794ms
        Thread calibration: mean lat.: 5154.761ms, rate sampling interval: 14934ms
        Thread calibration: mean lat.: 5249.303ms, rate sampling interval: 15040ms
        Thread Stats   Avg      Stdev     Max   +/- Stdev
          Latency    27.40s    11.12s   46.37s    57.64%
          Req/Sec    11.29      0.45    12.00    100.00%
        5526 requests in 1.00m, 2.55MB read
        Socket errors: connect 0, read 0, write 0, timeout 16
        Non-2xx or 3xx responses: 5526
      Requests/sec:     92.07
      Transfer/sec:     43.42KB

For a list of products, Saleor handles 98.31 requests per second with a 64.99KB per second transfer.

====================
benchmark: All products
  --------------------
  candidate: AllProducts on python-graphene at https://ecommerce-saleor.herokuapp.com/graphql/
      ++++++++++++++++++++
400Req/s Duration:60s open connections:20
      Running 1m test @ https://ecommerce-saleor.herokuapp.com/graphql/
        8 threads and 20 connections
        Thread calibration: mean lat.: 3983.652ms, rate sampling interval: 13484ms
        Thread calibration: mean lat.: 4041.357ms, rate sampling interval: 13737ms
        Thread calibration: mean lat.: 4137.041ms, rate sampling interval: 13852ms
        Thread calibration: mean lat.: 4095.248ms, rate sampling interval: 13754ms
        Thread calibration: mean lat.: 3972.554ms, rate sampling interval: 13467ms
        Thread calibration: mean lat.: 4081.227ms, rate sampling interval: 13778ms
        Thread calibration: mean lat.: 4024.856ms, rate sampling interval: 13524ms
        Thread calibration: mean lat.: 4075.532ms, rate sampling interval: 13688ms
        Thread Stats   Avg      Stdev     Max   +/- Stdev
          Latency    26.23s    10.91s   45.45s    57.57%
          Req/Sec    11.88      0.33    12.00    100.00%
        5900 requests in 1.00m, 3.81MB read
        Non-2xx or 3xx responses: 5900
      Requests/sec:     98.31
      Transfer/sec:     64.99KB

Since all the above tests were performed on a free Heroku account, they translate into superb performance on paid cloud hosting.

Summary

All three platforms are superstars in their own category.

Vendure.io will not choke up handling requests in any small or medium-sized shops. Although the out-of-the-box functionalities and existing plugins are rather basic, the process of creating new ones and implementing them to your shop is developer-friendly.

Spree is the Usain Bolt of our comparison. It handles requests so fast that they barely have the time to leave the user’s machine (quantum computing anyone?). Creating plugins here might be more cumbersome than in other platforms, but Spree has a lot to offer out-of-the-box.

Saleor provides solid performance with a developer-friendly way of extending the core platform. It’s perfect for creating custom and user-friendly ecommerce platforms.

There you have it! If you want to be up-to-date with the posts in this series, simply sign up to our newsletter.

Also, here’s a list of all posts we’ve written in this topic so far:

• Open Source eCommerce Platform Comparison Part 1 — The Best Ruby, Node, and Python Ones [Summer 2020]
• eCommerce Platforms Comparison Part 3 — Shopify, Shopify PLUS vs Open Source Alternatives [Autumn 2020]

Originally published at https://www.monterail.com.