DOC-12489 magma default storage engine #3813
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,95 +1,119 @@ | ||
| = Storage Engines | ||
| :description: pass:q[Couchbase supports two different backend storage engines: Couchstore and Magma.] | ||
|
|
||
| [abstract] | ||
| {description} | ||
| These storage engines organize the data both on disk ad in memory. | ||
|
There was a problem hiding this comment. Typo: organize the data both on disk ad in memory |
||
| This page explains how to choose a backend storage to suit your needs. | ||
|
|
||
|
|
||
| [#storage-engine-magma] | ||
| == Magma | ||
|
|
||
| [.edition]#{enterprise}# | ||
|
|
||
| Magma is designed for high performance with large datasets that do not fit in memory. | ||
| It's ideal for use cases that rely primarily on data stored on disk. | ||
| The performance of disk access is as good as the underlying disk sub-systems. | ||
| For example, using NVMe SSDs gives better performance than spinning drives. | ||
|
|
||
| When you create a bucket using the Magma storage engine, you choose the number of xref:learn:buckets-memory-and-storage/vbuckets.adoc[vBuckets] a bucket uses. | ||
| Magma supports two vBucket configurations: 128 or 1024. | ||
| This choice affects the minimum memory quota for the bucket. | ||
| The 128 vBucket configuration is the default for new buckets in Couchbase Server Enterprise Edition 8.0 and later. | ||
| It has a minimum memory quota of 100{nbsp}MiB per node. | ||
|
|
||
| NOTE: Magma buckets with 128 vBuckets are only available in clusters that have fully migrated to Couchbase Server 8.0 or later. | ||
|
|
||
| Magma using 1024 vBuckets has a minimum memory quota of 1{nbsp}GiB per node. | ||
|
There was a problem hiding this comment. I liked the "Which Storage Engine Should You Use?", but since that section is further down in the page, I'm thinking that this may be a good place to just note here: If you can allocate at least 1 GiB memory per node to your bucket, you should choose the 1024 vBucket option for Magma as you will get better performance at scale. |
||
|
|
||
| As your dataset grows, Magma has a minimum memory-to-data ratio of 1%. | ||
| For example, a node with 5{nbsp}TiB data in a Magma bucket must allocate at least 51{nbsp}GiB of RAM for the bucket. | ||
|
|
||
| === Magma Writer Thread Settings | ||
|
|
||
| Couchbase Server lets you set the number of threads for reading and writing data to the disk. | ||
| Under some circumstances, you may want to set the Writer Threads to `Disk i/o optimized` for Magma buckets. | ||
| This setting makes sure there are enough threads to sustain high write rates. | ||
|
|
||
| To learn more about how you should configure the Writer Thread settings for your Magma bucket, see xref:manage:manage-settings/general-settings.adoc#data-settings[Data Settings] | ||
|
|
||
| == Couchstore | ||
|
|
||
| Couchstore is the original storage engine for Couchbase Server. | ||
| It's the only storage engine available in Couchbase Server Community Edition. | ||
| Couchstore is designed for high performance with datasets that fit in memory. | ||
| It has a minimum memory requirement of 100{nbsp}MB per node, and a minimum memory-to-data ratio of 10%. | ||
| If you have a small dataset whose working set (frequently accessed data) can fit in memory, then you should consider using Couchstore. | ||
|
|
||
| == Couchstore verses Magma at a Glance | ||
|
|
||
| The following table summarizes the differences between Couchstore and Magma storage engines. | ||
|
|
||
| |=== | ||
| | {empty} | Couchstore |Magma 128 vBuckets | Magma 1024 vBuckets | ||
|
|
||
| | Minimum bucket memory quota | ||
| | 100{nbsp}MiB | ||
| | 100{nbsp}MiB | ||
| | 1{nbsp}GiB | ||
|
|
||
| | Minimum memory to data ratio | ||
| | 10% | ||
| | 1% | ||
| | 1% | ||
|
|
||
| | Maximum data per node | ||
| | 3{nbsp}TB | ||
| | 10{nbsp}TB, but with much higher possibility of data imbalance. | ||
| | 10{nbsp}TB | ||
|
|
||
| |=== | ||
|
|
||
| == Which Storage Engine Should You Use? | ||
|
|
||
| The choice of which storage engine to use depends on your use case and the size of your dataset. | ||
| As of Couchbase Serve 8.0, the Magma is the recommended storage engine for most use cases. | ||
| The choice between 128 and 1024 vBuckets gives you flexibility when configuring your bucket. | ||
|
|
||
| When to choose Magma with 128 vBuckets:: | ||
| You should use the Magma backend with 128 vBuckets if any of the following are true: | ||
|
|
||
| + | ||
| * You want to minimize memory use and have a working dataset that does not fit in the available memory or is more than 20% of the total dataset. | ||
| Magma with 128 vBuckets is more efficient in this case than Couchstore. | ||
| * You want to conserve memory as your dataset grows. | ||
| Because Magma has a 1% memory-to-data ratio, its memory requirements grow more slowly than Couchstore's 10% memory-to-data ratio. | ||
| * You have a dataset size that's less than 100{nbsp}GiB. | ||
| With this small dataset size, the 128 vBucket configuration saves you memory verses the minimum 1{nbsp}GiB memory required by the 1024 vBucket Magma configuration. | ||
| * You're configuring a POC or testing system and plan to use Magma with 1024 vBuckets in production. | ||
|
|
||
| When to choose Magma with 1024 vBuckets:: | ||
| You should use the Magma backend with 1024 vBuckets if any of the following are true: | ||
|
|
||
| + | ||
| * Your working set is larger than the available memory (or you expect it to outgrow memory), and you need high disk-access speed. | ||
| * You need to store and access large amounts of data (multiple terabytes) using a small amount of memory. | ||
| In this case, the 1024 vBucket configuration is less likely to have data imbalance issues than the 128 vBucket configuration. | ||
| Also, Magma with 1024 vBuckets uses less CPU for operations such as compaction verse the 128 vBucket configuration. | ||
| * Your applications make heavy use of transactions with persistence-based durability. | ||
| * Your dataset will grow beyond Couchstore's 3{nbsp}TB limit. | ||
|
|
||
| When should you choose Couchstore?:: | ||
| You should use the Couchstore backend if any of the following are true: | ||
|
|
||
| + | ||
| * You're using Couchbase Server Community Edition which only supports the Couchstore storage engine. | ||
| * You have a dataset with a working set that fits into available memory and the working set is less than 20% of the total dataset. | ||
| * You're running Couchbase Server on low-end hardware. | ||
| * You're running the legacy xref:learn:views/views-intro.adoc[] Service, which does not run on Magma storage. | ||
|
|
||
|
|
||
| == Migrating between Storage Engines | ||
|
|
||
| You can migrate a bucket to use a different storage engine. | ||
| Consider migrating a bucket if it no longer meets the criteria explained in the previous sections. | ||
| For example, suppose you have a bucket using the Couchstore backend whose working dataset has grown to the point where it no longer fits in memory. | ||
| In this case, you should migrate the bucket to use Magma as a storage backend. | ||
|
|
||
| To learn how to migrate a bucket's storage backend, see xref:manage:manage-buckets/migrate-bucket.adoc[]. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we are giving a link to the MapReduce Views page here, could we have the same note that we have in Views intro page saying that Views are deprecated and will be removed in a future version in the MapReduce Views page?
Link to Views intro page -- https://docs.couchbase.com/server/current/learn/views/views-intro.html