Hazelcast in Depth

What Hazelcast is, in one minute

Hazelcast is an in-memory data grid: a cluster of machines (called members) that together hold your data in RAM and spread it across themselves. To your code, a Hazelcast map (IMap) looks like an ordinary key-value map; behind it, each entry lives on one member, with a backup copy on another.

Teams use it as a distributed cache, a shared session store, a fast store for state that many services read, and a place to run computation next to the data. Compared with Redis, it is built for the JVM, can run inside your application, and leans on partitioned data with automatic backups rather than a primary with replicas.

How data is spread

• Every key is hashed into one of 271 partitions (the default count).
• Partitions are spread across members. Each partition has one owner (primary) and, by default, one synchronous backup on a different member.
• When a member joins or leaves, partitions migrate so the load is even again, and backups are re-created.

Embedded or client-server?

• Embedded: Hazelcast members run inside your application's JVM. Reads of keys owned by that instance need no network hop, and there is nothing extra to deploy. The cost: your app and your data scale and restart together. Every deploy or autoscaling event migrates partitions, and the data shares the app's memory and garbage collector.
• Client-server: the cluster runs on its own; applications connect as clients (Java, .NET, Go, Python, Node.js and more). One extra network hop, but the data tier scales, deploys and fails independently of the apps.

A common rule: embedded for a small, stable set of JVM services; client-server once apps scale often, use several languages, or the data must outlive app deploys.

The settings that matter

Setting	Default	What it decides
Partition count	271	how finely data is spread; set before the cluster starts
backup-count	1 (synchronous)	how many members must have a write before it returns
async-backup-count	0	extra copies made in the background
read-backup-data	false	whether embedded members may read their local backup (faster, possibly stale)
Eviction policy	NONE	whether entries are ever removed when a size limit is reached
time-to-live-seconds	0 (forever)	how long entries live

Note the eviction default: a map used as a cache, with no size limit and no TTL, grows until members run out of memory.

Backups: what survives a failure

With one synchronous backup, a write returns only after both the owner and the backup have it. If one member dies, its backups become owners and new backups are created on the survivors: no data lost. If two members die at the same time, some partitions may lose both copies. Raise backup-count for data that must survive that (each extra copy costs memory and write latency), or make sure the data can be reloaded from a database.

Asynchronous backups make writes faster, but a write acknowledged just before the owner dies may not have reached its backup yet.

Near cache: fast and slightly stale

A near cache keeps a local copy of frequently read entries in the client (or member), so repeated reads skip the network entirely. When an entry changes, the cluster sends invalidations, but they arrive asynchronously. Hazelcast is explicit about the trade: a near cache does not keep strong consistency, so you may read stale data for a short time. Use it for read-mostly data where slightly old values are acceptable (product details, configuration), not for balances or stock counts.

Patterns you will use

• Entry processors: instead of get, modify, put from several instances (which races and moves the value over the network twice), send a small function to the member that owns the key. It runs there one at a time for that entry, so the update is atomic.
• MapStore: connect a map to a database. Read-through loads missing keys on demand. With write-delay-seconds at 0 it is write-through (the database write happens before the put returns); above 0 it is write-behind: writes are batched and sent later, optionally keeping only the latest value per key (write-coalescing). Write-behind is fast, but the database lags behind the map.
• Queries and SQL: query maps by value with predicates or SQL; add indexes on fields you filter by.

Going deeper: split-brain and the CP Subsystem

If the network splits a cluster in two, each side can carry on as a smaller cluster, accepting writes to the same keys. When the network heals, the clusters merge and a merge policy decides which value wins for each key (the default, PutIfAbsentMergePolicy, adds an entry from the smaller cluster only if the larger one does not have that key). Some writes from the losing side are discarded. Split-brain protection can make a side with too few members refuse operations instead, choosing consistency over availability.

The regular data structures are built for availability (AP). For operations that must be strongly consistent (locks, counters, leader election), Hazelcast has the CP Subsystem, which runs Raft among a set of CP members. Without CP members configured, those structures run in an unsafe mode the documentation says is only for development and testing.

How it fails

• Memory exhaustion: no eviction and no TTL on a growing map.
• Data loss with simultaneous failures: two members down with one backup.
• Stale reads: near caches, read-backup-data, or asynchronous backups.
• Migration storms: frequent membership changes (embedded mode with autoscaling) keep partitions moving.
• Long GC pauses: a member that pauses too long is suspected dead, triggering migration. Keep heaps sensible.

What to say in a design review

• "Client-server topology, so our services scale and deploy without moving data."
• "One synchronous backup; anything critical is also in the database through MapStore, so losing two members at once means a reload, not lost data."
• "A near cache for product data only, where a few seconds of staleness is acceptable."
• "Counters are updated with entry processors, not get-then-put."

Remember this

• 271 partitions by default, each with one synchronous backup on another member.
• Embedded mode couples the data to your app's lifecycle; client-server separates them.
• A near cache trades strong consistency for speed: invalidations are asynchronous.
• Eviction is off by default. Set a size limit or TTL for caches.
• Use entry processors for atomic updates, and know whether your MapStore is write-through or write-behind.