Making cluster-safe plugins

In clustered Bitbucket Server, plugins largely just work. However, there are a few things to be aware of in more advanced plugins. Testing your plugin in a cluster is also more involved and requires some additional configuration.

Home directory

Bitbucket Server has a local home and a shared home for all instances, not just clustered instances. This is intended to make it simpler for plugin developers to write their plugins, knowing that BITBUCKET_HOME will be laid out consistently on standalone and clustered instances. The home directory is laid out as follows:

1
BITBUCKET_HOME
2
|-- bin
3
|-- caches
4
|-- export
5
|-- lib
6
|-- log
7
|-- shared (BITBUCKET_SHARED_HOME)
8
|   |-- config
9
|   |-- data
10
|   |   |-- attachments
11
|   |   |-- avatars
12
|   |   |-- repositories
13
|   |-- plugins
14
|   |   |-- installed-plugins
15
|   |-- bitbucket.properties
16
|-- tmp
 
 

BITBUCKET_SHARED_HOME, by default, is BITBUCKET_HOME/shared. Plugin developers should not rely on this, however; the location of BITBUCKET_SHARED_HOME can be overridden using environment variables or system properties. Instead, plugin developers should use:

• ApplicationPropertiesService.getHomeDir() => BITBUCKET_HOME
• ApplicationPropertiesService.getSharedHomeDir() => BITBUCKET_SHARED_HOME

In a clustered environment, BITBUCKET_SHARED_HOME is guaranteed to be the same filesystem on every node, allowing data that is stored there to be accessed by all nodes.

Warning:

BITBUCKET_SHARED_HOME will generally be a network mount, such as an NFS partition. This imposes some special considerations:

• Performance is likely to be slower than a local disk
• Some filesystem-level behaviors, like locking and renaming, may not work as expected (or at all)
• NFS configuration issues may trigger unexpected/unsafe behavior

Where possible, plugins should minimize their use of the filesystem if they need to use BITBUCKET_SHARED_HOME.

A note about dependencies

You can ensure a compatible version of shared libraries, like Atlassian Beehive, Atlassian Cache and Atlassian Scheduler, is used by importing Bitbucket Server's parent POM, like this:

1
<dependencyManagement>
2
     <dependencies>
3
         <dependency>
4
             <groupId>com.atlassian.bitbucket.server</groupId>
5
             <artifactId>bitbucket-parent</artifactId>
6
             <version>${bitbucket.version}</version>
7
             <type>pom</type>
8
             <scope>import</scope>
9
         </dependency>
10
     </dependencies>
11
 </dependencyManagement>
 
 

Where bitbucket.version is defined as the minimum version of Bitbucket Server you want your plugin to support.

A note about serialization

Communication among cluster nodes is facilitated by Java Serialization. This means, when using distributed types such as remote() Atlassian Cache Caches, job data in Atlassian Scheduler (even if you're using RunMode.RUN_LOCALLY!), or the BucketedExecutor, the objects you use must be Serializable. Externalizable extends Serializable and is also supported.

Most Bitbucket Server types, such as Project and Repository, are not Serializable and cannot be used directly in these contexts. Instead, their IDs (Project.getId(), Repository.getId(), the pair of [PullRequest.getToRef().getRepository().getId(), PullRequest.getId()], etc.) should be serialized and then their respective services (ProjectService.getById(int), RepositoryService.getById(int), PullRequestService.getById(int, long), etc.) should be used to re-retrieve the full objects as necessary. The services themselves are also not Serializable.

Because objects must be re-retrieved prior to processing, plugin code should account for the fact that the state of the objects may have changed:

• Projects and repositories can be deleted, so getById(int) calls should appropriately handle null returns
• Pull requests may be updated, referencing new commits or target branches or being merged or declined
• Etc.

This is not intended as an exhaustive list–rather to promote good programming practice and more robust processing. If any of the exact state of the object at serialization time matters, the relevant state should be extracted and included in the object's serialized form. This should be kept to a minimum, however, to keep the serialized representation of objects as small as possible. Large serialized blobs have impacts both on Bitbucket Server's memory footprint and on the efficiency of inter-node communication.

Note:

Bitbucket Server standalone instances are considered one-node clusters. That means the same Serializable rules apply regardless of whether multiple nodes are actually present. This is intended to make plugin developer's lives simpler. Bitbucket Server behaves consistently both clustered and standalone, so plugins written for a cluster will work correctly standalone.

Caching in a cluster

In simple plugins, it is common to cache data using ConcurrentMaps or Guava Caches. This caching will not work correctly in a cluster because updating the data on one node will leave stale data in the caches on other nodes.

Plugins should use Atlassian Cache, an API provided by Bitbucket Server for plugins. You can add Atlassian Cache to your plugin with the following Maven dependency:

1
<dependency>
2
     <groupId>com.atlassian.cache</groupId>
3
     <artifactId>atlassian-cache-api</artifactId>
4
     <scope>provided</scope>
5
 </dependency>
 
 

To use Atlassian Cache, you:

• Add <component-import key="cacheFactory" interface="com.atlassian.cache.CacheFactory"/> in atlassian-plugin.xml
• Add the CacheFactory to the relevant component's constructor
• Create the cache. You can also pass CacheSettings (created through the CacheSettingsBuilder class) to control many aspects of how the cache works. Caches are remote() by default.

1
cacheFactory.getCache("com.example.plugin:example-plugin-key:Example  Cache", 
2
             new CacheLoader<String, String>() {
3
                 @Nonnull
4
                 @Override
5
                 public String load(@Nonnull String key) {
6
                     return "Value";
7
                 }
8
             }
9
     );
 
 

• You should create your Cache once, in your constructor, and use the same instance afterward
  • Continuously re-fetching the cache from the CacheFactory is inefficient

Note:

If you are using a remote() cache (the default), your keys and values must be Serializable. Externalizable extends Serializable and is also acceptable.

Scheduling jobs in a cluster

Without any intervention, scheduled tasks will execute independently on each Bitbucket Server instance in a cluster. In some circumstances, this is desirable behavior. In other situations, you will need to use cluster-wide locking to ensure that jobs are only executed once per cluster. This is accomplished by using Atlassian Scheduler.

You can add Atlassian Scheduler to your plugin with the following Maven dependency:

1
<dependency>
2
     <groupId>com.atlassian.scheduler</groupId>
3
     <artifactId>atlassian-scheduler-api</artifactId>
4
     <scope>provided</scope>
5
 </dependency>
 
 

To use Atlassian Scheduler you:

• Add <component-import key="schedulerService" interface="com.atlassian.scheduler.SchedulerService"/> to your atlassian-plugin.xml
• Add the SchedulerService to the relevant component's constructor
• Register your JobRunner
• Schedule your job, assigning it an ID and providing its JobConfig which describes:
  • How often the job should run
  • The delay for the initial run
  • Whether the job should run on each cluster node or once across the cluster
• Unregister your JobRunner during shutdown

A JobRunner handles JobRunnerRequests and performs the actual processing. Generally each node in a cluster will register its own JobRunner. This allows all of the nodes in the cluster to run the job, allowing the cluster to more efficiently distribute load.

1
public class MyJobRunner implements JobRunner {
2
     @Override
3
     public JobRunnerResponse runJob(JobRunnerRequest request) {
4
         //Do some meaningful work
5
 
6
         return JobRunnerResponse.success();
7
     }
8
 }
9
 
10
 schedulerService.registerJobRunner("com.example.plugin:example-plugin-key:ExampleJobRunner", new MyJobRunner());
 
 

When a job is scheduled, the key assigned to the JobRunner when it is registered is used to associate the job with its runner:

1
schedulerService.scheduleJob( 
2
         JobId.of("com.example.plugin:example-plugin-key:ExampleJob"), 
3
         JobConfig.forJobRunnerKey("com.example.plugin:example-plugin-key:ExampleJobRunner") 
4
                 .withRunMode(RunMode.RUN_ONCE_PER_CLUSTER) 
5
                 .withSchedule(Schedule.forInterval(intervalInMillis, new Date(System.currentTimeMillis() + intervalInMillis))));
 
 

During application shutdown, you should unregister the JobRunner so that the node shutting down is no longer considered a candidate for running the job:

schedulerService.unregisterJobRunner("com.example.plugin:example-plugin-key:ExampleJobRunner");
 
 

The easiest way to put together the register and unregister lifecycle is to use the Spring InitializingBean and DisposableBean interfaces on your component:

1
public class ExampleComponent implements DisposableBean, InitializingBean { 
2
 
3
     private static final JobId JOB_ID = JobId.of("com.example.plugin:example-plugin-key:ExampleJob"); 
4
     private static final long JOB_INTERVAL = TimeUnit.MINUTES.toMillis(30L); 
5
     private static final String JOB_RUNNER_KEY = "com.example.plugin:example-plugin-key:ExampleJobRunner"; 
6
 
7
     private final SchedulerService scheduler; 
8
 
9
     public ExampleComponent(SchedulerService schedulerService) { 
10
         this.schedulerService = schedulerService; 
11
     } 
12
 
13
     @Override 
14
     public void afterPropertiesSet() throws SchedulerServiceException { 
15
         //The JobRunner could be another component injected in the constructor, a 
16
         //private nested class, etc. It just needs to implement JobRunner 
17
         schedulerService.registerJobRunner(JOB_RUNNER_KEY, new MyJobRunner()); 
18
         schedulerService.scheduleJob(JOB_ID, JobConfig.forJobRunnerKey(JOB_RUNNER_KEY) 
19
                 .withRunMode(RunMode.RUN_ONCE_PER_CLUSTER) 
20
                 .withSchedule(Schedule.forInterval(JOB_INTERVAL, new Date(System.currentTimeMillis() + JOB_INTERVAL)))); 
21
     } 
22
 
23
     @Override 
24
     public void destroy() { 
25
         schedulerService.unregisterJobRunner(JOB_RUNNER_KEY); 
26
     } 
27
 }
 
 

Note:

Job data provided in JobConfig is required to be Serializable, regardless of RunMode. The backing store for job data may serialize objects even for RunMode.RUN_LOCALLY jobs.

Warning:

Generally you should not unregister the job itself. Unregistering a job unregisters it across the cluster, not just on the node shutting down.

When multiple nodes schedule the same job but with a different schedule (even differing by milliseconds) then the last registration will win and replace the old job configuration and schedule. If the schedule is eligible to run immediately and multiple nodes take this action at close to the same time, then the job might run more than once as the instances replace one another.

Also note that scheduled jobs in Bitbucket Server are not persistent. They must be rescheduled each time the application starts.

Locking in a cluster

Java's locking primitives, like Lock, synchronized, etc., only apply to a single JVM and will not properly serialize operations in a cluster. Instead, you need to use the cluster-wide lock. This is accomplished by using:

• Atlassian Beehive's ClusterLockService
  • Atlassian Beehive is cross-product and also works in Confluence and JIRA
• LockService, part of the bitbucket-api module
  • LockService is specific to Bitbucket Server

Atlassian Beehive

You can add Atlassian Beehive to your plugin with the following Maven dependency:

1
<dependency> 
2
     <groupId>com.atlassian.beehive</groupId> 
3
     <artifactId>beehive-api</artifactId> 
4
     <scope>provided</scope> 
5
 </dependency>
 
 

To use Atlassian Beehive's ClusterLockService you:

• Add <component-import key="clusterLockService" interface="com.atlassian.beehive.ClusterLockService"/> to your atlassian-plugin.xml
• Add the ClusterLockService to the relevant component's constructor
• Create your ClusterLock (which extends the standard Java Lock interface):

1
public class ExampleComponent { 
2
 
3
     private final ClusterLock taskLock; 
4
 
5
     public ExampleComponent(ClusterLockService lockService) { 
6
         taskLock = lockService.getLockForName(getClass().getName() + ":TaskLock"); 
7
     } 
8
 
9
     public void performTask() { 
10
         if (taskLock.tryLock()) { 
11
             try { 
12
                 //Do something, knowing no other node in the cluster is accessing 
13
                 //whatever resource you're protecting 
14
             } finally { 
15
                 taskLock.unlock(); 
16
             } 
17
         } else { 
18
             //Another node in the cluster holds the lock already 
19
         } 
20
     } 
21
 }
 
 

LockService

You can use the LockService by adding a dependency on bitbucket-api, generally already a dependency of any Bitbucket Server plugin:

1
<dependency> 
2
     <groupId>com.atlassian.bitbucket.server</groupId>
3
     <artifactId>bitbucket-api</artifactId>
4
     <scope>provided</scope> 
5
 </dependency>
 
 

The LockService is used in a similar way to Atlassian Beehive's ClusterLockService:

• Add <component-import key="lockService" interface="com.atlassian.bitbucket.concurrent.LockService"/> to atlassian-plugin.xml
• Add the LockService to the relevant component's constructor
• Create your Lock:

1
public class ExampleComponent { 
2
 
3
     private final Lock taskLock; 
4
 
5
     public ExampleComponent(LockService lockService) { 
6
         taskLock = lockService.getLock(getClass().getName() + ":TaskLock"); 
7
     } 
8
 
9
     public void performTask() { 
10
         if (taskLock.tryLock()) { 
11
             try { 
12
                 //Do something, knowing no other node in the cluster is accessing 
13
                 //whatever resource you're protecting 
14
             } finally { 
15
                 taskLock.unlock(); 
16
             } 
17
         } else { 
18
             //Another node in the cluster holds the lock already 
19
         } 
20
     } 
21
 }
 
 

In addition to Locks, the LockService provides access to more specialized RepositoryLocks and PullRequestLocks.

• RepositoryLock allows concurrent operations on different Repository instances, but serializes operations on the same instance
• PullRequestLock allows concurrent operations on different PullRequest instances, but serializes operations on the same instance

These locks can be used to reduce contention, by allowing concurrent operations on different instances, while still ensuring each instance is acted on serially. These locks are cluster-safe, meaning only one node in the cluster will operate on a given Repository or PullRequest at once.

Note:

ClusteredLock, Lock, PullRequestLock and RepositoryLock are not Serializable and cannot be transferred between nodes.

• Locks can only be unlocked by the thread that acquired them
• Locks cannot be used as job data with Atlassian Scheduler

RepositoryLock and PullRequestLock are namespaced. The same Repository or PullRequest can be locked simultaneously in multiple RepositoryLock or PullRequestLock instances, respectively, which have different names.

It is not possible, from a plugin, to access the locks the host application uses to protect its own processing. They are intentionally stored in an unreachable namespace.

Executors in a cluster

ExecutorServices are useful for managing threaded jobs. Bitbucket Server provides a ScheduledExecutorService which can be imported by plugins to use a standard thread pool. However, ExecutorServices are local to the node where they are created. In a cluster, to efficiently distribute processing, it is sometimes desirable to allow scheduling a task on one node and processing it on another. To facilitate this, Bitbucket Server provides a BucketedExecutor in bitbucket-api, which is generally a dependency of any Bitbucket Server plugin.

1
<dependency> 
2
     <groupId>com.atlassian.bitbucket.server</groupId>
3
     <artifactId>bitbucket-api</artifactId>
4
     <scope>provided</scope> 
5
 </dependency>
 
 

To use the BucketedExecutor you:

• Add <component-import key="concurrencyService" interface="com.atlassian.bitbucket.concurrent.ConcurrencyService"/> to atlassian-plugin.xml
• Add ConcurrentService to the relevant component's constructor
• Create your BucketedExecutor:

1
public class MyTaskRequest implements Serializable { 
2
     //Repository is not Serializable 
3
     private final int repositoryId; 
4
 
5
     public MyTaskRequest(Repository repository) { 
6
         repositoryId = repository.getId(); 
7
     } 
8
 
9
     public int getRepositoryId() { 
10
         return repositoryId; 
11
     } 
12
 } 
13
 
14
 Function<MyTaskRequest, String> bucketFunction = new Function<MyTaskRequest, String>() { 
15
     @Override 
16
     public String apply(MyTaskRequest task) { 
17
         return String.valueOf(task.getRepositoryId()); 
18
     } 
19
 } 
20
 
21
 BucketProcessor<MyTaskRequest> processor = new BucketProcessor<MyTaskRequest>() { 
22
     @Override 
23
     public void process(@Nonnull String bucketId, @Nonnull List<MyTaskRequest> tasks) { 
24
         for (MyTaskRequest task : tasks) { 
25
             Repository repository = repositoryService.getById(task.getRepositoryId()); 
26
             if (repository == null) { 
27
                 log.info("Repository {} was deleted", task.getRepositoryId()); 
28
                 continue; 
29
             } 
30
             //Do some processing 
31
         } 
32
     } 
33
 } 
34
 
35
 BucketedExecutor<MyTaskRequest> executor = concurrencyService.getBucketedExecutor( 
36
         "com.example.plugin:example-plugin-key:ExampleBucketedExecutor", 
37
         new BucketedExecutorSettings.Builder<>(bucketFunction, processor) 
38
                 //How many tasks to process at once? Integer.MAX_VALUE processes the 
39
                 //whole bucket, 1 will receive one task at a time
40
                 .batchSize(Integer.MAX_VALUE) 
41
                 //How many retries, if processing fails? After the retries are 
42
                 //exhausted, the requests that failed will be discarded 
43
                 .maxAttempts(1) 
44
                 //How many threads can process tasks (from different buckets) at the 
45
                 //same time? Concurrency can be PER_NODE or PER_CLUSTER 
46
                 .maxConcurrency(config.getThreadCount(), ConcurrencyPolicy.PER_CLUSTER) 
47
                 .build());
 
 

Each BucketedExecutor is given a Guava Function which is used to divide the buckets. The plugin developer is free to define buckets as coarse or fine as desired. The BucketedExecutor offers two very useful guarantees:

• Tasks will always be passed to the BucketProcessor in the same order they were submitted in
• Exactly one thread may process a given bucket a time, regardless of concurrency
  • Concurrency allows multiple buckets to be processed simultaneously
  • This means BucketProcessors generally do not require locking, if the buckets are well-defined

Warning:

The task type used to specialize the BucketedExecutor generic must be Serializable. Even standalone instances (which are considered one-node clusters) will serialize the tasks as they are submitted, prior to invoking the BucketProcessor.

An bucket's concurrency policy can be either ConcurrencyPolicy.PER_CLUSTER or ConcurrencyPolicy.PER_NODE. PER_CLUSTER is used if you need to throttle concurrency because of a global resource (e.g. a remote service or shared file system). PER_NODE is used if you need to throttle concurrency because of a local resource (e.g. CPU or memory on the node)

When ConcurrencyPolicy.PER_CLUSTER is used, the concurrency limit is divided by number nodes in the cluster to determine how many buckets each node can process concurrently. The result is rounded up, such that every node in the cluster is always allowed to process at least one bucket.

• maxConcurrency(2, ConcurrencyPolicy.PER_CLUSTER) in a three-node cluster behaves like maxConcurrency(1, ConcurrencyPolicy.PER_NODE) - 2/3 = .667 ~ 1 per node
• maxConcurrency(3, ConcurrencyPolicy.PER_CLUSTER) in a two-node cluster behaves like maxConcurrency(2, ConcurrencyPolicy.PER_NODE) - 3/2 = 1.5 ~ 2 per node

Event handling in a cluster

Bitbucket Server does not offer cluster-wide events. Events, such as RepositoryPushEvent, are handled only on the node that raised them. In other words, whichever node processed the push will be the only node that processes events for that push. This is an intentional design decision. The development team feels that this makes implementing a clustered plugin simpler, because plugin developers are not required to prevent re-processing the same event on each node.

Plugin installation in a cluster

Installation for the Bitbucket Server cluster administrator is the same as with a single instance. Uploading a plugin through the web interface will store the plugin in BITBUCKET_SHARED_HOME and ensure that it is installed on all instances of the cluster.

Currently, cluster instances must be homogeneous. However, future plans may introduce support for rolling upgrades and other features that introduce disparities, whether temporary or permanent, between cluster nodes. Plugin developers can assume all nodes will:

• Have compatible versions of all exported APIs
• Have consistent home directory layouts

For the best forward compatibility, plugin developers should not assume all nodes are running the same version of Bitbucket Server.

Plugin testing in a cluster

It is important to test your plugin in a cluster. When running Bitbucket Server via the Atlassian SDK (AMPS) a clustered license is used, so multiple instances started via the Atlassian SDK can be clustered.

Alternatively, you can install the following timebombed license, which is cluster-enabled. This license is only valid for 3 hours, after which you will be unable to push to Bitbucket Server without restarting the servers:

1
AAABAA0ODAoPeNptUE1Lw0AUvPdXLHhOybZGsbCgpiE2aJrSDZ6
2
f8dUuJJvy3ibYf2+M22LF63wxM1ev+C6yzgopRXi3iORiFopkq8
3
UslNcTdsD7adJD3YEzrVU7qBk9HBOO4BIcqm95EN4EUp7Y1jqoX
4
A4NqgdXA7MBKzSyQ/KSFzDWoQVbYfJ5MHQck4r5k+cHu+lROerw
5
MjQZnLVyY9Y9nMKnVdt43bOp0DLq4wHHAnpYtMrTyRapR1ot1WN
6
WboLNPNZBWmZRoKPb1FsLIGeRRpuH1vQB1vDPA+ctnhw6Q4zDDv
7
pdNO+aN6T1rmQkVnJ22evftUVH1R4Y/975BcASkF4wLQIVAJHuX
8
Zz1SsymUm2B5V7p7Pap48xzAhROyzM1l9a1OqcWzxseRNmnZ4Xq
9
mQ==X02d9
 
 

The two easiest way to start a cluster of Bitbucket Server nodes are:

• Run your integration tests (which will start and stop your cluster automatically in the process of deploying your plugin and running tests on it)
• Use atlas-run to spin up a cluster which you can use to iteratively develop your plugin, deploy it and test its cluster safety

Both methods require Maven configuration but the same can be used for both.

To configure an N-node cluster you must specify N <product/> elements, one per node. Each node will need slightly different configuration to ensure it can independently start up (e.g. so each listens on different ports if running on the same machine), find other nodes in the cluster and finally join the cluster.

Specifically, each node will need:

• Its own HTTP port (supplied through the httpPort element)
• Its own SSH port (supplied through the plugin.ssh.port entry in the systemPropertyVariables element)

All nodes will need to share:

• A common BITBUCKET_SHARED_HOME directory (supplied through the bitbucket.shared.home entry in the systemPropertyVariables element)
• An external database (connection details supplied through the jdbc.* entries in the systemPropertyVariables element)

Each node will also need a way to find other nodes. This is supplied through the hazelcast.network.tcpip entry of the systemPropertyVariables element for TCP/IP and/or the hazelcast.network.multicast entry in the systemPropertyVariables element for IP multicast. Without one of these settings set to true (both are false by default) a node will never look for other nodes and thus never join a cluster.

The following Maven pom.xml configuration will start up a cluster of two Bitbucket Server nodes. Node 1 uses port 7991 for HTTP and 7997 for SSH and node 2 uses port 7992 for HTTP and 7998 for SSH. Both nodes use TCP/IP to find each other and use the default TCP/IP settings. They both use a BITBUCKET_SHARED_HOME of /opt/bamboo-agent/bamboo-agent-home/xml-data/build-dir/BSERV-BR47RELEASE-PERFORM/target/checkout/docs/target/bitbucket-node-1/home/shared and connect to the same MySQL database called bitbucket. Also note that because they are connecting to a MySQL database the MySQL JDBC driver jar must be made available to Bitbucket Server. This is achieved through the libArtifact entry for mysql:mysql-connector-java.

1
<build>
2
    <plugins>
3
        <plugin>
4
            <groupId>com.atlassian.maven.plugins</groupId>
5
            <artifactId>bitbucket-maven-plugin</artifactId>
6
            <version>${amps.version}</version>
7
            <extensions>true</extensions>
8
            <configuration>
9
                <products>
10
                    <!-- Node 1 -->
11
                    <product>
12
                        <id>bitbucket</id>
13
                        <instanceId>bitbucket-node-1</instanceId>
14
                        <version>${bitbucket.version}</version>
15
                        <dataVersion>${bitbucket.data.version}</dataVersion>
16
                        <!-- override the HTTP port used for this node -->
17
                        <httpPort>7991</httpPort>
18
                        <systemPropertyVariables>
19
                            <bitbucket.shared.home>/opt/bamboo-agent/bamboo-agent-home/xml-data/build-dir/BSERV-BR47RELEASE-PERFORM/target/checkout/docs/target/bitbucket-node-1/home/shared</bitbucket.shared.home>
20
                            <!-- override the SSH port used for this node -->
21
                            <plugin.ssh.port>7997</plugin.ssh.port>
22
                            <!-- override database settings so both nodes use a single database -->
23
                            <jdbc.driver>com.mysql.jdbc.Driver</jdbc.driver>
24
                            <jdbc.url>jdbc:mysql://localhost:3306/bitbucket?characterEncoding=utf8&amp;useUnicode=true&amp;sessionVariables=storage_engine%3DInnoDB</jdbc.url>
25
                            <jdbc.user>bitbucketuser</jdbc.user>
26
                            <jdbc.password>password</jdbc.password>
27
                            <!-- allow this node to find other nodes via TCP/IP -->
28
                            <hazelcast.network.tcpip>true</hazelcast.network.tcpip>
29
                            <!-- set to true if your load balancer supports stick sessions -->
30
                            <hazelcast.http.stickysessions>false</hazelcast.http.stickysessions>
31
                        </systemPropertyVariables>
32
                        <libArtifacts>
33
                            <!-- ensure MySQL drivers are available -->
34
                            <libArtifact>
35
                                <groupId>mysql</groupId>
36
                                <artifactId>mysql-connector-java</artifactId>
37
                                <version>5.1.32</version>
38
                            </libArtifact>
39
                        </libArtifacts>
40
                    </product> 
41
                    <!-- Node 2 -->
42
                    <product>
43
                        <id>bitbucket</id>
44
                        <instanceId>bitbucket-node-2</instanceId>
45
                        <version>${bitbucket.version}</version>
46
                        <dataVersion>${bitbucket.data.version}</dataVersion>
47
                        <!-- override the HTTP port used for this node -->
48
                        <httpPort>7992</httpPort>
49
                        <systemPropertyVariables>
50
                            <bitbucket.shared.home>/opt/bamboo-agent/bamboo-agent-home/xml-data/build-dir/BSERV-BR47RELEASE-PERFORM/target/checkout/docs/target/bitbucket-node-1/home/shared</bitbucket.shared.home>
51
                            <!-- override the SSH port used for this node -->
52
                            <plugin.ssh.port>7998</plugin.ssh.port>
53
                            <!-- override database settings so both nodes use a single database -->
54
                            <jdbc.driver>com.mysql.jdbc.Driver</jdbc.driver>
55
                            <jdbc.url>jdbc:mysql://localhost:3306/bitbucket?characterEncoding=utf8&amp;useUnicode=true&amp;sessionVariables=storage_engine%3DInnoDB</jdbc.url>
56
                            <jdbc.user>bitbucketuser</jdbc.user>
57
                            <jdbc.password>password</jdbc.password>
58
                            <!-- allow cluster nodes to find each other over TCP/IP thus enabling clustering for this node -->
59
                            <hazelcast.network.tcpip>true</hazelcast.network.tcpip>
60
                            <!-- set to true if your load balancer supports stick sessions -->
61
                            <hazelcast.http.stickysessions>false</hazelcast.http.stickysessions>
62
                        </systemPropertyVariables>
63
                        <libArtifacts>
64
                            <!-- ensure MySQL drivers are available -->
65
                            <libArtifact>
66
                                <groupId>mysql</groupId>
67
                                <artifactId>mysql-connector-java</artifactId>
68
                                <version>5.1.32</version>
69
                            </libArtifact>
70
                        </libArtifacts>
71
                    </product> 
72
                </products>
73
                <testGroups>
74
                    <!-- tell AMPS / Maven which products ie nodes to run for the named testGroup 'clusterTestGroup' -->
75
                    <testGroup>
76
                        <id>clusterTestGroup</id>
77
                        <productIds>
78
                            <productId>bitbucket-node-1</productId>
79
                            <productId>bitbucket-node-2</productId>
80
                        </productIds>
81
                    </testGroup>
82
                </testGroups>
83
            </configuration>
84
        </plugin>
85
 
86
        ...
87
 
88
    </plugins>
89
</build>
90
 
91
...
92
 
93
<properties>
94
    <bitbucket.version>4.0.0</bitbucket.version>
95
    <bitbucket.data.version>4.0.0</bitbucket.data.version>
96
    <amps.version>6.1.0</amps.version>
97
</properties>
 
 

Warning:

amps.version should be set to the same version as the one the minimum supported Bitbucket Server for your plugin uses. Use of dependencyManagement and <scope>import</scope> in you pom.xml as discussed earlier will only import dependencies, not properties or plugins so this value will need to be manually synchronised with Bitbucket Server's as you change your minimum supported Bitbucket Server.

To run the cluster configured above via Atlassian AMPS you would run:

atlas-run --testGroup clusterTestGroup
 
 

To run your integration tests in Maven against the cluster configured above, the following would normally suffice:

atlas-mvn clean install
 
 

For both methods above you will almost always want a load-balancer running to balance HTTP and SSH traffic between the nodes (and so that you can use a single port per protocol to communicate with the cluster). Taking the above configuration as an example you would want your load-balancer to balance HTTP traffic on port 7990 (standalone Bitbucket Server's HTTP default) to ports 7991 and 7992. For SSH traffic you would want it to balance SSH traffic on port 7999 (standalone Bitbucket Server's SSH default) to ports 7997 and 7998.

Atlassian provides a simple Maven plugin which you can configure and run as a load balancer. Again taking the above configuration as an example, you would add the following to your Maven POM:

1
<build>
2
    <plugins>
3
      <plugin>
4
        <groupId>com.atlassian.maven.plugins</groupId>
5
        <artifactId>load-balancer-maven-plugin</artifactId>
6
        <version>1.1</version>
7
        <executions>
8
            <execution>
9
                <id>start-load-balancer</id>
10
                <phase>pre-integration-test</phase>
11
                <goals>
12
                    <goal>start</goal>
13
                </goals>
14
            </execution>
15
            <execution>
16
                <id>stop-load-balancer</id>
17
                <phase>post-integration-test</phase>
18
                <goals>
19
                    <goal>stop</goal>
20
                </goals>
21
            </execution>
22
        </executions>
23
        <configuration>
24
          <balancers>
25
              <balancer>
26
                <port>7990</port>
27
                <targets>
28
                  <target>
29
                    <port>7991</port>
30
                  </target>
31
                  <target>
32
                    <port>7992</port>
33
                  </target>
34
                </targets>
35
              </balancer>
36
              <balancer>
37
                <port>7999</port>
38
                <targets>
39
                  <target>
40
                    <port>7997</port>
41
                  </target>
42
                  <target>
43
                    <port>7998</port>
44
                  </target>
45
                </targets>
46
              </balancer>
47
          </balancers>
48
        </configuration>
49
      </plugin>
50
    </plugins>
51
</build>
 
 

When you run your integration tests from Maven, before starting the cluster, this plugin will start a load balancer as configured and stop it once your tests have finished and the cluster has been shut down.

If you start your Bitbucket Server cluster via atlas-run --testGroup clusterTestGroup, you can run the load balancer separately via:

atlas-mvn com.atlassian.maven.plugins:load-balancer-maven-plugin:1.1:run
 
 

Marking your plugin as cluster compatible for the Marketplace

When you list your first cluster-compatible plugin version in the Marketplace, modify your atlassian-plugin.xml descriptor file. This tells the Marketplace and UPM that your plugin is cluster compatible. Add the following parameter inside the plugin-info section:

<param name="atlassian-data-center-compatible">true</param>
 
 

Here's an example of a generic plugin-info block with this param:

1
<plugin-info>
2
    <description>Base POM for Atlassian projects</description>
3
    <version>4.7.1</version>
4
    <vendor name="Atlassian" url="http://www.atlassian.com" />
5
    <param name="atlassian-data-center-compatible">true</param>
6
</plugin-info>