Cluster Systems

This manual is for an operator standing up an OpenVMX node in a VMScluster. V0.6 is the cluster-correctness release: an OpenVMX node forms a cluster over the real interconnect, reports its membership from the executive, and takes part in a cross-node distributed lock manager. This manual covers exactly what ships — the cluster identity parameters, how a two-node cluster forms, what SHOW CLUSTER and $GETSYI report, and what cross-node locking does for applications. It is just as explicit about what a real VMScluster does that OpenVMX does not do yet; those gaps are listed in Appendix B.

1Before you begin

1.1What this manual covers

A VMScluster is a set of systems that share resources and coordinate through a common lock manager. V0.6 makes an OpenVMX node a cluster participant: it carries a cluster identity, forms a cluster with a peer over the real interconnect, reports its membership from the executive, and takes part in a cross-node distributed lock manager (DLM). This manual is the operator view of that. The wire-protocol reverse engineering and the clean-room provenance of every frame live in the repository's cluster protocol specification (see Appendix A).

Everything described as working here runs over the real executive at /dev/vms. There is no per-process fake behind any cluster fact this manual shows: where a facility is absent, the system returns an honest error, not a fabricated success (INV-6).

1.2What V0.6 gives you

V0.6 is the cluster-correctness milestone. It gives you the following, all real:

• A cluster identity carried in the SYSGEN parameter store and read by the cluster daemon at boot (Chapter 2).
• Two-node cluster formation over genuine raw-Ethernet SCA transport, not a tunnel (Chapter 3).
• Executive-backed membership, read by SHOW CLUSTER and $GETSYI from the real executive membership block (Chapter 4).
• A cross-node distributed lock manager: grant, block-then-grant, blocking ASTs, lock value blocks, mastering and remastering, directory-ownership refusal, and distributed deadlock detection (Chapter 5).

What it does not give you matters just as much, so no one designs against a capability that is not there. In particular, there is no quorum enforcement and no split-brain protection at V0.6, MSCP-served volumes are absent, and cluster-wide logical names degrade to system-wide. These and the rest are listed in Appendix B.

1.3The reference lab

The cluster behaviour OpenVMX reproduces is grounded against a reference lab of genuine VMS VAX systems that the project operates. That lab is an observation oracle only. Nothing OpenVMX ships is copied from VMS: see the clean-room statement in Appendix A.

2Cluster identity parameters

An OpenVMX node's cluster identity lives in the SYSGEN parameter store, SYS$SYSTEM:OVMXVMSSYS.PAR — the OpenVMX analogue of the VMS VAXVMSSYS.PAR. The cluster daemon reads it at boot.

2.1The identity parameters

Table 2–1 lists the parameters that carry cluster identity, and states honestly which ones the cluster daemon reads at V0.6. Two of them — SCSNODE and SCSSYSTEMID — are the fatal identity pair and must be cluster-wide unique.

2.2Authoring identity the VMS way

You author a node's cluster identity the way a VMScluster administrator expects — by running the cluster configuration procedure, not by editing a file:

$ @SYS$MANAGER:CLUSTER_CONFIG_LAN.COM

                    Cluster Configuration Procedure

        1. ADD this node to a cluster
        2. CHANGE this node's cluster characteristics
        3. REMOVE a node from the cluster
        4. CREATE a duplicate system disk
        5. EXIT from this procedure

Enter choice [1]: 2
    What is the node's SCSNODE name: OVMXA
    What is the node's SCSSYSTEMID: 1030
    ...
    CHANGE complete. The new cluster identity is written to
    SYS$SYSTEM:OVMXVMSSYS.PAR and takes effect on the next reboot.

ADD configures this node as a cluster member (and enables participation); CHANGE reconfigures an existing member. The procedure drives SYSGEN to write a new version of SYS$SYSTEM:OVMXVMSSYS.PAR through the executive Files-11 ACP, the same store the cluster daemon reads its identity from at boot. Reboot to adopt it.

3Standing up a two-node cluster

Forming a cluster at V0.6 is three steps: give each node a unique identity, put both nodes on the same cluster group and LAN segment, and boot them. As the cluster forms, the cluster daemon on each node populates the executive membership block, which you then read with SHOW CLUSTER (Chapter 4).

3.1Give each node a unique identity

On each node, run @SYS$MANAGER:CLUSTER_CONFIG_LAN.COM (Section 2.2) and give it a distinct SCSNODE (six characters or fewer) and a distinct SCSSYSTEMID. Reboot each node to adopt its new identity.

3.2Match the cluster group

OpenVMX joins the reference lab's group 1 by default (CLUSTER_AUTHORIZE is a minimal stand-in — see Appendix B). Both nodes must be on the same LAN segment carrying the LAVC/SCA ethertype: the transport is genuine raw Ethernet, not a UDP tunnel. (Raw-Ethernet access is arranged by the runtime, not something you configure as an operator.)

3.3Boot both nodes

Boot both nodes (see the Installation Guide). As the cluster forms, the cluster daemon on each node populates the executive membership block. Formation takes on the order of a minute. Confirm membership with SHOW CLUSTER, in Chapter 4.

4Reading cluster state

Cluster state is read from the real executive, not from a file or a per-process flag. Every process that reads /dev/vms sees the same member set (INV-6).

4.1SHOW CLUSTER

SHOW CLUSTER reads the executive membership block through /dev/vms, which the cluster daemon populates as members join and depart. When the cluster has formed, it shows the cluster view: a banner naming the system the view is taken from, and a table of the member systems with their software family and status.

$ SHOW CLUSTER
View of Cluster from system ID 1030 node: OVMXA

  SYSTEMS                    MEMBERS
  NODE     SOFTWARE          STATUS
  OVMXA    VMS               MEMBER
  OVMXB    VMS               MEMBER

Node names and system IDs are the ones you set in each node's OVMXVMSSYS.PAR (Chapter 2); the values above are illustrative. A peer whose SCSNODE name has not yet been learned is shown by its SCSSYSTEMID. Peer SOFTWARE shows the family VMS without a version OpenVMX cannot vouch for.

4.2The three outcomes

SHOW CLUSTER gives one of three distinct outcomes, never conflated:

• Members present → the cluster view above.
• Executive reachable, no cluster → the genuine standalone-node answer: %SYSTEM-I-NOTMEMBER, this system is not a member of a VMScluster (status SS$_NORMAL).
• Executive unreachable (no /dev/vms) → %SYSTEM-W-NOSUCHDEV (status SS$_NOSUCHDEV). This is a transport failure, not a cluster fact. On the real runtime /dev/vms is always present.

! a standalone node -- executive reachable, not clustered
$ SHOW CLUSTER
%SYSTEM-I-NOTMEMBER, this system is not a member of a VMScluster

4.3$GETSYI agrees with SHOW CLUSTER

The $GETSYI system service reads the same executive membership block through /dev/vms. SYI$_CLUSTER_MEMBER and SYI$_CLUSTER_NODES agree with SHOW CLUSTER item for item. If the executive is unreachable, the item is left honestly unretrieved rather than answered from a file or a fabricated flag.

5Cross-node locking ($ENQ / DLM)

The ENQ-class distributed lock manager is real and complete on a real /dev/vms executive. Between OpenVMX nodes it does, today, over the SCS wire, everything in Section 5.1. There is one important limitation in how an ordinary application reaches it, in Section 5.2.

5.1What the DLM does between nodes

• Cross-node $ENQ grant on the mastering node, held for the remote requester's CSID.
• Block-then-grant: an incompatible request queues on the real waiting queue and grants on a real $DEQ.
• Blocking AST (BLKAST) delivered over the wire, firing a genuine user-mode AST on the remote holder.
• Lock value block replication both ways, on write and read crossings.
• Mastering and remastering to a survivor on graceful departure, rebuilding lock state from the survivors' real origin records.
• Directory-ownership refusal: a node will not master a resource it is not the directory for.
• Distributed deadlock detection by edge-chasing the real distributed wait-for graph, aborting a single globally-deterministic victim with SS$_DEADLOCK.

All of this is real executive state — resource blocks, granted and waiting queues, origin records. No per-process fake ever answers; absence is always an honest SS$_UNSUPPORTED (INV-6).

5.2How applications reach it

Cross-node locking today is daemon-choreographed and CSID-keyed. The DLM engine and its wire are proven between nodes, but a general application does not yet transparently acquire a remotely-mastered lock the way it would on VMS.

5.3The DLM wire is an OVMX design choice

The semantic field values the DLM carries are authentic $LCKDEF — the grant modes, the $ENQ flags, and the 16-byte lock value block — and the routing algorithm is the documented directory/master resolution. But the on-wire lock-manager SCS opcodes and body byte offsets are an OpenVMX design choice, not VMS-authentic: the vendor does not publish the lock manager's SCS byte layout, so OpenVMX defines its own and labels it as such. They will be replaced with the authentic layout once a real VAX DLM capture grounds it. Every byte-level detail, and the GROUNDED / inferred / unknown labeling of each field, is in the cluster protocol specification cited in Appendix A.

AAppendix A. Clean-room provenance

Everything OpenVMX knows about the VMScluster wire — SCS/NISCA framing, the membership handshake, the DLM message class — is derived only from (a) observing the wire on the project's own reference labs and (b) public VMS documentation (the Cluster Systems manual, $ENQ/$DEQ/$LCKDEF, and documented SDA/SYSGEN/SYSMAN tool output). No vendor VMS source or binary was ever disassembled, decompiled, or consulted.

Where the public documentation does not publish a byte-level layout, OpenVMX defines its own representation and labels it an OVMX design choice — it is never presented as VMS-authentic. This applies to the DLM SCS message opcodes and body offsets (Section 5.3) and to internal identifiers such as connection-ID assignments. If a future capture contradicts anything OpenVMX currently emits, the wire wins and OpenVMX changes.

The full provenance statement, and the GROUNDED / inferred / unknown labeling of every field, is the repository's cluster protocol specification, docs/cluster-protocol-spec.md §0.

BAppendix B. Not yet available at V0.6

A real VMScluster does the following; OpenVMX at V0.6 does not. These are stated plainly so no one designs against a capability that is not there.

B.1Quorum and votes are not enforced

There is no split-brain protection at V0.6. Be precise about why:

• OpenVMX always joins non-voting: the cluster daemon advertises VOTES=0 so it can never affect a VAX cluster's quorum. The local VOTES and EXPECTED_VOTES in your .PAR are not read.
• A quorum model is present and does run — the daemon folds each peer's wire-advertised VOTES into a connection-manager quorum computation and logs quorum PRESENT/LOST. But the gate result is only logged; it is never wired to suspend I/O or reconfigure the cluster. Quorum loss does not block anything.
• EXPECTED_VOTES is an open reverse-engineering gap on the wire (held at 1 in every capture), so the model seeds each peer's expected value from its advertised VOTES rather than reconciling a real value.

Net effect for an operator: do not rely on OpenVMX for quorum arbitration or split-brain avoidance.

B.2MSCP-served volumes — absent

A node cannot serve a local disk to the cluster over MSCP, and cannot mount a volume served by a peer. This is post-0.6 work. (The MSCP disk read/write path exists at the protocol level in the compatibility register, but serving a volume to the cluster as an operator flow is not available.)

B.3Cluster-wide logical names — degrade to system-wide

$CRELNM with LNM$M_CLUSTERWIDE does not replicate a logical-name table across members; the scope degrades to system-wide. Cluster-wide global sections ($MGBLSC/$CRMPSC with cluster scope) are absent.

B.4CLUSTER_AUTHORIZE is a lab-only stand-in

CLUSTER_AUTHORIZE is a minimal OpenVMX stand-in: a tiny typed file holding a group number and a cleartext password, defaulting to the reference lab's group 1 only. There is no real CLUSTER_AUTHORIZE.DAT on-disk format, no credential hashing, and no wire authentication. Joining an arbitrary VMScluster (any group or password) is 1.0 work.

B.5VMS-way authoring — partial

You can now author cluster identity the VMS way and reboot into it, with @SYS$MANAGER:CLUSTER_CONFIG_LAN.COM driving SYSGEN to write a new version of OVMXVMSSYS.PAR over the executive ACP (Section 2.2). What the broader VMS authoring pipeline still lacks at this edition: no SYSMAN PARAMETERS shell for string parameters, no conversational SYSBOOT, no AUTOGEN / MODPARAMS.DAT feedback loop, and the configuration procedure covers only the local node's identity — it does not provision a remote satellite, remove a remote member, or build a duplicate system disk (those items decline honestly). Enforcing the authored VOTES and joining an arbitrary cluster's group (Appendix B.1, B.4) are the remaining 1.0 work.

B.6DECnet — essentially greenfield

Cluster interconnect and application use of DECnet Phase IV is not implemented.

For the current status of every facility, see the compatibility surface.