HDFS HA using Journal Nodes

04/11/2023 Copyright 2013 Trend Micro Inc.

Introducing Journal Nodes

Manual

Failover

04/11/2023 Copyright 2013 Trend Micro Inc.

ArchitectureJN1

JN2

JN3

NNActive

NNStandby

DN DNDNDN

Block locations map

04/11/2023 Copyright 2013 Trend Micro Inc.

• When any namespace modification is performed it durably logs a record of the modification to JNs

• The Standby reads the edits from the JNs and applies them to its own namespace

JournalNodes’ job

JN1

JN2

JN3

NNActive

NNStandby

Edits Edits Edits

Edits Edits EditsEdits Edits Edits

SafeMode

04/11/2023 Copyright 2013 Trend Micro Inc.

• Specify path on local disk

• tolerate at most (N - 1) / 2 failures 

JournalNodes’ storage

04/11/2023 Copyright 2013 Trend Micro Inc.

• JournalNodes will only allow a single NameNode to be a writer at a time.

• no potential for corrupting the file system metadata from a split-brain scenario.

JournalNodes’ fencing 

JN1

JN2

JN3

NNActive

NNStandby

WRITE READ

04/11/2023 Copyright 2013 Trend Micro Inc.

• Whenever a NameNode becomes active, it first generate an epoch number.

• first active NameNode after the namespace is initialized starts with epoch number 1

• any failovers or restarts result in an increment of the epoch number

JournalNodes’ fencing 

04/11/2023 Copyright 2013 Trend Micro Inc.

• When a new NameNode becomes active, it has an epoch number higher than any previous NameNode

• Call JournalNodes to increment their promised epochs

• Fencing: – JNs receive newer epoch

update majority of JNs’ promised epochs accept– JNs receive older epoch

reject

JournalNodes’ fencing 

04/11/2023 Copyright 2013 Trend Micro Inc.

• previous Active NameNode could serve read requests to clients which may be out of date until a write access performed

• You can specify some fencing method to avoid this happened

But…

04/11/2023 Copyright 2013 Trend Micro Inc.

Fencing

Method

04/11/2023 Copyright 2013 Trend Micro Inc.

• sshfenceSSH to the Active NameNode and kill the process

Fencing Method

04/11/2023 Copyright 2013 Trend Micro Inc.

• shellrun a shell command to fence the Active NameNode

• The script may have properties with the '_' character replacing any '.' ex : dfs_namenode_rpc-address 

Fencing Method

04/11/2023 Copyright 2013 Trend Micro Inc.

• Additional environment variable

Fencing Method

04/11/2023 Copyright 2013 Trend Micro Inc.

Automatic

Failover

04/11/2023 Copyright 2013 Trend Micro Inc.

JN1

JN2

JN3

04/11/2023 Copyright 2013 Trend Micro Inc.

• Health monitoring– the ZKFC pings its local NameNode on a periodic basis with a

health-check command. (healthy/unhealthy)

• ZooKeeper session management– when the local NameNode is healthy, the ZKFC holds a session

open in ZooKeeper. – If the local NameNode is active, it also holds a special "lock"

znode. – if the session expires, the lock node will be automatically

deleted.

ZKFailoverController 

04/11/2023 Copyright 2013 Trend Micro Inc.

• ZooKeeper-based election– if the local NameNode is healthy, and no other node currently

holds the lock znode, it will itself try to acquire the lock. – If it succeeds, then it has "won the election“

• Failover– the previous active is fenced– local NameNode transitions to active state.

ZKFailoverController 

04/11/2023 Copyright 2013 Trend Micro Inc.

JN1

JN2

JN3

NNActive

1

2

3

4

5 6

7

04/11/2023 Copyright 2013 Trend Micro Inc.

Client

Side

04/11/2023 Copyright 2013 Trend Micro Inc.

• Client connect to Active Namenode via proxy

• When Active Namenode down, client receive Exception retry and send RPC to another namenode (implement by ConfiguredFailoverProxyProvider)

Client Failover

04/11/2023 Copyright 2013 Trend Micro Inc.

Steps to Apply

HDFS HA

04/11/2023 Copyright 2013 Trend Micro Inc.

• If setting up a fresh HDFS cluster,hdfs namenode –format

• copy over the contents of your NameNode metadata directories to the otherhdfs namenode –bootstrapStandby./format-failover-namenode.sh

• hdfs –initializeSharedEdits to initialize edits log in journalnode

• Startup both Namenode

converting a non-HA-enabled cluster to be HA-enabled

04/11/2023 Copyright 2013 Trend Micro Inc.

• http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithQJM.html#Deployment

• http://blog.cloudera.com/blog/2012/10/quorum-based-journaling-in-cdh4-1/

• https://issues.apache.org/jira/secure/attachment/12547598/qjournal-design.pdf

Reference