🌱 Infra/Hadoop_HDFS

[Hadoop 3.3] HDFS μ„œλΉ„μŠ€ 포트 및 μ„€μ • μ΄ν•΄ν•˜κ³  λ„˜μ–΄κ°€κΈ°!

mini_world 2021. 4. 29. 19:29

 

HDFS ν”„λ‘œμ„ΈμŠ€ ν™•μΈν•˜κΈ°

 

1) Master Node

ν˜„μž¬ Master μ„œλ²„μ—μ„œλŠ” NameNode ν”„λ‘œμŠ€μ„Έλ§Œ 운영되고 μžˆμŠ΅λ‹ˆλ‹€.
λ§ˆμŠ€ν„° λ…Έλ“œλŠ” 8020, 9870 μ΄λ ‡κ²Œ λ‘κ°œμ˜ 포트λ₯Ό LISTENμ€‘μž…λ‹ˆλ‹€.

 

2) Worker01 Node (Secondary NameNode + DataNode)

ν˜„μž¬ Worker01 μ„œλ²„μ—μ„œλŠ” 세컨더리 λ„€μž„λ…Έλ“œ ν”„λ‘œμ„ΈμŠ€μ™€, λ°μ΄ν„°λ…Έλ“œ ν”„λ‘œμŠ€μ„Έκ°€ μš΄μ˜μ€‘μž…λ‹ˆλ‹€.

 

3) Worker02 Node (DataNode)

ν˜„μž¬ Worker 02 μ„œλ²„μ—μ„œλŠ” λ°μ΄ν„°λ…Έλ“œ ν”„λ‘œμ„ΈμŠ€ μš΄μ˜μ€‘μž…λ‹ˆλ‹€. 

 

4) Worker03 Node (DataNode)

ν˜„μž¬ Worker 03 μ„œλ²„μ—μ„œλŠ” λ°μ΄ν„°λ…Έλ“œ ν”„λ‘œμ„ΈμŠ€ μš΄μ˜μ€‘μž…λ‹ˆλ‹€. 

 

 

 

HDFS Service Port 

HDFS의 μ„œλΉ„μŠ€ ν¬νŠΈμ™€ νŒŒλΌλ©”ν„°λ₯Ό ν™•μΈν•˜κ² μŠ΅λ‹ˆλ‹€.

ꡬ뢄 μ„œλ²„ 포트 ν”„λ‘œν† μ½œ νŒŒλΌλ©”ν„° μ„€μ •νŒŒμΌ
NameNode Web UI Master 9870 http dfs.http.address hdfs-site.xml
NameNode HDFS API 8020 IPC dfs.defaultFS core-site.xml
Secondary NameNode
Web UI		 Worker01 9869 http dfs.secondary.http.address hdfs-site.xml
DataNode Web UI Worker01~03 9864 http dfs.datanode.http.address hdfs-site.xml
DetaNode Data Transfer 9866 http dfs.datanode.address hdfs-site.xml
DataNode Metadata 
Operation		 9867 IPC dfs.datanode.ipc.address hdfs-site.xml

 

HDFS Service Port  Configureation / Describe

ꡬ뢄 μ„€μ •νŒŒμΌ μ„€μ • 방법 μ„€λͺ…
NameNode Web UI hdfs-site.xml

<property>
    <name>dfs.namenode.http-address</name>
    <value>0.0.0.0:9870</value> 
</property>

 HDFS λ§ˆμŠ€ν„°λ…Έλ“œ μ›ΉUI
NameNode HDFS API core-site.xml

<property>
    <name>fs.defaultFS</name>
    <value>hdfs://master:8020</value>
</property>

 HDFS API ν¬νŠΈ
Secondary
NameNode
Web UI		 hdfs-site.xml

<property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>0.0.0.0:9869</value>
<property>

 HDFS λ³΄μ‘°
DataNode Web UI hdfs-site.xml

<property>
    <name>dfs.datanode.http.address</name>
    <value>0.0.0.0:9864</value>
</property>

 λ°μ΄ν„°λ…Έλ“œ μ›ΉUI
데이터 λ…Έλ“œμ˜ μƒνƒœ, λ‘œκ·Έλ“±μ„ ν™•μΈν•  μˆ˜ μžˆμŒ
DetaNode 
Data Transfer		 hdfs-site.xml

<property>
    <name>dfs.datanode.address</name>
    <value>0.0.0.0:9866</value>
</property>

 데이터 μ „솑 ν¬νŠΈ
DataNode Metadata 
Operation		 hdfs-site.xml

<property>
    <name>dfs.datanode.ipc.address</name>
    <value>0.0.0.0:9867</value>
</property>

 HDFS λ©”타데이터 μš΄μ˜

 

μ €μž‘μžν‘œμ‹œ