2.5 Start hdfs
2.5.1 configuration files
- hdfs-site.xml
- core-site.xml
- mapred-site.xml
- yarn-site.xml
2.5.2 modify essential configuration files
etc/hadoop/conf/yarn-site.xml
etc/hadoop/conf/core-site.xml
etc/hadoop/conf/hdfs-site.xml
2.5.3 format namenode
This is for only the first time starting hdfs on the namenode
2.5.4 jps: Java virtual machine (JVM) Process Status
check jps and use jps to monitor the starting procedure to ensure all required services are up and running
jps location: JVM under bin
2.5.5 start namenode
- sbin/hadoop-daemon.sh start namenode
- sbin/hadoop-daemon.sh start datanode
- sbin/yarn-daemon.sh start resourcemanager
- sbin/yarn-daemon.sh start nodemanager
- sbin/mr-jobhistory-daemon.sh start historyserver
go to sbin folder of hadoop home, run start-dfs.sh start namenode, and use jps to check the status
2.5.6 start datanode
2.5.7 start Resource Manager
2.5.8 start Node Manager
2.5.9 start Job History Server
2.5.10 check HDFS status
2.5.11 check yarn status
2.5.12 list hdfs files
2.5.13 view hdfs files
2.5.14 create hdfs directory
hdfs dfs -mkdir hdfs://localhost:9000/Data
delete hdfs directory
method 1: hdfs dfs -rm -r hdfs://localhost:9000/Data1
method 2: hadoop fs -rm -r hdfs://localhost:9000/Data1
2.5.15 load data to hdfs
hdfs dfs -put ./opt/hadoop/pig-0.15.0/test/student_data.txt hdfs://localhost:9000/Data