HDFS File System Commands - Examples

 
 
Here are some of the examples of performing various file and directory operations in the HDFS file system. 
Commands File : files/kir-hdfs-commands.txt
 
 
=> Create New directory
    The following command creates new directory in the HDFS file system
 
   hduser@kirlin>hdfs dfs -mkdir /kiran
 
=> List the contents of the directory
 
   hduser@kirlin>hdfs dfs -ls /
   Found 1 items
   drwxr-xr-x   - hduser supergroup          0 2014-02-20 18:21 /kiran
=> Copy the files from local file system to HDFS
   In this example, files are copied from the local file system /tmp directory to the HDFS /kiran directory
 
   $ls -lart /tmp/*.txt
   -rw-r--r--. 1 oracle dba 22 Feb 20 18:26 /tmp/names.txt
   -rw-r--r--. 1 oracle dba 51 Feb 20 18:27 /tmp/addr.txt
 
   hduser@kirlin>hdfs dfs -put /tmp/*.txt /kiran/.
 
   hduser@kirlin>hdfs dfs -ls /kiran
   Found 2 items
   -rw-r--r--   3 hduser supergroup         51 2014-02-20 18:29 /kiran/addr.txt
   -rw-r--r--   3 hduser supergroup         22 2014-02-20 18:29 /kiran/names.txt
   hduser@kirlin>
 
 
=> Display the HDFS file contents
 
   hduser@kirlin>hdfs dfs -cat /kiran/*.txt
   1 richmond, USA
   2 hyderabad, INDIA
   3 mumbai, INDIA
   1 kiran
   2 raj
   3 radhi
   hduser@kirlin>
 
 
=> Change HDFS Files' owner, group and permissions
 
   hduser@kirlin>hdfs dfs -chgrp dba /kiran/*.txt
 
   hduser@kirlin>hdfs dfs -chown kiran /kiran/*.txt
 
   hduser@kirlin>hdfs dfs -chmod 750 /kiran/*.txt
 
   hduser@kirlin>hdfs dfs -ls /kiran
   Found 2 items
   -rwxr-x---   3 kiran dba         51 2014-02-20 18:29 /kiran/addr.txt
   -rwxr-x---   3 kiran dba         22 2014-02-20 18:29 /kiran/names.txt
   hduser@kirlin>
 
 
=> create and rename a file
 
   hduser@kirlin>hdfs dfs -touchz /kiran/archive
 
   hduser@kirlin>hdfs dfs -ls /kiran
   Found 3 items
   -rwxr-x---   3 kiran  dba                51 2014-02-20 18:29 /kiran/addr.txt
   -rw-r--r--   3 hduser supergroup          0 2014-02-20 18:56 /kiran/archive
   -rwxr-x---   3 kiran  dba                22 2014-02-20 18:29 /kiran/names.txt
 
 
   hduser@kirlin>hdfs dfs -mv /kiran/archive /kiran/1.txt
   hduser@kirlin>hdfs dfs -ls /kiran
   Found 3 items
   -rw-r--r--   3 hduser supergroup          0 2014-02-20 18:56 /kiran/1.txt
   -rwxr-x---   3 kiran  dba                51 2014-02-20 18:29 /kiran/addr.txt
   -rwxr-x---   3 kiran  dba                22 2014-02-20 18:29 /kiran/names.txt
   hduser@kirlin>
 
 
 
==> Testing if it is a hdfs file or a hdfs directory from a shell script
 
 
   hduser@kirlin>
   hduser@kirlin>TST=/kiran/addr.txt
   hduser@kirlin>if hdfs dfs -test -e $TST; then
   > echo $TST "is  a file"
   >  elif hdfs dfs -test -d  $TST; then
   > echo $TST "is  a directory"
   > fi
   /kiran/addr.txt is  a file
   hduser@kirlin>
 
 
==> Do the recursive delete for the entire directory
 
   hduser@kirlin>hdfs dfs -rm -r /kiran
   Deleted /kiran
   hduser@kirlin>hdfs dfs -ls /
   Found 1 items
   -rw-r--r--   3 hduser supergroup          0 2014-02-20 18:56 /tmp
   hduser@kirlin>
 
 
 
==> merge files..
    In this example, contents of hdfs file are merged to file on the local file system
 
   $ cat /tmp/names.txt
   1 kiran
   2 raj
   $
 
 
   hduser@kirlin>hdfs dfs -cat /kiran/names.txt
   1 kiran
   2 raj
   3 radhi
   hduser@kirlin
 
   hduser@kirlin>hdfs dfs -getmerge /kiran/names.txt /tmp/names.txt
   hduser@kirlin>
 
   hduser@kirlin>hdfs dfs -cat /kiran/names.txt
   1 kiran
   2 raj
   3 radhi
 
   $ cat /tmp/names.txt
   1 kiran
   2 raj
   3 radhi
 
For complete command reference, please visit the Apache Hadoop Page
 
 
 
 

Cheers!

rajkiran

 

 

Copyright ©2018 Rajkiran Ghanta (deegeplanet.com)