{"id":239,"date":"2014-04-02T22:38:49","date_gmt":"2014-04-03T02:38:49","guid":{"rendered":"http:\/\/www.mbeckler.org\/blog\/?p=239"},"modified":"2014-04-02T22:38:49","modified_gmt":"2014-04-03T02:38:49","slug":"report-space-used-in-ftp-directories","status":"publish","type":"post","link":"https:\/\/www.mbeckler.org\/blog\/?p=239","title":{"rendered":"Report space used in FTP directories"},"content":{"rendered":"<p>A friend was looking for a way to list the space usage on a windows server that only had FTP access. I had written something similar for a project long ago, and polished up to do the job.<\/p>\n<p>This python script will walk an FTP directory in a top-down, depth-first pattern. It uses the <a href=\"https:\/\/docs.python.org\/2\/library\/ftplib.html\">ftplib<\/a> library, which I believe is built-in to most or all python distributions. Configure the FTP_* variables near the top to set the server, port, user, password, and the delay between each FTP operation (to avoid hammering the server). The script recursively processes directories, creating a dirStruct tuple that contains the following items:<\/p>\n<pre>(pwd, subdirList, fileList, sizeInFilesHere, sizeTotal)\r\n    pwd is a string like \"\/debian\/dists\/experimental\"\r\n    subdirList is a list of tuples just like this one\r\n    fileList is a list of (filename, sizeInBytes) tuples\r\n    sizeInFilesHere is a sum of all the files in this directory\r\n    sizeTotal is a sum of all the files in this directory and all subdirectories\r\n<\/pre>\n<p>It also writes data to two CSV files:<\/p>\n<ul>\n<li><strong>dirStruct_only_folders.csv<\/strong>\n<ul>\n<li>Contains entries for just the directories.<\/li>\n<li>Local size is the total size of files in that folder (does not count subdirs).<\/li>\n<li>Total size is the sum of local size and total size of all subdirs.<\/li>\n<\/ul>\n<\/li>\n<li><strong>dirStruct_complete.csv<\/strong>\n<ul>\n<li>Contains entries for both files and folders.<\/li>\n<li>Files do not have a total size, only a local size.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<pre>\r\n#!\/usr\/bin\/env python\r\n#\r\n# A script to recursively walk an FTP server directory structure, recording information\r\n# about the file and directory sizes as it traverses the folders.\r\n#\r\n# Stores output in two CSV files:\r\n#  dirStruct_only_folders.csv\r\n#     Contains entries for just the directories.\r\n#     Local size is the total size of files in that folder (does not count subdirs).\r\n#     Total size is the sum of local size and total size of all subdirs.\r\n#  dirStruct_complete.csv\r\n#     Contains entries for both files and folders.\r\n#     Files do not have a total size, only a local size.\r\n#\r\n# Customize the FTP_* variables below.\r\n#\r\n# Basically does a depth-first search.\r\n#\r\n# Written by Matthew L Beckler, matthew at mbeckler dot org.\r\n# Released into the public domain, do whatever you like with this.\r\n# Email me if you like the script or have suggestions to improve it.\r\n\r\nfrom ftplib import FTP\r\nfrom time import sleep\r\n\r\n\r\nFTP_SERVER = \"ftp.debian.org\"\r\nFTP_PORT = \"21\" # 21 is the default\r\nFTP_USER = \"\" # leave empty for anon FTP server\r\nFTP_PASS = \"\"\r\nFTP_DELAY = 1 # how long to wait between calls to the ftp server\r\n\r\ndef parseListLine(line):\r\n   # Files look like          \"-rw-r--r--    1 1176     1176       176158 Mar 30 01:52 README.mirrors.html\"\r\n   # Directories look like    \"drwxr-sr-x   15 1176     1176         4096 Feb 15 09:22 dists\"\r\n   # Returns (name, isDir, sizeBytes)\r\n   items = line.split()\r\n   return (items[8], items[0][0] == \"d\", int(items[4]))\r\n\r\n# Since the silly ftp library makes us use a callback to handle each line of text from the server,\r\n# we have a global lines buffer. Clear the buffer variable before doing each call.\r\nlines = []\r\ndef appendLine(line):\r\n   global lines\r\n   lines.append(line)\r\ndef getListingParsed(ftp):\r\n   \"\"\" This is a sensible interface to the silly line getting system. Returns a copy of the directory listing, parsed. \"\"\"\r\n   global lines\r\n   lines = []\r\n   ftp.dir(appendLine)\r\n   myLines = lines[:]\r\n   parsedLines = map(parseListLine, myLines)\r\n   return parsedLines\r\n   \r\ndef descendDirectories(ftp):\r\n   # Will return a tuple for the current ftp directory, like this:\r\n   # (pwd, subdirList, fileList, sizeInFilesHere, sizeTotal)\r\n   #     pwd is a string like \"\/debian\/dists\/experimental\"\r\n   #     subdirList is a list of tuples just like this one\r\n   #     fileList is a list of (filename, sizeInBytes) tuples\r\n   #     sizeInFilesHere is a sum of all the files in this directory\r\n   #     sizeTotal is a sum of all the files in this directory and all subdirectories\r\n\r\n   sleep(FTP_DELAY) # be a nice client\r\n\r\n   # make our directory structure to return\r\n   pwd = ftp.pwd()\r\n   subdirList = []\r\n   fileList = []\r\n   sizeInFilesHere = 0\r\n   sizeTotal = 0\r\n\r\n   print pwd + \"\/\"\r\n   items = getListingParsed(ftp)\r\n   for name, isDir, sizeBytes in items:\r\n      if not isDir:\r\n         fileList.append( (name, sizeBytes) )\r\n         sizeInFilesHere += sizeBytes\r\n      else:\r\n         # is a directory, so recurse\r\n         ftp.cwd(name)\r\n         struct = descendDirectories(ftp)\r\n         ftp.cwd(\"..\")\r\n         subdirList.append(struct)\r\n         sizeTotal += struct[4]\r\n\r\n   # add in the size of all files here to sizeTotal\r\n   sizeTotal += sizeInFilesHere\r\n   return (pwd, subdirList, fileList, sizeInFilesHere, sizeTotal)\r\n\r\ndef pprintBytes(b):\r\n   \"\"\" Pretty prints a number of bytes with a proper suffix, like K, M, G, T. \"\"\"\r\n   suffixes = [\"\", \"K\", \"M\", \"G\", \"T\", \"?\"]\r\n   ix = 0\r\n   while (b > 1024):\r\n      b \/= 1024.0\r\n      ix += 1\r\n   s = suffixes[min(len(suffixes) - 1, ix)]\r\n   if int(b) == b:\r\n      return \"%d%s\" % (b, s)\r\n   else:\r\n      return \"%.1f%s\" % (b, s)\r\n\r\ndef pprintDirStruct(dirStruct):\r\n   \"\"\" Pretty print the directory structure. RECURSIVE FUNCTION! \"\"\"\r\n   print \"{}\/ ({} in {} files here, {} total)\".format(dirStruct[0], pprintBytes(dirStruct[3]), len(dirStruct[2]), pprintBytes(dirStruct[4]))\r\n   for ds in dirStruct[1]:\r\n      pprintDirStruct(ds)\r\n\r\ndef saveDirStructToCSV(dirStruct, fid, includeFiles):\r\n   \"\"\" Save the directory structure to a CSV file. RECURSIVE FUNCTION! \"\"\"\r\n   # Info about this directory itself\r\n   fid.write(\"\\\"{}\/\\\",{},{}\\n\".format(dirStruct[0], dirStruct[3], dirStruct[4]))\r\n   pwd = dirStruct[0]\r\n\r\n   # Info about files here\r\n   if includeFiles:\r\n      for name, size in dirStruct[2]:\r\n         fid.write(\"\\\"{}\\\",{},\\n\".format(pwd + \"\/\" + name, size))\r\n\r\n   # Info about dirs here, recurse\r\n   for ds in dirStruct[1]:\r\n      saveDirStructToCSV(ds, fid, includeFiles)\r\n\r\nprint \"Connecting to FTP server '%s' port %s...\" % (FTP_SERVER, FTP_PORT)\r\nftp = FTP()\r\nftp.connect(FTP_SERVER, FTP_PORT)\r\nif FTP_USER == \"\":\r\n   ftp.login()\r\nelse:\r\n   ftp.login(FTP_USER, FTP_PASS)\r\n\r\nprint \"Walking directory structure...\"\r\ndirStruct = descendDirectories(ftp)\r\n\r\nprint \"\"\r\nprint \"Finished descending directories, here is the info:\"\r\npprintDirStruct(dirStruct)\r\nprint \"\"\r\n\r\nFILENAME = \"dirStruct_complete.csv\"\r\nprint \"Saving complete directory info (files and folders) to a CSV file: '%s'\" % FILENAME\r\nwith open(FILENAME, \"w\") as fid:\r\n   fid.write(\"\\\"Path\\\",\\\"Local size\\\",\\\"Total size\\\"\\n\")\r\n   saveDirStructToCSV(dirStruct, fid, includeFiles=True)\r\n\r\nFILENAME = \"dirStruct_only_folders.csv\"\r\nprint \"Saving directory info (only folders) to a CSV file: '%s'\" % FILENAME\r\nwith open(FILENAME, \"w\") as fid:\r\n   fid.write(\"\\\"Path\\\",\\\"Local size\\\",\\\"Total size\\\"\\n\")\r\n   saveDirStructToCSV(dirStruct, fid, includeFiles=False)\r\n<\/pre>\n<p>Sample CSV output:<\/p>\n<pre>\r\n\"Path\",\"Local size\",\"Total size\"\r\n\"\/plugins\/\",5426535,7594527\r\n\"\/plugins\/foo-1.1.jar\",7774,\r\n\"\/plugins\/CHANGELOG.txt\",45169,\r\n<\/pre>\n<p>Local size is just the size of the file itself, or the size of all files in a directory. Total size is the total size of the files in a directory plus the total sizes of all subdirectories. Files do not have a total size entry.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A friend was looking for a way to list the space usage on a windows server that only had FTP access. I had written something similar for a project long ago, and polished up to do the job. This python script will walk an FTP directory in a top-down, depth-first pattern. It uses the ftplib [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[1],"tags":[86,84,16,85],"class_list":["post-239","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-clients","tag-ftp","tag-python","tag-servers"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p2BznB-3R","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.mbeckler.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/239","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.mbeckler.org\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.mbeckler.org\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.mbeckler.org\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.mbeckler.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=239"}],"version-history":[{"count":6,"href":"https:\/\/www.mbeckler.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/239\/revisions"}],"predecessor-version":[{"id":246,"href":"https:\/\/www.mbeckler.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/239\/revisions\/246"}],"wp:attachment":[{"href":"https:\/\/www.mbeckler.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=239"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.mbeckler.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=239"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.mbeckler.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=239"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}