How to fix "No space left on device" error? df -h,df -i all ok, thanks

flyaga · 2015-11-05 09:21:53

When I run a python script to read a big file and write to many little files(about thousand files), most times it can work ok, but sometimes it showes "No space left on device" error, I had used df -h and df -i command to check system setting, they all are ok, please see below results:

df -lh
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 97G 47G 46G 51% /
tmpfs 32G 296K 32G 1% /dev/shm
/dev/sda2 985M 43M 892M 5% /boot
/dev/sda1 200M 256K 200M 1% /boot/efi
/dev/sda4 97G 225M 92G 1% /home
/dev/bigpfs 64T 35T 29T 55% /gpfs01

df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda3 6406144 1986724 4419420 32% /
tmpfs 8243694 9 8243685 1% /dev/shm
/dev/sda2 64000 22 63978 1% /boot
/dev/sda1 0 0 0 - /boot/efi
/dev/sda4 6406144 492 6405652 1% /home
/dev/bigpfs 70714880 1011439 669703441 1% /gpfs01

I had googled many pages, but no idea, how to fix it???

Awebb · 2015-11-05 09:44:45

1. [ code] tags for code will help us read your outputs.
2. I'd say your python code is broken.

jasonwryan · 2015-11-05 17:10:43

Not a Sysadmin issue, moving to Newbie Corner...

pypi · 2015-11-05 18:14:59

If you are using btrfs, see the FAQs.

flyaga · 2015-11-06 02:32:39

The file system isn't btrfs, below is my file system infotmation
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda3 ext4 97G 46G 46G 51% /
tmpfs tmpfs 32G 296K 32G 1% /dev/shm
/dev/sda2 ext4 985M 43M 892M 5% /boot
/dev/sda1 vfat 200M 256K 200M 1% /boot/efi
/dev/sda4 ext4 97G 225M 92G 1% /home
/dev/bigpfs gpfs 64T 36T 29T 56% /gpfs01

Rethil · 2015-11-06 04:32:48

Better post your script.

Awebb · 2015-11-06 07:24:57

We can't help you if you don't start providing useful information.

flyaga · 2015-11-10 02:52:02

Sorry for delay reply. Below is my code

import errno,shutil,re,datetime,os,time,sys,gzip,mypylib

#get yyyymm from time
#get 201509 from 2015-09-11 12:12:12
def GetYYYYMMFromTimeString(sTime):
	if len(sTime)!=19:
		return ''
	return sTime[0:4]+sTime[5:7]
	
#get wt part from wt_number
#get 00 from WT05500
def GetWindPartFromWTNumber(wtNumber):
	if len(wtNumber)!=7:
		return ''
	return wtNumber[5:7]

startTime=datetime.datetime.now()
returnCode=0
	
#open allinone file
type=sys.argv[1]

#/gpfs01/sec-temp/other/downloaded/thread-1.txt
sourceFile=sys.argv[2]

#/gpfs01/secdex/
destFilePath=sys.argv[3]


#base source file path
#thread-1.txt
baseSourceFileName=os.path.basename(sourceFile)

#source file path
sourceFilePath=os.path.dirname(sourceFile)

#hbase file path
#/gpfs01/sec-temp/other/hbase
hbaseFilePath=''
if len(sys.argv)>4:
	hbaseFilePath=sys.argv[4]

#dest file format
destFileFormat='text'
if len(sys.argv)>5:
	destFileFormat=sys.argv[5]
	
#public hbase file name
#/gpfs01/sec-temp/other/hbase/pub-thread-1.txt
hbaseFileNamePublic=''
if hbaseFilePath!='':
	hbaseFileNamePublic=sourceFilePath+os.path.sep+'pub-'+baseSourceFileName
	hbaseFileNamePublicNew=hbaseFilePath+os.path.sep+'pub-'+baseSourceFileName

#self hbase file name 
#/gpfs01/sec-temp/other/hbase/self-thread-1.txt
hbaseFileNameSelf=''
if hbaseFilePath!='':
	hbaseFileNameSelf=sourceFilePath+os.path.sep+'self-'+baseSourceFileName
	hbaseFileNameSelfNew=hbaseFilePath+os.path.sep+'self-'+baseSourceFileName

iTimeNo=-1
iItemNo=-1
if type=='other':
	iTimeNo=2
	iItemNo=1
	iDataSourceNo=0
	publicFieldCount=25
	publicFieldStartNo=3
	hbaseFileNameSelf=''

if type=='all':
	iTimeNo=3
	iItemNo=1
	iDataSourceNo=0
	iDeviceTypeNumberNo=2
	iSwitchValueFieldsNo=4
	publicFieldCount=100
	publicFieldStartNo=5
	selfFieldCount=500
	selfFieldStartNo=105

try:
	input=open(sourceFile,'r')
except Exception,e:
	print "Error, can't open source file %s, reason="%(sourceFile)
	print "Error '%s' happened on line %d" % (sys.exc_info()[1],sys.exc_info()[2].tb_lineno)
	exit(1)

destFileHandleDict={}
iLineNo=0
subLine=''
hbaseFileHandlePublic=None
hbaseFileHandlePublicSelf=None

#read line from file
try:
	while 1==1:
		subLine=input.readline()
		if len(subLine)==0:
			break

		if iLineNo==0:
			if hbaseFileNamePublic!='':
				hbaseFileHandlePublic=open(hbaseFileNamePublic,'w+')

			if hbaseFileNameSelf!='':
				hbaseFileHandlePublicSelf=open(hbaseFileNameSelf,'w+')

		iLineNo=iLineNo+1		
		subLine=subLine.strip()

#get wt_number and time
		fieldArray=subLine.split(',')
		if len(fieldArray)<=max(iItemNo,iTimeNo):
			print "Warning, No.%d line field count is less=%s"%(iLineNo,subLine)
			continue
		itemNo=fieldArray[iItemNo]
		sTime=fieldArray[iTimeNo]
		
		windPart=GetWindPartFromWTNumber(itemNo)
		yyyymm=GetYYYYMMFromTimeString(sTime)

#output to dest file path per wt_number using text or gzip compress
		#get location
		location=type
		if type!='other':
			location='all'
		destFileNamePath=destFilePath+os.path.sep+yyyymm+os.path.sep+windPart+os.path.sep
		destFileName=destFileNamePath+itemNo+"-"+location+"5sr"
		if destFileFormat=="gz":
			destFileName=destFileName+".gz"

		if not os.path.exists(destFileNamePath):
			try:
				os.makedirs(destFileNamePath)
			except OSError as exc:
				if exc.errno == errno.EEXIST and os.path.isdir(path):
					pass
				else: 
					raise

		#if iLineNo>0:
		#	print destFileName
		#	break
		if iLineNo%10000==1:
			print 'Processing No.%d,%s'%(iLineNo,destFileName)		

		if not (destFileName in destFileHandleDict.keys()):
			#print 'opening %s'%(destFileName)
			if destFileFormat=="gz":
				destFileHandleDict[destFileName]=gzip.open(destFileName,'ab')
			if destFileFormat=="text":
				destFileHandleDict[destFileName]=open(destFileName,'a+')

		destFileHandleDict[destFileName].write(subLine+'\n')

		#write to hbase format file
		if hbaseFileNamePublic!='':
			sTime1=re.sub('[\s\-\:]','',sTime)
			rowKey=('%s_%s')%(itemNo[::-1],sTime1)
			dataSource=fieldArray[iDataSourceNo]

			if type=='other':
				line='%s#%s'%(rowKey,dataSource)
				for i in range(0, publicFieldCount-1 + 1):
					line='%s#%s'%(line,fieldArray[publicFieldStartNo+i])
				hbaseFileHandlePublic.write(line+'\n')
				#print line

			if type=='all':
				line='%s#%s#%s'%(rowKey,dataSource,fieldArray[iDeviceTypeNumberNo])
				for i in range(0, publicFieldCount-1 + 1):
					try:
						sTemp=fieldArray[publicFieldStartNo+i]
					except:
						sTemp=''

					try:
						sTemp1=fieldArray[publicFieldStartNo+i+1]
					except:
						sTemp1=''

					if (sTemp=='') and (sTemp1==''):
						break
					line='%s#%s'%(line,sTemp)
				hbaseFileHandlePublic.write(line+'\n')
				#print 'pub='+line
			
		if hbaseFileNameSelf!='':
			sTime1=re.sub('[\s\-\:]','',sTime)
			rowKey=('%s_%s')%(itemNo[::-1],sTime1)
			dataSource=fieldArray[iDataSourceNo]
			
			line='%s#%s#%s#%s#'%(rowKey,dataSource,fieldArray[iDeviceTypeNumberNo],fieldArray[iSwitchValueFieldsNo])
			sTemp=''
			for i in range(0, selfFieldCount-1 + 1):
				if sTemp=='':
					sTemp=fieldArray[selfFieldStartNo+i]
				else:
					sTemp='%s,%s'%(sTemp,fieldArray[selfFieldStartNo+i])
			hbaseFileHandlePublicSelf.write(line+sTemp+'\n')
			#print 'self='+line
			
except Exception, e:
	print "Error, No.%d line format error=%s reason="%(iLineNo,subLine)
	print "Error '%s' happened on line %d" % (sys.exc_info()[1],sys.exc_info()[2].tb_lineno)
	returnCode=1
#close
finally:
	totalSeconds=mypylib.GetTimeChaSeconds(startTime,datetime.datetime.now())
	print 'alldone=%s,%d lines,%d s,%s'%(sourceFile,iLineNo,totalSeconds,time.strftime('%Y-%m-%d %H:%M:%S', time.localtime()))

	iNo=0
	for item in destFileHandleDict:
		try:
			iNo=iNo+1
			destFileHandleDict[item].close()
			if iNo%100==1:
				print 'close No.%d dest file %s ok'%(iNo,item)
		except Exception, e:
			print "Error, close No.%d file %s error="%(iNo,item)
			print "Error '%s' happened on line %d" % (sys.exc_info()[1],sys.exc_info()[2].tb_lineno)
			returnCode=1

	if hbaseFileHandlePublic!=None:
		try:
			hbaseFileHandlePublic.close()
			print 'close hbase public %s ok'%(hbaseFileNamePublic)
			shutil.move(hbaseFileNamePublic,hbaseFileNamePublicNew)
			print 'move hbase public to %s ok'%(hbaseFileNamePublicNew)
		except Exception, e:
			print "Error, close or move hbase public file %s error="%(hbaseFileNamePublic)
			print "Error '%s' happened on line %d" % (sys.exc_info()[1],sys.exc_info()[2].tb_lineno)
			returnCode=1

	if hbaseFileHandlePublicSelf!=None:
		try:
			hbaseFileHandlePublicSelf.close()
			print 'close hbase self %s ok'%(hbaseFileNameSelf)
			shutil.move(hbaseFileNameSelf,hbaseFileNameSelfNew)
			print 'move hbase self to %s ok'%(hbaseFileNameSelfNew)
		except Exception, e:
			print "Error, close or move hbase self file %s error="%(hbaseFileNameSelf)
			print "Error '%s' happened on line %d" % (sys.exc_info()[1],sys.exc_info()[2].tb_lineno)
			returnCode=1

	try:
		input.close()
		print 'close source file %s ok'%(sourceFile)
	except Exception, e:
		print "Error, close source file %s error="%(sourceFile)
		print "Error '%s' happened on line %d" % (sys.exc_info()[1],sys.exc_info()[2].tb_lineno)
		returnCode=1

	totalSeconds=mypylib.GetTimeChaSeconds(startTime,datetime.datetime.now())
	print 'allclose=%d s,%s'%(totalSeconds,time.strftime('%Y-%m-%d %H:%M:%S', time.localtime()))
	exit(returnCode)

Last edited by flyaga (2015-11-10 04:22:44)

pypi · 2015-11-10 04:17:56

Please consider using code tags!

jasonwryan · 2015-11-10 04:19:40

That's the second, and last, time OP has been asked to paste using code tags:
https://wiki.archlinux.org/index.php/Fo … s_and_code

flyaga · 2015-11-10 04:23:50

Ok, it is ok, jasonwryan and pypi, thanks!

pypi · 2015-11-10 04:49:24

Hmm, so you are writing to a path under a gpfs filesystem?
Without checking whether or not the python script is indeed "broken", a quick google suggested that GPFS can have quotas; something I hadn't thought of. Have you checked whether you are exceeding any quota set up?
Also, google suggested mmdf for gpfs filesystems; have you tried that?

flyaga · 2015-11-10 05:00:12

>Hmm, so you are writing to a path under a gpfs filesystem?
Yes

>Without checking whether or not the python script is indeed "broken", a quick google suggested that GPFS can have quotas; something I hadn't thought of. Have you checked whether you are exceeding any quota set up?
>Also, google suggested mmdf for gpfs filesystems; have you tried that?
Sorry, I don't, I haven't any knowledge of gpfs quotas, how to do it?

pypi · 2015-11-10 05:21:17

flyaga wrote:

>Without checking whether or not the python script is indeed "broken", a quick google suggested that GPFS can have quotas; something I hadn't thought of. Have you checked whether you are exceeding any quota set up?
>Also, google suggested mmdf for gpfs filesystems; have you tried that?
Sorry, I don't, I haven't any knowledge of gpfs quotas, how to do it?

5 minutes of googling will tell you as much as it did me. Or using the links in the quoted post. Or both.

flyaga · 2015-11-10 05:24:57

pypi, thanks!

Arch Linux

#1 2015-11-05 09:21:53

How to fix "No space left on device" error? df -h,df -i all ok, thanks

#2 2015-11-05 09:44:45

Re: How to fix "No space left on device" error? df -h,df -i all ok, thanks

#3 2015-11-05 17:10:43

Re: How to fix "No space left on device" error? df -h,df -i all ok, thanks

#4 2015-11-05 18:14:59

Re: How to fix "No space left on device" error? df -h,df -i all ok, thanks

#5 2015-11-06 02:32:39

Re: How to fix "No space left on device" error? df -h,df -i all ok, thanks

#6 2015-11-06 04:32:48

Re: How to fix "No space left on device" error? df -h,df -i all ok, thanks

#7 2015-11-06 07:24:57

Re: How to fix "No space left on device" error? df -h,df -i all ok, thanks

#8 2015-11-10 02:52:02

Re: How to fix "No space left on device" error? df -h,df -i all ok, thanks

#9 2015-11-10 04:17:56

Re: How to fix "No space left on device" error? df -h,df -i all ok, thanks

#10 2015-11-10 04:19:40

Re: How to fix "No space left on device" error? df -h,df -i all ok, thanks

#11 2015-11-10 04:23:50

Re: How to fix "No space left on device" error? df -h,df -i all ok, thanks

#12 2015-11-10 04:49:24

Re: How to fix "No space left on device" error? df -h,df -i all ok, thanks

#13 2015-11-10 05:00:12

Re: How to fix "No space left on device" error? df -h,df -i all ok, thanks

#14 2015-11-10 05:21:17

Re: How to fix "No space left on device" error? df -h,df -i all ok, thanks

#15 2015-11-10 05:24:57

Re: How to fix "No space left on device" error? df -h,df -i all ok, thanks

Board footer