You are not logged in.
When I run a python script to read a big file and write to many little files(about thousand files), most times it can work ok, but sometimes it showes "No space left on device" error, I had used df -h and df -i command to check system setting, they all are ok, please see below results:
df -lh
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 97G 47G 46G 51% /
tmpfs 32G 296K 32G 1% /dev/shm
/dev/sda2 985M 43M 892M 5% /boot
/dev/sda1 200M 256K 200M 1% /boot/efi
/dev/sda4 97G 225M 92G 1% /home
/dev/bigpfs 64T 35T 29T 55% /gpfs01
df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda3 6406144 1986724 4419420 32% /
tmpfs 8243694 9 8243685 1% /dev/shm
/dev/sda2 64000 22 63978 1% /boot
/dev/sda1 0 0 0 - /boot/efi
/dev/sda4 6406144 492 6405652 1% /home
/dev/bigpfs 70714880 1011439 669703441 1% /gpfs01
I had googled many pages, but no idea, how to fix it???
Offline
1. [ code] tags for code will help us read your outputs.
2. I'd say your python code is broken.
Offline
Not a Sysadmin issue, moving to Newbie Corner...
Offline
If you are using btrfs, see the FAQs.
Offline
The file system isn't btrfs, below is my file system infotmation
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda3 ext4 97G 46G 46G 51% /
tmpfs tmpfs 32G 296K 32G 1% /dev/shm
/dev/sda2 ext4 985M 43M 892M 5% /boot
/dev/sda1 vfat 200M 256K 200M 1% /boot/efi
/dev/sda4 ext4 97G 225M 92G 1% /home
/dev/bigpfs gpfs 64T 36T 29T 56% /gpfs01
Offline
Better post your script.
Offline
We can't help you if you don't start providing useful information.
Offline
Sorry for delay reply. Below is my code
import errno,shutil,re,datetime,os,time,sys,gzip,mypylib
#get yyyymm from time
#get 201509 from 2015-09-11 12:12:12
def GetYYYYMMFromTimeString(sTime):
if len(sTime)!=19:
return ''
return sTime[0:4]+sTime[5:7]
#get wt part from wt_number
#get 00 from WT05500
def GetWindPartFromWTNumber(wtNumber):
if len(wtNumber)!=7:
return ''
return wtNumber[5:7]
startTime=datetime.datetime.now()
returnCode=0
#open allinone file
type=sys.argv[1]
#/gpfs01/sec-temp/other/downloaded/thread-1.txt
sourceFile=sys.argv[2]
#/gpfs01/secdex/
destFilePath=sys.argv[3]
#base source file path
#thread-1.txt
baseSourceFileName=os.path.basename(sourceFile)
#source file path
sourceFilePath=os.path.dirname(sourceFile)
#hbase file path
#/gpfs01/sec-temp/other/hbase
hbaseFilePath=''
if len(sys.argv)>4:
hbaseFilePath=sys.argv[4]
#dest file format
destFileFormat='text'
if len(sys.argv)>5:
destFileFormat=sys.argv[5]
#public hbase file name
#/gpfs01/sec-temp/other/hbase/pub-thread-1.txt
hbaseFileNamePublic=''
if hbaseFilePath!='':
hbaseFileNamePublic=sourceFilePath+os.path.sep+'pub-'+baseSourceFileName
hbaseFileNamePublicNew=hbaseFilePath+os.path.sep+'pub-'+baseSourceFileName
#self hbase file name
#/gpfs01/sec-temp/other/hbase/self-thread-1.txt
hbaseFileNameSelf=''
if hbaseFilePath!='':
hbaseFileNameSelf=sourceFilePath+os.path.sep+'self-'+baseSourceFileName
hbaseFileNameSelfNew=hbaseFilePath+os.path.sep+'self-'+baseSourceFileName
iTimeNo=-1
iItemNo=-1
if type=='other':
iTimeNo=2
iItemNo=1
iDataSourceNo=0
publicFieldCount=25
publicFieldStartNo=3
hbaseFileNameSelf=''
if type=='all':
iTimeNo=3
iItemNo=1
iDataSourceNo=0
iDeviceTypeNumberNo=2
iSwitchValueFieldsNo=4
publicFieldCount=100
publicFieldStartNo=5
selfFieldCount=500
selfFieldStartNo=105
try:
input=open(sourceFile,'r')
except Exception,e:
print "Error, can't open source file %s, reason="%(sourceFile)
print "Error '%s' happened on line %d" % (sys.exc_info()[1],sys.exc_info()[2].tb_lineno)
exit(1)
destFileHandleDict={}
iLineNo=0
subLine=''
hbaseFileHandlePublic=None
hbaseFileHandlePublicSelf=None
#read line from file
try:
while 1==1:
subLine=input.readline()
if len(subLine)==0:
break
if iLineNo==0:
if hbaseFileNamePublic!='':
hbaseFileHandlePublic=open(hbaseFileNamePublic,'w+')
if hbaseFileNameSelf!='':
hbaseFileHandlePublicSelf=open(hbaseFileNameSelf,'w+')
iLineNo=iLineNo+1
subLine=subLine.strip()
#get wt_number and time
fieldArray=subLine.split(',')
if len(fieldArray)<=max(iItemNo,iTimeNo):
print "Warning, No.%d line field count is less=%s"%(iLineNo,subLine)
continue
itemNo=fieldArray[iItemNo]
sTime=fieldArray[iTimeNo]
windPart=GetWindPartFromWTNumber(itemNo)
yyyymm=GetYYYYMMFromTimeString(sTime)
#output to dest file path per wt_number using text or gzip compress
#get location
location=type
if type!='other':
location='all'
destFileNamePath=destFilePath+os.path.sep+yyyymm+os.path.sep+windPart+os.path.sep
destFileName=destFileNamePath+itemNo+"-"+location+"5sr"
if destFileFormat=="gz":
destFileName=destFileName+".gz"
if not os.path.exists(destFileNamePath):
try:
os.makedirs(destFileNamePath)
except OSError as exc:
if exc.errno == errno.EEXIST and os.path.isdir(path):
pass
else:
raise
#if iLineNo>0:
# print destFileName
# break
if iLineNo%10000==1:
print 'Processing No.%d,%s'%(iLineNo,destFileName)
if not (destFileName in destFileHandleDict.keys()):
#print 'opening %s'%(destFileName)
if destFileFormat=="gz":
destFileHandleDict[destFileName]=gzip.open(destFileName,'ab')
if destFileFormat=="text":
destFileHandleDict[destFileName]=open(destFileName,'a+')
destFileHandleDict[destFileName].write(subLine+'\n')
#write to hbase format file
if hbaseFileNamePublic!='':
sTime1=re.sub('[\s\-\:]','',sTime)
rowKey=('%s_%s')%(itemNo[::-1],sTime1)
dataSource=fieldArray[iDataSourceNo]
if type=='other':
line='%s#%s'%(rowKey,dataSource)
for i in range(0, publicFieldCount-1 + 1):
line='%s#%s'%(line,fieldArray[publicFieldStartNo+i])
hbaseFileHandlePublic.write(line+'\n')
#print line
if type=='all':
line='%s#%s#%s'%(rowKey,dataSource,fieldArray[iDeviceTypeNumberNo])
for i in range(0, publicFieldCount-1 + 1):
try:
sTemp=fieldArray[publicFieldStartNo+i]
except:
sTemp=''
try:
sTemp1=fieldArray[publicFieldStartNo+i+1]
except:
sTemp1=''
if (sTemp=='') and (sTemp1==''):
break
line='%s#%s'%(line,sTemp)
hbaseFileHandlePublic.write(line+'\n')
#print 'pub='+line
if hbaseFileNameSelf!='':
sTime1=re.sub('[\s\-\:]','',sTime)
rowKey=('%s_%s')%(itemNo[::-1],sTime1)
dataSource=fieldArray[iDataSourceNo]
line='%s#%s#%s#%s#'%(rowKey,dataSource,fieldArray[iDeviceTypeNumberNo],fieldArray[iSwitchValueFieldsNo])
sTemp=''
for i in range(0, selfFieldCount-1 + 1):
if sTemp=='':
sTemp=fieldArray[selfFieldStartNo+i]
else:
sTemp='%s,%s'%(sTemp,fieldArray[selfFieldStartNo+i])
hbaseFileHandlePublicSelf.write(line+sTemp+'\n')
#print 'self='+line
except Exception, e:
print "Error, No.%d line format error=%s reason="%(iLineNo,subLine)
print "Error '%s' happened on line %d" % (sys.exc_info()[1],sys.exc_info()[2].tb_lineno)
returnCode=1
#close
finally:
totalSeconds=mypylib.GetTimeChaSeconds(startTime,datetime.datetime.now())
print 'alldone=%s,%d lines,%d s,%s'%(sourceFile,iLineNo,totalSeconds,time.strftime('%Y-%m-%d %H:%M:%S', time.localtime()))
iNo=0
for item in destFileHandleDict:
try:
iNo=iNo+1
destFileHandleDict[item].close()
if iNo%100==1:
print 'close No.%d dest file %s ok'%(iNo,item)
except Exception, e:
print "Error, close No.%d file %s error="%(iNo,item)
print "Error '%s' happened on line %d" % (sys.exc_info()[1],sys.exc_info()[2].tb_lineno)
returnCode=1
if hbaseFileHandlePublic!=None:
try:
hbaseFileHandlePublic.close()
print 'close hbase public %s ok'%(hbaseFileNamePublic)
shutil.move(hbaseFileNamePublic,hbaseFileNamePublicNew)
print 'move hbase public to %s ok'%(hbaseFileNamePublicNew)
except Exception, e:
print "Error, close or move hbase public file %s error="%(hbaseFileNamePublic)
print "Error '%s' happened on line %d" % (sys.exc_info()[1],sys.exc_info()[2].tb_lineno)
returnCode=1
if hbaseFileHandlePublicSelf!=None:
try:
hbaseFileHandlePublicSelf.close()
print 'close hbase self %s ok'%(hbaseFileNameSelf)
shutil.move(hbaseFileNameSelf,hbaseFileNameSelfNew)
print 'move hbase self to %s ok'%(hbaseFileNameSelfNew)
except Exception, e:
print "Error, close or move hbase self file %s error="%(hbaseFileNameSelf)
print "Error '%s' happened on line %d" % (sys.exc_info()[1],sys.exc_info()[2].tb_lineno)
returnCode=1
try:
input.close()
print 'close source file %s ok'%(sourceFile)
except Exception, e:
print "Error, close source file %s error="%(sourceFile)
print "Error '%s' happened on line %d" % (sys.exc_info()[1],sys.exc_info()[2].tb_lineno)
returnCode=1
totalSeconds=mypylib.GetTimeChaSeconds(startTime,datetime.datetime.now())
print 'allclose=%d s,%s'%(totalSeconds,time.strftime('%Y-%m-%d %H:%M:%S', time.localtime()))
exit(returnCode)
Last edited by flyaga (2015-11-10 04:22:44)
Offline
Please consider using code tags!
Offline
That's the second, and last, time OP has been asked to paste using code tags:
https://wiki.archlinux.org/index.php/Fo … s_and_code
Offline
Ok, it is ok, jasonwryan and pypi, thanks!
Offline
Hmm, so you are writing to a path under a gpfs filesystem?
Without checking whether or not the python script is indeed "broken", a quick google suggested that GPFS can have quotas; something I hadn't thought of. Have you checked whether you are exceeding any quota set up?
Also, google suggested mmdf for gpfs filesystems; have you tried that?
Offline
>Hmm, so you are writing to a path under a gpfs filesystem?
Yes
>Without checking whether or not the python script is indeed "broken", a quick google suggested that GPFS can have quotas; something I hadn't thought of. Have you checked whether you are exceeding any quota set up?
>Also, google suggested mmdf for gpfs filesystems; have you tried that?
Sorry, I don't, I haven't any knowledge of gpfs quotas, how to do it?
Offline
>Without checking whether or not the python script is indeed "broken", a quick google suggested that GPFS can have quotas; something I hadn't thought of. Have you checked whether you are exceeding any quota set up?
>Also, google suggested mmdf for gpfs filesystems; have you tried that?
Sorry, I don't, I haven't any knowledge of gpfs quotas, how to do it?
5 minutes of googling will tell you as much as it did me. Or using the links in the quoted post. Or both.
Offline
pypi, thanks!
Offline