메모리 부족(MemoryError) : numpy._core._exceptions._ArrayMemoryError:

numpy._core._exceptions._ArrayMemoryError:

Unable to allocate 4.43 GiB메모리 부족(MemoryError)

NumPy가 한 번에 약 4.43GB의 메모리를 할당하려 했는데 실패했다는 의미

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1783998730.213805    5088 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1783998732.754191    5088 port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
Data Processing..
train
Traceback (most recent call last):
  File "D:\study\sk_playdata\book\Intelligent-Projects-Using-Python\Chapter02\TransferLearning.py", line 295, in <module>
    obj.main()
    ~~~~~~~~^^
  File "D:\study\sk_playdata\book\Intelligent-Projects-Using-Python\Chapter02\TransferLearning.py", line 279, in main
    train_X,train_y = self.read_data(self.class_folders,self.path,self.num_class,self.dim,train_val='train')
                      ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\study\sk_playdata\book\Intelligent-Projects-Using-Python\Chapter02\TransferLearning.py", line 131, in read_data
    return np.array(train_X),train_y
           ~~~~~~~~^^^^^^^^^
numpy._core._exceptions._ArrayMemoryError: Unable to allocate 4.43 GiB for an array with shape (31614, 224, 224, 3) and data type uint8
(base) PS D:\study\sk_playdata\book\Intelligent-Projects-Using-Python>

 

 

 

read할때 슬라이스. file_list = file_list[:1000]

	def read_data(self,class_folders,path,num_class,dim,train_val='train'):
		print(train_val)
		train_X,train_y = [],[] 
		for c in class_folders:
			# path_class = path + str(train_val) + '/' + str(c)
			path_class = os.path.join(path, train_val, c)

			file_list = os.listdir(path_class) 

			# 메모리 부족(MemoryError)추가
			file_list = file_list[:1000]

			for f in file_list:
				img = self.get_im_cv2(path_class + '/' + f)
				img = self.pre_process(img)
				train_X.append(img)
				label = int(c.split('class')[1])
				train_y.append(int(label))

		# train_y = keras.utils.np_utils.to_categorical(np.array(train_y),num_class) 
		train_y = to_categorical(np.array(train_y), num_class)

		return np.array(train_X),train_y
		print("train_X shape:", np.array(train_X).shape)
		print("train_y shape:", train_y.shape)

 

 

 

 

 

완료

968/87910968 ━━━━━━━━━━━━━━━━━━━━ 8s 0us/step 046546944/87910968 ━━━━━━━━━━━━━━━━━━━━ 4s 0us/step
Traceback (most recent call last):
  File "D:\study\sk_playdata\book\Intelligent-Projects-Using-Python\Chapter02\TransferLearning.py", line 299, in <module>
    obj.main()
    ~~~~~~~~^^
  File "D:\study\sk_playdata\book\Intelligent-Projects-Using-Python\Chapter02\TransferLearning.py", line 284, in main
                self.model_save_dest = self.train_model(train_X,train_y,n_fold=self.folds,batch_size=self.batch_size,
                             ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                                            epochs=self.epochs,dim=self.dim,lr=self.lr,model=self.model)
                                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\study\sk_playdata\book\Intelligent-Projects-Using-Python\Chapter02\TransferLearning.py", line 210, in train_model
    model_final = self.inception_pseudo(dim=224,freeze_layers=10,full_freeze='N')
  File "D:\study\sk_playdata\book\Intelligent-Projects-Using-Python\Chapter02\TransferLearning.py", line 149, in inception_pseudo
    model_final = Model(input = model.input,outputs=out)
  File "C:\Users\playdata2\anaconda3\Lib\site-packages\keras\src\models\model.py", line 162, in __init__
    Layer.__init__(self, *args, **kwargs)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\playdata2\anaconda3\Lib\site-packages\keras\src\layers\layer.py", line 295, in __init__
    raise ValueError(
    ...<2 lines>...
    )
ValueError: Unrecognized keyword arguments passed to Model: {'input': <KerasTensor shape=(None, None, None, 3), dtype=float32, sparse=False, ragged=False, name=keras_tensor>, 'outputs': <KerasTensor shape=(None, 5), dtype=float32, sparse=False, ragged=False, name=keras_tensor_316>}
(base) PS D:\study\sk_playdata\book\Intelligent-Projects-Using-Python>