본문 바로가기
dev_AI_framework

한 사이클 연산 완료, 출력 로그 기반 오류 확인 및 수정

명징직조지훈 2025. 8. 2. 05:22 
=== [Epoch 1] 시작 ===
=== [DEBUG] All backward ops ===
op_type=5 input=input param= output=flatten_1839845566432_out
op_type=0 input=flatten_1839845566432_out param=dense_1839845565568_W output=dense_1839845565568_linear
op_type=1 input=dense_1839845565568_linear param=dense_1839845565568_b output=dense_1839845565568_out
op_type=2 input=dense_1839845565568_out param= output=activation_1839844972816_out
[activation_backward] grad_out[0]=1.000000, out=0.567298, grad_in=1.000000
[activation_backward] grad_out[1]=1.000000, out=2.749292, grad_in=1.000000
??run_graph_backward finished.
INFO:dev.models.sequential:[Batch 완료] 손실: 3.518179
INFO:dev.models.sequential:
=== [Epoch 2] 시작 ===
=== [DEBUG] All backward ops ===
op_type=5 input=input param= output=flatten_1839845566432_out
op_type=0 input=flatten_1839845566432_out param=dense_1839845565568_W output=dense_1839845565568_linear
op_type=1 input=dense_1839845565568_linear param=dense_1839845565568_b output=dense_1839845565568_out
op_type=2 input=dense_1839845565568_out param= output=activation_1839844972816_out
[activation_backward] grad_out[0]=1.000000, out=0.000000, grad_in=0.000000
[activation_backward] grad_out[1]=1.000000, out=2.739292, grad_in=1.000000
??run_graph_backward finished.
INFO:dev.models.sequential:[Batch 완료] 손실: 3.727931
INFO:dev.models.sequential:
=== [Epoch 3] 시작 ===
=== [DEBUG] All backward ops ===
op_type=5 input=input param= output=flatten_1839845566432_out
op_type=0 input=flatten_1839845566432_out param=dense_1839845565568_W output=dense_1839845565568_linear
op_type=1 input=dense_1839845565568_linear param=dense_1839845565568_b output=dense_1839845565568_out
op_type=2 input=dense_1839845565568_out param= output=activation_1839844972816_out
[activation_backward] grad_out[0]=1.000000, out=0.000000, grad_in=0.000000
[activation_backward] grad_out[1]=1.000000, out=0.070000, grad_in=1.000000
??run_graph_backward finished.
INFO:dev.models.sequential:[Batch 완료] 손실: 0.245450
INFO:dev.models.sequential:
=== [Epoch 4] 시작 ===
=== [DEBUG] All backward ops ===
op_type=5 input=input param= output=flatten_1839845566432_out
op_type=0 input=flatten_1839845566432_out param=dense_1839845565568_W output=dense_1839845565568_linear
op_type=1 input=dense_1839845565568_linear param=dense_1839845565568_b output=dense_1839845565568_out
op_type=2 input=dense_1839845565568_out param= output=activation_1839844972816_out
[activation_backward] grad_out[0]=1.000000, out=0.000000, grad_in=0.000000
[activation_backward] grad_out[1]=1.000000, out=2.699292, grad_in=1.000000
??run_graph_backward finished.
INFO:dev.models.sequential:[Batch 완료] 손실: 3.623159
INFO:dev.models.sequential:
=== [Epoch 5] 시작 ===
=== [DEBUG] All backward ops ===
op_type=5 input=input param= output=flatten_1839845566432_out
op_type=0 input=flatten_1839845566432_out param=dense_1839845565568_W output=dense_1839845565568_linear
op_type=1 input=dense_1839845565568_linear param=dense_1839845565568_b output=dense_1839845565568_out
op_type=2 input=dense_1839845565568_out param= output=activation_1839844972816_out
[activation_backward] grad_out[0]=1.000000, out=0.000000, grad_in=0.000000
[activation_backward] grad_out[1]=1.000000, out=9.930000, grad_in=1.000000
??run_graph_backward finished.
INFO:dev.models.sequential:[Batch 완료] 손실: 48.559448
INFO:dev.models.sequential:
=== [Epoch 6] 시작 ===
=== [DEBUG] All backward ops ===
op_type=5 input=input param= output=flatten_1839845566432_out
op_type=0 input=flatten_1839845566432_out param=dense_1839845565568_W output=dense_1839845565568_linear
op_type=1 input=dense_1839845565568_linear param=dense_1839845565568_b output=dense_1839845565568_out
op_type=2 input=dense_1839845565568_out param= output=activation_1839844972816_out
[activation_backward] grad_out[0]=1.000000, out=0.000000, grad_in=0.000000
[activation_backward] grad_out[1]=1.000000, out=2.659292, grad_in=1.000000
??run_graph_backward finished.
INFO:dev.models.sequential:[Batch 완료] 손실: 3.519988
INFO:dev.models.sequential:
=== [Epoch 7] 시작 ===
=== [DEBUG] All backward ops ===
op_type=5 input=input param= output=flatten_1839845566432_out
op_type=0 input=flatten_1839845566432_out param=dense_1839845565568_W output=dense_1839845565568_linear
op_type=1 input=dense_1839845565568_linear param=dense_1839845565568_b output=dense_1839845565568_out
op_type=2 input=dense_1839845565568_out param= output=activation_1839844972816_out
[activation_backward] grad_out[0]=1.000000, out=0.000000, grad_in=0.000000
[activation_backward] grad_out[1]=1.000000, out=9.890000, grad_in=1.000000
??run_graph_backward finished.
INFO:dev.models.sequential:[Batch 완료] 손실: 48.167049
INFO:dev.models.sequential:
=== [Epoch 8] 시작 ===
=== [DEBUG] All backward ops ===
op_type=5 input=input param= output=flatten_1839845566432_out
op_type=0 input=flatten_1839845566432_out param=dense_1839845565568_W output=dense_1839845565568_linear
op_type=1 input=dense_1839845565568_linear param=dense_1839845565568_b output=dense_1839845565568_out
op_type=2 input=dense_1839845565568_out param= output=activation_1839844972816_out
[activation_backward] grad_out[0]=1.000000, out=0.000000, grad_in=0.000000
[activation_backward] grad_out[1]=1.000000, out=2.619292, grad_in=1.000000
??run_graph_backward finished.
INFO:dev.models.sequential:[Batch 완료] 손실: 3.418416
INFO:dev.models.sequential:
=== [Epoch 9] 시작 ===
=== [DEBUG] All backward ops ===
op_type=5 input=input param= output=flatten_1839845566432_out
op_type=0 input=flatten_1839845566432_out param=dense_1839845565568_W output=dense_1839845565568_linear
op_type=1 input=dense_1839845565568_linear param=dense_1839845565568_b output=dense_1839845565568_out
op_type=2 input=dense_1839845565568_out param= output=activation_1839844972816_out
[activation_backward] grad_out[0]=1.000000, out=0.000000, grad_in=0.000000
[activation_backward] grad_out[1]=1.000000, out=9.850000, grad_in=1.000000
??run_graph_backward finished.
INFO:dev.models.sequential:[Batch 완료] 손실: 47.776249
INFO:dev.models.sequential:
=== [Epoch 10] 시작 ===
=== [DEBUG] All backward ops ===
op_type=5 input=input param= output=flatten_1839845566432_out
op_type=0 input=flatten_1839845566432_out param=dense_1839845565568_W output=dense_1839845565568_linear
op_type=1 input=dense_1839845565568_linear param=dense_1839845565568_b output=dense_1839845565568_out
op_type=2 input=dense_1839845565568_out param= output=activation_1839844972816_out
[activation_backward] grad_out[0]=1.000000, out=0.000000, grad_in=0.000000
[activation_backward] grad_out[1]=1.000000, out=2.579292, grad_in=1.000000
??run_graph_backward finished.
INFO:dev.models.sequential:[Batch 완료] 손실: 3.318444
INFO:dev.models.sequential:📊 평가 손실: 47.387051, 메트릭(mse): 47.387051

📊 최종 평가 메트릭 (MSE): 47.387051
🔍 예측 출력:
 [[0.   9.81]]

 

dense - activation 의 아주 간단한 구조, 출력이 0~1 사이의 값이 아닌 다른 값이 나오므로 sigmoid, activatoin function 의 확인 필요

  1. CUDA 내에서 정의한 OpStruct 와 Python 에서 정의한 OpStruct 의 불일치
  2. CUDA 내 sigmoid 연산에서 사용한 __expf 에서의 오류 발생 가능성, 속도는 빠르지만 정확성 부족 및 NaN 발생 가능성
__device__ float safe_sigmoid(float x) {
    if (isnan(x)) return 0.5f;
    if (x < -30.0f) return 0.0f;
    if (x > 30.0f) return 1.0f;
    return 1.0f / (1.0f + expf(-x));  // 정확성을 위해 expf 사용
}

__global__ void activation_sigmoid(const float* input, const float* bias, float* output, int rows, int cols) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;
    if (idx < rows * cols) {
        int col = idx % cols;
        float val = input[idx] + (bias ? bias[col] : 0.0f);
        output[idx] = safe_sigmoid(val);
    }
}

수정 후 출력

=== [TEST] Sequential 모델 학습 + 평가 (metrics 포함) ===
INFO:dev.models.sequential:✅ 레이어 추가됨: Flatten (input_shape=(1, 2, 2), output_shape=(1, 4))
INFO:dev.models.sequential:✅ 레이어 추가됨: Dense (input_shape=(1, 4), output_shape=(1, 2))
INFO:dev.models.sequential:✅ 레이어 추가됨: Activation (input_shape=(1, 2), output_shape=(1, 2))
[INFO] Dense 초기화 완료: weights=0.5, bias=0.1
INFO:dev.models.sequential:
=== [Epoch 1] 시작 ===
=== [DEBUG] All backward ops ===
op_type=5 input=input param= output=flatten_2443705994368_out
op_type=0 input=flatten_2443705994368_out param=dense_2442952969200_W output=dense_2442952969200_linear
op_type=1 input=dense_2442952969200_linear param=dense_2442952969200_b output=dense_2442952969200_out
op_type=3 input=dense_2442952969200_out param= output=activation_2442954032624_out
[activation_backward] grad_out[0]=1.000000, out=0.001266, grad_in=0.001265, act_type=3
[activation_backward] grad_out[1]=1.000000, out=0.999071, grad_in=0.000928, act_type=3
??run_graph_backward finished.
INFO:dev.models.sequential:[Batch 완료] 손실: 0.648279
INFO:dev.models.sequential:
=== [Epoch 2] 시작 ===
=== [DEBUG] All backward ops ===
op_type=5 input=input param= output=flatten_2443705994368_out
op_type=0 input=flatten_2443705994368_out param=dense_2442952969200_W output=dense_2442952969200_linear
op_type=1 input=dense_2442952969200_linear param=dense_2442952969200_b output=dense_2442952969200_out
op_type=3 input=dense_2442952969200_out param= output=activation_2442954032624_out
[activation_backward] grad_out[0]=1.000000, out=0.500000, grad_in=0.250000, act_type=3
[activation_backward] grad_out[1]=1.000000, out=0.999071, grad_in=0.000928, act_type=3
??run_graph_backward finished.
INFO:dev.models.sequential:[Batch 완료] 손실: 0.424165
INFO:dev.models.sequential:
=== [Epoch 3] 시작 ===
=== [DEBUG] All backward ops ===
op_type=5 input=input param= output=flatten_2443705994368_out
op_type=0 input=flatten_2443705994368_out param=dense_2442952969200_W output=dense_2442952969200_linear
op_type=1 input=dense_2442952969200_linear param=dense_2442952969200_b output=dense_2442952969200_out
op_type=3 input=dense_2442952969200_out param= output=activation_2442954032624_out
[activation_backward] grad_out[0]=1.000000, out=0.500000, grad_in=0.250000, act_type=3
[activation_backward] grad_out[1]=1.000000, out=0.524972, grad_in=0.249376, act_type=3
??run_graph_backward finished.
INFO:dev.models.sequential:[Batch 완료] 손실: 0.110301
INFO:dev.models.sequential:
=== [Epoch 4] 시작 ===
=== [DEBUG] All backward ops ===
op_type=5 input=input param= output=flatten_2443705994368_out
op_type=0 input=flatten_2443705994368_out param=dense_2442952969200_W output=dense_2442952969200_linear
op_type=1 input=dense_2442952969200_linear param=dense_2442952969200_b output=dense_2442952969200_out
op_type=3 input=dense_2442952969200_out param= output=activation_2442954032624_out
[activation_backward] grad_out[0]=1.000000, out=0.500000, grad_in=0.250000, act_type=3
[activation_backward] grad_out[1]=1.000000, out=0.999067, grad_in=0.000932, act_type=3
??run_graph_backward finished.
INFO:dev.models.sequential:[Batch 완료] 손실: 0.424160
INFO:dev.models.sequential:
=== [Epoch 5] 시작 ===
=== [DEBUG] All backward ops ===
op_type=5 input=input param= output=flatten_2443705994368_out
op_type=0 input=flatten_2443705994368_out param=dense_2442952969200_W output=dense_2442952969200_linear
op_type=1 input=dense_2442952969200_linear param=dense_2442952969200_b output=dense_2442952969200_out
op_type=3 input=dense_2442952969200_out param= output=activation_2442954032624_out
[activation_backward] grad_out[0]=1.000000, out=0.500000, grad_in=0.250000, act_type=3
[activation_backward] grad_out[1]=1.000000, out=0.999954, grad_in=0.000046, act_type=3
??run_graph_backward finished.
INFO:dev.models.sequential:[Batch 완료] 손실: 0.424959
INFO:dev.models.sequential:
=== [Epoch 6] 시작 ===
=== [DEBUG] All backward ops ===
op_type=5 input=input param= output=flatten_2443705994368_out
op_type=0 input=flatten_2443705994368_out param=dense_2442952969200_W output=dense_2442952969200_linear
op_type=1 input=dense_2442952969200_linear param=dense_2442952969200_b output=dense_2442952969200_out
op_type=3 input=dense_2442952969200_out param= output=activation_2442954032624_out
[activation_backward] grad_out[0]=1.000000, out=0.500000, grad_in=0.250000, act_type=3
[activation_backward] grad_out[1]=1.000000, out=0.999067, grad_in=0.000932, act_type=3
??run_graph_backward finished.
INFO:dev.models.sequential:[Batch 완료] 손실: 0.424160
INFO:dev.models.sequential:
=== [Epoch 7] 시작 ===
=== [DEBUG] All backward ops ===
op_type=5 input=input param= output=flatten_2443705994368_out
op_type=0 input=flatten_2443705994368_out param=dense_2442952969200_W output=dense_2442952969200_linear
op_type=1 input=dense_2442952969200_linear param=dense_2442952969200_b output=dense_2442952969200_out
op_type=3 input=dense_2442952969200_out param= output=activation_2442954032624_out
[activation_backward] grad_out[0]=1.000000, out=0.500000, grad_in=0.250000, act_type=3
[activation_backward] grad_out[1]=1.000000, out=0.999954, grad_in=0.000046, act_type=3
??run_graph_backward finished.
INFO:dev.models.sequential:[Batch 완료] 손실: 0.424959
INFO:dev.models.sequential:
=== [Epoch 8] 시작 ===
=== [DEBUG] All backward ops ===
op_type=5 input=input param= output=flatten_2443705994368_out
op_type=0 input=flatten_2443705994368_out param=dense_2442952969200_W output=dense_2442952969200_linear
op_type=1 input=dense_2442952969200_linear param=dense_2442952969200_b output=dense_2442952969200_out
op_type=3 input=dense_2442952969200_out param= output=activation_2442954032624_out
[activation_backward] grad_out[0]=1.000000, out=0.500000, grad_in=0.250000, act_type=3
[activation_backward] grad_out[1]=1.000000, out=0.999067, grad_in=0.000932, act_type=3
??run_graph_backward finished.
INFO:dev.models.sequential:[Batch 완료] 손실: 0.424160
INFO:dev.models.sequential:
=== [Epoch 9] 시작 ===
=== [DEBUG] All backward ops ===
op_type=5 input=input param= output=flatten_2443705994368_out
op_type=0 input=flatten_2443705994368_out param=dense_2442952969200_W output=dense_2442952969200_linear
op_type=1 input=dense_2442952969200_linear param=dense_2442952969200_b output=dense_2442952969200_out
op_type=3 input=dense_2442952969200_out param= output=activation_2442954032624_out
[activation_backward] grad_out[0]=1.000000, out=0.500000, grad_in=0.250000, act_type=3
[activation_backward] grad_out[1]=1.000000, out=0.999954, grad_in=0.000046, act_type=3
??run_graph_backward finished.
INFO:dev.models.sequential:[Batch 완료] 손실: 0.424959
INFO:dev.models.sequential:
=== [Epoch 10] 시작 ===
=== [DEBUG] All backward ops ===
op_type=5 input=input param= output=flatten_2443705994368_out
op_type=0 input=flatten_2443705994368_out param=dense_2442952969200_W output=dense_2442952969200_linear
op_type=1 input=dense_2442952969200_linear param=dense_2442952969200_b output=dense_2442952969200_out
op_type=3 input=dense_2442952969200_out param= output=activation_2442954032624_out
[activation_backward] grad_out[0]=1.000000, out=0.500000, grad_in=0.250000, act_type=3
[activation_backward] grad_out[1]=1.000000, out=0.999067, grad_in=0.000932, act_type=3
??run_graph_backward finished.
INFO:dev.models.sequential:[Batch 완료] 손실: 0.424160
INFO:dev.models.sequential:📊 평가 손실: 0.424959, 메트릭(mse): 0.424959

📊 최종 평가 메트릭 (MSE): 0.424959
🔍 예측 출력:
 [[0.5        0.99995434]]

정상 작동의 확인

'dev_AI_framework' 카테고리의 다른 글

CUDA, CUPY, Python 버전 세팅 및 확인  (0) 2025.08.03
간단한 모델의 학습으로 필요한 추가 사항의 확인  (0) 2025.08.02
optimizer 의 backend 내 구현 완료, 학습 과정 정리  (0) 2025.08.02
E 행렬 기반 forward, backward 연산과 loss, metrics 연산 통합  (1) 2025.07.31
각 cu, cuh 파일별 역할  (3) 2025.07.20

'dev_AI_framework' Related Articles

티스토리툴바