각 출력 유닛의 루트 노드 접근
루트 노드에서 자식 노드로 접근하면서 기울기 계산,
역전파
입력값 변화량에 대한 기울기
가중치 변화량에 대한 기울기
노드 값에 적용
자식 노드 탐색
자식 노드 리스트 추가 및 재귀 수행
node.grad_input ,node.grad_weight 값을 누적, 해당 값을 통해 가중치 갱신량을 계산하는데 배치 단위 연산이 끝난 후 누적된 두 값에 배치 크기를 나눠 갱신량을 계산한다.
grad_input : 입력값의 변화에 대한 비용 함수의 변화량 ( weight 값과 동일 )
grad_weight : 가중치 변화에 대한 비용 함수의 변화량 ( input 값과 동일 ) - 해당 값을 통해 가중치를 갱신해야 함
operation - add 에 weight 가 있는 경우는 bias 에 해당,
multiply 의 경우 모두 weight 가 존재함...
square 0.004093346993654268 0.0 1.6755489610458433e-05
subtract 0.7763381162903117 0.7722447692966574 0.004093346993654268
reciprocal 1.0 1.2880985475483855 0.7763381162903117
add 0.2880985475483856 1.0 1.2880985475483855
exp -1.2444526783886194 0.0 0.2880985475483856
negate 1.2444526783886194 0.0 -1.2444526783886194
add 1.1980022656686218 0.046450412719997725 1.2444526783886194
add 0.0 0.0 1.1980022656686218
multiply 0.7451204239923259 -0.4280460641762345 -0.3189458648272422
reciprocal 1.0 1.3420649438677836 0.7451204239923259
add 0.34206494386778374 1.0 1.3420649438677836
exp -1.0727546655969369 0.0 0.34206494386778374
negate 1.0727546655969369 0.0 -1.0727546655969369
add 0.4609017708745575 0.6118528947223795 1.0727546655969369
add 0.0 0.0 0.4609017708745575
multiply 0.6075448519014384 0.4967141530112327 0.30177612652855773
multiply 0.17052412368729153 -0.23415337472333597 -0.03992879903311886
multiply 0.06505159298527952 -0.4694743859349521 -0.030540056670854538
multiply 0.9488855372533332 0.24196227156603412 0.2295945000499732
multiply 0.21546977971011916 -0.7033438017074073 -0.1515493340143728
reciprocal 1.0 4.641022055832346 0.21546977971011916
add 3.6410220558323467 1.0 4.641022055832346
exp 1.2922644268053656 0.0 3.6410220558323467
negate -1.2922644268053656 0.0 1.2922644268053656
add -1.904117321527745 0.6118528947223795 -1.2922644268053656
add 0.0 0.0 -1.904117321527745
multiply 0.6075448519014384 -0.13826430117118466 -0.08400176437830326
multiply 0.17052412368729153 -0.23413695694918055 -0.039925999406568116
multiply 0.06505159298527952 0.5425600435859647 0.03529439512542969
multiply 0.9488855372533332 -1.913280244657798 -1.8154839528683033
multiply 0.4031760099104639 -0.6294749609242508 -0.2537892030839846
reciprocal 1.0 2.48030630647413 0.4031760099104639
add 1.48030630647413 1.0 2.48030630647413
exp 0.39224903019583557 0.0 1.48030630647413
negate -0.39224903019583557 0.0 0.39224903019583557
add -1.004101924918215 0.6118528947223795 -0.39224903019583557
add 0.0 0.0 -1.004101924918215
multiply 0.6075448519014384 0.6476885381006925 0.3934998369586443
multiply 0.17052412368729153 1.5792128155073915 0.26929388148013833
multiply 0.06505159298527952 -0.46341769281246226 -0.030146059135013586
multiply 0.9488855372533332 -1.7249178325130328 -1.6367495842219841
multiply 0.7510434291090605 2.5594880310377928 1.9222866675942214
reciprocal 1.0 1.3314809253923823 0.7510434291090605
add 0.33148092539238216 1.0 1.3314809253923823
exp -1.1041850113061993 0.0 0.33148092539238216
negate 1.1041850113061993 0.0 -1.1041850113061993
add 0.4923321165838197 0.6118528947223795 1.1041850113061993
add 0.0 0.0 0.4923321165838197
multiply 0.6075448519014384 1.5230298564080254 0.9253089485528827
multiply 0.17052412368729153 0.7674347291529088 0.1308661346759937
multiply 0.06505159298527952 -0.46572975357025687 -0.03029646237038688
multiply 0.9488855372533332 -0.5622875292409727 -0.5335465042746697
square 0.19524756715603103 0.0 0.03812161248034885
subtract 0.39396324869020344 0.1987156815341724 0.19524756715603103
reciprocal 1.0 2.538307832836354 0.39396324869020344
add 1.538307832836354 1.0 2.538307832836354
exp 0.430683002444882 0.0 1.538307832836354
negate -0.430683002444882 0.0 0.430683002444882
add -0.4771334151648797 0.046450412719997725 -0.430683002444882
add 0.0 0.0 -0.4771334151648797
multiply 0.7451204239923259 -0.7424068371191724 -0.5531824972490395
reciprocal 1.0 1.3420649438677836 0.7451204239923259
add 0.34206494386778374 1.0 1.3420649438677836
exp -1.0727546655969369 0.0 0.34206494386778374
negate 1.0727546655969369 0.0 -1.0727546655969369
add 0.4609017708745575 0.6118528947223795 1.0727546655969369
add 0.0 0.0 0.4609017708745575
multiply 0.6075448519014384 0.4967141530112327 0.30177612652855773
multiply 0.17052412368729153 -0.23415337472333597 -0.03992879903311886
multiply 0.06505159298527952 -0.4694743859349521 -0.030540056670854538
multiply 0.9488855372533332 0.24196227156603412 0.2295945000499732
multiply 0.21546977971011916 -2.1396206560762394 -0.46102359142796795
reciprocal 1.0 4.641022055832346 0.21546977971011916
add 3.6410220558323467 1.0 4.641022055832346
exp 1.2922644268053656 0.0 3.6410220558323467
negate -1.2922644268053656 0.0 1.2922644268053656
add -1.904117321527745 0.6118528947223795 -1.2922644268053656
add 0.0 0.0 -1.904117321527745
multiply 0.6075448519014384 -0.13826430117118466 -0.08400176437830326
multiply 0.17052412368729153 -0.23413695694918055 -0.039925999406568116
multiply 0.06505159298527952 0.5425600435859647 0.03529439512542969
multiply 0.9488855372533332 -1.913280244657798 -1.8154839528683033
multiply 0.4031760099104639 0.5977204669126083 0.24098655289164483
reciprocal 1.0 2.48030630647413 0.4031760099104639
add 1.48030630647413 1.0 2.48030630647413
exp 0.39224903019583557 0.0 1.48030630647413
negate -0.39224903019583557 0.0 0.39224903019583557
add -1.004101924918215 0.6118528947223795 -0.39224903019583557
add 0.0 0.0 -1.004101924918215
multiply 0.6075448519014384 0.6476885381006925 0.3934998369586443
multiply 0.17052412368729153 1.5792128155073915 0.26929388148013833
multiply 0.06505159298527952 -0.46341769281246226 -0.030146059135013586
multiply 0.9488855372533332 -1.7249178325130328 -1.6367495842219841
multiply 0.7510434291090605 0.3942330218796011 0.2960861206204829
reciprocal 1.0 1.3314809253923823 0.7510434291090605
add 0.33148092539238216 1.0 1.3314809253923823
exp -1.1041850113061993 0.0 0.33148092539238216
negate 1.1041850113061993 0.0 -1.1041850113061993
add 0.4923321165838197 0.6118528947223795 1.1041850113061993
add 0.0 0.0 0.4923321165838197
multiply 0.6075448519014384 1.5230298564080254 0.9253089485528827
multiply 0.17052412368729153 0.7674347291529088 0.1308661346759937
multiply 0.06505159298527952 -0.46572975357025687 -0.03029646237038688
multiply 0.9488855372533332 -0.5622875292409727 -0.5335465042746697자식 노드를 방문하면서 operation 값과 input_a, input_b, output 의 값
가중치 업데이트 완료~
'dev_AI_framework' 카테고리의 다른 글
| pytorch 가 layer 를 쌓는 과정에 대해, activation layer 의 구현 (0) | 2024.09.30 |
|---|---|
| pytorch 에 기반, 계산 그래프의 생성 방식을 바꿔보자... (0) | 2024.09.30 |
| 계산 그래프는 한 번만 생성해야 해, 완료~ (0) | 2024.09.19 |
| 계산 그래프는 한 번만 생성해야 해, 완료~ (1) | 2024.09.13 |
| 계산 그래프는 한 번만 생성해야 해, 생성 확인~ (0) | 2024.09.13 |