계산 그래프 연결이 이상해, 여기서 정리하고 확인해보자
입력 데이터의 길이 3,
입력 데이터 차원 4,
은닉 유닛 수 5 의 가정
연산 수행 중에 이전 활성화 노드에 대한 계산 그래프 출력,
0.0890103 의 활성화 노드 출력 값이 반복해서 나오는 것을 확인할 수 있음,
두 번째 스텝에서 5 번 등장,
세 번째 스텝에서 25 번 등장,
이에 대한 이유로 RNN 은 recurrent 연산에 있어 ( hidden_unit, hidden_unit ) 의 동일한 가중치를 사용,
첫 번째 스텝에서의 h_0 (0 번째 recurrent 값) 은 0 으로 채워진 (1,5) 행렬임 (1,5) @ (5,5) 의 연산 결과는 0으로 채워진 (1,5) 의 결과와
입력 데이터와 input_weght 값과의 연산 결과인 (1,4) @ (4,5) 의 연산 출력, (1,5) 의 값이 사용된다.
recurrent (1,5) + input (1,5)
activation (recurrent + input) (1,5) 의 결과가 0.0890103 이라는 것
다시 계산 그래프로 돌아와서 출력하는 activation_recurrent 의 개수, 25개가 의미하는 것에 대해 생각
첫 번째 반복에서는 이전의 은닉 입력 값이 0으로 이에 대한 계산 그래프 구현이 없었고,
두 번째 반복에서 부터 이전의 은닉 입력의 활성화 함수의 계산 그래프가 출력되는 것으로 입력의 크기는 (1,5) 이다.
따라서 직관적으로 생각했을 때 5개의 계산 그래프만 출력되어야 한다고 생각이 들고 해당 계산 그래프에 포함되어야 하는 내용으로 그 값이 완성되기 위해 구성에 필요한 activation (recurrent + input) 요 값들이 포함되어야 한다.
Node: reciprocal, Weight: 11.2346, Grad Total: 0, Output: 0.0890103
Children:
Node: add, Weight: 1, Grad Total: 0, Output: 11.2346
Children:
Node: exp, Weight: 0, Grad Total: 0, Output: 10.2346
Children:
Node: negate, Weight: 0, Grad Total: 0, Output: 2.32578
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -2.32578
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -2.32578
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -2.32578
Children:
Node: multiply, Weight: -1.41537, Grad Total: 0, Output: -0.773798
Leaf node
Node: multiply, Weight: 0.404051, Grad Total: 0, Output: 0.0746906
Leaf node
Node: multiply, Weight: -1.91877, Grad Total: 0, Output: -1.86041
Leaf node
Node: multiply, Weight: 0.301547, Grad Total: 0, Output: 0.233739
Leaf node
Node: add, Weight: 0, Grad Total: 0, Output: 0
Children:
Node: multiply, Weight: 0.791032, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: 2.19046, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: -1.55066, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: 1.54993, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: 0.22746, Grad Total: 0, Output: 0
Leaf node하나의 recurrent 출력을 보면, 해당 출력을 구성하는 요소가
input 에 대한 가중치 연산과 (1,4) (4,1)
recurrent 에 대한 연산 (1,5) (5,1) 이 합쳐진 것,
input 에 대한 연산이 (1,4)(4,5-1) (1,4)(4,5-2) (1,4)(4,5-3) (1,4)(4,5-4) (1,4)(4,5-5)
recurrent 에 대한 연산 (1,5)(5-1) (1,5)(5-2) (1,5)(5-3) (1,5)(5-4) (1,5)(5-5)
로 총 25 개의 조합이 나온다.
그런데 여기서 recurrent 에 대한 연산의 결과가 첫 번째 time_step 에서는 0으로 동일하므로
input 에 대한 연산 결과에 activation 연산의 결과만 출력, 여기서 5개의 동일한 값이 5번씩 반복되어 출력되는 형태임
// 순환 구조 연결: 현재 타임스텝에서 이전 타임스텝의 activation_nodes와 연결
// 이것만 잘 수정하면 돼
if (t > 0) { // 첫 번째 타임스텝 제외
for (int u = 0; u < units; u++) {
// 현재 유닛의 recurrent_multiply_node에서 리프 노드를 가져옴
auto recurrent_leaf_nodes = recurrent_multiply_node_list[u]->find_leaf_nodes();
// 리프 노드의 수와 previous_activation_nodes의 크기를 비교하여 반복
int num_leaf_nodes = recurrent_leaf_nodes.size();
for (int j = 0; j < num_leaf_nodes && j < units; j++) {
// 출력 확인
std::cout << t << "check" << std::endl;
previous_activation_nodes[j]->print_tree(previous_activation_nodes[j]);
std::cout << t << "check" << std::endl;
// 이전 타임스텝의 activation_nodes[j]를 현재 타임스텝의 리프 노드들과 연결
recurrent_leaf_nodes[j]->add_child(previous_activation_nodes[j]);
previous_activation_nodes[j]->add_parent(recurrent_leaf_nodes[j]);
}
}
}위 알고리즘으로 recurrent 노드와 연결했는데
필요한 것으로 previous_activation 의 루트 노드와 recurrent_multiply 의 리프 노드를 연결했어야 했음
recurrent_multiply 의 리프노드의 형태에 대해 출력해보자
아 지금 25개가 출력되어야 하는게 맞는건지도 헷갈리네
출력을 previous_activation_nodes 를 확인하는 방법으로 보자
이렇게 하면 직관적으로 각 5개의 출력을 확인할 수 있음
Node: reciprocal, Weight: 11.2346, Grad Total: 0, Output: 0.0890103
Children:
Node: add, Weight: 1, Grad Total: 0, Output: 11.2346
Children:
Node: exp, Weight: 0, Grad Total: 0, Output: 10.2346
Children:
Node: negate, Weight: 0, Grad Total: 0, Output: 2.32578
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -2.32578
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -2.32578
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -2.32578
Children:
Node: multiply, Weight: -1.41537, Grad Total: 0, Output: -0.773798
Leaf node
Node: multiply, Weight: 0.404051, Grad Total: 0, Output: 0.0746906
Leaf node
Node: multiply, Weight: -1.91877, Grad Total: 0, Output: -1.86041
Leaf node
Node: multiply, Weight: 0.301547, Grad Total: 0, Output: 0.233739
Leaf node
Node: add, Weight: 0, Grad Total: 0, Output: 0
Children:
Node: multiply, Weight: 0.791032, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: 2.19046, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: -1.55066, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: 1.54993, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: 0.22746, Grad Total: 0, Output: 0
Leaf node
Node: reciprocal, Weight: 1.93605, Grad Total: 0, Output: 0.516515
Children:
Node: add, Weight: 1, Grad Total: 0, Output: 1.93605
Children:
Node: exp, Weight: 0, Grad Total: 0, Output: 0.936051
Children:
Node: negate, Weight: 0, Grad Total: 0, Output: -0.0660851
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 0.0660851
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 0.0660851
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 0.0660851
Children:
Node: multiply, Weight: -0.420645, Grad Total: 0, Output: -0.229971
Leaf node
Node: multiply, Weight: 1.88619, Grad Total: 0, Output: 0.34867
Leaf node
Node: multiply, Weight: -0.0265139, Grad Total: 0, Output: -0.0257074
Leaf node
Node: multiply, Weight: -0.0347118, Grad Total: 0, Output: -0.0269062
Leaf node
Node: add, Weight: 0, Grad Total: 0, Output: 0
Children:
Node: multiply, Weight: -0.909387, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: -0.990536, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: 0.068563, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: -0.783253, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: 1.30714, Grad Total: 0, Output: 0
Leaf node
Node: reciprocal, Weight: 3.72529, Grad Total: 0, Output: 0.268435
Children:
Node: add, Weight: 1, Grad Total: 0, Output: 3.72529
Children:
Node: exp, Weight: 0, Grad Total: 0, Output: 2.72529
Children:
Node: negate, Weight: 0, Grad Total: 0, Output: 1.00258
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -1.00258
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -1.00258
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -1.00258
Children:
Node: multiply, Weight: -0.342715, Grad Total: 0, Output: -0.187366
Leaf node
Node: multiply, Weight: 0.174578, Grad Total: 0, Output: 0.0322715
Leaf node
Node: multiply, Weight: 0.0602302, Grad Total: 0, Output: 0.0583983
Leaf node
Node: multiply, Weight: -1.16868, Grad Total: 0, Output: -0.905881
Leaf node
Node: add, Weight: 0, Grad Total: 0, Output: 0
Children:
Node: multiply, Weight: 1.40279, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: -0.566298, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: -1.0623, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: -0.322062, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: -1.60748, Grad Total: 0, Output: -0
Leaf node
Node: reciprocal, Weight: 1.05596, Grad Total: 0, Output: 0.947008
Children:
Node: add, Weight: 1, Grad Total: 0, Output: 1.05596
Children:
Node: exp, Weight: 0, Grad Total: 0, Output: 0.0559578
Children:
Node: negate, Weight: 0, Grad Total: 0, Output: -2.88316
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 2.88316
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 2.88316
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 2.88316
Children:
Node: multiply, Weight: -0.802277, Grad Total: 0, Output: -0.438613
Leaf node
Node: multiply, Weight: 0.25755, Grad Total: 0, Output: 0.0476093
Leaf node
Node: multiply, Weight: 2.46324, Grad Total: 0, Output: 2.38832
Leaf node
Node: multiply, Weight: 1.14282, Grad Total: 0, Output: 0.885839
Leaf node
Node: add, Weight: 0, Grad Total: 0, Output: 0
Children:
Node: multiply, Weight: -1.40185, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: 0.0996514, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: 0.473592, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: 0.813517, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: 0.184634, Grad Total: 0, Output: 0
Leaf node
Node: reciprocal, Weight: 1.74498, Grad Total: 0, Output: 0.573073
Children:
Node: add, Weight: 1, Grad Total: 0, Output: 1.74498
Children:
Node: exp, Weight: 0, Grad Total: 0, Output: 0.744979
Children:
Node: negate, Weight: 0, Grad Total: 0, Output: -0.2944
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 0.2944
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 0.2944
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 0.2944
Children:
Node: multiply, Weight: -0.161286, Grad Total: 0, Output: -0.0881766
Leaf node
Node: multiply, Weight: -0.0744459, Grad Total: 0, Output: -0.0137617
Leaf node
Node: multiply, Weight: -0.192361, Grad Total: 0, Output: -0.18651
Leaf node
Node: multiply, Weight: 0.751933, Grad Total: 0, Output: 0.582848
Leaf node
Node: add, Weight: 0, Grad Total: 0, Output: 0
Children:
Node: multiply, Weight: 0.586857, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: -0.503476, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: -0.919424, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: -1.23086, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: 0.259883, Grad Total: 0, Output: 0
Leaf nodeprevious_unit 별 결과를 확인할 수 있음
여기서 입력 데이터에 사용되는 input_weight (4,5) 와 recurrent_weight (5,5) 의 모든 값을 확인
두 번째 타임 스텝의 경우에도 정상적인 출력 형태를 확인 가능
recurrent_multiply_node_list size: 5
recurrent_multiply_node_list_leaf_node size: 5
1check
1check
Node: reciprocal, Weight: 1.5791, Grad Total: 0, Output: 0.633273
Children:
Node: add, Weight: 1, Grad Total: 0, Output: 1.5791
Children:
Node: exp, Weight: 0, Grad Total: 0, Output: 0.579098
Children:
Node: negate, Weight: 0, Grad Total: 0, Output: -0.546284
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 0.546284
Children:
Node: add, Weight: 2.38371, Grad Total: 0, Output: 0.546284
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -1.83743
Children:
Node: multiply, Weight: -1.41537, Grad Total: 0, Output: -1.32974
Leaf node
Node: multiply, Weight: 0.404051, Grad Total: 0, Output: 0.361556
Leaf node
Node: multiply, Weight: -1.91877, Grad Total: 0, Output: -1.14723
Leaf node
Node: multiply, Weight: 0.301547, Grad Total: 0, Output: 0.277989
Leaf node
Node: add, Weight: 0, Grad Total: 0, Output: 2.38371
Children:
Node: multiply, Weight: 0.791032, Grad Total: 0, Output: 0.07041
Children:
Node: reciprocal, Weight: 11.2346, Grad Total: 0, Output: 0.0890103
Children:
Node: add, Weight: 1, Grad Total: 0, Output: 11.2346
Children:
Node: exp, Weight: 0, Grad Total: 0, Output: 10.2346
Children:
Node: negate, Weight: 0, Grad Total: 0, Output: 2.32578
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -2.32578
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -2.32578
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -2.32578
Children:
Node: multiply, Weight: -1.41537, Grad Total: 0, Output: -0.773798
Leaf node
Node: multiply, Weight: 0.404051, Grad Total: 0, Output: 0.0746906
Leaf node
Node: multiply, Weight: -1.91877, Grad Total: 0, Output: -1.86041
Leaf node
Node: multiply, Weight: 0.301547, Grad Total: 0, Output: 0.233739
Leaf node
Node: add, Weight: 0, Grad Total: 0, Output: 0
Children:
Node: multiply, Weight: 0.791032, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: 2.19046, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: -1.55066, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: 1.54993, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: 0.22746, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: 2.19046, Grad Total: 0, Output: 1.1314
Children:
Node: reciprocal, Weight: 1.93605, Grad Total: 0, Output: 0.516515
Children:
Node: add, Weight: 1, Grad Total: 0, Output: 1.93605
Children:
Node: exp, Weight: 0, Grad Total: 0, Output: 0.936051
Children:
Node: negate, Weight: 0, Grad Total: 0, Output: -0.0660851
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 0.0660851
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 0.0660851
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 0.0660851
Children:
Node: multiply, Weight: -0.420645, Grad Total: 0, Output: -0.229971
Leaf node
Node: multiply, Weight: 1.88619, Grad Total: 0, Output: 0.34867
Leaf node
Node: multiply, Weight: -0.0265139, Grad Total: 0, Output: -0.0257074
Leaf node
Node: multiply, Weight: -0.0347118, Grad Total: 0, Output: -0.0269062
Leaf node
Node: add, Weight: 0, Grad Total: 0, Output: 0
Children:
Node: multiply, Weight: -0.909387, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: -0.990536, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: 0.068563, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: -0.783253, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: 1.30714, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: -1.55066, Grad Total: 0, Output: -0.416253
Children:
Node: reciprocal, Weight: 3.72529, Grad Total: 0, Output: 0.268435
Children:
Node: add, Weight: 1, Grad Total: 0, Output: 3.72529
Children:
Node: exp, Weight: 0, Grad Total: 0, Output: 2.72529
Children:
Node: negate, Weight: 0, Grad Total: 0, Output: 1.00258
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -1.00258
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -1.00258
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -1.00258
Children:
Node: multiply, Weight: -0.342715, Grad Total: 0, Output: -0.187366
Leaf node
Node: multiply, Weight: 0.174578, Grad Total: 0, Output: 0.0322715
Leaf node
Node: multiply, Weight: 0.0602302, Grad Total: 0, Output: 0.0583983
Leaf node
Node: multiply, Weight: -1.16868, Grad Total: 0, Output: -0.905881
Leaf node
Node: add, Weight: 0, Grad Total: 0, Output: 0
Children:
Node: multiply, Weight: 1.40279, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: -0.566298, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: -1.0623, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: -0.322062, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: -1.60748, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: 1.54993, Grad Total: 0, Output: 1.4678
Children:
Node: reciprocal, Weight: 1.05596, Grad Total: 0, Output: 0.947008
Children:
Node: add, Weight: 1, Grad Total: 0, Output: 1.05596
Children:
Node: exp, Weight: 0, Grad Total: 0, Output: 0.0559578
Children:
Node: negate, Weight: 0, Grad Total: 0, Output: -2.88316
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 2.88316
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 2.88316
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 2.88316
Children:
Node: multiply, Weight: -0.802277, Grad Total: 0, Output: -0.438613
Leaf node
Node: multiply, Weight: 0.25755, Grad Total: 0, Output: 0.0476093
Leaf node
Node: multiply, Weight: 2.46324, Grad Total: 0, Output: 2.38832
Leaf node
Node: multiply, Weight: 1.14282, Grad Total: 0, Output: 0.885839
Leaf node
Node: add, Weight: 0, Grad Total: 0, Output: 0
Children:
Node: multiply, Weight: -1.40185, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: 0.0996514, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: 0.473592, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: 0.813517, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: 0.184634, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: 0.22746, Grad Total: 0, Output: 0.130351
Children:
Node: reciprocal, Weight: 1.74498, Grad Total: 0, Output: 0.573073
Children:
Node: add, Weight: 1, Grad Total: 0, Output: 1.74498
Children:
Node: exp, Weight: 0, Grad Total: 0, Output: 0.744979
Children:
Node: negate, Weight: 0, Grad Total: 0, Output: -0.2944
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 0.2944
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 0.2944
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 0.2944
Children:
Node: multiply, Weight: -0.161286, Grad Total: 0, Output: -0.0881766
Leaf node
Node: multiply, Weight: -0.0744459, Grad Total: 0, Output: -0.0137617
Leaf node
Node: multiply, Weight: -0.192361, Grad Total: 0, Output: -0.18651
Leaf node
Node: multiply, Weight: 0.751933, Grad Total: 0, Output: 0.582848
Leaf node
Node: add, Weight: 0, Grad Total: 0, Output: 0
Children:
Node: multiply, Weight: 0.586857, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: -0.503476, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: -0.919424, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: -1.23086, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: 0.259883, Grad Total: 0, Output: 0
Leaf node
2번째 time_step 의 첫 번째 unit 에 대한 그래프, time_step_num, 2의 recurret_weight 는 (5,1) 에 해당하는 가중치들이, time_step_num 1 의 recurrent_weight 에는 (5,5) 의 모든 가중치들이 사용되고 있음을 확인할 수 있으며,
이러한 unit 이 5개가 있음
그런데, 3번째 time_step 에서의 문제점 발생
Node: reciprocal, Weight: 1.03456, Grad Total: 0, Output: 0.966594
Children:
Node: add, Weight: 1, Grad Total: 0, Output: 1.03456
Children:
Node: exp, Weight: 0, Grad Total: 0, Output: 0.03456
Children:
Node: negate, Weight: 0, Grad Total: 0, Output: -3.36506
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 3.36506
Children:
Node: add, Weight: 3.3998, Grad Total: 0, Output: 3.36506
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -0.034741
Children:
Node: multiply, Weight: -1.41537, Grad Total: 0, Output: -0.12525
Leaf node
Node: multiply, Weight: 0.404051, Grad Total: 0, Output: 0.079187
Leaf node
Node: multiply, Weight: -1.91877, Grad Total: 0, Output: -0.0867808
Leaf node
Node: multiply, Weight: 0.301547, Grad Total: 0, Output: 0.0981025
Leaf node
Node: add, Weight: 0, Grad Total: 0, Output: 3.3998
Children:
Node: multiply, Weight: 0.791032, Grad Total: 0, Output: 0.500939
Children:
Node: reciprocal, Weight: 1.5791, Grad Total: 0, Output: 0.633273
Children:
Node: add, Weight: 1, Grad Total: 0, Output: 1.5791
Children:
Node: exp, Weight: 0, Grad Total: 0, Output: 0.579098
Children:
Node: negate, Weight: 0, Grad Total: 0, Output: -0.546284
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 0.546284
Children:
Node: add, Weight: 2.38371, Grad Total: 0, Output: 0.546284
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -1.83743
Children:
Node: multiply, Weight: -1.41537, Grad Total: 0, Output: -1.32974
Leaf node
Node: multiply, Weight: 0.404051, Grad Total: 0, Output: 0.361556
Leaf node
Node: multiply, Weight: -1.91877, Grad Total: 0, Output: -1.14723
Leaf node
Node: multiply, Weight: 0.301547, Grad Total: 0, Output: 0.277989
Leaf node
Node: add, Weight: 0, Grad Total: 0, Output: 2.38371
Children:
Node: multiply, Weight: 0.791032, Grad Total: 0, Output: 0.07041
Children:
Node: reciprocal, Weight: 11.2346, Grad Total: 0, Output: 0.0890103
Children:
Node: add, Weight: 1, Grad Total: 0, Output: 11.2346
Children:
Node: exp, Weight: 0, Grad Total: 0, Output: 10.2346
Children:
Node: negate, Weight: 0, Grad Total: 0, Output: 2.32578
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -2.32578
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -2.32578
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -2.32578
Children:
Node: multiply, Weight: -1.41537, Grad Total: 0, Output: -0.773798
Leaf node
Node: multiply, Weight: 0.404051, Grad Total: 0, Output: 0.0746906
Leaf node
Node: multiply, Weight: -1.91877, Grad Total: 0, Output: -1.86041
Leaf node
Node: multiply, Weight: 0.301547, Grad Total: 0, Output: 0.233739
Leaf node
Node: add, Weight: 0, Grad Total: 0, Output: 0
Children:
Node: multiply, Weight: 0.791032, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: 2.19046, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: -1.55066, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: 1.54993, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: 0.22746, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: 2.19046, Grad Total: 0, Output: 1.1314
Children:
Node: reciprocal, Weight: 1.93605, Grad Total: 0, Output: 0.516515
Children:
Node: add, Weight: 1, Grad Total: 0, Output: 1.93605
Children:
Node: exp, Weight: 0, Grad Total: 0, Output: 0.936051
Children:
Node: negate, Weight: 0, Grad Total: 0, Output: -0.0660851
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 0.0660851
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 0.0660851
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 0.0660851
Children:
Node: multiply, Weight: -0.420645, Grad Total: 0, Output: -0.229971
Leaf node
Node: multiply, Weight: 1.88619, Grad Total: 0, Output: 0.34867
Leaf node
Node: multiply, Weight: -0.0265139, Grad Total: 0, Output: -0.0257074
Leaf node
Node: multiply, Weight: -0.0347118, Grad Total: 0, Output: -0.0269062
Leaf node
Node: add, Weight: 0, Grad Total: 0, Output: 0
Children:
Node: multiply, Weight: -0.909387, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: -0.990536, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: 0.068563, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: -0.783253, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: 1.30714, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: -1.55066, Grad Total: 0, Output: -0.416253
Children:
Node: reciprocal, Weight: 3.72529, Grad Total: 0, Output: 0.268435
Children:
Node: add, Weight: 1, Grad Total: 0, Output: 3.72529
Children:
Node: exp, Weight: 0, Grad Total: 0, Output: 2.72529
Children:
Node: negate, Weight: 0, Grad Total: 0, Output: 1.00258
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -1.00258
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -1.00258
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -1.00258
Children:
Node: multiply, Weight: -0.342715, Grad Total: 0, Output: -0.187366
Leaf node
Node: multiply, Weight: 0.174578, Grad Total: 0, Output: 0.0322715
Leaf node
Node: multiply, Weight: 0.0602302, Grad Total: 0, Output: 0.0583983
Leaf node
Node: multiply, Weight: -1.16868, Grad Total: 0, Output: -0.905881
Leaf node
Node: add, Weight: 0, Grad Total: 0, Output: 0
Children:
Node: multiply, Weight: 1.40279, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: -0.566298, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: -1.0623, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: -0.322062, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: -1.60748, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: 1.54993, Grad Total: 0, Output: 1.4678
Children:
Node: reciprocal, Weight: 1.05596, Grad Total: 0, Output: 0.947008
Children:
Node: add, Weight: 1, Grad Total: 0, Output: 1.05596
Children:
Node: exp, Weight: 0, Grad Total: 0, Output: 0.0559578
Children:
Node: negate, Weight: 0, Grad Total: 0, Output: -2.88316
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 2.88316
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 2.88316
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 2.88316
Children:
Node: multiply, Weight: -0.802277, Grad Total: 0, Output: -0.438613
Leaf node
Node: multiply, Weight: 0.25755, Grad Total: 0, Output: 0.0476093
Leaf node
Node: multiply, Weight: 2.46324, Grad Total: 0, Output: 2.38832
Leaf node
Node: multiply, Weight: 1.14282, Grad Total: 0, Output: 0.885839
Leaf node
Node: add, Weight: 0, Grad Total: 0, Output: 0
Children:
Node: multiply, Weight: -1.40185, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: 0.0996514, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: 0.473592, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: 0.813517, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: 0.184634, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: 0.22746, Grad Total: 0, Output: 0.130351
Children:
Node: reciprocal, Weight: 1.74498, Grad Total: 0, Output: 0.573073
Children:
Node: add, Weight: 1, Grad Total: 0, Output: 1.74498
Children:
Node: exp, Weight: 0, Grad Total: 0, Output: 0.744979
Children:
Node: negate, Weight: 0, Grad Total: 0, Output: -0.2944
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 0.2944
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 0.2944
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 0.2944
Children:
Node: multiply, Weight: -0.161286, Grad Total: 0, Output: -0.0881766
Leaf node
Node: multiply, Weight: -0.0744459, Grad Total: 0, Output: -0.0137617
Leaf node
Node: multiply, Weight: -0.192361, Grad Total: 0, Output: -0.18651
Leaf node
Node: multiply, Weight: 0.751933, Grad Total: 0, Output: 0.582848
Leaf node
Node: add, Weight: 0, Grad Total: 0, Output: 0
Children:
Node: multiply, Weight: 0.586857, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: -0.503476, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: -0.919424, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: -1.23086, Grad Total: 0, Output: -0
Leaf node
Node: multiply, Weight: 0.259883, Grad Total: 0, Output: 0
Leaf node
Node: multiply, Weight: 2.19046, Grad Total: 0, Output: 1.45288
Children:
Node: reciprocal, Weight: 1.50766, Grad Total: 0, Output: 0.663278
Children:
Node: add, Weight: 1, Grad Total: 0, Output: 1.50766
Children:
Node: exp, Weight: 0, Grad Total: 0, Output: 0.507664
Children:
Node: negate, Weight: 0, Grad Total: 0, Output: -0.677936
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 0.677936
Children:
Node: add, Weight: -0.566826, Grad Total: 0, Output: 0.677936
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 1.24476
Children:
Node: multiply, Weight: -0.420645, Grad Total: 0, Output: -0.395196
Leaf node
Node: multiply, Weight: 1.88619, Grad Total: 0, Output: 1.68781
Leaf node
Node: multiply, Weight: -0.0265139, Grad Total: 0, Output: -0.0158526
Leaf node
Node: multiply, Weight: -0.0347118, Grad Total: 0, Output: -0.0319999
Leaf node
Node: add, Weight: 0, Grad Total: 0, Output: -0.566826
Children:
Node: multiply, Weight: -0.909387, Grad Total: 0, Output: -0.0809449
Children:
Node: reciprocal (already visited)
Node: multiply, Weight: -0.990536, Grad Total: 0, Output: -0.511627
Children:
Node: reciprocal (already visited)
Node: multiply, Weight: 0.068563, Grad Total: 0, Output: 0.0184047
Children:
Node: reciprocal (already visited)
Node: multiply, Weight: -0.783253, Grad Total: 0, Output: -0.741747
Children:
Node: reciprocal (already visited)
Node: multiply, Weight: 1.30714, Grad Total: 0, Output: 0.749088
Children:
Node: reciprocal (already visited)
Node: multiply, Weight: -1.55066, Grad Total: 0, Output: -0.081943
Children:
Node: reciprocal, Weight: 18.9237, Grad Total: 0, Output: 0.0528438
Children:
Node: add, Weight: 1, Grad Total: 0, Output: 18.9237
Children:
Node: exp, Weight: 0, Grad Total: 0, Output: 17.9237
Children:
Node: negate, Weight: 0, Grad Total: 0, Output: 2.88612
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -2.88612
Children:
Node: add, Weight: -1.679, Grad Total: 0, Output: -2.88612
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -1.20713
Children:
Node: multiply, Weight: -0.342715, Grad Total: 0, Output: -0.32198
Leaf node
Node: multiply, Weight: 0.174578, Grad Total: 0, Output: 0.156217
Leaf node
Node: multiply, Weight: 0.0602302, Grad Total: 0, Output: 0.0360116
Leaf node
Node: multiply, Weight: -1.16868, Grad Total: 0, Output: -1.07737
Leaf node
Node: add, Weight: 0, Grad Total: 0, Output: -1.679
Children:
Node: multiply, Weight: 1.40279, Grad Total: 0, Output: 0.124863
Children:
Node: reciprocal (already visited)
Node: multiply, Weight: -0.566298, Grad Total: 0, Output: -0.292501
Children:
Node: reciprocal (already visited)
Node: multiply, Weight: -1.0623, Grad Total: 0, Output: -0.28516
Children:
Node: reciprocal (already visited)
Node: multiply, Weight: -0.322062, Grad Total: 0, Output: -0.304995
Children:
Node: reciprocal (already visited)
Node: multiply, Weight: -1.60748, Grad Total: 0, Output: -0.921205
Children:
Node: reciprocal (already visited)
Node: multiply, Weight: 1.54993, Grad Total: 0, Output: 1.4716
Children:
Node: reciprocal, Weight: 1.05323, Grad Total: 0, Output: 0.949457
Children:
Node: add, Weight: 1, Grad Total: 0, Output: 1.05323
Children:
Node: exp, Weight: 0, Grad Total: 0, Output: 0.0532332
Children:
Node: negate, Weight: 0, Grad Total: 0, Output: -2.93307
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 2.93307
Children:
Node: add, Weight: 0.930037, Grad Total: 0, Output: 2.93307
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 2.00304
Children:
Node: multiply, Weight: -0.802277, Grad Total: 0, Output: -0.753739
Leaf node
Node: multiply, Weight: 0.25755, Grad Total: 0, Output: 0.230463
Leaf node
Node: multiply, Weight: 2.46324, Grad Total: 0, Output: 1.47277
Leaf node
Node: multiply, Weight: 1.14282, Grad Total: 0, Output: 1.05354
Leaf node
Node: add, Weight: 0, Grad Total: 0, Output: 0.930037
Children:
Node: multiply, Weight: -1.40185, Grad Total: 0, Output: -0.124779
Children:
Node: reciprocal (already visited)
Node: multiply, Weight: 0.0996514, Grad Total: 0, Output: 0.0514715
Children:
Node: reciprocal (already visited)
Node: multiply, Weight: 0.473592, Grad Total: 0, Output: 0.127129
Children:
Node: reciprocal (already visited)
Node: multiply, Weight: 0.813517, Grad Total: 0, Output: 0.770407
Children:
Node: reciprocal (already visited)
Node: multiply, Weight: 0.184634, Grad Total: 0, Output: 0.105809
Children:
Node: reciprocal (already visited)
Node: multiply, Weight: 0.22746, Grad Total: 0, Output: 0.0563257
Children:
Node: reciprocal, Weight: 4.0383, Grad Total: 0, Output: 0.247629
Children:
Node: add, Weight: 1, Grad Total: 0, Output: 4.0383
Children:
Node: exp, Weight: 0, Grad Total: 0, Output: 3.0383
Children:
Node: negate, Weight: 0, Grad Total: 0, Output: 1.1113
Children:
Node: add, Weight: 0, Grad Total: 0, Output: -1.1113
Children:
Node: add, Weight: -1.47133, Grad Total: 0, Output: -1.1113
Children:
Node: add, Weight: 0, Grad Total: 0, Output: 0.360031
Children:
Node: multiply, Weight: -0.161286, Grad Total: 0, Output: -0.151528
Leaf node
Node: multiply, Weight: -0.0744459, Grad Total: 0, Output: -0.0666162
Leaf node
Node: multiply, Weight: -0.192361, Grad Total: 0, Output: -0.115013
Leaf node
Node: multiply, Weight: 0.751933, Grad Total: 0, Output: 0.693188
Leaf node
Node: add, Weight: 0, Grad Total: 0, Output: -1.47133
Children:
Node: multiply, Weight: 0.586857, Grad Total: 0, Output: 0.0522364
Children:
Node: reciprocal (already visited)
Node: multiply, Weight: -0.503476, Grad Total: 0, Output: -0.260053
Children:
Node: reciprocal (already visited)
Node: multiply, Weight: -0.919424, Grad Total: 0, Output: -0.246806
Children:
Node: reciprocal (already visited)
Node: multiply, Weight: -1.23086, Grad Total: 0, Output: -1.16564
Children:
Node: reciprocal (already visited)
Node: multiply, Weight: 0.259883, Grad Total: 0, Output: 0.148932
Children:
Node: reciprocal (already visited)3 번째 time_step 의 첫 번째 hidden_unit 에 대해 연결된 2 번째 time_step 의 첫 번째 hidden_unit 아래에만 이러한 연결이 제대로 되어 있고, 2 번째 time_step 의 두 번째 hidden_unit 에는 그 하위 정보들이 저장되어 있지 않음
// 순환 구조 연결: 현재 타임스텝에서 이전 타임스텝의 activation_nodes와 연결
// 이것만 잘 수정하면 돼
if (t > 0) { // 첫 번째 타임스텝 제외
for (int u = 0; u < units; u++) {
// 현재 유닛의 recurrent_multiply_node에서 리프 노드를 가져옴
auto recurrent_leaf_nodes = recurrent_multiply_node_list[u]->find_leaf_nodes();
// 리프 노드의 수와 previous_activation_nodes의 크기를 비교하여 반복
int num_leaf_nodes = recurrent_leaf_nodes.size();
if (num_leaf_nodes != units) {
std::cerr << "Error: Leaf nodes size mismatch. Expected: " << units << ", Found: " << num_leaf_nodes << std::endl;
throw std::runtime_error("Leaf nodes size mismatch.");
}
for (int j = 0; j < units; j++) {
// 이전 타임스텝의 activation_nodes[j]를 현재 타임스텝의 리프 노드들과 연결
recurrent_leaf_nodes[j]->add_child(previous_activation_nodes[j]);
previous_activation_nodes[j]->add_parent(recurrent_leaf_nodes[j]);
}
}
}유닛을 연결하는 코드를 다시 살펴보면, 2중 for 문으로 구현해서 3번째 time_step 이후에는 계산 그래프의 연결을 제대로 못하는 것을 확인할 수 있었음
재귀로 구현해야 함
'dev_AI_framework' 카테고리의 다른 글
| batch_data 의 처리 (1) | 2024.10.21 |
|---|---|
| RNN 계산 그래프 수정 - recurrent 를 recursion 으로! (0) | 2024.10.21 |
| RNN 계산 그래프 구현 (3) | 2024.10.14 |
| pooling 연산의 구현 (0) | 2024.10.04 |
| layer 쌓기 add_layer 에서 가변 인수로의 확장 수행 (0) | 2024.09.30 |