記 Spring Cloud Gateway 內存溢出查詢過程
環境配置：
- org.springframework.boot ： 2.1.4.RELEASE
- org.springframework.cloud ：Greenwich.SR1
事故記錄：
由于網關存在 RequestBody 丟失的情況，顧采用了網上的通用解決方案，使用如下方式解決：
|
1
2
3
4
5
6
7
8
9
10
11
|
@Beanpublic RouteLocator tpauditRoutes(RouteLocatorBuilder builder) { return builder.routes().route("gateway-post", r -> r.order(1) .method(HttpMethod.POST) .and() .readBody(String.class, requestBody -> {return true;}) # 重點在這 .and() .path("/gateway/**") .filters(f -> {f.stripPrefix(1);return f;}) .uri("lb://APP-API")).build();} |
測試環境，Spring Cloud Gateway 網關功能編寫完成。開始進行測試環境壓測。
正常采用梯度壓測方式，最高用戶峰值設置為400并發。經歷兩輪時長10分鐘左右壓測，沒有異常情況出現。
中午吃飯時間，設置了1個小時的時間進行測試。
回來的時候系統報出如下異常
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
2019-08-12 15:06:07,296 1092208 [reactor-http-server-epoll-12] WARN io.netty.channel.AbstractChannelHandlerContext.warn:146 - An exception '{}' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 503316487, max: 504889344) at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:640) at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:594) at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:764) at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:740) at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244) at io.netty.buffer.PoolArena.allocate(PoolArena.java:214) at io.netty.buffer.PoolArena.allocate(PoolArena.java:146) at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:324) at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185) at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176) at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:137) at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114) at io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:72) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:793) at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:382) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:315) at io. |
當時一臉懵逼，馬上開始監控 Jvm 堆棧，減少jvm的內存空間，提升并發數以后，重啟項目重新壓測，
項目啟動參數如下：
|
1
2
3
|
java -jar -Xmx1024M /opt/deploy/gateway-appapi/cloud-employ-gateway-0.0.5-SNAPSHOT.jar↓↓↓↓修改為↓↓↓↓java -jar -Xmx512M /opt/deploy/gateway-appapi/cloud-employ-gateway-0.0.5-SNAPSHOT.jar |
縮減了一半內存啟動，等待問題復現。等待3分鐘問題再次復現，但是同時Jvm卻的進行了Full GC。
|
1
2
3
|
EC EU OC OU MC MU CCSC CCSU YGC YGCT FGC FGCT 275456.0 100103.0 484864.0 50280.2 67672.0 64001.3 9088.0 8463.2 501 11.945 3 0.262275968.0 25072.3 484864.0 47329.3 67672.0 63959.4 9088.0 8448.8 502 11.970 4 0.429 |
沒錯，在出現問題的時候，系統出現了Full Gc，但是OU并沒有達到觸發的原因。
結合日志中的 direct memory，想到了Jvm 中的堆外內存。
使用 -XX:MaxDirectMemorySize 可以進行設置 Jvm 堆外內存大小，當 Direct ByteBuffer 分配的堆外內存到達指定大小后，即觸發Full GC。
該值是有上限的，默認是64M，最大為 sun.misc.VM.maxDirectMemory()。
結合所有情況，表明堆外內存使用存在內存溢出的情況。
報錯內容為Netty框架，新增以下配置，開啟Netty錯誤日志打印：
|
1
2
|
-Dio.netty.leakDetection.targetRecords=40 #設置Records 上限-Dio.netty.leakDetection.level=advanced #設置日志級別 |
項目啟動，沒任何問題，開啟壓測后服務報出如下異常：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
|
2019-08-13 14:59:01,656 18047 [reactor-http-nio-7] ERROR io.netty.util.ResourceLeakDetector.reportTracedLeak:317 - LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.Recent access records: #1: org.springframework.core.io.buffer.NettyDataBuffer.release(NettyDataBuffer.java:301) org.springframework.core.io.buffer.DataBufferUtils.release(DataBufferUtils.java:420) org.springframework.core.codec.StringDecoder.decodeDataBuffer(StringDecoder.java:208) org.springframework.core.codec.StringDecoder.decodeDataBuffer(StringDecoder.java:59) org.springframework.core.codec.AbstractDataBufferDecoder.lambda$decodeToMono$1(AbstractDataBufferDecoder.java:68) reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:107) reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:103) reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:287) reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:331) reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1505) reactor.core.publisher.MonoCollectList$MonoBufferAllSubscriber.onComplete(MonoCollectList.java:123) reactor.core.publisher.FluxJust$WeakScalarSubscription.request(FluxJust.java:101) reactor.core.publisher.MonoCollectList$MonoBufferAllSubscriber.onSubscribe(MonoCollectList.java:90) reactor.core.publisher.FluxJust.subscribe(FluxJust.java:70) reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54) reactor.core.publisher.MonoCollectList.subscribe(MonoCollectList.java:59) reactor.core.publisher.MonoFilterFuseable.subscribe(MonoFilterFuseable.java:44) reactor.core.publisher.MonoMapFuseable.subscribe(MonoMapFuseable.java:56) reactor.core.publisher.MonoSubscriberContext.subscribe(MonoSubscriberContext.java:47) reactor.core.publisher.MonoMapFuseable.subscribe(MonoMapFuseable.java:59) reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44) reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44) reactor.core.publisher.MonoPeek.subscribe(MonoPeek.java:71) reactor.core.publisher.MonoMap.subscribe(MonoMap.java:55) reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150) reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:103) reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:287) reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:331) reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1505) reactor.core.publisher.MonoCollectList$MonoBufferAllSubscriber.onComplete(MonoCollectList.java:123) reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136) reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:252) reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136) reactor.netty.channel.FluxReceive.terminateReceiver(FluxReceive.java:372) reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:196) reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:337) reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:333) reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:453) reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:141) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337) reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:191) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337) io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:677) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:612) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:529) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:491) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) java.lang.Thread.run(Unknown Source)#2: io.netty.buffer.AdvancedLeakAwareByteBuf.nioBuffer(AdvancedLeakAwareByteBuf.java:712) org.springframework.core.io.buffer.NettyDataBuffer.asByteBuffer(NettyDataBuffer.java:266) org.springframework.core.codec.StringDecoder.decodeDataBuffer(StringDecoder.java:207) org.springframework.core.codec.StringDecoder.decodeDataBuffer(StringDecoder.java:59) org.springframework.core.codec.AbstractDataBufferDecoder.lambda$decodeToMono$1(AbstractDataBufferDecoder.java:68) reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:107) reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:103) reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:287) reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:331) reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1505) reactor.core.publisher.MonoCollectList$MonoBufferAllSubscriber.onComplete(MonoCollectList.java:123) reactor.core.publisher.FluxJust$WeakScalarSubscription.request(FluxJust.java:101) reactor.core.publisher.MonoCollectList$MonoBufferAllSubscriber.onSubscribe(MonoCollectList.java:90) reactor.core.publisher.FluxJust.subscribe(FluxJust.java:70) reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54) reactor.core.publisher.MonoCollectList.subscribe(MonoCollectList.java:59) reactor.core.publisher.MonoFilterFuseable.subscribe(MonoFilterFuseable.java:44) reactor.core.publisher.MonoMapFuseable.subscribe(MonoMapFuseable.java:56) reactor.core.publisher.MonoSubscriberContext.subscribe(MonoSubscriberContext.java:47) reactor.core.publisher.MonoMapFuseable.subscribe(MonoMapFuseable.java:59) reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44) reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44) reactor.core.publisher.MonoPeek.subscribe(MonoPeek.java:71) reactor.core.publisher.MonoMap.subscribe(MonoMap.java:55) reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150) reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:103) reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:287) reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:331) reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1505) reactor.core.publisher.MonoCollectList$MonoBufferAllSubscriber.onComplete(MonoCollectList.java:123) reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136) reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:252) reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136) reactor.netty.channel.FluxReceive.terminateReceiver(FluxReceive.java:372) reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:196) reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:337) reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:333) reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:453) reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:141) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337) reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:191) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337) io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:677) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:612) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:529) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:491) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) java.lang.Thread.run(Unknown Source)#3: io.netty.buffer.AdvancedLeakAwareByteBuf.slice(AdvancedLeakAwareByteBuf.java:82) org.springframework.core.io.buffer.NettyDataBuffer.slice(NettyDataBuffer.java:260) org.springframework.core.io.buffer.NettyDataBuffer.slice(NettyDataBuffer.java:42) org.springframework.cloud.gateway.handler.predicate.ReadBodyPredicateFactory.lambda$null$0(ReadBodyPredicateFactory.java:102) reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:46) reactor.core.publisher.MonoCollectList.subscribe(MonoCollectList.java:59) reactor.core.publisher.MonoFilterFuseable.subscribe(MonoFilterFuseable.java:44) reactor.core.publisher.MonoMapFuseable.subscribe(MonoMapFuseable.java:56) reactor.core.publisher.MonoSubscriberContext.subscribe(MonoSubscriberContext.java:47) reactor.core.publisher.MonoMapFuseable.subscribe(MonoMapFuseable.java:59) reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44) reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44) reactor.core.publisher.MonoPeek.subscribe(MonoPeek.java:71) reactor.core.publisher.MonoMap.subscribe(MonoMap.java:55) reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150) reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:103) reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:287) reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:331) reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1505) reactor.core.publisher.MonoCollectList$MonoBufferAllSubscriber.onComplete(MonoCollectList.java:123) reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136) reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:252) reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136) reactor.netty.channel.FluxReceive.terminateReceiver(FluxReceive.java:372) reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:196) reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:337) reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:333) reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:453) reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:141) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337) reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:191) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337) io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:677) io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:612) io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:529) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:491) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) java.lang.Thread.run(Unknown Source) |
在 #3 中，我發現了一個眼熟的類，ReadBodyPredicateFactory.java ，還記得最開始的時候使用 readbody 配置么？
這里就是進行 cachedRequestBodyObject 的寫入類，
追蹤一下Readbody源碼
|
1
2
3
4
5
6
7
8
9
10
11
12
|
/** * This predicate is BETA and may be subject to change in a future release. A * predicate that checks the contents of the request body * @param inClass the class to parse the body to * @param predicate a predicate to check the contents of the body * @param <T> the type the body is parsed to * @return a {@link BooleanSpec} to be used to add logical operators */public <T> BooleanSpec readBody(Class<T> inClass, Predicate<T> predicate) { return asyncPredicate(getBean(ReadBodyPredicateFactory.class) .applyAsync(c -> c.setPredicate(inClass, predicate)));} |
異步調用的 ReadBodyPredicateFactory.applyAsync() 和 錯誤日志中的
|
1
|
org.springframework.cloud.gateway.handler.predicate.ReadBodyPredicateFactory.lambda$null$0(ReadBodyPredicateFactory.java:102) |
指向方法一致。查看源碼102行：
|
1
2
3
|
Flux<DataBuffer> cachedFlux = Flux.defer(() -> Flux.just(dataBuffer.slice(0, dataBuffer.readableByteCount()))); |
此處 Spring Cloud Gateway 通過 dataBuffer.slice 切割出了新的 dataBuffer，但是通過 Netty 的內存檢測工具判斷，此處的 dataBuffer 并沒有被回收。
錯誤如下，日志很多容易被忽視。
ERROR io.netty.util.ResourceLeakDetector.reportTracedLeak:317 - LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
找到問題那就要解決才行，嘗試修改源碼
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
|
@Override@SuppressWarnings("unchecked")public AsyncPredicate<ServerWebExchange> applyAsync(Config config) { return exchange -> { Class inClass = config.getInClass(); Object cachedBody = exchange.getAttribute(CACHE_REQUEST_BODY_OBJECT_KEY); Mono<?> modifiedBody; // We can only read the body from the request once, once that // happens if we // try to read the body again an exception will be thrown. The below // if/else // caches the body object as a request attribute in the // ServerWebExchange // so if this filter is run more than once (due to more than one // route // using it) we do not try to read the request body multiple times if (cachedBody != null) { try { boolean test = config.predicate.test(cachedBody); exchange.getAttributes().put(TEST_ATTRIBUTE, test); return Mono.just(test); } catch (ClassCastException e) { if (LOGGER.isDebugEnabled()) { LOGGER.debug("Predicate test failed because class in predicate " + "does not match the cached body object", e); } } return Mono.just(false); } else { // Join all the DataBuffers so we have a single DataBuffer for // the body return DataBufferUtils.join(exchange.getRequest().getBody()).flatMap(dataBuffer -> { // Update the retain counts so we can read the body twice, // once to parse into an object // that we can test the predicate against and a second time // when the HTTP client sends // the request downstream // Note: if we end up reading the body twice we will run // into // a problem, but as of right // now there is no good use case for doing this DataBufferUtils.retain(dataBuffer); // Make a slice for each read so each read has its own // read/write indexes Flux<DataBuffer> cachedFlux = Flux .defer(() -> Flux.just(dataBuffer.slice(0, dataBuffer.readableByteCount()))); ServerHttpRequest mutatedRequest = new ServerHttpRequestDecorator(exchange.getRequest()) { @Override public Flux<DataBuffer> getBody() { return cachedFlux; } }; # 新增如下代碼 DataBufferUtils.release(dataBuffer); return ServerRequest.create(exchange.mutate().request(mutatedRequest).build(), messageReaders) .bodyToMono(inClass).doOnNext(objectValue -> { exchange.getAttributes().put(CACHE_REQUEST_BODY_OBJECT_KEY, objectValue); exchange.getAttributes().put(CACHED_REQUEST_BODY_KEY, cachedFlux); }).map(objectValue -> config.predicate.test(objectValue)); }); } };} |
Spring Cloud Gateway 在配置的架構中，版本為2.1.1，修改以上代碼后，啟動項目測試，問題沒有復現，正常運行。
同樣這個問題，也可以選擇升級 Spring Cloud Gateway 版本，在官方2.1.2版本中，此處代碼已被重構，升級后測試也完全正常。
以上為個人經驗，希望能給大家一個參考，也希望大家多多支持服務器之家。
原文鏈接：https://blog.csdn.net/live501837145/article/details/99446673
