The Practice to Monitor Sharing-Sphere Application Performance
Sharding-Sphere is an ecosystem of open source distributed database middleware solutions that provides multiple forms of solutions for distributed data storage. Sharding-JDBC, a sub-project of Sharding-Sphere, has been widely used in the technical architecture of Orange Finance and has been one of the most important solutions in the industry. During using Sharding-JDBC, performance test and operational performance monitoring are absolutely necessary. As strong supporters of Sharding-Sphere, we — a team from Technology Innovation Center of Orange Finance — feel glad to be responsible for monitoring Sharing-JDBC application performance. This article is mainly about the exploration and practice of the combination of Sharding-Sphere and APM (Application Performance Monitor).
- Introduction of Sharding-Opentracing
Sharding-Opentracing is an Opentracing Specification-based APM toolkit provided by Sharding-Sphere. It adopts publish-subscribe-style communication between components to trace and record the core link methods, including SQL parsing routing method, SQL Query Execution method, and ResultSet Merging method.
The Opentracing (https://github.com/opentracing) specification was created to address the API incompatibilities in different distributed tracing systems. It is a lightweight normalization layer between applications/class libraries and tracing/log analysis programs.
Sharding-Sphere uses Opentracing to trace methods, making it easy for system developers to add or replace tracing system. Tracer injection can be done easily through using the ShardingJDBCTracer class.
public final class ShardingJDBCTracer {
/**
* 通过读取系统属性进行注册
* -Dshardingjdbc.opentracing.tracer.class=*****
*/
public static void init() {...}
/**
* 通过方法参数进行注册
*/
public static void init(final Tracer tracer) {...}
}It then subscribes core method events through event listener, completes method interception and implements method tracing. The implemented event listeners are as follows:
SqlRoutingEventListener
ExecuteEventListener
MergeEventListener
2. Performance Tracing Based on Opentracing Specification — Practice of SkyWalking
SkyWalking (https://github.com/apache/incubator-skywalking) is an open source APM system for distributed systems created by Wu Sheng, which supports Opentracing Specification. Next we will talk about how to use the Sharding-Opentracing toolkit in applications through SkyWalking.
(1) Introduce related dependencies into application system:
<dependency>
<groupId>io.shardingsphere</groupId>
<artifactId>sharding-opentracing</artifactId>
<version>3.0.0.M2-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.apache.skywalking</groupId>
<artifactId>apm-toolkit-opentracing</artifactId>
<version>5.0.0-beta</version>
</dependency>(2) SkyWalkingTracer in apm-toolkit-opentracing package of SkyWalking implements interface Tracer, completing the injection of Tracer when application starts.
ShardingJDBCTracer.init(new SkywalkingTracer());SkyWalking access is completed in two simple steps. You can check the call chain information of service request through Trace view of SkyWalking, including execution information of Sharding-JDBC on SQL Routing, SQL Execution, and ResultSet Merging. SQL Execution adopts a strategy, namely parallel execution of multiple SQL. Here is the execution condition of each thread.
You can see the related parameter information of each tracing Span through the Span Info view of SkyWalking, including route library information, SQL parsing statements, execution and incoming parameters, etc.
3. Performance Tracing Based on Bytecode Enhancement — Pinpoint Practice
We have learned how to use Opentracing Specification-based APM framework to monitor the performance of Sharding-JDBC. How do we get started with the APM framework that uses the bytecode enhancement technology without implementing Opentracing Specification? As for the APM framework using bytecode enhancement technology, our strategy is to write corresponding plugin to enhance the core method of Sharding-Sphere. Let’s look at how to write a plugin for the Sharding-Sphere component in the Pinpoint framework.
Implement the Interceptor interface and complete the component method interceptor. Start tracing before the component method is executed and stop tracing after the component method is executed. Here is the implementation of ResultSet Merging method tracing.
public class ResultSetMergeInterceptor implements AroundInterceptor1 {
private final TraceContext traceContext;
private final MethodDescriptor descriptor;
private final PLogger logger = PLoggerFactory.getLogger(getClass());
public ResultSetMergeInterceptor(TraceContext traceContext, MethodDescriptor descriptor) {
this.traceContext = traceContext;
this.descriptor = descriptor;
}
@Override
public void before(Object target, Object arg0) {
if (logger.isDebugEnabled()) {
logger.beforeInterceptor(target, new Object[]{arg0});
}
final Trace trace = traceContext.currentTraceObject();
if (trace == null) {
return;
}
trace.traceBlockBegin();
}
@Override
public void after(Object target, Object arg0, Object result, Throwable throwable) {
if (logger.isDebugEnabled()) {
logger.afterInterceptor(target, new Object[]{arg0}, result, throwable);
}
final Trace trace = traceContext.currentTraceObject();
if (trace == null) {
return;
}
try {
SpanEventRecorder recorder = trace.currentSpanEventRecorder();
recorder.recordServiceType(ShardingSphereConstants.SHARDING_SPHERE_MERGE);
recorder.recordApi(descriptor);
recorder.recordException(throwable);
} finally {
trace.traceBlockEnd();
}
}
}Implement ProfilerPlugin interface and TransformTemplateAware interface and complete the registration of plugin ID and transform template. The binding between tracing method and method interceptor should be completed through the transform method of TransformTemplate. For example, the merge method of ShardingPreparedStatement class should be bound to the ResultSetMergeInterceptor implemented above.
public class ShardingSpherePlugin implements ProfilerPlugin, TransformTemplateAware {
private final PLogger logger = PLoggerFactory.getLogger(this.getClass());
private static final String SHARDINGSPHERE_SCOPE = "SHARDINGSPHERE_SCOPE";
private TransformTemplate transformTemplate;
@Override
public void setup(ProfilerPluginSetupContext context) {
ShardingSphereConfig config = new ShardingSphereConfig(context.getConfig());
logger.debug("[ShardingSphere] pluginEnable={}", config.isPluginEnable());
if (config.isPluginEnable()) {
addSqlRouteTransformer();
addSqlExecutorTransformer();
addResultMergeTransformer();
}
}
@Override
public void setTransformTemplate(TransformTemplate transformTemplate) {
this.transformTemplate = transformTemplate;
}
private void addResultMergeTransformer() {
TransformCallback transformCallback = new TransformCallback() {
@Override
public byte[] doInTransform(Instrumentor instrumentor, ClassLoader classLoader, String className, Class<?> classBeingRedefined, ProtectionDomain protectionDomain, byte[] classfileBuffer) throws InstrumentException {
InstrumentClass target = instrumentor.getInstrumentClass(classLoader, className, classfileBuffer);
InstrumentMethod method = target.getDeclaredMethod("merge", "io.shardingsphere.core.merger.MergeEngine");
method.addScopedInterceptor("com.navercorp.pinpoint.plugin.shardingsphere.interceptor.ResultSetMergeInterceptor", SHARDINGSPHERE_SCOPE);
return target.toBytecode();
}
};
transformTemplate.transform("io.shardingsphere.core.jdbc.core.statement.ShardingPreparedStatement", transformCallback);
transformTemplate.transform("io.shardingsphere.core.jdbc.core.statement.ShardingStatement", transformCallback);
}
}For complete plugin code, you can refer to：
https://github.com/beckhampu/pinpoint/tree/sharding-sphere-1.7.2/plugins
We can clearly see the effect of the Sharding-JDBC core method in a service process through CallTree of Pinpoint.
What I have introduced is related practice of Sharding-Sphere and APM, and I hope you have got something from this article. If you have any ideas, comments and suggestions, please leave a message and communicate with me. What’s more, we warmly welcome you to contribute codes to make Sharding-Sphere better and better.
Sharding-Sphere, consisting of three independent products, namely Sharding-JDBC, Sharding-Proxy, and Sharding-Sidecar, is an ecosystem of open source distributed database middleware solutions. They all provide standardized data sharding, read-write splitting, flexible transactions and data management functions, which can be applied to various application scenarios such as Java isomorphic and heterogeneous languages, containers and cloud natives.
For more information, you could visit our new website:
or Twitter: @ShardingSphere
If you like it, please give us a star on Github as encouragement.
The project address:
https://github.com/sharding-sphere/sharding-sphere/
https://gitee.com/sharding-sphere/sharding-sphere/
Thank you very much.
