基于腾讯云CLS封装日志starter组件

基于腾讯云CLS封装日志starter组件

杰子学编程 117 2022-06-30

基于腾讯云CLS封装日志starter组件

目的

​ 为公司业务日志打点记录,完善调用连信息,封装统一日志组件,各业务系统服务引用该组件,通过注解方式记录日志,日志可写入系统日志文件也支持写入腾讯云日志服务cls.

使用方式

2.1、在工程pom文件中增加以下依赖:

<dependency>
    <groupId>com.jdd</groupId>
    <artifactId>spring-boot-starter-jdd-log</artifactId>
    <version>1.0.0</version>
</dependency>

注:需配置公司私服

2.2、在工程配置文件中增加以下配置:

jdd:
  log:
    endpoint: ap-nanjing.cls.tencentcs.com
    secretId: XX
    secretKey: XX
    topicId: 69041ade-9177-49ec-97c0-95bd63db25ee 可联系运维获取
    instance: cls 日志记录方式:cls 为腾讯云日志存储,file 为项目日志存储

2.3、使用日志注解

 @GetMapping("/query")
 @AuditLog(module = "应用管理", action = "应用查询")
 public ObjResponse<ResultPage<AppModel>> query(AppModel appModel) {
     return ObjResponse.of(appService.query(appModel));
 }

​ 在需要增加日志的Controller中增加@AuditLog注解。其中module为服务模块名称,action为方法动作名称。

2.4、日志记录格式

日志记录为json格式,记录内容如下:

{
  "schema": "http",
  "targetServer": "pay",
  "requestTime": "2022-06-30 10:23:29",
  "code": "200",
  "requestBody": "{\"accounts\":[\"600202206280004\"]}",
  "responseTime": "2022-06-30 10:23:29",
  "ip": "119.29.81.32, 10.206.111.93",
  "requestMethod": "POST",
  "responseData": "{\"code\":200,\"message\":\"成功\",\"data\":[{\"keyword\":null,\"startCreateTime\":null,\"endCreateTime\":null,\"createTime\":\"2022-06-28 16:22:11\",\"createUser\":null,\"updateTime\":\"2022-06-28 16:22:12\",\"updateUser\":null,\"logicDel\":false,\"enterpriseId\":206667,\"channelNo\":null,\"createUserName\":null,\"id\":41,\"account\":\"600202206280004\",\"enterpriseCode\":\"18004412661\",\"enterpriseName\":\"小猫咪\",\"quota\":20000,\"useQuota\":-770800,\"availableQuota\":790800,\"status\":1,\"applyUserId\":1082,\"applyUserName\":\"x x\",\"expansion\":null,\"billingDate\":30,\"billingCycle\":30}]}",
  "message": "成功",
  "requestPath": "/pay/credit/getAccountsByCodes",
  "executeTime": "39",
  "__SOURCE__": "10.206.111.113",
  "__FILENAME__": "",
  "__HOSTNAME__": "",
  "__PKG_LOGID__": 262146
}

截图

该日志可在腾讯云中搜索查看。

组件实现

3.1、创建配置文件类

创建配置文件类,接收配置文件信息,主要内容如下:

@Data
@ConfigurationProperties(prefix = "jdd.log")
public class LogProperties {
    /**
     * 请求地址
     */
    private String endpoint;
    /**
     * secretId
     */
    private String secretId;
    /**
     * secretKey
     */
    private String secretKey;
    /**
     * topicId
     */
    private String topicId;
    /**
     * 服务实例:
     * file-本地文件
     * cls-腾讯云日志服务
     */
    private String instance;
}

3.2、创建配置类对象

创建配置类对象JddLogAntoConfiguration,该配置主要进行对象实例化操作。

@Slf4j
@Data
@Configuration
@EnableConfigurationProperties({LogProperties.class})
public class JddLogAntoConfiguration {
    /**
     * 本地日志文件
     */
    private static final String FILE = "file";
    /**
     * 腾讯云存储
     */
    private static final String CLS = "cls";

    private final Map<String, SaveLogService> logServiceMap = new HashMap<>();

    private final LogProperties logProperties;

    public JddLogAntoConfiguration(LogProperties logProperties) {
        this.logProperties = logProperties;
    }

    /**
     * 初始化logService实现
     */
    @PostConstruct
    public void init() {
        log.info("JddLogAntoConfiguration init");
        Assert.notNull(logProperties.getEndpoint(), "endpoint is not null !");
        Assert.notNull(logProperties.getSecretId(), "secretId is not null !");
        Assert.notNull(logProperties.getSecretKey(), "secretKey is not null !");
        Assert.notNull(logProperties.getTopicId(), "topicId is not null !");
        logServiceMap.put(FILE, new FileSaveLogServiceImpl());
        logServiceMap.put(CLS, new ClsSaveLogServiceImpl(logProperties));
    }

    /**
     * 根据配置文件内容不同选择不同的实现类对象
     *
     * @param logProperties 配置文件
     * @return
     */
    @Bean
    public SaveLogService getSaveLogService(LogProperties logProperties) {
        String instance = logProperties.getInstance();
        log.info("getSaveLogService bean is start");
        if (Objects.isNull(instance)) {
            throw new RuntimeException("logProperties instance not null !");
        }
        return logServiceMap.get(logProperties.getInstance());
    }

}

3.3、自定义日志注解

这里我们进行自定义注解类AuditLog,后续主要通过该注解进行日志打点。

/**
 * 在controller方法加上该注解,指定模块名称、操作名称,主要用来记录用户对系统更新相关的操作,对于重要的查询也可以加上
 *
 * @author wanghongjie
 */
@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface AuditLog {
    /**
     * 模块名称
     *
     * @return
     */
    String module() default "";

    /**
     * 操作名称
     *
     * @return
     */
    String action() default "";
}

3.4、创建注解实现类(切面类)

这里我们通过切面方式实现,扫描AuditLog注解,获取标记该注解的方法信息,包含请求参数和响应参数。

@Component
@Aspect
@Slf4j
public class AuditLogAspect implements ApplicationContextAware {
    private static final String UNKNOWN = "unknown";
    private static final String LOCALHOST = "127.0.0.1";
    private static final String SEPARATOR = ",";
    @Resource
    private SaveLogService logService;

    private ApplicationContext applicationContext;

    @Pointcut("@annotation(com.jdd.starter.log.annotation.AuditLog)")
    public void auditRecord() {
    }

    @Around(value = "auditRecord()")
    public Object record(ProceedingJoinPoint joinPoint) throws Throwable {
        Environment environment = applicationContext.getBean(Environment.class);
        long startTime = System.currentTimeMillis();
        LogBean webLog = new LogBean();
        Object result = null;
        try {
            // 获取访问的方法
            Method method = ((MethodSignature) joinPoint.getSignature()).getMethod();
            // 获取请求对象
            HttpServletRequest request = ((ServletRequestAttributes) Objects.requireNonNull(RequestContextHolder.getRequestAttributes())).getRequest();
            // 获取自定义参数
            AuditLog auditLog = method.getAnnotation(AuditLog.class);
            // 收集请求和响应数据
            String ipAddress = getIpAddr(request);
            webLog.setIp(ipAddress);
            webLog.setStartTime(new Date(startTime));
            webLog.setMethod(request.getMethod());
            webLog.setParameter(getParameter(method, joinPoint.getArgs()));
            webLog.setUrl(URLUtil.getPath(request.getRequestURI()));
            webLog.setModule(auditLog.module());
            webLog.setAction(auditLog.action());
            result = joinPoint.proceed();
            webLog.setResult(JSONUtil.toJsonStr(result));
            webLog.setExecuteSuccess(true);
            webLog.setSpendTime((int) (System.currentTimeMillis() - startTime));
            webLog.setApplicationName(environment.getProperty("spring.application.name"));
            webLog.setEnvironment(environment.getProperty("spring.profiles.active"));
        } catch (Exception e) {
            webLog.setResult(e.getMessage());
            webLog.setExecuteSuccess(false);
            webLog.setSpendTime((int) (System.currentTimeMillis() - startTime));
            throw e;
        } finally {
            logService.saveLog(webLog);
        }
        return result;
    }

    /**
     * 根据方法和传入的参数获取请求参数
     */
    private String getParameter(Method method, Object[] args) {
        List<Object> argList = new ArrayList<>();
        Parameter[] parameters = method.getParameters();
        for (int i = 0; i < parameters.length; i++) {
            //将RequestBody注解修饰的参数作为请求参数
            RequestBody requestBody = parameters[i].getAnnotation(RequestBody.class);
            //将RequestParam注解修饰的参数作为请求参数
            RequestParam requestParam = parameters[i].getAnnotation(RequestParam.class);
            if (requestBody != null) {
                argList.add(args[i]);
            } else if (requestParam != null) {
                Map<String, Object> map = new HashMap<>();
                String key = parameters[i].getName();
                if (!StringUtils.isEmpty(requestParam.value())) {
                    key = requestParam.value();
                }
                map.put(key, args[i]);
                argList.add(map);
            } else {
                argList.add(args[i]);
            }
        }
        if (argList.size() == 0) {
            return null;
        } else if (argList.size() == 1) {
            return JSONUtil.toJsonStr(argList.get(0));
        } else {
            return JSONUtil.toJsonStr(argList);
        }
    }


    public static String getIpAddr(HttpServletRequest request) {
        String ipAddress;
        try {
            ipAddress = request.getHeader("x-forwarded-for");
            if (ipAddress == null || ipAddress.length() == 0 || UNKNOWN.equalsIgnoreCase(ipAddress)) {
                ipAddress = request.getHeader("Proxy-Client-IP");
            }
            if (ipAddress == null || ipAddress.length() == 0 || UNKNOWN.equalsIgnoreCase(ipAddress)) {
                ipAddress = request.getHeader("WL-Proxy-Client-IP");
            }
            if (ipAddress == null || ipAddress.length() == 0 || UNKNOWN.equalsIgnoreCase(ipAddress)) {
                ipAddress = request.getRemoteAddr();
                if (LOCALHOST.equals(ipAddress)) {
                    InetAddress inet = null;
                    try {
                        inet = InetAddress.getLocalHost();
                    } catch (UnknownHostException e) {
                        e.printStackTrace();
                    }
                    ipAddress = inet.getHostAddress();
                }
            }
            if (ipAddress != null && ipAddress.length() > 15) {
                if (ipAddress.indexOf(SEPARATOR) > 0) {
                    ipAddress = ipAddress.substring(0, ipAddress.indexOf(","));
                }
            }
        } catch (Exception e) {
            ipAddress = "";
        }
        return ipAddress;
    }

    @Override
    public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
        this.applicationContext = applicationContext;
    }
}

主要数据采集全在AuditLogAspect中,包括获取IP地址等,通过LogBean对象,构建数据结构,通过logService.saveLog()将日志信息存储。

logService通过JddLogAntoConfiguration配置实现。

LogBean对象内容如下:

@Data
public class LogBean implements Serializable {
    /**
     * 操作时间
     */
    private Date startTime;

    /**
     * 消耗时间
     */
    private Integer spendTime;

    /**
     * URL
     */
    private String url;

    /**
     * 请求类型
     */
    private String method;

    /**
     * IP地址
     */
    private String ip;

    /**
     * 归属地
     */
    private String address;

    /**
     * 业务模块
     */
    private String module;

    /**
     * 操作类型
     */
    private String action;

    /**
     * 执行结果:成功or失败
     */
    private boolean isExecuteSuccess;

    /**
     * 请求参数
     */
    private String parameter;

    /**
     * 返回结果
     */
    private String result;

    /**
     * 创建时间
     */
    private Date createTime;

    /**
     * 操作用户
     */
    private String username;
    /**
     * 创建人
     */
    private Integer createUser;
    /**
     * 更新时间
     */
    private Date updateTime;
    /**
     * 更新人
     */
    private Integer updateUser;
    /**
     * 客户名称
     */
    private String enterpriseName;

    /**
     * 客户ID
     */
    private Long enterpriseId;
    /**
     * 渠道编号
     */
    private String channelNo;
    /**
     * 日志采集环境
     */
    private String environment;
    /**
     * 应用名称
     */
    private String applicationName;
}

3.5、logService实现类

logService的实现类有目前实现了两个,分别是ClsSaveLogServiceImplFileSaveLogServiceImpl,分别对应配置文件中的cls和file。目前暂时不支持双写。

  • ClsSaveLogServiceImpl实现类:
@Slf4j
@Service
public class ClsSaveLogServiceImpl implements SaveLogService {

    private final LogProperties logProperties;
    private AsyncProducerClient client;

    public ClsSaveLogServiceImpl(LogProperties logProperties) {
        this.logProperties = logProperties;
        init();
    }

    private void init() {
        final AsyncProducerConfig config = new AsyncProducerConfig(logProperties.getEndpoint(), logProperties.getSecretId(), logProperties.getSecretKey(), NetworkUtils.getLocalMachineIP());
        // 构建一个客户端实例
        client = new AsyncProducerClient(config);
        log.info("spring-boot-starter-jdd-log is init , topic is {}", logProperties.getTopicId());
    }

    @Override
    public void saveLog(LogBean logBean) {
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        List<LogItem> logItems = new ArrayList<>();
        int ts = (int) (System.currentTimeMillis() / 1000);
        LogItem logItem = new LogItem(ts);
        logItem.PushBack(new LogContent("status", notNull(String.valueOf(logBean.isExecuteSuccess()))));
        logItem.PushBack(new LogContent("method", notNull(logBean.getMethod())));
        logItem.PushBack(new LogContent("action", notNull(logBean.getAction())));
        logItem.PushBack(new LogContent("username", notNull(logBean.getUsername())));
        logItem.PushBack(new LogContent("requestPath", notNull(logBean.getUrl())));
        logItem.PushBack(new LogContent("module", notNull(logBean.getModule())));
        logItem.PushBack(new LogContent("ip", notNull(logBean.getIp())));
        logItem.PushBack(new LogContent("startTime", notNull(sdf.format(logBean.getStartTime()))));
        logItem.PushBack(new LogContent("executeTime", notNull(String.valueOf(logBean.getSpendTime()))));
        logItem.PushBack(new LogContent("requestBody", notNull(logBean.getParameter())));
        logItem.PushBack(new LogContent("responseData", notNull(logBean.getResult())));
        logItem.PushBack(new LogContent("applicationName", notNull(logBean.getApplicationName())));
        logItem.PushBack(new LogContent("environment", notNull(logBean.getEnvironment())));
        logItems.add(logItem);
        try {
            client.putLogs(logProperties.getTopicId(), logItems, result -> {
            });
        } catch (Exception ignored) {
        }
    }

    private String notNull(String value) {
        return StringUtils.isEmpty(value) ? "" : value;
    }

    @PreDestroy
    private void destroy() {
        if (Objects.nonNull(client)) {
            try {
                client.close();
            } catch (Exception e) {
                log.error("destroy error :", e);
            }
        }
    }
}

非常简单,通过配置文件类获取配置文件中的参数信息,并获取构建AsyncProducerClient对象,通过BeanLog获取采集到信息,上传到腾讯云的cls中。

  • FileSaveLogServiceImpl实现类:
@Service
@Slf4j
public class FileSaveLogServiceImpl implements SaveLogService {
    @Override
    public void saveLog(LogBean logBean) {
        log.info("save log is {}", JSONUtil.toJsonStr(logBean));
    }
}

FileSaveLog实现类直接依赖业务系统的日志功能,将采集信息记录在日志文件中,后续跟进ELK进行数据采集。

3.6、增加spring.factories配置文件

这里不详细讲解SpringBoot自动配置流程,我们定义spring.factories文件,内容如下:

org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
com.jdd.starter.log.config.JddLogAntoConfiguration,\
com.jdd.starter.log.aspect.AuditLogAspect

通过配置com.jdd.starter.log.config.JddLogAntoConfigurationcom.jdd.starter.log.aspect.AuditLogAspect将整个组件的对象加载到Spring容器中。

最后附上整个组件的目录结构:

目录结构

更多精彩内容欢迎关注公众号:[杰子学编程](


# SpringBoot # Starter # 日志 # 组件