大神可以添加动态建表建库吗？

Question

大神可以添加动态建表建库吗？

roy00008 opened this issue 7 months ago · comments

roy00008 commented 7 months ago

因为有些业务需要到动态建表，建库呢

xuejmnet · Answer 1 · Mon Nov 13 2023 17:37:00 GMT+0800 (China Standard Time)

@roy00008 原则上是可以的目前已sql的写法可以做到动态表和分表

xuejmnet · Answer 2 · Mon Nov 13 2023 22:39:49 GMT+0800 (China Standard Time)

demo已经上传,这个对象的这个路由支持你传入任意时间他会创建任意表名,并且支持复杂查询

roy00008 · Answer 3 · Tue Nov 14 2023 09:37:21 GMT+0800 (China Standard Time)

这个demo在哪里呀大神

xuejmnet · Answer 4 · Tue Nov 14 2023 09:41:26 GMT+0800 (China Standard Time)

@roy00008 就是easy-query的源码里面 https://github.com/xuejmnet/easy-query/tree/main/samples/spring-sharding-demo

roy00008 · Answer 5 · Wed Nov 15 2023 10:35:22 GMT+0800 (China Standard Time)

大神按你的写法，我这边运行调用的就报错呢。

[ERROR][11-15 10:11:57.234][jlhttp-1][*][c.e.q.c.u.EasyJdbcExecutorUtil.insert - 260]: INSERT INTO lottery (lottery_num,lottery_result,lottery_type,table_key) VALUES (?,?,?,?)
java.sql.BatchUpdateException: Table 'lucky-game-lottery.lottery' doesn't exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.cj.util.Util.handleNewInstance(Util.java:192)
at com.mysql.cj.util.Util.getInstance(Util.java:167)
at com.mysql.cj.util.Util.getInstance(Util.java:174)
at com.mysql.cj.jdbc.exceptions.SQLError.createBatchUpdateException(SQLError.java:224)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeBatchWithMultiValuesClause(ClientPreparedStatement.java:718)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeBatchInternal(ClientPreparedStatement.java:409)
at com.mysql.cj.jdbc.StatementImpl.executeBatch(StatementImpl.java:795)
at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:128)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)
at com.easy.query.core.util.EasyJdbcExecutorUtil.insert(EasyJdbcExecutorUtil.java:226)
at com.easy.query.core.basic.jdbc.executor.internal.unit.impl.EasyInsertExecutor.executeCommandUnit(EasyInsertExecutor.java:44)
at com.easy.query.core.basic.jdbc.executor.internal.unit.impl.EasyInsertExecutor.executeCommandUnit(EasyInsertExecutor.java:27)
at com.easy.query.core.basic.jdbc.executor.internal.unit.abstraction.AbstractExecutor.groupExecute(AbstractExecutor.java:117)
at com.easy.query.core.basic.jdbc.executor.internal.unit.abstraction.AbstractExecutor.executeSingle0(AbstractExecutor.java:75)
at com.easy.query.core.basic.jdbc.executor.internal.unit.abstraction.AbstractExecutor.execute0(AbstractExecutor.java:67)
at

xuejmnet · Answer 6 · Wed Nov 15 2023 10:40:12 GMT+0800 (China Standard Time)

@roy00008 看你的错误应该是使用了schema,所以你先确定下创建表的时候有没有指定schema就是lucky-game-lottery比如原来是

create table lottery(....)
应该改成
create table lucky-game-lottery.lottery(....)

如果已经在create table上添加了schema那么应该是你的表没有添加schema

@Table(value = "lottery",schema = "lucky-game-lottery")

 private synchronized void addShardingTail(String tail) {
        if (!tails.containsKey(tail)) {

            try {
                String sql = "create table order_test_" + tail + "\n" +
                        "(\n" +
                        "    id        varchar(128) not null\n" +
                        "        primary key,\n" +
                        "    name       varchar(128)          not null,\n" +
                        "    create_time   datetime not null\n" +
                        ");";
                easyQuery.sqlExecute(sql);
            } catch (Exception ex) {
                log.error("创建表失败:" + ex.getMessage(), ex);
            }

            EasyQueryOption easyQueryOption = easyQuery.getRuntimeContext().getQueryConfiguration().getEasyQueryOption();
            EntityMetadata entityMetadata = easyQuery.getRuntimeContext().getEntityMetadataManager().getEntityMetadata(entityClass());
            entityMetadata.addActualTableWithDataSource(easyQueryOption.getDefaultDataSourceName(), entityMetadata.getTableName() + tableSeparator() + tail);
            tails.put(tail, def);
        }
    }

roy00008 · Answer 7 · Wed Nov 15 2023 10:49:25 GMT+0800 (China Standard Time)

添加了schema还是会出错。

xuejmnet · Answer 8 · Wed Nov 15 2023 10:50:27 GMT+0800 (China Standard Time)

什么错误呢
@roy00008

roy00008 · Answer 9 · Wed Nov 15 2023 10:51:34 GMT+0800 (China Standard Time)

java.sql.BatchUpdateException: Table 'lucky-game-lottery.lottery' doesn't exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.cj.util.Util.handleNewInstance(Util.java:192)
at com.mysql.cj.util.Util.getInstance(Util.java:167)
at com.mysql.cj.util.Util.getInstance(Util.java:174)
at com.mysql.cj.jdbc.exceptions.SQLError.createBatchUpdateException(SQLError.java:224)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeBatchWithMultiValuesClause(ClientPreparedStatement.java:718)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeBatchInternal(ClientPreparedStatement.java:409)
at com.mysql.cj.jdbc.StatementImpl.executeBatch(StatementImpl.java:795)

roy00008 · Answer 10 · Wed Nov 15 2023 10:51:55 GMT+0800 (China Standard Time)

package com.example.data.sharding;

import com.easy.query.api4j.client.EasyQuery;
import com.easy.query.core.configuration.EasyQueryOption;
import com.easy.query.core.logging.Log;
import com.easy.query.core.logging.LogFactory;
import com.easy.query.core.metadata.EntityMetadata;
import com.easy.query.core.sharding.api.route.time.AbstractMonthTableRoute;
import com.easy.query.solon.annotation.Db;
import com.example.data.entity.LotteryDa;
import org.noear.solon.annotation.Component;
import org.noear.solon.annotation.Inject;

import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.stream.Collectors;

/**

赛果分表片区
*/
@component
public class LotteryTableRoute extends AbstractMonthTableRoute {
private static final Log log = LogFactory.getLog(LotteryTableRoute.class);
public static final ConcurrentHashMap<String, Object> tails = new ConcurrentHashMap<>();
private final Object def = new Object();
private boolean inited = false;

@db("main-db")
private EasyQuery easyQuery;
@OverRide
protected LocalDateTime convertLocalDateTime(Object o) {
return (LocalDateTime) o;
}

@OverRide
protected String formatShardingValue(LocalDateTime time) {
String shardingValue = time.format(DateTimeFormatter.ofPattern("yyyyMMdd"));
if (!inited) {
initTails();
}
checkShardingTail(shardingValue);
return shardingValue;
}

private synchronized void initTails() {
if (!inited) {
initTails0();
inited = true;
}
}

private void initTails0() {
EntityMetadata entityMetadata = easyQuery.getRuntimeContext().getEntityMetadataManager().getEntityMetadata(entityClass());
EasyQueryOption easyQueryOption = easyQuery.getRuntimeContext().getQueryConfiguration().getEasyQueryOption();
List<Map<String, Object>> maps = easyQuery.sqlQueryMap("SHOW TABLES");
List tables = maps.stream().flatMap(o -> o.values().stream()).map(o -> o.toString())
.filter(o -> o.startsWith("lottery")).collect(Collectors.toList());
for (String table : tables) {
entityMetadata.addActualTableWithDataSource(easyQueryOption.getDefaultDataSourceName(), table);
tails.put(table.substring(8), def);
}
}

private void checkShardingTail(String tail) {
if (!tails.containsKey(tail)) {
addShardingTail(tail);
}
}

private synchronized void addShardingTail(String tail) {
if (!tails.containsKey(tail)) {
try {
String sql = "CREATE TABLE lottery(\n" +
" ID INT NOT NULL AUTO_INCREMENT COMMENT 'ID ID' ,\n" +
" LOTTERY_NUM BIGINT NOT NULL COMMENT '期数期数' ,\n" +
" LOTTERY_RESULT VARCHAR(1024) NOT NULL COMMENT '赛果赛果' ,\n" +
" LOTTERY_TYPE INT NOT NULL COMMENT '票种 100=澳8;200=澳5' ,\n" +
" TABLE_KEY VARCHAR(32) NOT NULL COMMENT '表键值' ,\n" +
" REVISION INT COMMENT '乐观锁' ,\n" +
" CREATED_TIME DATETIME COMMENT '创建时间' ,\n" +
" UPDATED_TIME DATETIME COMMENT '更新时间' ,\n" +
" PRIMARY KEY (ID)\n" +
");";
easyQuery.sqlExecute(sql);
} catch (Exception ex) {
log.error("创建表失败:" + ex.getMessage(), ex);
}
```
     EasyQueryOption easyQueryOption = easyQuery.getRuntimeContext().getQueryConfiguration().getEasyQueryOption();
     EntityMetadata entityMetadata = easyQuery.getRuntimeContext().getEntityMetadataManager().getEntityMetadata(entityClass());
     entityMetadata.addActualTableWithDataSource(easyQueryOption.getDefaultDataSourceName(), entityMetadata.getTableName() + tableSeparator() + tail);
     tails.put(tail, def);
 }
```
}
}

roy00008 · Answer 11 · Wed Nov 15 2023 10:52:05 GMT+0800 (China Standard Time)

package com.example.data.entity;

import com.easy.query.core.annotation.Column;
import com.easy.query.core.annotation.Table;
import com.easy.query.core.annotation.Version;
import com.easy.query.core.basic.extension.version.VersionLongStrategy;
import lombok.Data;

@table(value = "lottery",schema = "lucky-game-lottery")
@DaTa
public class LotteryDa {
/** ID;ID /
@column(primaryKey = true)
private Integer id ;
/* 期数;期数 /
private Long lotteryNum ;
/* 赛果;赛果 /
private String lotteryResult ;
/* 票种;100=澳8;200=澳5 /
private Integer lotteryType ;
/* 表键值 /
private String tableKey ;
/* 乐观锁 */
@Version(strategy = VersionLongStrategy.class)
private Integer revision ;
}

roy00008 · Answer 12 · Wed Nov 15 2023 10:52:22 GMT+0800 (China Standard Time)

package com.example.controller;

import com.easy.query.api.proxy.client.EasyProxyQuery;
import com.easy.query.solon.annotation.Db;
import com.example.data.entity.LotteryDa;
import com.example.data.sharding.LotteryTableRoute;
import org.noear.solon.annotation.Controller;
import org.noear.solon.annotation.Inject;
import org.noear.solon.annotation.Mapping;
import org.noear.solon.cache.jedis.RedisCacheService;
import org.noear.solon.core.handle.Context;
import org.noear.solon.core.handle.Result;
import org.noear.solon.core.handle.SessionState;

import java.util.ArrayList;

@controller
public class TestController {

@Inject("${youcan.cache}")
private RedisCacheService redissonClient;

@Db("main-db")
private EasyProxyQuery easyProxyQuery;

@Inject
private LotteryTableRoute lotteryTableRoute;



@Mapping(value = "/get/{key}")
private Result get(Context ctx,String key){
    ArrayList<LotteryDa> orderEntities = new ArrayList<>();
    LotteryDa lotteryDa = new LotteryDa();
    lotteryDa.setLotteryNum(31094502l);
    lotteryDa.setLotteryType(100);
    lotteryDa.setLotteryResult("9,1,6,20,13,16,14,12");
    lotteryDa.setTableKey("20231115");
    orderEntities.add(lotteryDa);
    easyProxyQuery.insertable(orderEntities).executeRows();
    return Result.succeed(redissonClient.get(key,String.class));
}

}

xuejmnet · Answer 13 · Wed Nov 15 2023 10:53:48 GMT+0800 (China Standard Time)

@roy00008

create table lottery(....)
应该改成
create table lucky-game-lottery.lottery(....)

String sql = "CREATE TABLE lottery(\n" + --这边把schema加上我之前说了
" ID INT NOT NULL AUTO_INCREMENT COMMENT 'ID ID' ,\n" +
" LOTTERY_NUM BIGINT NOT NULL COMMENT '期数 期数' ,\n" +
" LOTTERY_RESULT VARCHAR(1024) NOT NULL COMMENT '赛果 赛果' ,\n" +
" LOTTERY_TYPE INT NOT NULL COMMENT '票种 100=澳8;200=澳5' ,\n" +
" TABLE_KEY VARCHAR(32) NOT NULL COMMENT '表键值' ,\n" +
" REVISION INT COMMENT '乐观锁' ,\n" +
" CREATED_TIME DATETIME COMMENT '创建时间' ,\n" +
" UPDATED_TIME DATETIME COMMENT '更新时间' ,\n" +
" PRIMARY KEY (ID)\n" +
");";

而且你好像没有用到tail这个属性,demo里面我是进行了分表处理

xuejmnet · Answer 14 · Wed Nov 15 2023 10:55:47 GMT+0800 (China Standard Time)

xuejmnet commented 7 months ago

roy00008 · Answer 15 · Wed Nov 15 2023 11:02:01 GMT+0800 (China Standard Time)

我想问一下 LotteryTableRoute 这个类是在哪调用的？

roy00008 · Answer 16 · Wed Nov 15 2023 11:02:32 GMT+0800 (China Standard Time)

因为我的代码里面都没有地方调用 LotteryTableRoute 所以我就很奇怪

roy00008 · Answer 17 · Wed Nov 15 2023 11:03:44 GMT+0800 (China Standard Time)

调用这句 easyProxyQuery.insertable(orderEntities).executeRows(); 框架是怎么知道先去调用LotteryTableRoute创建表的

xuejmnet · Answer 18 · Wed Nov 15 2023 11:07:08 GMT+0800 (China Standard Time)

@component 注解会将其注入到默认的easy-query框架内部

框架会获取到route路由然后添加到 TableRouteManager里面在执行的时候会判断当前是否是分片对象如果是并且是分表对象那么就会去获取对应的路由然后执行响应的方法,（在格式化尾巴后缀的时候判断这个后缀是否已经被创建了没有就创建表 **这部分就是你自己实现的代码**）

xuejmnet · Answer 19 · Wed Nov 15 2023 11:08:26 GMT+0800 (China Standard Time)

@roy00008 执行逻辑你可以打这个断点 EntityExpressionExecutor 实现类是 DefaultEntityExpressionExecutor

roy00008 · Answer 20 · Wed Nov 15 2023 11:08:49 GMT+0800 (China Standard Time)

LotteryTableRoute 这个类我打了断点都没有进去的

xuejmnet · Answer 21 · Wed Nov 15 2023 11:11:52 GMT+0800 (China Standard Time)

@roy00008 你的easy-query是starter构建的吗还是自己构建的bean如果是自己构建的需要自己添加route到tableRouteManager

 TableRouteManager tableRouteManager = runtimeContext.getTableRouteManager();
        tableRouteManager.addRoute(new TopicShardingTableRoute());

可以直接看单元测试

https://github.com/xuejmnet/easy-query/blob/f89fa54629d38875fc503e075be884837765dd64/sql-test/src/main/java/com/easy/query/test/BaseTest.java#L179

xuejmnet · Answer 22 · Wed Nov 15 2023 11:12:28 GMT+0800 (China Standard Time)

starter构建的bean会根据@component注解自己去注册,本质还是tableRouteManager.addRoute(new TopicShardingTableRoute());

roy00008 · Answer 23 · Wed Nov 15 2023 11:13:02 GMT+0800 (China Standard Time)

import org.noear.solon.annotation.Component;

roy00008 · Answer 24 · Wed Nov 15 2023 11:13:18 GMT+0800 (China Standard Time)

会不会是import org.noear.solon.annotation.Component; 这个的识别不了

xuejmnet · Answer 25 · Wed Nov 15 2023 11:15:28 GMT+0800 (China Standard Time)

@roy00008 你是solon吗,如果是solon那么看文档里面的 https://xuejm.gitee.io/easy-query-doc/guide/config/config-solon.html#solon%E6%89%80%E6%9C%89%E9%85%8D%E7%BD%AE

//    /**
//     * 添加分表或者分库的路由,分库数据源
//     * @param runtimeContext
//     */
//    @Bean
//    public void db1QueryRuntimeContext(@Db("db1") QueryRuntimeContext runtimeContext){
//        TableRouteManager tableRouteManager = runtimeContext.getTableRouteManager();
//        DataSourceRouteManager dataSourceRouteManager = runtimeContext.getDataSourceRouteManager();
//        tableRouteManager.addRoute(...);
//        dataSourceRouteManager.addRoute(...);
//
//        DataSourceManager dataSourceManager = runtimeContext.getDataSourceManager();
//
//        dataSourceManager.addDataSource(key, dataSource, poolSize);
//    }

solon的操作都需要手动添加,因为solon支持多数据源,如果只是@component那么无法知晓具体到哪个数据源所以都是手动加的

roy00008 · Answer 26 · Wed Nov 15 2023 13:58:26 GMT+0800 (China Standard Time)

@bean
public void db1QueryRuntimeContext(@db("main-db") QueryRuntimeContext runtimeContext){
TableRouteManager tableRouteManager = runtimeContext.getTableRouteManager();
tableRouteManager.addRoute(new LotteryTableRoute());
}

roy00008 · Answer 27 · Wed Nov 15 2023 13:59:09 GMT+0800 (China Standard Time)

写成这样以后，还是提示Table 'lucky-game-lottery.lottery' doesn't exist

roy00008 · Answer 28 · Wed Nov 15 2023 14:12:36 GMT+0800 (China Standard Time)

大神，应该怎么写吖，能不能用solon写一个demo，可以动态建表的

xuejmnet · Answer 29 · Mon Nov 20 2023 13:08:44 GMT+0800 (China Standard Time)

如果可以了的话我就关闭了：）