Allow creating table again if the table create failed before

Question

Allow creating table again if the table create failed before

ShiKaiWi opened this issue 10 months ago · comments

Description

Currently, another table creation is disallowed if it fails to create the table firstly, which is unfriendly for use.

Proposal

The reason to disallow the retry to create the same table again is that the failure of creation may result from the ceresdb-server, that is to say, the inconsistency may already exist after the first create failure, and CeresMeta can't do more about the such inconsistent state.

Here is the current procedure to create a table:

The client sends create table sql to a ceresdb-server;
ceresdb-server sends it to CeresMeta;
CeresMeta picks some shard for this table;
CeresMeta tries to allocate a table id for the table name, and it fails if the table name has been used;
CeresMeta demands the ceresdb-server owning the shard to create the table, and waits for the response, it fails if the creation by ceresdb-server fails;
CeresMeta persists the update about the shard topology (mapping between shards and tables);
...
...

┌──────────┐             ┌───────────────┐             ┌──────────────┐              ┌───────────────┐
│  client  │             │ceresdb-server0│             │  ceresmeta   │              │ceresdb-server1│
└──────────┘             └───────────────┘             └──────────────┘              └───────────────┘
     │                           │                             │                             │        
     │   1.Send create table                                                                          
     │   request                 │                             │                             │        
     │──────────────────────────▶                                                                     
     │                           │ 2.Send request to ceresmeta │                             │        
     │                            ────────────────────────────▶                                       
     │                           │                             │                             │        
     │                                                          ─────────┐                            
     │                           │                             │ 3.Pick shard for table      │        
     │                                                           4.Alloc table id                     
   Time                          │                             │         │                   │        
     │                                                          ◀────────┘                            
     │                           │                             │                             │        
     │                                                             5.Demand ceresdb-server            
     │                           │                             │   to create table           │        
     │                                                          ────────────────────────────▶         
     │                           │                             │                             │        
     │                                                             6.Respond to ceresmeta             
     │                           │                             │◀────────────────────────────│        
     │                                                                                                
     │                           │                             │─────────┐                   │        
     │                                                           7.Update shard topology              
     │                           │                             │         │                   │        
     │                                                          ◀────────┘                            
     │                           │◀─8.Respond──────────────────│                             │        
     │                                                                                                
     │ ◀────9.Respond────────────│                             │                             │        
     ▼

And in most cases, the step 5 may fail, and it will lead to the following create requests' failure at the step 4;

Here is a proposal to address this problem:
Let's persist the create table params in the meta data (in etcd, with the table name as the key), and the following table creation should utilize this information and ensure the table always lie at the same shard.

Here is the new procedure to create a table:

┌──────────┐             ┌───────────────┐     ┌──────────────┐                          ┌───────────────┐
│  client  │             │ceresdb-server0│     │  ceresmeta   │                          │ceresdb-server1│
└──────────┘             └───────────────┘     └──────────────┘                          └───────────────┘
     │                           │                     │                                          │       
     │   1.Send create table                                                                              
     │   request                 │                     │                                          │       
     │──────────────────────────▶  2.Send request to                                                      
     │                           │ ceresmeta           │                                          │       
     │                            ───────────────────▶                                                    
     │                           │                     │                                          │       
     │                                                  ─────────┐                                        
     │                           │                     │ 3.Pick shard for table as a hint         │       
     │                                                   4.According to the `PrepareCreate`               
     │                           │                     │ mapping and shard hint to decide the     │       
     │                                                   table id and shard id.                           
   Time                          │                     │ 4a.If table is not found in the          │       
     │                                                   `PrepareCreate`, persist the table id            
     │                           │                     │ and shard to it.                         │       
     │                                                  ◀────────┘                                        
     │                           │                     │                                          │       
     │                                                                                                    
     │                           │                     │ 5.Demand ceresdb-server to create table  │       
     │                                                  ─────────────────────────────────────────▶        
     │                           │                     │                                          │       
     │                                                   6.Respond to ceresmeta                           
     │                           │                     │◀─────────────────────────────────────────│       
     │                                                                                                    
     │                           │                     │ ─────────┐                               │       
     │                                                    7.Update shard topology                         
     │                           │                     │  7a.Remove the entry from                │       
     │                                                    the `PrepareCreate`                             
     │                           │                     │ ◀────────┘                               │       
     │                            ◀─8.Respond─────────                                                    
     │                           │                     │                                          │       
     │◀────9.Respond────────────                                                                          
     ▼                           │                     │                                          │

In the flow graph, it can be found that:

The step 3 modifies the determined shard id as a hint;
The step 4a and 7a is added;

In the new procedure, another create table will be allowed because it is ensured to be created at the same shard. However, some other cases should be taken into considerations. Let's discuss these problems in the following section.

Concurrent Creation

The current procedure avoids the concurrent creation when allocating the table id for the table name (it fails if the table name is already allocated to an ID).
And in the new procedure, it won't fail if it is found that the table name has been allocated to an ID. However, it doesn't lead to any harm because it is idempotent.