num records 10000 and bulk size 1000 only writes 1000 keys in crate

Question

num records 10000 and bulk size 1000 only writes 1000 keys in crate

asicoe opened this issue 7 years ago · comments

El Dorado131 commented 7 years ago

Running this:
cr8 insert-fake-data --hosts crate-1:4200 --table t1 --bulk-size 1000 --num-records 10000

results in only 1000 records in a 1 node CrateDB 2.1.5.

Expected behaviour is to find 10000 records.

Mathias Fußenegger · Answer 1 · Mon Sep 04 2017 14:36:02 GMT+0800 (China Standard Time)

Could you post the schema of the table and show the output that you get with cr8? I can't reproduce this.

El Dorado131 · Answer 2 · Thu Sep 07 2017 05:31:24 GMT+0800 (China Standard Time)

Hi,

Sure, sorry for not doing it already.

I am using Python 3.6.2.

git clone https://github.com/mfussenegger/cr8.git
cd cr8
python3.6 -m pip install -e .
docker run -d -p 4200:4200 -p 4300:4300 -p 5432:5432 crate:2.1.5
docker exec -ti crate crash --command "CREATE TABLE t1 (a STRING, b STRING, c STRING, d STRING, PRIMARY KEY (a, b)) CLUSTERED BY (a) with (number_of_replicas = 0);"

cr8 insert-fake-data --table t1 --bulk-size 100 --num-records 1000 //Expect 1000 rows in t1.
Found schema:
{
"a": "string",
"b": "string",
"c": "string",
"d": "string"
}
Using insert statement:
insert into "doc"."t1" ("a", "b", "c", "d") values (?, ?, ?, ?)
Will make 10 requests with a bulk size of 100
Generating fake data and executing inserts
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:01<00:00, 9.96 requests/s]

Thanks

Mathias Fußenegger · Answer 3 · Thu Sep 07 2017 15:35:13 GMT+0800 (China Standard Time)

I was able to reproduce it with your schema. The problem is the primary key
definition. The default random value provider for string columns doesn't
deliver enough unique values.

If you remove the primary key definition you'll get 1000 records inserted, but
for example, if you then do a select distinct a from t1 you'll see that you
get a much lower number of records (74 in my case)

So you can either:

Not use a primary key at all
Add a column named "id" of type string as primary key. cr8 will then pick a uuid provider.
Make use of the --mapping-file option to choose a different provider for the PK columns. (See cr8 insert-fake-data --help)

El Dorado131 · Answer 4 · Thu Sep 07 2017 23:28:36 GMT+0800 (China Standard Time)

Hi,

Thanks for the reply.

Ok, I see your point but if it's like you say and the culprit is the randomness of the keys, why do I always get exactly the same number of records in CrateDb as the --bulk-size I pass in. For instance the exact same example above ran with --bulk-size 10 and --num-records 100 would give 10 records in CrateDb.

As for your suggestions:

I have to use a primary key so I cannot change the schema.
I now have id in the schema and indeed uuids are generated but it still generates exactly bulk-size records in crate:

cr8 insert-fake-data --table t1 --bulk-size 10 --num-records 100
Found schema:
{
"b": "string",
"c": "string",
"d": "string",
"id": "string"
}
Using insert statement:
insert into "doc"."t1" ("b", "c", "d", "id") values (?, ?, ?, ?)
Will make 10 requests with a bulk size of 10
Generating fake data and executing inserts
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:00<00:00, 58.71 requests/s]

This same thing happens if I remove b from the primary key and also then I remove the CLUSTERED BY clause.

I haven't tried option 3 yet as option 2 did not work.

Thanks

El Dorado131 · Answer 5 · Tue Sep 12 2017 14:09:24 GMT+0800 (China Standard Time)

Hi Mathias,

Were you able to reproduce the above?

Thanks

Mathias Fußenegger · Answer 6 · Tue Sep 12 2017 16:05:52 GMT+0800 (China Standard Time)

Yep I'm able to reproduce it. Thanks.

It seems that in https://github.com/joke2k/faker the seed handling changed slightly. Now in cr8 each bulk requests appears to generate data based on the same starting seed.

As a workaround you could downgrade faker until I've adopted cr8

(to downgrade you can use e.g. pip install Faker==0.7.1)

Mathias Fußenegger · Answer 7 · Wed Sep 13 2017 04:40:10 GMT+0800 (China Standard Time)

Should be fixed via #175 - although the data generation will be a bit slower.

El Dorado131 · Answer 8 · Tue Sep 19 2017 18:43:11 GMT+0800 (China Standard Time)

Thanks Mathias.
pip install Faker==0.7.1
worked for me.