Choose Hashing function

Question

Choose Hashing function

j2ghz opened this issue 7 years ago · comments

Tried BenchmarkDotNet on https://github.com/brandondahler/Data.HashFunction/tree/DotNetCore/src/System.Data.HashFunction.xxHash

BenchmarkDotNet=v0.10.8, OS=Windows 10.0.16232
Processor=Intel Core i7-4710HQ CPU 2.50GHz (Haswell), ProcessorCount=8
Frequency=2435765 Hz, Resolution=410.5486 ns, Timer=TSC
dotnet cli version=2.0.0-preview2-006497
  [Host]     : .NET Core 4.6.25211.01, 64bit RyuJITDEBUG [AttachedDebugger]
  DefaultJob : .NET Core 4.6.25211.01, 64bit RyuJIT

Method	Mean	Error	StdDev
Sha256	49.65 us	0.3757 us	0.3330 us
Md5	20.88 us	0.0838 us	0.0784 us
xxH	18.86 us	0.1706 us	0.1512 us

Jozef Hollý · Answer 1 · Sun Jul 09 2017 17:32:45 GMT+0800 (China Standard Time)

Ran again, in release, without debug, on byte[1026]

BenchmarkDotNet=v0.10.8, OS=Windows 10.0.16232
Processor=Intel Core i7-4710HQ CPU 2.50GHz (Haswell), ProcessorCount=8
Frequency=2435765 Hz, Resolution=410.5486 ns, Timer=TSC
dotnet cli version=2.0.0-preview2-006497
  [Host]     : .NET Core 4.6.25211.01, 64bit RyuJIT
  DefaultJob : .NET Core 4.6.25211.01, 64bit RyuJIT

Method	Mean	Error	StdDev
Sha256	5.721 us	0.0302 us	0.0267 us
Md5	2.500 us	0.0102 us	0.0095 us
xxH	2.194 us	0.0156 us	0.0145 us

Jozef Hollý · Answer 2 · Sun Jul 09 2017 19:50:08 GMT+0800 (China Standard Time)

with variable data size

BenchmarkDotNet=v0.10.8, OS=Windows 10.0.16232
Processor=Intel Core i7-4710HQ CPU 2.50GHz (Haswell), ProcessorCount=8
Frequency=2435765 Hz, Resolution=410.5486 ns, Timer=TSC
dotnet cli version=2.0.0-preview2-006497
  [Host]     : .NET Core 4.6.25211.01, 64bit RyuJITDEBUG [AttachedDebugger]
  DefaultJob : .NET Core 4.6.25211.01, 64bit RyuJIT

Method	N	Mean	Error	StdDev	Scaled
Sha256	1024	5.613 μs	0.0149 μs	0.0132 μs	1.00
xxHash32	1024	3.992 μs	0.0248 μs	0.0232 μs	0.71
xxHash64	1024	2.083 μs	0.0139 μs	0.0123 μs	0.37
Sha256	1048576	4,993.792 μs	30.2022 μs	28.2511 μs	1.00
xxHash32	1048576	3,896.981 μs	29.3475 μs	26.0158 μs	0.78
xxHash64	1048576	1,866.801 μs	8.5724 μs	8.0187 μs	0.37
Sha256	1073741824	5,210,652.113 μs	7,179.6743 μs	6,715.8712 μs	1.00
xxHash32	1073741824	4,033,018.419 μs	6,619.3987 μs	6,191.7891 μs	0.77
xxHash64	1073741824	1,911,168.663 μs	3,330.8290 μs	3,115.6592 μs	0.37

Jozef Hollý · Answer 3 · Sun Jul 09 2017 20:25:31 GMT+0800 (China Standard Time)

Diferrent sizes, just xxHash64, final

BenchmarkDotNet=v0.10.8, OS=Windows 10.0.16232
Processor=Intel Core i7-4710HQ CPU 2.50GHz (Haswell), ProcessorCount=8
Frequency=2435765 Hz, Resolution=410.5486 ns, Timer=TSC
dotnet cli version=2.0.0-preview2-006497
  [Host]     : .NET Core 4.6.25211.01, 64bit RyuJIT
  DefaultJob : .NET Core 4.6.25211.01, 64bit RyuJIT

Method	dataSizeBytes	Mean	Error	StdDev
xxHash64	1073	2.215 us	0.0078 us	0.0073 us
xxHash64	10737	19.021 us	0.0625 us	0.0585 us
xxHash64	107374	186.177 us	0.7968 us	0.7453 us
xxHash64	1073741	1,849.961 us	7.2826 us	6.8121 us
xxHash64	10737418	18,621.881 us	62.7286 us	58.6764 us
xxHash64	107374182	186,785.877 us	614.1825 us	574.5067 us
xxHash64	1073741824	1,859,136.249 us	1,594.9764 us	1,331.8774 us

Jozef Hollý · Answer 4 · Sun Jul 09 2017 20:50:49 GMT+0800 (China Standard Time)

source: https://github.com/j2ghz/xxHash64Benchmark

Jozef Hollý · Answer 5 · Sun Jul 09 2017 21:26:09 GMT+0800 (China Standard Time)

Another system

BenchmarkDotNet=v0.10.8, OS=rancheros v1.0.3
Processor=Intel Core i5-2400 CPU 3.10GHz (Sandy Bridge), ProcessorCount=4
Frequency=1000000000 Hz, Resolution=1.0000 ns, Timer=UNKNOWN
dotnet cli version=1.0.4
  [Host]     : .NET Core 4.6.25211.01, 64bit RyuJIT
  DefaultJob : .NET Core 4.6.25211.01, 64bit RyuJIT

Method	dataSizeBytes	Mean	Error	StdDev
xxHash64	1073	2.769 us	0.0063 us	0.0059 us
xxHash64	10737	23.970 us	0.0174 us	0.0154 us
xxHash64	107374	234.532 us	0.1415 us	0.1324 us
xxHash64	1073741	2,345.825 us	3.1974 us	2.9909 us
xxHash64	10737418	23,643.866 us	34.2171 us	32.0067 us
xxHash64	107374182	236,312.796 us	660.3241 us	617.6675 us
xxHash64	1073741824	2,368,209.885 us	10,650.5131 us	9,962.4956 us

Jozef Hollý · Answer 6 · Sun Jul 09 2017 23:51:33 GMT+0800 (China Standard Time)

Another system

BenchmarkDotNet=v0.10.8, OS=Windows 10 Redstone 2 (10.0.15063)
Processor=Intel Core i7-3770 CPU 3.40GHz (Ivy Bridge), ProcessorCount=8
Frequency=3312806 Hz, Resolution=301.8589 ns, Timer=TSC
dotnet cli version=1.0.4
  [Host]     : .NET Core 4.6.25211.01, 64bit RyuJIT
  DefaultJob : .NET Core 4.6.25211.01, 64bit RyuJIT

Method	dataSizeBytes	Mean	Error	StdDev	Median
xxHash64	1073	2.230 us	0.0360 us	0.0337 us	2.224 us
xxHash64	10737	19.140 us	0.2777 us	0.2598 us	19.102 us
xxHash64	107374	187.574 us	2.2302 us	2.0862 us	187.603 us
xxHash64	1073741	1,957.777 us	53.7768 us	99.6787 us	1,927.732 us
xxHash64	10737418	19,471.398 us	228.5603 us	213.7954 us	19,464.001 us
xxHash64	107374182	191,651.000 us	1,210.4030 us	1,132.2116 us	191,645.380 us
xxHash64	1073741824	1,963,685.790 us	38,975.5368 us	53,350.1547 us	1,927,262.200 us

Jozef Hollý · Answer 7 · Mon Jul 10 2017 05:07:26 GMT+0800 (China Standard Time)

Another system

BenchmarkDotNet=v0.10.8, OS=Windows 10 Redstone 2 (10.0.15063)
Processor=Intel Core i7-4770K CPU 3.50GHz (Haswell), ProcessorCount=8
Frequency=3417965 Hz, Resolution=292.5717 ns, Timer=TSC
dotnet cli version=1.0.4
  [Host]     : .NET Core 4.6.25211.01, 64bit RyuJIT
  DefaultJob : .NET Core 4.6.25211.01, 64bit RyuJIT

Method	dataSizeBytes	Mean	Error	StdDev
xxHash64	1073	1.936 us	0.0431 us	0.0576 us
xxHash64	10737	16.312 us	0.3144 us	0.3088 us
xxHash64	107374	159.035 us	3.0921 us	2.8923 us
xxHash64	1073741	1,576.769 us	31.2196 us	24.3742 us
xxHash64	10737418	15,979.093 us	318.1048 us	297.5554 us
xxHash64	107374182	164,444.001 us	3,577.5001 us	7,226.7267 us
xxHash64	1073741824	1,621,551.210 us	31,098.8432 us	30,543.2194 us