There may be some Sql cause memcached to cache data can not be successful.
GoogleCodeExporter opened this issue · comments
I'm using @ReadThroughMultiCache annotation. I found some type of sql like
below this, SSM will return a message. Then memcached is not successful cache
data.
----
com.google.code.ssm.aop.ReadThroughMultiCacheAdvice: Did not receive a
correlated amount of data from the target method.
If My Sql so below, then memcached is not successful cache data.
----
mysql> SELECT u.id, u.name, r.id as "role.id", r.name as "role.name"
-> FROM user u, user_role ur, role r
-> WHERE u.id = ur.userid and r.id = ur.roleid and ur.roleid in (1,2);
+------+-------+---------+------------+
| id | name | role.id | role.name |
+------+-------+---------+------------+
| 1000 | Tom | 1 | ROLE_ADMIN |
| 1001 | Jerry | 2 | ROLE_USER |
| 1002 | Jack | 1 | ROLE_ADMIN |
+------+-------+---------+------------+
3 rows in set (0.00 sec)
If My Sql modified so below, then memcached is successful cache data.
----
mysql> SELECT u.id, u.name, r.id as "role.id", r.name as "role.name"
-> FROM user u, user_role ur, role r
-> WHERE u.id = ur.userid and r.id = ur.roleid and ur.roleid in (1);
+------+------+---------+------------+
| id | name | role.id | role.name |
+------+------+---------+------------+
| 1000 | Tom | 1 | ROLE_ADMIN |
| 1002 | Jack | 1 | ROLE_ADMIN |
+------+------+---------+------------+
2 rows in set (0.00 sec)
If My Sql modified so below, then memcached is successful cache data.
----
mysql> SELECT u.id, u.name
-> FROM user u
-> WHERE u.id in (1000,1001,1002);
+------+-------+
| id | name |
+------+-------+
| 1000 | Tom |
| 1001 | Jerry |
| 1002 | Jack |
+------+-------+
My evn: Spring 3, Mybatis 2.3.5, SSM 3.0.2, MySQL5.5
Test Code:
https://github.com/batizhao/spring-mybatis-memcached/tree/master/ssm3-mybatis2-m
emcached
Original issue reported on code.google.com by zhaob...@gmail.com on 25 Sep 2012 at 8:23
Thank you for the report.
Unfortunately I cannot run your project. You are using mysql db that I don't
have. I've also some problems with importing project in eclipse. There are some
errors due to wrong package name and usage of internal sun's class.
I assume that the problem you've described is with this method:
@Override
@ReadThroughMultiCache(namespace = "user/getUsersByUserIds", expiration = 600)
public List<User> getUsersByUserIds(@ParameterValueKeyProvider final List<Long> ids) {
return (List<User>) sqlMapClientTemplate.queryForList("getUsersByUserIds", ids);
}
The problem occur only when result (list of user) contains different amount of
elements than the ids list. In such case SSM cannot associate id on the ids
list with corresponding User object in result list.
Let's see on example how @ReadThroughMultiCache works by default. Let's assume
that all users are defined in DB and cache is empty:
List<Long> ids = Arrays.asList(1000L, 1001L, 1002L);
List<User> users = getUsersByUserIds(ids);
SSM builds cache keys for each id on ids list:
1000 -> user/getUsersByUserIds:1000
1001 -> user/getUsersByUserIds:1001
1002 -> user/getUsersByUserIds:1002
and checks if there're data in cache. There isn't any data in cache so
intercepted method is invoked with the same list of ids. As a result we get
list with 3 elements in particular order: {User(Tom), User(Jerry), User(Jack)}.
According to the order SSM will associate first element on the ids list with
first on the result, second with the second and so on:
1000L -> User(Tom)
1001L -> User(Jerry)
1002L -> User(Jack)
Using such association SSM generates cache keys and stores user objects under
keys:
user/getUsersByUserIds:1000 -> User(Tom)
user/getUsersByUserIds:1001 -> User(Jerry)
user/getUsersByUserIds:1002 -> User(Jack)
Next time someone invokes method with getUsersByUserIds(Arrays.asList(1000L,
1001L, 1002L)) all objects are returned from cache and intercepted method is
not invoked at all.
As you see the amount of elements in result and the order of elements are very
important. Using 'in' sql command you cannot achieved both (order and fixed
amount). This is why in SSM I've implemented @ReadThroughMultiCacheOption. You
can use it together with @ReadThroughMultiCache and set generateKeysFromResult
to true. Then instead of generating cache keys based on method's parameters
annotated with @ParameterValueKeyProvider (in your case ids), cache keys are
generated directly from elements on result list using toString() or method
annotated with @CacheKeyMethod. This generation of cache key from result occurs
only when some of the elements are not in the cache. So in this case the order
of elements is not important also the amount of elements in result may be
different than amount of ids. What is crucial is that cache key generated from
result must be the same as the one generated from method parameters, otherwise
object won't be accessible. Again example should explain it:
@ReadThroughMultiCacheOption(generateKeysFromResult=true)
@ReadThroughMultiCache(namespace = "user/getUsersByUserIds", expiration = 600)
public List<User> getUsersByUserIds(@ParameterValueKeyProvider final List<Long> ids) {
..
}
To make this work User should have @CacheKeyMethod which returns id, the same
id which is used on ids parameter. Let's assume that only users Tom (1000) and
Jack(1002) are defined in DB and cache is empty. When first time someone invoke
this code
List<Long> ids = Arrays.asList(1000L, 1001L, 1002L);
List<User> users = getUsersByUserIds(ids);
SSM builds cache keys for each id on ids list:
1000 -> user/getUsersByUserIds:1000
1001 -> user/getUsersByUserIds:1001
1002 -> user/getUsersByUserIds:1002
and checks if there're data in cache. There isn't any data in cache so
intercepted method is invoked with the same list of ids. As a result we get
list with 2 elements: {User(Tom), User(Jack)}. According to the
@ReadThroughMultiCacheOption annotation SSM creates cache key base on result
(User object), the user object as its key returns "1000" for Tom and "1002" for
Jack so SSM adds to cache:
user/getUsersByUserIds:1000 -> User(Tom)
user/getUsersByUserIds:1002 -> User(Jack)
Next time someone invokes method with getUsersByUserIds(Arrays.asList(1000L,
1001L, 1002L)) Tom and Jack objects are returned from cache and intercepted
method is invoked with missed id: getUsersByUserIds(Arrays.asList(1001L)).
Together with generateKeysFromResult you may use addNullsToCache then in cache
User with id 1001 (user/getUsersByUserIds:1001) will be marked as null and null
will be returned immediately without invoking intercepted method.
Is it clear for you now?
Original comment by ragno...@gmail.com on 25 Sep 2012 at 8:48
Sorry, that module is a sub-module of the spring-mybatis-memcached
project(https://github.com/batizhao/spring-mybatis-memcached). You need the
clone root directory before the run. I have replaced the mysql to hsqldb.
that method signature is a bit ambiguous, I modified a bit. Change
testGetUsersByUserIds to testGetUsersByRoleIds.
I tried it @ReadThroughMultiCacheOption, that question is no.
I run that UserDaoTest.testGetUsersByRoleIds(), log4j log is as follows.
But I hope like get, with role.id as the key, rather than user.id
Or at least get and set the key consistent.
----
DEBUG 2012-09-26 10:25:06,874 net.rubyeye.xmemcached.impl.Optimizer: Optimieze
merge buffer:get user/getUsersByRoleIds:1 user/getUsersByRoleIds:2
DEBUG 2012-09-26 10:25:06,990 net.rubyeye.xmemcached.impl.Optimizer: Optimieze
merge buffer:set user/getUsersByRoleIds:1000 8 60 120
{"v":{"me.batizhao.model.User":{"id":1000,"name":"Tom","role":{"me.batizhao.mode
l.Role":{"id":1,"name":"ROLE_ADMIN"}}}}}
DEBUG 2012-09-26 10:25:06,993 net.rubyeye.xmemcached.impl.Optimizer: Optimieze
merge buffer:set user/getUsersByRoleIds:1001 8 60 121
{"v":{"me.batizhao.model.User":{"id":1001,"name":"Jerry","role":{"me.batizhao.mo
del.Role":{"id":2,"name":"ROLE_USER"}}}}}
DEBUG 2012-09-26 10:25:06,995 net.rubyeye.xmemcached.impl.Optimizer: Optimieze
merge buffer:set user/getUsersByRoleIds:1002 8 60 121
{"v":{"me.batizhao.model.User":{"id":1002,"name":"Jack","role":{"me.batizhao.mod
el.Role":{"id":1,"name":"ROLE_ADMIN"}}}}}
I need this result set:
----
mysql> SELECT u.id, u.name, r.id as "role.id", r.name as "role.name"
-> FROM user u, user_role ur, role r
-> WHERE u.id = ur.userid and r.id = ur.roleid and ur.roleid in (1,2);
+------+-------+---------+------------+
| id | name | role.id | role.name |
+------+-------+---------+------------+
| 1000 | Tom | 1 | ROLE_ADMIN |
| 1001 | Jerry | 2 | ROLE_USER |
| 1002 | Jack | 1 | ROLE_ADMIN |
+------+-------+---------+------------+
3 rows in set (0.00 sec)
Original comment by zhaob...@gmail.com on 26 Sep 2012 at 3:08
Add a few.
About @ReadThroughMultiCacheOption , this is not:
----
@ReadThroughMultiCacheOption(generateKeysFromResult=true)
@ReadThroughMultiCache(namespace = "user/getUsersByUserIds", expiration = 600)
public List<User> getUsersByUserIds(@ParameterValueKeyProvider final List<Long>
ids)
This is ok:
----
@ReadThroughMultiCache(namespace = "user/getUsersByRoleIds", expiration = 600,
option = @ReadThroughMultiCacheOption(generateKeysFromResult = true))
public List<User> getUsersByRoleIds(@ParameterValueKeyProvider final List<Long>
ids)
Original comment by zhaob...@gmail.com on 26 Sep 2012 at 9:35
Yes, sorry for the mistake it was late when I wrote the answer.
So as I understand the @ReadThroughMultiCache(namespace =
"user/getUsersByRoleIds", expiration = 600, option =
@ReadThroughMultiCacheOption(generateKeysFromResult = true)) works as you want?
Original comment by ragno...@gmail.com on 26 Sep 2012 at 10:37
[deleted comment]
[deleted comment]
No, not what I want. That usage does not compile in my environment. follow this
----
@ReadThroughMultiCacheOption(generateKeysFromResult=true)
@ReadThroughMultiCache(namespace = "user/getUsersByUserIds", expiration = 600)
Later I modified in such a way
----
@ReadThroughMultiCache(namespace = "user/getUsersByRoleIds", expiration = 600,
option = @ReadThroughMultiCacheOption(generateKeysFromResult = true))
But the result is not what I want.
I hope like get, with role.id as the key, rather than user.id
Or at least get and set the key consistent.
Now, Memcached Log:
----
<21 new auto-negotiating client connection
21: Client using the ascii protocol
<21 get user/getUsersByRoles:1 user/getUsersByRoles:2
>21 END
<21 set user/getUsersByRoles:1000 8 60 120
>21 STORED
<21 set user/getUsersByRoles:1002 8 60 121
>21 STORED
<21 set user/getUsersByRoles:1001 8 60 121
>21 STORED
<21 connection closed.
Log4j:
----
DEBUG 2012-09-26 10:25:06,874 net.rubyeye.xmemcached.impl.Optimizer: Optimieze
merge buffer:get user/getUsersByRoleIds:1 user/getUsersByRoleIds:2
DEBUG 2012-09-26 10:25:06,990 net.rubyeye.xmemcached.impl.Optimizer: Optimieze
merge buffer:set user/getUsersByRoleIds:1000 8 60 120
{"v":{"me.batizhao.model.User":{"id":1000,"name":"Tom","role":{"me.batizhao.mode
l.Role":{"id":1,"name":"ROLE_ADMIN"}}}}}
DEBUG 2012-09-26 10:25:06,993 net.rubyeye.xmemcached.impl.Optimizer: Optimieze
merge buffer:set user/getUsersByRoleIds:1001 8 60 121
{"v":{"me.batizhao.model.User":{"id":1001,"name":"Jerry","role":{"me.batizhao.mo
del.Role":{"id":2,"name":"ROLE_USER"}}}}}
DEBUG 2012-09-26 10:25:06,995 net.rubyeye.xmemcached.impl.Optimizer: Optimieze
merge buffer:set user/getUsersByRoleIds:1002 8 60 121
{"v":{"me.batizhao.model.User":{"id":1002,"name":"Jack","role":{"me.batizhao.mod
el.Role":{"id":1,"name":"ROLE_ADMIN"}}}}}
Original comment by zhaob...@gmail.com on 27 Sep 2012 at 12:58
Take a look at you method: public List<User>
getUsersByRoleIds(@ParameterValueKeyProvider final List<Long> ids). The input
ids is a list of ROLE ids and the ids are used to create cache key and fetch
data from cache. When some of data are missing in cache underlying method is
invoked and returned objects of type USER are stored into cache under key
generated from USER object (generateKeysFromResult=true). The USER object
returns id as its cache key ( @CacheKeyMethod). This is way role.id is not used
when putting object into cache.
The relationship User-Role is many-to-one, isn't? The returned result
(List<User>) may contain two or more USER objects with the same ROLE, so also
the same key if role.id will be used as a cache key. Under given cache key only
one java object can be stored by SSM, there is no append logic. In such case
the last USER object on the list with given role.id will overwrite earlier USER
objects with the same role.id.
As I understand what you want to do is to store list of users with given
role.id under single cache key. Then you should use @ReadThroughSingleCache and
annotate method that returns list of users of given role.id (getUsersByRoleId).
There is no way to somehow group objects from result list, split into sublists
and put into cache. SSM allows only to put the result of the method under
single cache key or if the result is a list put each object from the list under
different cache key (overwriting previous value in cache)
Is it clear now?
Original comment by ragno...@gmail.com on 27 Sep 2012 at 6:00
If you want to store given object under different cache key according to needs
(e.g. once by id another time by name) then please create dedicated feature
request. I will add @CacheKeyBean annotation which allows such customization
per method invocation. But as I said before it won't solve your issue with
getUsersByRoleIds method.
Original comment by ragno...@gmail.com on 27 Sep 2012 at 6:15
Thank you, and I know how to do. I think I may need to modify the query
(getUsersByRoleIds).
Option 1: through the loop call getUsersByRoleId method to get the desired
results.
Option 2: use @ReadThroughAssignCache returns the entire cache block.
Option 3: Use queryForMap return a Map (not verified).
Please close it, thx.
Original comment by zhaob...@gmail.com on 27 Sep 2012 at 6:24
Tried to return to the Map, SSM is not yet supported.
Original comment by zhaob...@gmail.com on 27 Sep 2012 at 10:11
Map should work with @ReadThroughAssignCache and @ReadThroughSingleCache. It
stores whole map under single cache key.
There is no support for storing Map while using @ReadThroughMultiCache.
Original comment by ragno...@gmail.com on 27 Sep 2012 at 10:30
Yes, I see. I got it.
Thx.
Original comment by zhaob...@gmail.com on 28 Sep 2012 at 1:01
Original comment by ragno...@gmail.com on 7 Dec 2012 at 6:18
- Changed state: Invalid