Simple tool to extract queries from a Solr log
GoogleCodeExporter opened this issue · comments
It would be nice to have a small tool inside SolrMeter to extract queries from
a Solr log file and store them into a new file.
The generated file can then be used by SolrMeter to run a stress test thanks to
Issue 55 (
http://code.google.com/p/solrmeter/issues/detail?id=55 ).
There is a screenshot attached to give a rough idea of how it should look like.
Original issue reported on code.google.com by e.to...@sourcesense.com on 7 Oct 2011 at 8:39
Attachments:
- [SolrMeter Extract From Log Files.jpg](https://storage.googleapis.com/google-code-attachments/solrmeter/issue-90/comment-0/SolrMeter Extract From Log Files.jpg)
This is a first cut. I'm submitting it just to have your feedback.
At the moment you can select a solr log file and you have to type an output
file name.
There isn't any error handling yet.
This is a list of issues that should be fixed:
- the output file is generated into /tmp directory
solution: add an extra field to let a user specify the output directory. (Default directory will be the same of the Solr log file)
- at the moment the regex that extracts queries is inside the Java code. It
would be nice to have it customizable. In that case I could introduce another
UI field so a user can write his own regex depending on his log file.
- the regex now is not able to distinguish between select / update queries.
Probably we would like just to have select queries and not ping or update
queries.
- improve UI and add error handling
Any feedback is really appreciated
edo
Original comment by e.to...@sourcesense.com on 7 Oct 2011 at 8:56
Attachments:
This is great Edoardo! I added some internationalization. Added also a file
selection for the output file. Default output file is called "output.txt" and
is located under the same directory as the source file.
I also added a details panel, to show the location of the generated file or the
message of the exception if there is an error.
Still needs to be done:
- regex has to be customizable, probably by adding another input in the same
dialog.
- regex still doesn't recognise updates / selectrs
- Still needs some more error handling.
Original comment by tflo...@gmail.com on 18 Oct 2011 at 1:20
Attachments:
Apparently I forgot to add some files. Here is the patch again.
Original comment by tflo...@gmail.com on 18 Oct 2011 at 1:41
Attachments:
Hi all,
today i had some spare time and i checked out your patch!
i also added a few more things:
- added an input textfield for the regular expression. The regex is now
customizable from the UI and prefilled with the default regex
- fixed regex: now it gets queries executed only on the select query handler
- added some error handling
I also added some padding on the progress bar and on the textarea.
Finally i added another option called "remove duplicate queries". Basically
with this option enabled the query file generated will not contain duplicated
queries.
please give it a try and report possible bugs or improvements :)
edo
Original comment by edoardo.tosca on 7 Nov 2011 at 5:23
Attachments:
It works great. I'll commit this patch.
Original comment by tflo...@gmail.com on 9 Nov 2011 at 2:23
- Changed state: Started
Committed in revision 279. Updates and improvements to this tool should be done
in different issues, I'm closing this one.
Thanks for your work Edoardo.
Original comment by tflo...@gmail.com on 10 Nov 2011 at 12:09
- Changed state: Resolved
- Added labels: Milestone-Release-0.2.1
Original comment by tflo...@gmail.com on 9 Oct 2012 at 10:48
- Added labels: Milestone-Release-0.3.0
- Removed labels: Milestone-Release-0.2.1