Giters

openjournals / joss-reviews

Reviews for the Journal of Open Source Software

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[REVIEW]: A MATLAB-based Instrument Control (MIC) package for fluorescence imaging

editorialbot opened this issue · comments

commented

Submitting author: @sajjad88 (Sajjad Khan)
Repository: https://github.com/LidkeLab/matlab-instrument-control
Branch with paper.md (empty if default branch):
Version: v0.1.0
Editor: @adamltyson
Reviewers: @bencardoen, @raacampbell
Archive: Pending

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/145f04b4b9a1d7dcb68f8e7d5bbe665b"><img src="https://joss.theoj.org/papers/145f04b4b9a1d7dcb68f8e7d5bbe665b/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/145f04b4b9a1d7dcb68f8e7d5bbe665b/status.svg)](https://joss.theoj.org/papers/145f04b4b9a1d7dcb68f8e7d5bbe665b)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@bencardoen & @raacampbell, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review.
First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @adamltyson know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @raacampbell

📝 Checklist for @bencardoen

commented

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf
commented 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.1364/OPEX.13.007052 is OK
- 10.1126/science.1127344 is OK
- 10.1038/nmeth929 is OK
- 10.1126/science.1137395 is OK
- 10.1021/acs.chemrev.6b00815 is OK
- 10.1016/j.bpj.2017.11.2912 is OK
- 10.21105/joss.05563 is OK
- 10.1371/journal.pone.0203291 is OK
- 10.1016/j.celrep.2023.113603 is OK
- 10.1038/s41467-022-34894-2 is OK
- 10.1002/0471142727.mb1420s92 is OK
- 10.5281/zenodo.4289803 is OK

🟡 SKIP DOIs

- None

❌ MISSING DOIs

- None

❌ INVALID DOIs

- None
commented

Software report:

github.com/AlDanial/cloc v 1.90  T=0.31 s (1324.8 files/s, 185930.6 lines/s)
---------------------------------------------------------------------------------------
Language                             files          blank        comment           code
---------------------------------------------------------------------------------------
MATLAB                                 140           3471           6950          17329
XML                                     88              0              0          14990
C++                                     90           1049            856           3823
Markdown                                71            677              0           2530
C/C++ Header                             7            436            324           1599
Python                                   7            414            416           1574
Visual Studio Solution                   4              4              4           1006
TeX                                      1             16              0            180
YAML                                     1              6              7             18
Windows Module Definition                1              0              0              2
JSON                                     1              0              0              1
---------------------------------------------------------------------------------------
SUM:                                   411           6073           8557          43052
---------------------------------------------------------------------------------------

Commit count by author:

   580	dschodt
    91	Hanieh
    70	Sandeep
    59	sajjad88
    28	ellyse-taylor
    18	Michael Wester
    12	HMFarsibaf
    12	kiwibogo
    11	Keith Lidke
     8	Sandeep Pallikkuth
     8	TIRF
     7	Keith A. Lidke
     5	MohamadFazel
     3	kalidke
     2	Hanieh Mazloom Farsibaf
     2	Sajjad Khan
     2	Sheng Liu
     1	Ali
     1	Gert-Jan Bakker
     1	MJWester
commented

Paper file info:

📄 Wordcount for paper.md is 1135

✅ The paper includes a Statement of need section

commented

License info:

✅ License found: MIT License (Valid open source OSI approved license)

commented

@bencardoen, @raacampbell, this is the review thread for the paper. All of our communications will happen here from now on.

As a reviewer, the first step is to create a checklist for your review by entering:

@editorialbot generate my checklist

as the top of a new comment in this thread.

These checklists contain the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. The first comment in this thread also contains links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention #7275 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

Hardware submissions can be tricky to review, as not all reviewers will necessarily be able to review all functionality. I hope with three reviewers we can cover anything. However, if there are aspects of the submission you can't review, please let me know.

We aim for reviews to be completed within about 2-4 weeks. Please let me know if any of you require some more time. We can also use EditorialBot (our bot) to set automatic reminders if you know you'll be away for a known period of time.

Please feel free to ping me (@adamltyson) if you have any questions/concerns.

commented

Review checklist for @raacampbell

Conflict of interest

  • I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

General checks

  • Repository: Is the source code for this software available at the https://github.com/LidkeLab/matlab-instrument-control?
  • License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
  • Contribution and authorship: Has the submitting author (@sajjad88) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
  • Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
  • Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
  • Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
  • Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

  • Installation: Does installation proceed as outlined in the documentation?
  • Functionality: Have the functional claims of the software been confirmed?
  • Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
  • Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
  • Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
  • Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
  • Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

  • Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
  • A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
  • State of the field: Do the authors describe how this software compares to other commonly-used packages?
  • Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
  • References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

COMMENTS

General

The idea behind this project is excellent. Having an organised collection of classes that control hardware is definitely useful. Structuring it such that classes inherit from abstract base classes to enforce consistency is exactly the correct thing to do. I commend the authors for this. My main reservation a present is that documentation is far too sparse. There really needs to be a good documentation with extensive examples, links to projects that have used the system, etc, etc. Without this, it is very hard envisage the scope of the project. My next reservation is that the organisation and design of project at present confines it to being a useful set of hardware control examples, rather than a well-integrated platform for building hardware-based software projects.

Code Quality

PM100D class

I have not been able to look at all the code and test everything. I looked at a few things and focussed on the ThorLabs PM100D power meter code. I was pleased to see this and did not know it was so easy to talk to the device via VISA. The interface provided by the authors seems rushed, though, and not well thought out. For example:

  • After object instantiation, PM.measure does not produce a reading. The user must first do PM.Ask=power (the other option being 'temp). Why not have measureTempandmeasurePower` methods? I submitted a PR.
  • The class has hard-coded limits for the range of wavelengths it can be set to. It has a method for reading the wavelength range of the sensor but does nothing with it. I submitted PR for this too.
  • The setWavelength method does not have an input argument. Instead the user must set the Lambda property and then run the setWavelength method. This very unexpected. I submitted a PR for this too.
  • The class is also a bit basic and a lot of things are not implemented. e.g. checking and setting auto range. I added a few in a PR, but there are more missing.

In summary, to effectively use the PM100D class I had to perform extensive modifications. There are still more changes I would make, but my PR addresses the main problems.

Lack of a model/view system

One basic thing a project of this sort should fulfil is that it should constitute a well-thought through model/view system such that users can easily use the hardware control classes with any arbitrary GUI. Thus, hardware control logic and GUI code should be separate and independent. Hardware control code should have clearly defined and documented standards and behaviours across classes to make it predictable to work with. Hardware control code should have well designed Observable properties to which GUIs can attach listeners in order to facilitate building views in top of the models (hardware control classes). These features are not fulfilled by MIC. The presence of GUI control code in hardware classes particularly problematic. For example, in MIC_ThorlabsLED.m at line 134 setting the power runs a GUI update; this sort of thing is present throughout. To be a useful general-purpose hardware control system MIC must have a well thought through model/view structure.

Hardware settings

One problem that is encountered when building software systems of this sort is the need to handle hardware settings that might differ between rigs and projects. e.g. stage limits, camera frame rates, etc. This allows software to start up with safe and useful default values or to maintain settings across settings by caching them in a text file. It would have been great, therefore, if MIC included a class for handling hardware settings. e.g. writing them to a file. checking they are within range and valid, maybe linking property changes to values in the file, etc. I have code that largely does these things if the authors would like to adapt it...

Tests

All classes seem to have a unitTest method. However, these are not true unit tests as they must be run interactively by the user and they do no contain pass/fail assertions. These methods are closer to demos of the classes' behavior. See: LidkeLab/matlab-instrument-control#125

I realise it can be hard to unit test hardware control code, but I feel that it should be clear what the so-called "unitTest" methods are in the code. Their naming creates an expectation that is not fulfilled. I have suggestions in the Issue, above.

Overall

Building a coherent software toolset for that works together well for hardware control is hard. It requires a lot of planning and really well written classes so users have faith that everything will work as expected. I do not feel that MIC in its current state fulfils these requirements. Some classes do prove to be useful examples for how to do something (e.g. I didn't know I could communicate with the power meter via VISA), but I would take the key information from MIC and roll my own solution instead. I wouldn't try to build a system using MC. The main reasons for this are:

  1. Insufficient documentation. That includes insufficient documentation on what level of standardisation I can expect across different classes. e.g. I can only discover if all camera classes are intended to behave the same way if I read through the code. There is not documentation confirming this or unit tests to verify it (I don't consider the unitTest methods in the classes to be real unit tests).
  2. Lack of a model/view concept that would make integration of hardware control code into arbitrary GUIs easy. Also presence of GUI code in some control classes. I would have to re-write MIC to use it effectively in my own GUIs.
  3. Lack of a system for handling hardware settings.
  4. I see to many things like: hardcoded device serial numbers in MIC_NanoMaxPiezos.m or no ability handle the presence of multiple VISA devices in MIC_PM100D. I therefore would have to solve lots of problems within MIC in order to use it.

I am sorry if this comes across as negative. I enjoyed reviewing this submission and learned things from it. To my knowledge MATLAB is missing an effective solution of the sort proposed here, but such a solution is actually very hard to do well. MIC represents a first step but is not the whole journey.

Issues associated with this review

Issues highlighted as "BLOCKER" represent items that require fixing before acceptance. The remaining Issues mostly fall into the "nice to have" category, but in some cases not addressing them will substantially reduce the potential impact of the work.

Pull requests

commented

Review checklist for @bencardoen

Conflict of interest

  • I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

General checks

  • Repository: Is the source code for this software available at the https://github.com/LidkeLab/matlab-instrument-control?
  • License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
  • Contribution and authorship: Has the submitting author (@sajjad88) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
  • Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
  • Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
  • Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
  • Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

  • Installation: Does installation proceed as outlined in the documentation?
  • Functionality: Have the functional claims of the software been confirmed?
  • Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
  • Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
  • Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
  • Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
  • Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

  • Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
  • A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
  • State of the field: Do the authors describe how this software compares to other commonly-used packages?
  • Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
  • References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?
commented

I am finished with my review.

commented

Thanks @raacampbell.

@sajjad88 you can start looking into addressing these issues now, or you may choose to wait until @bencardoen has had a chance to complete their review.