Giters

readthedocs / readthedocs-build

Work in Progress builder

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Add submodule configuration

agjohnson opened this issue · comments

commented

Seems a number of projects are hitting issues with submodule clone steps, and this should be configurable in our config. Because git doesn't provide the mechanism for optional submodules, we can do something like:

submodules:
  include: []
  recursive: false

as implicit default. And:

submodules:
  include:
    - foo/
    - bar/

As include example.

submodules:
  exclude:
    - foo/

^ is the exclude example.

Legacy projects implemented before this change would have an implicit configuration of:

submodules:
  include: all
  recursive: true

It's not possible to include and exclude submodules, this is a validation error.

This would eventually alter the code that rtfd/readthedocs.org uses for submodule cloning, and would only selectively clone things? The changes there are likely moving to specific submodule clone commands.

  • Currently submodule cloning takes place in the git checkout step. This will need to be moved to a new step after we parse the configuration
  • New behavior will be to not clone submodules automatically
  • If possible, we won't recursively clone as well. Maybe this is another option on the spec?
  • We'll add a new project.Feature, plus migration, to lock old users into the old behavior (all submodules are cloned)
  • We can perhaps also detect that a project has submodules that won't be cloned and raise a warning. This warning shouldn't be a build failure, but a notification on-site/through email?
commented

+1 we hit that currently and this is a blocker to us

commented

We'll add a new project.Feature, plus migration, to lock old users into the old behavior (all submodules are cloned)

I think this is kind of related to readthedocs/readthedocs.org#3821

What about versioning the spec of the yaml? I have raised an issue for that readthedocs/readthedocs.org#3806

So, in that case the users without a version key on the yaml, have a default version (1.0?). I know this would be a big change on the code base, but would allow us to make more changes in the future without worrying about the other projects.

For getting the submodules path we can parse the git submodule status command output or maybe parse the .gitmodules file so we can implement the exclude option.

include: all

I like the idea of the all keyword here, this keyword could be used on other parts of the spec. On formats for example.

commented

Yeah, i think we need a version in our config, definitely.

@haubourg we have a stop gap measure in place for skipping all submodule checkout. What's your project? I can enable the feature if you don't want any submodules at all.

commented

@agjohnson Hi! our project is http://qwat.readthedocs.io/en/latest/.
If you can enable the submodule fence, that will help a lot thanks!

commented

@haubourg I just added that flag in qwat project. Let me know if that worked for you. Thanks.

commented

@humitos that made the trick, many thanks !

commented

@bochen87 done! Let me know if it's OK :)

commented

@humitos works, thanks! :-)

commented

Are we going to support this on the v1 of the yaml file? Or maybe to do this only on the v2.

commented

Sounds like we should probably support it in the v1. Seems like a useful option.

commented

@humitos Could you disable submodule cloning on https://readthedocs.org/projects/firesim/

We have similar issues with submodules that are currently private (and also very large), but unnecessary for building our documentation. For now, I've made a branch with .gitmodules ripped out to get our documentation up, but that's not ideal. Thanks a bunch!

commented

@sagark done. It should skip the submodules now.

commented

Thanks, works for us now!

commented

@humitos May I also get the submodule skip magic?
http://rematch.readthedocs.io/en/latest/

Thanks!!

commented

Hi, I was wondering if someone could share an update on this? any workaround perhaps? I'd like to resume using readthedocs but cannot as I have a submodule not related to the docs that causes the same error.

commented

@nirizr we are planning this for the v2 of the config file (would be available soon), meanwhile a core maintainer needs to add a flag to your project to skip submodules.

commented

@stsewd thanks for the response. How can I get a maintainer to add that flag?

commented

@nirizr can your provide a link to your rtd project? I think @humitos can help you here

commented

I already did, a few comments above. Anyway, here it is again: http://rematch.readthedocs.io/
Thanks for your assistance!

commented

@nirizr done! I triggered a build for your latest version and it did pass.

commented

Thanks! Much appreciated!

commented

Migrating this discussion to readthedocs/readthedocs.org#4464. We'll be implementing this feature there instead of this repo.

commented

@humitos Can you also disable submodules for https://github.com/keystone-enclave/keystone ?
I think the yaml feature is not being used in readthedocs.org yet is it?
Thanks!

Dayeol

commented

@stsewd IIRC there is a setting to enable YAML v2 in some projects, right? If so, maybe this is a good opportunity to add keystone and let the user to use the skip submodules options from there. What do you think?

commented

I'm running into the same issue with a private submodule. However, I'm a complete newbie and just followed basic tutorial steps. My projects isn't very advanced and there could be hundreds of other things that have gone wrong, but it would be helpful to get the submodule problem out of the way to start fixing everything else that is potentially wrong - any advice would be appreciated. :-)
https://readthedocs.org/projects/multimatch/ from https://github.com/AdinaWagner/multimatch

commented

@humitos yes we have submodules configuration working in v2, but the current schema will change soon, so I'm not sure if users want to keep an eye on the changes. And also the docs are not deployed yet.

commented

@stsewd I was thinking about them helping us to test our V2 config as "beta tester users". I don't think they will need to use many of the configs though and just probably be enough with the simple ones + the exclude submodules. I don't know, just an idea.

commented

That would be great to have test users. Basically, the config would be something similar to what we use on the rtd repo https://github.com/rtfd/readthedocs.org/blob/master/.readthedocs.yml

version: 2
sphinx:
  configuration: docs/conf.py
python:
  requirements: requirements.txt

Submodules are disabled by default in v2

commented

I gladly help as a beta tester, @stsewd & @humitos, and try out what you propose. I probably would need a bit a guidance, though. I have now included a .readthedocs.yml based on @stsewd 's suggestion at the root of my directory (https://github.com/AdinaWagner/multimatch/blob/master/.readthedocs.yml).
Currently, the build is failing with the error: "Problem in your project's configuration. Invalid "format": expected list".
The conf.py file is here (https://github.com/AdinaWagner/multimatch/blob/master/docs/source/conf.py) and was made with the quickstart wizard.
Could you give me some insights on how to fix this?

commented

We need to activate a flag from our side to enable v2 for your project.

commented

sure - is there something I need to do at this point?

commented

Hi @AdinaWagner and @dayeol!

I enabled YAML configuration V2 for your projects (mulitmatch and keystone). So, now you will need to create/edit the YAML file to something similar to what @stsewd mentioned at #30 (comment)

Note that you will need to remove/add the settings you need, but the key point is to use version: 2 at the top of the file.

Hope it works!

commented

In case there is something wrong with this, open a new issue at https://github.com/rtfd/readthedocs.org/issues/new explaining this and linking this one there.

commented

Thanks, the docs built successfully! :-)

commented

Awesome!

commented

I'll let you know here when the schema change, here are some docs about the current implementation readthedocs/readthedocs.org#4451

commented

Hi, I added .readthedocs.yml and added version: 2 at the top, but still readthedocs is checking out all the submodules. Can you help?
I'm testing in test-rtfd-v2 branch:
yml: https://github.com/keystone-enclave/keystone/blob/test-rtfd-v2/.readthedocs.yml
conf.py: https://github.com/keystone-enclave/keystone/blob/test-rtfd-v2/docs/source/conf.py

commented

@dayeol can you please provide your project url?

commented

@stsewd Another problem is that SSL certificate is invalid, so the website shows red flag.
Thank you for your help!

commented

@humitos believe you added the flag to the wrong project (keystone instead of keystone-enclave), the other project shouldn't have any problem anyway.

I think you have misconfigured the cname record, please see https://docs.readthedocs.io/en/latest/custom_domains.html

$ dig +short docs.keystone-enclave.org
keystone-enclave.readthedocs.io.
137.116.78.48
commented

@stsewd Another problem is that SSL certificate is invalid, so the website shows red flag.

Please, open a new issue for this problem at https://github.com/rtfd/readthedocs.org/issues/new

commented

@humitos believe you added the flag to the wrong project (keystone instead of keystone-enclave), the other project shouldn't have any problem anyway.

Added the flag to keystone-enclave now.

Hope it works now.

commented

Hi @stsewd, as discussed on Gitter, could you enable v2 config on my site https://docs.pantheon.pegasys.tech/ ? Thanks a lot.

commented

Thanks I will test that and tell you.

commented

@NicolasMassart @AdinaWagner @dayeol

Hi, we just deployed some new changes to our v2 schema, so your projects will need to update the configuration file.

The main difference is in the requirements key.

What looks like this:

python:
   requirements: requirements.txt

You'll need to change it to

python:
    install:
        - requirements: requirements.txt

You can see the full updated docs in https://github.com/rtfd/readthedocs.org/blob/329286f2f8c811c6bda37f664546de73945dfe37/docs/config-file/v2.rst

or in the pull request readthedocs/readthedocs.org#4451

commented

thanks a lot for letting me know @stsewd - I've implemented the change!