Check for internet connection

[Guillaume Vandekerckhove]

• PyDOV version:2.0.0
• Python version:3.7.8
• Operating System: windows x64

Description

After I got the groundwater samples from DOV for a specific area and filtered them on the element As I wanted to add the aquifer code (specified in the groundwater filter) to the result. Due to the fact that I work from where a lot of people using the same internet connection and so the connection is not stable, I wanted to check the internet connection first so that I don't get the error that there is no connection.

-> I want to check the internet connection in a for loop.
-> If there is a connecting run the code
-> If there's no connection wait for 30 seconds and check again the connection

Eventually the code worked and I have the data that I wanted.
Without checking the internet connection, I got the error that maximum tries were reached.

What I Did

import inspect, sys
import pydov
import pandas as pd
import time
import requests

from pydov.search.grondwatermonster import GrondwaterMonsterSearch
from pydov.search.grondwaterfilter import GrondwaterFilterSearch
from owslib.fes import Or, Not, PropertyIsNull, PropertyIsLessThanOrEqualTo, And, PropertyIsLike, PropertyIsEqualTo
from pydov.util.location import Within, Box


gwmonster = GrondwaterMonsterSearch()
gwfilter = GrondwaterFilterSearch()

def internet():
    try:
        requests.get('https://www.google.com/').status_code
        return "Connected"
    except:
        return "Not Connected"

def loop():
    ac = []
    count = 0
    for pkey_filter in df.pkey_filter:
        def connected():
            query = PropertyIsEqualTo(propertyname='pkey_filter', literal=pkey_filter)
            df_filter = gwfilter.search(query=query)
            aquifer_code = df_filter.aquifer_code[0]
            ac.extend([aquifer_code])



        if internet() == "Connected":
            query = PropertyIsEqualTo(propertyname='pkey_filter',literal=pkey_filter)
            df_filter = gwfilter.search(query=query)
            aquifer_code = df_filter.aquifer_code[0]
            ac.extend([aquifer_code])
            count += 1
            print(count, "/", number_of_rows)

        else:
            print("waiting for internet connection")
            time.sleep(30)
            internet()
            while True:
                if internet() == "Not Connected":
                    print("waiting for internet connection")
                    time.sleep(30)
                    internet()
                elif internet() == "Connected":
                   connected()
                   break


    df['aquifer_code'] = ac
    df['datum_monstername'] = pd.to_datetime(df['datum_monstername'])



df1 = gwmonster.search(location=Within(Box(20000, 150000, 30000, 250000)))
df = df1[df1.parameter == 'As']
df = pd.DataFrame(df)
print(df)
index = df.index
number_of_rows = len(index)

loop()

df.to_csv(r'C:\Users\guill\Documents\UGent\stage dov 2021\python\data1.csv', header=True, index=None, sep='\t', mode='a')
df.to_csv(r'C:\Users\guill\Documents\UGent\stage dov 2021\python\data.csv', header=True, index=None, sep='\t', mode='a')

########################################################################################################################

df2 = gwmonster.search(location=Within(Box(30000, 150000, 40000, 250000)))
df = df2[df2.parameter == 'As']
df = pd.DataFrame(df)
print(df)
index = df.index
number_of_rows = len(index)

loop()

df.to_csv(r'C:\Users\guill\Documents\UGent\stage dov 2021\python\data2.csv', header=True, index=None, sep='\t', mode='a')
df.to_csv(r'C:\Users\guill\Documents\UGent\stage dov 2021\python\data.csv', header=None, index=None, sep='\t', mode='a')

########################################################################################################################

[Roel Huybrechts]

The problem might be that your connection is down for longer than we retry? That would mean that it is down for at least 17 minutes. I think that is a reasonable time to bail out?

If your connection is that bad, additional tricks like you implemented might be necessary but I don't think that should be part of every pydov install. Your script will run forever waiting for a connection, I don't think that is something we can implement in a library as this will stall peoples scripts forever in case of a network issue, without warning or error.

It seems you want to get the aquifer_code for each groundwater sample. This can be queried a lot easier than using a for loop, by using the Join operator:

df2 = gwfilter.search(query=Join(df_monsters, 'pkey_filter'),
                      return_fields=['pkey_filter', 'aquifer_code'])

This dataframe can subsequently be combined with the first one using pd.merge. This avoids the for loop and all the requests entirely, and should be a lot faster and more reliable.

[Guillaume Vandekerckhove]

Thanks for the extra tips. The internet connection is in my case interrupted for max 30 seconds, because I only get once the printed line that it's waiting for internet connection.

[Roel Huybrechts]

That should be no problem in the current master version. Can you try that instead of the 2.0.0 release?

[Roel Huybrechts]

I believe this should be fixed in the 2.1.0 release, can you retry this with the latest release?

[Roel Huybrechts]

I'm going to close this issue since I believe it should be fixed in the last release. If it still does not work for you, don't hesitate to reopen or create a new issue!