Unable to connect Raspberry Pi 3 to new WiFi network

So new Ubiquiti Access point installed and running very well. I’m moving everything over to the new combined network “RisingStars” (2.4GHz & 5GHz), previously I had the two networks “FinalCrisis” (2.4GHz) and “RisingStars” (5GHz). This Raspberry Pi, a model B, was connected to “FinalCrisis” as it doesn’t support 5GHz and was to far away as well.

With the Ubiquiti I wanted to just have the one network, SSID, of “RisingStars” and let the Access Point or the device decide what to connect to. So for the transisition stage I have the Ubiquiti running the “RisingStars” (2.4GHz & 5GHz) SSIDs and an old router running “FinalCrisis” (2.4GHz).

So it should have been just a case of running sudo raspi-config changing the network, rebooting and done. Should have been. One reboot later and it’s still connected to “FinalCrisis”. A check of the file /etc/wpa_supplicant/wpa_supplicant.conf shows that “FinalCrisis” is down as the “ssid” to use but, at the top of the file, is this comment: # Generated automatically by /usr/lib/piaware-support/generate-network-config. Aha, I run Flight Aware on this Pi as well, so I need to find what files are used by that script. One of them is the /boot/piaware-config.txt file and this has the entries wireless-ssid and wireless-password set to “FinalCrisis” and “JMS” respectively. So I changed them to the new values, ran the generate-network-config and rebooted…The Pi is still connecting to “FinalCrisis”! To make matters worse running iwlist scan only shows all WiFi networks on the first run, subsequent runs only list “FinalCrisis”. So more messing about with raspi-config, this time making sure the local settings are all correct, i.e. GB or UK. Reboot, run iwlist scan and all networks are there but it’s still connecting to “FinalCrisis”, run iwlist scan and only “FinalCrisis” is showing again! I found a reference to /etc/network/interfaces as being another possible file to change, so I edited that and once again the first line says: # Generated automatically by /usr/lib/piaware-support/generate-network-config! But there was a reference to another file: /etc/wpa_supplicant/wpa-roam.conf, which was similar to files I’ve looked at so I dived into there, that also says it’s generated by the aforementioned piaware script, arrgh!

In the end I made manual changes to all three files and ran the generate-network-config script, rebooted and voila “RisingStars” is now shown as the connected SSID by both iwgetid and the DHCP server I’m running. I still can’t get iwlist to show all networks after the first run it just shows “RisingStars” now, no idea why.

Hope this helps someone, and if anyone knows why please leave a comment.

Internet connection utility for Raspberry Pi (Or my first Python script)

I have a Raspberry Pi server, only a Pi 3, but it copes very well with what it has to do.

  • It’s my main webserver however, for various reasons, it handles all internet traffic via a 2.4Ghz Wi-Fi connection so very suprising that it copes.
  • It’s located in the loft space as it runs several flight tracking applications and needs an external aerial as high as possible.

Recently, either down to a router firmware update or an OS update, it’s become unstable and doesn’t always reconnect whent the Wi-Fi goes down – I’ve still not pinned this down exactly but Wi-Fi connection is the probable cause as another Raspberry Pi connected via ethernet handles this situation perfectly. Anyway, digressing, so I needed to create a job that would check the internet connection is present and if not reboot the Pi and hope that solves the issue, so a really simplistic fix as this doesn’t take into account issues with the router/network/ISP and just assumes it’s the local Wi-Fi connection that’s gone. This post is how I went about creating the job and then tweaking it, and using GPT to verify.

So the first draft of the code is below.

import os
import time
import subprocess

def check_wifi_connection(host="8.8.8.8"):
    try:
        response = subprocess.check_output(["ping", "-c", "1", host])
        return True
    except subprocess.CalledProcessError:
        return False

def reboot_raspberry_pi():
    print("Wi-Fi connection lost. Rebooting Raspberry Pi...")
    time.sleep(5)  # Delay before rebooting to allow any running processes to finish
    os.system("sudo reboot")

def main():
    print("Wi-Fi monitoring script started.")
    while True:
    if not check_wifi_connection():
        reboot_raspberry_pi()
        time.sleep(10)  # Adjust the interval as per your requirements

if __name__ == "__main__":
    main()

While generating the second draft I found out that Python doesn’t like the mixing of tabs and spaces, it generates runtime errors. Takes me back to the 80’s! Had to make changes to Notepad++ and the Tab Settings for python so that the Use default is unchecked, Tab size is set to 4 and Replace by space is ticked (checked).

So the working implementations now follow. First one is the most basic just check the connection and reboot the Pi if there isn’t one.

import os
import time
import subprocess

def check_wifi_connection(host="8.8.8.8"):
    try:
        response = subprocess.check_output(["ping", "-c", "1", host])
        return True
    except subprocess.CalledProcessError:
        return False

def reboot_raspberry_pi():
    print("Wi-Fi connection lost. Rebooting Raspberry Pi...")
    os.system("sudo reboot")

def main():
    print("Wi-Fi monitoring script started.")
    if not check_wifi_connection():
        reboot_raspberry_pi()
    print("Wi-Fi monitoring script finished.")

if __name__ == "__main__":
    main()

The above Python script, wifi_monitor.py, is run by a cron job every three hours:

33 */3 * * * python3 /home/pi/wifi_monitor.py

Now we’re going to add an email notification to the script, this will take a few iterations to get right. The first script is the example Python template to send an email, generated using Chat GPT (Interesting that it uses gmail):

import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart

# Replace these placeholders with your own information
sender_email = 'your_email@gmail.com'
receiver_email = 'receiver_email@example.com'
smtp_server = 'smtp.example.com'
smtp_port = 587  # Default SMTP port for TLS
smtp_username = 'your_email@gmail.com'
smtp_password = 'your_email_password'
subject = 'Sample Email Subject'
body = 'This is the body of the email.'

# Create a MIMEText object for the email body
email_body = MIMEText(body, 'plain')

# Create a MIMEMultipart object and attach the body
msg = MIMEMultipart()
msg.attach(email_body)
msg['From'] = sender_email
msg['To'] = receiver_email
msg['Subject'] = subject

try:
    # Connect to the SMTP server
    server = smtplib.SMTP(smtp_server, smtp_port)

    # Start TLS encryption (optional, but recommended)
    server.starttls()

    # Login to your email account
    server.login(smtp_username, smtp_password)

    # Send the email
    server.sendmail(sender_email, receiver_email, msg.as_string())

    # Close the SMTP server connection
    server.quit()

    print('Email sent successfully!')
except Exception as e:
    print(f'Error: {str(e)}')

But before I can implement it there is already a problem! How do you send an email to say the Internet connection has been lost when you have no Internet?! My stop gap solution is to create a file just before the reboot of the Raspberry Pi, when the loss of Internet is detected. Then on system startup, or after three hours as per the cron settings above, the Python script runs, checks for the presence of the file and deletes it, then sends an email saying a reboot has occurred because the Internet connection was lost, or it reboots the Pi again…

So my current monitoring script now looks like this:

import os
import os
import time
import subprocess
import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart

#Internet down marker
no_internet_file = "/tmp/no_internet.txt"

def is_internet_connected(host="8.8.8.8"):
    try:
        response = subprocess.check_output(["ping", "-c", "4", host])
        return True
    except subprocess.CalledProcessError:
    # Create a file to indicate no internet connection
        with open(no_internet_file, 'w') as file:
            file.write('No Internet connection detected.')
        return False

def reboot_raspberry_pi():
    print("Internet connection lost. Rebooting Raspberry Pi...")
    os.system("sudo reboot")

# Function to send an email
def send_email(subject, body):
    sender_email = 'RaspberryPi@example.com'  # Replace with your email
    receiver_email = 'RaspberryPi@example.com'  # Replace with the recipient's email
    smtp_server = 'smtp.example.com'  # Replace with your SMTP server
    smtp_port = 587  # Default SMTP port for TLS
    smtp_username = 'your_email@gmail.com'  # Replace with your email
    smtp_password = 'your_email_password'  # Replace with your email password

    # Create a MIMEText object for the email body
    email_body = MIMEText(body, 'plain')

    # Create a MIMEMultipart object and attach the body
    msg = MIMEMultipart()
    msg.attach(email_body)
    msg['From'] = sender_email
    msg['To'] = receiver_email
    msg['Subject'] = subject

    try:
        # Connect to the SMTP server
        server = smtplib.SMTP(smtp_server, smtp_port)

        # Start TLS encryption (optional, but recommended)
        server.starttls()

        # Login to your email account
        server.login(smtp_username, smtp_password)

        # Send the email
        server.sendmail(sender_email, receiver_email, msg.as_string())

        # Close the SMTP server connection
        server.quit()

        print('Email "', subject, '" sent successfully!', sep='')
    except Exception as e:
        print(f'Error: {str(e)}')


def main():
    print("Internet monitoring script started.")

# On startup, check if the file exists
    if os.path.exists(no_internet_file):
        # Send an email notification
        subject = 'Loft Raspberry Pi rebooted due to No Internet Connection'
        body = 'The Raspberry Pi, in the loft, has been rebooted due to a lack of Internet connection.'
        send_email(subject, body)

        # Delete the file
        try:
            os.remove(no_internet_file)
        except FileNotFoundError:
            pass
    
    if not is_internet_connected():
        reboot_raspberry_pi()

    print("Internet monitoring script finished.")

if __name__ == "__main__":
    main()

So the final leg now. As mentioned this Raspberry Pi also runs several flight tracking programs. One of them, Dump1090, has been failing recently but not on this Pi, on another Pi. My suspicion is that the USB antenna has a dodgy connection as restarting the Dump1090 process doesn’t resolve the issue but a full reboot of the Pi does. So now I want to expand the Internet checking process to also cover checking Dump1090. Simples. Off we go…

import os
import time
import subprocess
import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart

# Define the RaspberryPi location
raspberry_bush = "Window"

# Internet down marker
no_internet_file = "/tmp/no_internet.txt"

# Define the command to check the Dump1090 status
status_command = "sudo systemctl status dump1090-fa"

def is_internet_connected(host="8.8.8.8"):
    try:
        response = subprocess.check_output(["ping", "-c", "4", host])
        return True
    except subprocess.CalledProcessError:
    # Create a file to indicate no internet connection
        with open(no_internet_file, 'w') as file:
            file.write('No Internet connection detected.')
        return False

# Define a function to check the Dump1090 status
def check_dump_status():
    try:
        # Run the status command and capture its output
        status_output = subprocess.check_output(status_command, shell=True, text=True)
        
        # Check if the output contains "FAILURE" to determine if data is NOT being sent
        if "FAILURE" in status_output:
            return False
        else:
            return True
    except subprocess.CalledProcessError:
        return False

def reboot_raspberry_pi():
    print("Rebooting Raspberry Pi...")
    os.system("sudo reboot")

# Function to send an email
def send_email(subject, body):
    sender_email = 'RaspberryPi@example.com'  # Replace with your email
    receiver_email = 'RaspberryPi@example.com'  # Replace with the recipient's email
    smtp_server = 'smtp.example.com'  # Replace with your SMTP server
    smtp_port = 587  # Default SMTP port for TLS
    smtp_username = 'your_email@gmail.com'  # Replace with your email
    smtp_password = 'your_email_password'  # Replace with your email password

    # Create a MIMEText object for the email body
    email_body = MIMEText(body, 'plain')

    # Create a MIMEMultipart object and attach the body
    msg = MIMEMultipart()
    msg.attach(email_body)
    msg['From'] = sender_email
    msg['To'] = receiver_email
    msg['Subject'] = subject

    try:
        # Connect to the SMTP server
        server = smtplib.SMTP(smtp_server, smtp_port)

        # Start TLS encryption (optional, but recommended)
        server.starttls()

        # Login to your email account
        server.login(smtp_username, smtp_password)

        # Send the email
        server.sendmail(sender_email, receiver_email, msg.as_string())

        # Close the SMTP server connection
        server.quit()

        print('Email "', subject, '" sent successfully!', sep='')
    except Exception as e:
        print(f'Error: {str(e)}')


def main():
    print("Dump1090 and Internet monitoring script started.")

# On startup, check if the no Internet file exists
    if os.path.exists(no_internet_file):
        # Send an email notification
        print("Internet down file found")
        subject = raspberry_bush + ' Raspberry Pi rebooted due to No Internet Connection'
        body = 'The Raspberry Pi, on the " + raspberry_bush + ", has been rebooted due to a lack of Internet connectivity.'
        send_email(subject, body)

        # Delete the file
        try:
            os.remove(no_internet_file)
        except FileNotFoundError:
            print("File deletion failed")
            pass
    
    if not is_internet_connected():
        reboot_raspberry_pi()

    if not check_dump_status():
        subject = raspberry_bush + ' Raspberry Pi rebooted as Dump1090 not connected.'
        dump_status = subprocess.check_output(status_command, shell=True, text=True)
        body = str(dump_status)
        send_email(subject, body)
        reboot_raspberry_pi()

    print("Dump1090 and Internet monitoring script finished.")

if __name__ == "__main__":
    main()

I highlighted the latest changes for the Dump1090 checks. Hopefully it’s all clear, any questions please ask!

4-Jan-2024

Remember to add the file and command to the right cron entry though. For example sudo crontab -e doesn’t edit the same file as crontab -e, the latter edits settings for the current user the former, I believe, for root; then you can also edit /etc/crontab, which is for the system.

2023 Router tweaks

So time for the quarterly set of tweaks to see if I can improve my speeds, reliability etc. This time I’m trying to sort out an issue with Wi-Fi which is causing a server, located in my garage, to be unresponsive on the network.

The first set of changes were nothing to do with the router but to qBittorent, in my aim to maximise the upload speed I’d allowed the software to swamp the local network so the Server wasn’t actually able to connect to the network and get an IP address! Some of these tweaks are unique to the Billion BiPAC 8900AX I use while others are standard and so relate to all, I’ll try and identify which is which.

MMM  DD HH:MM:SS kern warn kernel: Missed Membership report from 169.254.149.241 for 224.0.0.252 1 times. Waiting..

So in the router logs I frequently saw the above warning. The 169.254.149.xxx means the Server hasn’t actually been assigned an IP address yet, have a search for that range to find out more details. Once I’d reduced the number of connections allowed for all torrents and active seeding torrents I started to see less of those issues, they still occur but at least the Server does get assigned a local IP address during the day now. For the Billion router you can set QoS rules, under Configuration->Quality of Service. For these to work for torrents I had to pre-configure the port to use on inbound and outbound connections, I don’t use UPNP which isn’t a bad thing it seems, again search for security issues round that and generally it seems turning it off is favoured. So my torrent connections are now set to a low priority on the outbound (upload/LAN to WAN) side and limited, on the inbound (download/WAN to LAN), side to 67% of the available bandwidth, these rules also only apply to the torrent server, which has a fixed IP address (This server has an ethernet connection to the network and so always got an IP address). This allows Streaming and browser to continue quite happily with no interruptions whereas before this stuttering, on streaming was common.


Now onto the generic Wi-Fi changes…

Bizzarely this involved reducing the bandwidth settings in the 2.4GHz/5GHz ranges. For the 2.4GHz I now have a bandwidth of 20MHz, previously I tried to use 40MHz but the local area just couldn’t sustain that because of neighbours networks, while the 5GHz is currently set to 40MHz. The result is that the 2.4GHz has greater stability, as does the 5GHz, but the range of the 5GHz has actually increased into more rooms in the house! This seems to be down to collisions with neighbouring networks, not many four at the worse case, but by trying to use the full bandwidth it was frequently causing issues. Now I’m using less of the bandwidth the quality of the connection, and range, has improved. Seems strange? Yes but I did find many sites that explain this in detail and why it can improve the quality of the connection.

I did also enable “MLD Multicast Proxy”, this is the IP6 version of “IGMP Multicast Proxy” it turns out, I wasn’t aware until I started searching for improvements. But has it made a difference? I can’t tell but it hasn’t made it worse and I do know that enabling IGMP solved our initial issues, when I first started using the Billion, when trying to use Youview and the online TV catchup services.

Lastly I reduced the “RTS Threshold” from the maximum 2347 down to 2040. Again no noticeable improvement but it’s not worse. This is my current focus though, so does it need to go lower? I’ve seen recommendations to drop it to 500 but these are accompanied with the caution to do so gradually as that doesn’t suit all. So, hopefully, I’ll get some updates to this later.

DNS lookup issues

So you’re using the internet and sites are loading, speed tests are reporting the normal speed but still there’s something amiss; pages seem sluggish to respond.
Try

nslookup www.google.com

from the command line and see what happens.
If you get some timeouts then maybe your DNS servers are slow to respond. Try the same command but use different DNS values, for example Google’s:

nslookup www.google.com 8.8.8.8

is that slower or faster?
Just had the same issue and had to revert back to using the ISP’s DNS’s then I could change back to openDNS and all was fine, once again? Why? No idea, I’d already rebooted the router and PC before then.

A tale of two drives

So just happened to spot, when running HWiNFO64 to check motherboard version, that my 250GB Crucial MX500 is down to 8% life remaining! What? Couple of months of go when I was checking that value it had 28% life remaining, so all good and plenty of time to plan my next upgrade. Now I’m in panic mode and ordering a new 500GB Crucial MX500 for £43, which is a good price as the 250GB cost me £63 just over four years ago.

HWiNFO64 SMART status
SMART Status from HWiNFO64 v7.26

So what happened? Seems that the Crucial drives have a potential with many small packet writes, this may just be on the model made all those years ago and not on the current models, Crucial have always had a good reputation from reviews. So in the SMART details we can see it has written a lot of data, compared to the other MX500. The failing drive is my boot disk so has the pagefile, core programs (Anit-virus, VirtualBox, Browsers) and applications (Office, VLC, Notepad++) on it.while the other SSD, only a couple of months newer and running the same firmware version, has all the games, documents, photographs and BOINC application.

So the failure drive presumably has all the small packet writes (temporary files, browser cache & cookies, pagefile) while the still okay SSD has the larger writes (photographs, documents).

So just about to install the new 500GB Crucial MX500 replacement drive, and we’ll see how that goes over the coming month. After that the PC will probably be passed on as I should have my new one built and moved into the world of NVMe drives.

10-Sep-22:
Quick update to this. The machine, Treacle, has now been retired and relegated to a hardware test role, admittedly old hardware. The new Crucial MX500 drive is at 100% life remaining while the old 1TB MX500 has only 64% life remaining, so still pretty good.

So is it aceptable to sell an SSD that has only 8% life remaining? It will need a secure erase, which presumably will reduce that further, cue another addition to this post.

18-Nov-22:
The secure erase, three passes using “British HMG IS5” via Eraser an overnight job, didn’t reduce the disk life any. So I put the drive on eBay with full disclosure of the remaining drive life, including screen grabs from HWInfo SMART data, and a starting price of £3 + £3 p&p. One week later it sold for £26 inc p&p, when you can buy a brand new one for £38.08 inc p&p! The buyer, one of two bidding, was happy with the drive and condition. So why pay, what I consider, over the odds? Did they expect to be able to retrieve data from the drive? I have had a close friend, computer illiterate it now seems, sell a computer after physically destroying the drive only to be contacted by the buyer to say they have his personal data now (The buyer did send the drive back).

So is there a market in second hand drives purely for what data you can retrieve from them? I know it’s possible to get data from incorrectly erased drives, I’ve had to recover said data for friends in the past, so is that data of use to someone? You’d need some serious time and software to be able to make use of any account/password data on the drive.

Facebook – Login Error – An unexpected error occurred

Having the above error? Getting the same thing on your phone, Firefox, Edge or Chrome desktop browsers? Tried the web and that suggests deleting your cache, deleting the app but still you get the error?
Do you have one machine that still can log into Facebook? If you do then go and turn off, I know this is unsafe, 2FA (In my case my mobile and email systems). Now, no need to clear your cache, just log in as normal and job done!
You can now turn 2FA back on. What’s the point of 2FA if you have to turn if off to log in? No idea! I received two SMS messages, from Facebook, while 2FA was on, but no opportunity to enter any of the codes received.
Maybe this was all just a “glitch” on the day and will go away tomorrow.

Logging “Audit Success” in Windows Logs

I noticed, while reviewing my logs, that I still get masses of “Audit Success” entries in the Security logs. What I mean is 30+ entries every second, seems an insane number to me, even more so as they were all the 4799 event. I mean so a membership was successfully enumerated? Okay move on, but these entries were now in the tens of thousands.

Much hunting round and I found that since Windows 7, I think, logging of successful events is now on by default. So unless you find the process/Service ID GUID of the services triggering the event and turning them off individually or setting them to “Failure”, which would take weeks trying to remove them you’re stuck, well unless your knowledge of audit policy commands is very good.

So welcome to this Superuser.com article, or rather question and answer, to help you out.

https://superuser.com/questions/1516725/how-to-disable-windows-10-system-log

Sneak peak is to run this command: auditpol /set /subcategory:"Filtering Platform Connection" /success:disable /failure:enable
To disable successful Credential Manager reads, another frequently logged event, use:
auditpol /set /subcategory:"Credential Validation" /success:disable /failure:enable

The longer version is to read the article and find out how to remove other event types. Either way I’m now down to four or six “Audit Success” events being logged every couple of minutes, and those 4799 events that hid a load of other information are gone now. Woohoo

Enabling WPS on a Billion BiPAC 8900AX

By default WPS is disabled on the 8900AX, so if you need to connect a device, TV, printer etc, using WPS then you will have to go into the Configuration menu and enable it.

  • This can be found under the Configuration->Wireless 2.4G->Security menu.
  • Select the “Enable” value for the “WPS” combo, under “WPS Setup”.
  • Ensure that the “Add Client” option is set to “Use STA PIN”.
  • Make sure that the “PIN” value is blank.
  • Leave all other options at their default settings.

Once this is done all you need to do, to connect the device using WPS, is to press the WPS button on the router, for around a second, and then perform the equivalent activity on the device you want connected.

Remote Desktop – Custom screen size

Sometimes you don’t want to run an RDP session full screen but you do want to want to make use of the real estate more than the default RDP settings allow you to choose. Using the RDP panel you can only select set values from the slider control, there isn’t the ability to be fully flexible.

So you have to customise your RDP session but this time using notepad instead. Generally your RDP session will load its default values from the Default.rdp file, held in your “My Documents” folder. So edit this and change the following two lines to whatever value you want, from the 1920×1080 defaults, in my case.

desktopwidth:i:1600
desktopheight:i:1100