Category Archives: python

Flask It

If you have worked around Python and Web technologies then you have heard of Flask. It is well known and there are a lot of examples for getting started with simple apps.

Simple/single apps are great if you are learning, but quickly bog down if you have even a little bit of complexity.

Here are some key things to know before you get too far into a monolithic application.

Modules

Blueprints is the Flask concept for bringing in sub-things or modules. The official documentation is clear and provides an example for rendering a page, but similar concept applies for APIs if your application has no web content.

Our recommendation is, as soon as you have the simple app understood, break it into modules (in your preferred pattern) before adding any more routes or pages.

Exceptions

If you leave the exception handling up to Flask, especially if you have an API only application, you will end up with the base errors being returned, like Method Not Allowed or Server Error and the calling client will have no idea what the cause may be.

To deal with this, you can use exception handlers. Create a MyApplicationException and when that exception occurs, return a json object with meaningful description which your client can then understand. In this Handling Application Errors example, instead of catching HTTPException, catch MyApplicationException, and adjust so that response = HTTPException().get_response() to setup the base exception to send onwards by returning response, e.code.

from flask import json
from werkzeug.exceptions import HTTPException

@app.errorhandler(MyApplicationException)
def handle_exception(e):
    """Return JSON instead of HTML for our application errors."""
    # start with an empty http exception as that is what we wish to return
    response = HTTPException().get_response()
    # replace the body with JSON
    response.data = json.dumps({
        "code": e.code,
        "description": e.description,
        #any other key values you want the client to have access to
    })
    response.content_type = "application/json"
    return response, e.code

PySide6 6.2 to 6.3 Upgrade

If you use PySide6 and have developed against PySide6 version 6.2.x, and you upgrade to 6.3 or later, you may find the following failures.

Module Import Error

ModuleNotFoundError: No module named 'PySide6.QtCore'

The Qt documentation hints at it, so if you performed just a pip install to the new version, then it is likely you’re issue. Instead uninstall and re-install.

In case you are updating your 6.2.x install, make sure the PySide6_Essentials and PySide6_Addons wheels are getting installed, otherwise, uninstall and install PySide6 to get the new structure.

From: https://www.qt.io/blog/qt-for-python-details-on-the-new-6.3-release

So in your virtual environment, or main environment

# ensure all previous installs uninstalled
python3 -m pip uninstall PySide6
python3 -m pip uninstall PySide6-Essentials
python3 -m pip uninstall PySide6-AddOns
# and finally re-install
python3 -m pip install PySide6

Could not load the Qt platform plugin “xcb”

After upgrading PySide6 to 6.5 on a Ubuntu 20.04, attempting to load an application functional with PySide6 6.2.4 you may find it fails on a missing plugin as follows.

qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: xcb, vnc, wayland, minimalegl, vkkhrdisplay, offscreen, wayland-egl, minimal, linuxfb, eglfs.

If you enable debug by adding the environment variable …

QT_DEBUG_PLUGINS=1

… you may see later versions of PySide6 are looking for -cursor xcb library also.

libxcb-cursor.so.0: cannot open shared object file: No such file or directory

This can be installed using apt install as follows

sudo apt install libxcb-cursor0

Setting up LabJack U3 On Linux for Python

Here’s a collation of steps to setup a U3 LabJack device on Linux, in this case Ubuntu 20.04.

Download tar ball

Downloaded the Linux x86_64 Release Linux 64-bit Installer Package from:

https://labjack.com/pages/support/?doc=/software-driver/installer-downloads/ljm-software-installers-t4-t7-digit/#header-three-3r1sj

Extract and Install

Extract

$ tar -xzf labjack_ljm_software_2019_07_16_x86_64tar.gz

$ ls -al
labjack_ljm_software_2019_07_16_x86_64 
labjack_ljm_software_2019_07_16_x86_64tar.gz

Install

$ cd labjack_ljm_software_2019_07_16_x86_64
$ sudo ./labjack_ljm_installer.run 
[sudo] password for user: 
Creating directory labjack_ljm_software
Verifying archive integrity... All good.
Uncompressing LabJack software for T4s, T7s, and Digits for Linux x86_64.......................................................................................................................................................................................................................................................................
Installing libLabJackM.so.1.20.1 to /usr/local/lib... done.
Installing LabJackM.h to /usr/local/include... done.
Installing constants files to /usr/local/share... done.
Installing labjack_kipling to /opt... done.
Installing command line shortcut labjack_kipling to /usr/local/bin... done.
Registering with application launcher... done.
/usr/local/lib >> /etc/ld.so.conf
Adding LabJack device rules... done.
Restarting the device rules... done.

Install finished. Please check out the README for usage help.

If you have any LabJack devices connected, please disconnect and
reconnect them now for device rule changes to take effect.

Verify

Plug in your device (U3 JabJack) to USB port and list /var/log/syslog looking for LabJack U3 to ensure it is found, as below.

$ tail /var/log/syslog
May  3 11:33:50 laptop systemd[5916]: tracker-extract.service: Succeeded.
May  3 11:33:51 laptop kernel: [15761.633639] usb 3-3: new full-speed USB device number 6 using xhci_hcd
May  3 11:33:51 laptop kernel: [15761.782937] usb 3-3: New USB device found, idVendor=0cd5, idProduct=0003, bcdDevice= 0.00
May  3 11:33:51 laptop-01-ub kernel: [15761.782942] usb 3-3: New USB device strings: Mfr=1, Product=2, SerialNumber=0
May  3 11:33:51 laptop kernel: [15761.782944] usb 3-3: Product: LabJack U3
May  3 11:33:51 laptop kernel: [15761.782947] usb 3-3: Manufacturer: LabJack

Install ExoDriver

The above software install does not install all requirements. The exodriver is also required for usb control from Python, and is detailed here:

https://labjack.com/pages/support?doc=/software-driver/installer-downloads/exodriver/#section-header-two-25lef

$ sudo apt-get install build-essential
... output removed
$ sudo apt-get install libusb-1.0-0-dev
... output removed
$ sudo apt-get install git-core 
... output removed
$ git clone https://github.com/labjack/exodriver.git
Cloning into 'exodriver'...
remote: Enumerating objects: 1044, done.
remote: Counting objects: 100% (48/48), done.
remote: Compressing objects: 100% (38/38), done.
remote: Total 1044 (delta 23), reused 19 (delta 10), pack-reused 996
Receiving objects: 100% (1044/1044), 378.59 KiB | 3.08 MiB/s, done.
Resolving deltas: 100% (615/615), done.
$ cd exodriver/
$ sudo ./install.sh 
Making..
rm -f liblabjackusb.so.2.7.0 *.o *~
cc -fPIC -g -Wall  -c labjackusb.c
cc -shared -Wl,-soname,liblabjackusb.so -o liblabjackusb.so.2.7.0 labjackusb.o -lusb-1.0 -lc
Installing..
test -z /usr/local/lib || mkdir -p /usr/local/lib
install liblabjackusb.so.2.7.0 /usr/local/lib
test -z /usr/local/include || mkdir -p /usr/local/include
install labjackusb.h /usr/local/include
ldconfig
Adding 90-labjack.rules to /lib/udev/rules.d..
Restarting the rules..
Install finished. Thank you for choosing LabJack.
If you have any LabJack devices connected, please disconnect and reconnect them now for device rule changes to take effect.

If you do not install or build it, you end up with this error if you try to open a device in Python

~$ python3
Python 3.8.10 (default, Mar 13 2023, 10:26:41) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import u3
<class 'LabJackPython.LabJackException'>: Could not load the Exodriver driver. Ethernet connectivity only.

Check that the Exodriver is installed, and the permissions are set correctly.
The error message was: liblabjackusb.so: cannot open shared object file: No such file or directory
>>> quit()

Install Python Library

Install the python library as per:

https://labjack.com/pages/support?doc=/software-driver/example-codewrappers/labjackpython-for-ud-exodriver-u12-windows-mac-linux/#header-three-qhnmb

$ python3 -m pip install LabJackPython
Collecting LabJackPython
  Downloading LabJackPython-2.1.0-py2.py3-none-any.whl (115 kB)
     |████████████████████████████████| 115 kB 2.2 MB/s 
Installing collected packages: LabJackPython
Successfully installed LabJackPython-2.1.0

If you installed without sudo, you’re library will be in your .local folder.

Find the u3.py file like this

$ find ~ | grep u3.py
/home/user/.local/lib/python3.8/site-packages/u3.py

Open it to have a look, or see it on the github repo

https://github.com/labjack/LabJackPython/blob/master/src/u3.py

Run python3 interpreter and verify you can communicate with your device by instantiating a U3() class, reading an analog input (default state of pins, so we can avoid configuring them at this point) and printing the temperature.

$ python3
Python 3.8.10 (default, Mar 13 2023, 10:26:41) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import u3
>>> d = u3.U3()
>>> print (d.getAIN(4))
0.305592048
>>> print(d.getTemperature())
303.5464268177748
>>> d.close()

PySide6 Snippets

These snippets may apply to PySide2 also but are untested there.

Plain Text to Rich Text Conversion

Handling special characters in Qt Rich Text Widget.

If you are using a rich text widget, then any special characters, like say the less than sign <, if written to the text object will cause your output to be broken. Qt provides a function for these cases: convertFromPlainText

from PySide6.QtGui import Qt
...
    def my_method(plain_text)
        myQmlModel.rich_text_property = Qt.convertFromPlainText(plain_text)

Carriage Returns and Line Feeds in Rich Text do not work with WordWrap

In addition to the convertFromPlainText method, you can not have line feeds in your text, so you need to find and convert those to <br/>. If you do not, WordWrap option will not work correctly and your lines will extend past your widget width. At least in PySide 6.2.3.

Reference: QtGui — Qt for Python


Qt ® is a registered trademark of The Qt Company Ltd. and its subsidiaries.

Python ® is a registered trademark of the Python Software Foundation.

Qt5 to Qt6 Qml/Pyside 2 to 6 Porting Adventures

Here are a few things noticed during a port of a Pyside2 project and associated Qml to Pyside6 (6.2.1).

Injection of parameters into signal handlers is deprecated.

If you hit the following warning, it is formatting quirk now being enforced by Qt 6.

Injection of parameters into signal handlers is deprecated. Use JavaScript functions with form al parameters instead.

In our case it came about in this snippet of Qt5/Qml code on the Main Window’s onClosing handling to avoid closing on a certain condition. As the documentations quotes, you are allowed to set closing.accepted to true or false: Window QML Type | Qt Quick 6.2.1

ApplicationWindow {
    id: application
    onClosing: {
        ...
        if (condition) {
            close.accepted = false
            userDialog.open()
        } else {
            close.accepted = true
        }
    }

The fix is fortunately simple through the addition of the function(close). Refer for more details: Signal and Handler Event System | Qt QML 6.3.0

ApplicationWindow {
    id: application
    onClosing: function(close) {
        ...
        if (condition) {
            close.accepted = false
            userDialog.open()
        } else {
            close.accepted = true
        }
    }

Qml ComboBox indicator is null

Leaving the indicator element as the “default” one by not providing it in Qml in Qt5 (5.12 at least) “worked” in that a sub component could reference its with indicator.width. However porting the Qml to PySide6 resulted in a null dereference.

The contentItem’s rightPadding below references control.indicator.width, which threw the null dereference in PySide6 but not PySide2.

import QtQuick.Controls 2.12
ComboBox {
    id: control
    ...
    contentItem: Text {
        leftPadding: 0
        rightPadding: control.indicator.width + control.spacing
        ...
    }
    delegate: ItemDelegate {
        width: control.width
        ...
    }
    background: Rectangle {
        border.width: 1
        ...
    }
}

Simply adding a dummy Canvas worked around this problem, but naturally that is a bad work around.

import QtQuick.Controls 2.12
ComboBox {
    id: control
    ...
    all the above content
    ...
    indicator: Canvas {
    }
}

It seems that this issue was specific to the 2.12 import, changing the import to leave out the version selection or using 2.5, which was the installed version for PySide6, resolved the null de-refence.

import QtQuick.Controls 2.5

QtQuick.Dialogs 1.2 not installed

The modal dialog import in the main window resulted in an error

module “QtQuick.Dialogs” version 1.2 is not installed

Removing the 1.2 version removes the error, but introduces a new Dialog which results in the next error with the dialog being opened in the Top Left of the parent.

import QtQuick.Dialogs

Qml Dialog is centered Top Left of window

QtQuick.Dialogs 1.2 seemed to center the Modal Dialog to your window. In moving to 2.x version, it inherits from Popup which has x,y coordinates, which default to top left. Alternatively you can use anchors to center the dialog.

anchors.centerIn: Overlay.overlay

Refer popup documentation for use of anchoring or x,y positioning: Popup QML Type | Qt Quick Controls 6.3.0

Qt 6 Qml seems not to have a TreeView

A small set back for our project was what seems to be a lack of the TreeView from QtQuick.Controls 1.4, which was dropped at least initially in version 6. It seems it can be obtained from the Qt Marketplace, but the license is GPLv3 or proprietary, which makes one concerned to use it, if your own project is one of the permissive free software licenses, which many QML projects seem to be.

In our particular case an alternate implementation using ListView was adopted to avoid licensing complications.


Qt ® is a registered trademark of The Qt Company Ltd. and its subsidiaries.

Python ® is a registered trademark of the Python Software Foundation.

Python Gems

This post holds some Python® snippets of common and useful coding scenarios.

Iterate through Python List in Reverse

# Traverse Python list in reverse
for item in reversed(list_of_items):
    print(item)

Print List without brackets

mylist = [1,2,3,4]
print(mylist)
[1, 2, 3, 4]
print(','.join(map(str, mylist)))
1,2,3,4

Adding to lists

Subtly different to many other languages, you can use append, insert to add a single object (of any type) or extend, to add a list to another to form one contiguous list. If you use append to add a list, the whole list will be placed as a single object.

list1 = ['one', 'two']
list2 = ['three', 'four']
# Then extend the first with the second list
list1.extend(list2)
# to get
['one','two','three','four']
# If you instead had
list1.append(list2)
# it would become
['one', 'two', ['three','four']]

List manipulations

# Sort your list
list_sort = sorted(['z', 'a', 'f', 'b'])

Iterate through Dictionary

for key,value in mydictionary.items()
    print(key, value)

for key in mydictionary.keys()
    print(key, mydictionary[key])

for value in mydictionary.values()
    print(value)

Array of Formatted Strings

items = [1,2,3,4]
fstring_items = [f"the number: {i}" for i in items]
print(fstring_items)
['the number: 1', 'the number: 2', 'the number: 3', 'the number: 4']

List of Files in a Directory

directory = "/path/to/dir"
files = [f for f in os.listdir(directory) if os.path.isfile(os.path.join(directory, f))]

Directory and File Detection

# Get the absolute path to this python file
current_path = os.path.dirname(os.path.abspath(__file__))

List Module Location

If you need to know where your current environment is getting a module from, then run python cli as below. Make sure you are in the correct environment. i.e if you are using a virtual environment, first activate it.

$ python3
>>> import module_name
>>> print(module_name.__file__)

Install from Repository

python3 -m pip install git+https://bitbucket.org/project-directory/repository.git@tag-or-branch-name

If you want to install globally use sudo.

If you get error error: invalid command 'bdist_wheel' then you need to install python -m pip wheel. This can likely happen in a virtual environment. You could add wheel as a dependency in your setup.py file.

Building your python package

For a long time to build a package the setup.py was called directly.

This is deprecated.

SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.

This docs page has a nice if detailed explanation.

In short replace your previous

python setup.py sdist bdist_wheel

with

python -m build

in the package directory containing the setup.py file.

Threading

What thread is that?

When dealing with thread’s inevitably you need to figure out what is what thread, which the name of the thread can help you with in debugging.

print(threading.current_thread().name, "your message")

Python ® is a registered trademark of the Python Software Foundation.

Python Date and Time

If there is something that can go wrong in programming, it is date and time. If you are using time, think, and do it right first time. It is amazing how many projects end up in an uncomfortable situation because time-zones or daylight saving or ambiguous time causes hard to reverse issues because copious amounts of data has the wrong time.

At Direkt EmbeddedTM we recommend a few simple rules with date and time.

Always save time in UTC

Always send time in UTC, or an ISO format including the Time Zone

Always print time in UTC, or an ISO format including the Time Zone

If you must use local time only do so on a UI, and always include a timezone somewhere, just to be clear!



Here are some recommended Python® snippets for date and time using the base installation (no libraries).

Date Time in ISO UTC

from datetime import datetime, timezone

now = datetime.now(timezone.utc)

# Always use timezone.utc to ensure isoformat returns +00:00.
print(now.isoformat())

2021-08-13T04:53:23.588839+00:00

Getting your timezone

zone = datetime.now(timezone.utc).astimezone().tzinfo
print(zone)
AEST

Date Formatted File Names

For logging, you want a reverse date, similar to ISO format, but without colons. The following format is pretty common. Add the Z at the end to indicate UTC time. Z if for zulu, aka UTC. It is not very common, but common enough and short and clear.

utc_str = now.strftime("%Y%m%dT%H%M%SZ")
app = "my-app"
file_name = f"{utc_str}-{app}.log"
print(file_name)

20210813T051726Z-my-app.log

Note: f formatted strings are dependent on python 3.6


Python ® is a registered trademark of the Python Software Foundation.